A method, device and equipment for detecting data quality

Abstract

The embodiments of this specification provide a data quality detection method, device and equipment, which relate to the field of artificial intelligence technology, wherein the method includes: generating a data lineage relationship information set based on query conditions; the data lineage relationship information set is used to characterize the relationship between data; using the data lineage relationship information set, determining the data range that needs to be detected; setting quality detection parameters based on the data range that needs to be detected; generating a target structured query statement according to the data range that needs to be detected and the quality detection parameters; using the target structured query statement to perform quality detection on the data within the data range that needs to be detected, and obtaining a quality detection result. In the embodiments of this specification, the target structured query statement can be automatically generated by setting the numerical values ​​of each quality detection parameter, thereby solving the problem that business personnel cannot code, and effectively improving the efficiency of quality detection, and can efficiently respond to the needs of business data governance.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular to a data quality detection method, device and equipment. Background Art

[0002] At present, the regulatory authorities' inspections of financial institutions are characterized by a wider scope, higher standards, stricter requirements, and heavier penalties. The supervision of data governance in financial institutions has entered a "high-pressure period". In the existing technology, due to the insufficient coding ability of business department personnel, data governance can only be carried out by solidifying the release method. Generally, the business department puts forward requirements, the developers of the financial technology department do the coding, and release it through version deployment. After the launch, each business department analyzes the data quality questions generated and issues governance measures. The whole process has a long communication path and poor analysis timeliness. It can be seen that the technical solutions in the existing technology cannot efficiently analyze the quality of business data.

[0003] To address the above problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of this specification provide a data quality detection method, device and equipment to solve the problem in the prior art that it is impossible to efficiently perform quality analysis on business data.

[0005] The embodiments of the present specification provide a data quality detection method, including: generating a data lineage relationship information set based on query conditions; wherein the data lineage relationship information set is used to characterize the relationship between data; using the data lineage relationship information set, determining a data range that needs to be detected; setting quality detection parameters based on the data range that needs to be detected; generating a target structured query statement according to the data range that needs to be detected and the quality detection parameters; using the target structured query statement to perform quality detection on the data within the data range that needs to be detected to obtain a quality detection result; wherein the quality detection result is used to indicate whether the data has passed the quality detection.

[0006] The embodiments of the present specification also provide a data quality detection device, including: a first generation module, used to generate a data lineage relationship information set based on a query condition; wherein the data lineage relationship information set is used to characterize the relationship between data; a determination module, used to determine the data range that needs to be detected using the data lineage relationship information set; a setting module, used to set quality detection parameters based on the data range that needs to be detected; a second generation module, used to generate a target structured query statement according to the data range that needs to be detected and the quality detection parameters; a processing module, used to perform quality detection on the data within the data range that needs to be detected using the target structured query statement to obtain a quality detection result; wherein the quality detection result is used to indicate whether the data has passed the quality detection.

[0007] The embodiments of the present specification also provide a data quality detection device, including a processor and a memory for storing processor executable instructions, and the processor implements the steps of any method embodiment in the embodiments of the present specification when executing the instructions.

[0008] The embodiments of the present specification also provide a computer-readable storage medium on which computer instructions are stored. When the instructions are executed, the steps of any method embodiment in the embodiments of the present specification are implemented.

[0009] The embodiments of this specification provide a data quality detection method, which can generate a data lineage relationship information set for characterizing the relationship between data based on query conditions, and use the data lineage relationship information set to determine the data range that needs to be detected. Further, quality detection parameters can be set based on the data range that needs to be detected, and a target structured query statement can be automatically generated based on the data range that needs to be detected and the quality detection parameters. The target structured query statement can be used to perform quality detection on the data within the data range that needs to be detected to obtain a quality detection result, wherein the quality detection result can be used to indicate whether the data has passed the quality detection. For users who cannot program, the target structured query statement can be automatically generated by setting the values ​​of each quality detection parameter, thereby solving the problem of business personnel not being able to code, and effectively improving the efficiency of quality detection, and can efficiently respond to the needs of business data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the embodiments of this specification, constitute a part of the embodiments of this specification, and do not constitute a limitation of the embodiments of this specification. In the drawings:

[0011] Figure 1 It is a schematic diagram of the steps of the data quality detection method provided in the embodiment of this specification;

[0012] Figure 2 It is a schematic diagram of a data blood relationship diagram provided according to an embodiment of this specification;

[0013] Figure 3 is a schematic diagram of an interface for selecting an output field according to an embodiment of this specification;

[0014] Figure 4 is a schematic diagram of an interface for selecting screening conditions provided according to an embodiment of this specification;

[0015] Figure 5 is a schematic diagram of an interface for defining sorting output provided according to an embodiment of this specification;

[0016] Figure 6 is a schematic diagram of an interface for defining group output according to an embodiment of this specification;

[0017] Figure 7 It is a schematic diagram of an interface for submitting SQL according to an embodiment of this specification;

[0018] Figure 8 is a structural schematic diagram of a data quality detection device provided according to an embodiment of this specification;

[0019] Fig. 9 It is a structural schematic diagram of a data quality detection device provided according to an embodiment of this specification. DETAILED DESCRIPTION

[0020] The principles and spirit of the embodiments of this specification will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of this specification, and are not intended to limit the scope of the embodiments of this specification in any way. On the contrary, these embodiments are provided to make the disclosure of the embodiments of this specification more thorough and complete, and to fully convey the scope of the disclosure to those skilled in the art.

[0021] Those skilled in the art know that the implementation of the embodiments of this specification can be implemented as a system, device, method or computer program product. Therefore, the embodiments disclosed in this specification can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0022] Although the processes described below include multiple operations that occur in a particular order, it should be clear that these processes may include more or fewer operations, which may be performed sequentially or in parallel (eg, using parallel processors or a multi-threaded environment).

[0023] See also Figure 1This embodiment can provide a data quality detection method. The data quality detection method can be used to automatically generate a target structured query statement by setting the values ​​of various quality detection parameters to solve the problem that business personnel cannot code. The above data quality detection method can include the following steps.

[0024] S101: Generate a data lineage relationship information set based on query conditions; wherein the data lineage relationship information set is used to represent the relationship between data.

[0025] In this embodiment, the query condition input by the target user can be received, and a query is performed based on the query condition to generate a data lineage relationship information set of the queried data. The data lineage relationship information set can be used to characterize the relationship between the data.

[0026] In this embodiment, the query conditions may be characteristic information of the data to be queried, such as system name, table name, field name, version information, etc. Of course, the query conditions are not limited to the above examples, and those skilled in the art may make other changes based on the technical essence of the embodiments of this specification, but as long as the functions and effects achieved are the same or similar to those of the embodiments of this specification, they should be covered within the protection scope of the embodiments of this specification.

[0027] In this embodiment, the above-mentioned data lineage relationship information set can be used to characterize the full-link relationship of end-to-end data generation and conversion and the processing logic of data processing and processing, and the above-mentioned data lineage relationship information set can be displayed or stored in the form of tables, images, etc. Preferably, the above-mentioned data lineage relationship information set can be an SVG vector image, so that the target user can intuitively and clearly understand the relationship between the data. Among them, SVG (scalable vector graphics) is a new two-dimensional vector graphics format formulated by the International Internet Standards Organization, and is also a network vector graphics standard in the specification. Of course, the form of the data lineage relationship information set is not limited to the above examples. Technical personnel in the relevant field may make other changes under the inspiration of the technical essence of the embodiments of this specification, but as long as the functions and effects achieved are the same or similar to the embodiments of this specification, they should be covered within the protection scope of the embodiments of this specification.

[0028] In this embodiment, while generating the data kinship information set, data analysis can also be performed to comprehensively analyze and display the current status of the queried data to help target users discover potential data quality issues. Among them, the above data analysis can include: data integrity analysis, data classification analysis, data distribution analysis, data attribute analysis, data format analysis, etc. Of course, it can be understood that the above data analysis can also include duplicate data analysis, etc., which can be determined according to actual conditions, and this specification embodiment does not limit this.

[0029] In this embodiment, the results of the above data analysis can be displayed in a fusion of the data lineage information set, or can be displayed separately, so that the target user can more accurately determine the data range that needs to be tested for data quality. Of course, it is understandable that only the data lineage information set can be displayed to the target user, and the specific amount can be determined according to the actual situation, and this embodiment of the specification does not limit this.

[0030] S102: Determine the data range that needs to be detected by using the data lineage information set.

[0031] In this embodiment, since the data lineage relationship information set can be used to represent the relationship between data, the data lineage relationship information set can be used to determine the data range that needs to be detected. In some embodiments, the data that needs to be detected can be directly selected from the data lineage relationship information set to obtain the data range, or the data range that needs to be detected can be independently input. The specific range can be determined according to actual conditions, and this specification embodiment does not limit this.

[0032] In this embodiment, the data range may be specific data (fields) to be detected, or a data table (table name) to be detected, etc. The specific range may be determined according to actual conditions, and this specification embodiment does not limit this.

[0033] In this embodiment, if there is a quality problem with the data associated with other data, it will affect the other data. Therefore, the data associated with other data can be used as the data to be detected. In some embodiments, all data can also be used as the object of detection, or the data with high frequency of use can be used as the object of detection. In some embodiments, if the data is analyzed at the same time, the data that may have quality problems can be used as the data to be detected in combination with the results of the analysis. The specific data can be determined according to the actual situation, and the embodiments of this specification are not limited to this.

[0034] S103: Setting quality detection parameters based on the data range to be detected.

[0035] In this embodiment, quality detection parameters can be set based on the data range to be detected according to the regulatory target requirements, wherein the above regulatory target requirements can be determined according to actual business needs.

[0036] In this embodiment, the above-mentioned quality inspection parameters can be used to characterize the parameters that need to be set for general quality inspection and business data quality inspection. Among them, the above-mentioned general quality inspection identifies the standards that must be checked for each type of field, including the number of records, primary key null, duplicate exceptions, field length, etc.; the above-mentioned business data quality inspection can be a quality inspection standard that is additionally customized for each field according to actual needs.

[0037] In this embodiment, since the requirements for general quality inspection and business data quality inspection are different, the general quality inspection and business data quality inspection can be performed separately. In some embodiments, the general quality inspection rules can be uniformly scheduled to perform full table scans in batches every night; the business data quality inspection rules perform full table scans on different data storage platforms (determined data ranges) according to the time frequency set by the target user. Of course, the method of data quality detection is not limited to the above examples. Technical personnel in the relevant field may make other changes under the inspiration of the technical essence of the embodiments of this specification, but as long as the functions and effects achieved are the same or similar to the embodiments of this specification, they should be covered within the protection scope of the embodiments of this specification.

[0038] In this embodiment, the above-mentioned quality detection parameters may include: output field, screening condition, output sorting method, output grouping method, etc. Of course, it can be understood that the above-mentioned quality detection parameters may also include: detection frequency, field type, etc. The specific parameters may be determined according to actual conditions, and the embodiments of this specification do not limit this.

[0039] S104: Generate a target structured query statement according to the data range and quality detection parameters that need to be detected.

[0040] In this embodiment, the corresponding SQL (Structured Query Language) scripts of various quality detection parameters can be maintained in the target rule base. After the data range and quality detection parameters are determined, the target structured query statement can be generated based on the SQL scripts maintained in the target rule base.

[0041] In this embodiment, target users who can program can directly write target structured statements in the corresponding interface. Target users who cannot program can select the values ​​of each quality inspection parameter in turn based on the navigation to automatically generate target structured query statements, thereby solving the problem of business personnel not being able to code.

[0042] S105: Using the target structured query statement, perform quality inspection on the data within the data range to be inspected, and obtain a quality inspection result; wherein the quality inspection result is used to indicate whether the data has passed the quality inspection.

[0043] In this embodiment, the target structured query statement can be executed to perform quality inspection on the data within the data range that needs to be inspected to obtain a quality inspection result; wherein the above quality inspection result can be used to indicate whether a single data has passed the quality inspection.

[0044] In this embodiment, a target structured query statement can be specified at a specified time point to perform quality inspection on data within the data range that needs to be inspected, or the target structured query statement can be repeatedly specified at a certain time interval to perform quality inspection on data within the data range that needs to be inspected. The specific details can be determined based on actual conditions, and the embodiments of this specification are not limited to this.

[0045] In this embodiment, the above-mentioned quality inspection results may include the results of whether each piece of data has passed the quality inspection, and the above-mentioned quality inspection results may be fed back to the target user. If there is data that fails the quality inspection, the target user can further combine the quality inspection results for data governance. The specific target user can determine the processing objects corresponding to different problems, and forward the corresponding quality inspection results to the processing objects for problem governance, closing the loop. Of course, it can be understood that it can also be directly processed by the target user, and the specific details can be determined according to the actual situation, and this specification embodiment does not limit this.

[0046] In this embodiment, quality control and evaluation can also be carried out based on the quality inspection results for data with high usage frequency and high attention, and a report can be generated, so that the results of the quality inspection are traceable and can be easily combined with other quality inspection results for overall management of the database.

[0047] From the above description, it can be seen that the embodiments of this specification achieve the following technical effects: a data lineage information set for characterizing the relationship between data can be generated based on the query conditions, and the data lineage information set can be used to determine the data range that needs to be detected. Further, quality detection parameters can be set based on the data range that needs to be detected, and a target structured query statement can be automatically generated based on the data range and quality detection parameters that need to be detected. The target structured query statement can be used to perform quality detection on the data within the data range that needs to be detected to obtain a quality detection result, wherein the quality detection result can be used to indicate whether the data has passed the quality detection. For users who cannot program, the target structured query statement can be automatically generated by setting the values ​​of each quality detection parameter, thereby solving the problem that business personnel cannot code, and effectively improving the efficiency of quality detection, and can efficiently respond to the needs of business data governance.

[0048] In one embodiment, generating a data lineage relationship information set based on a query condition may include: obtaining end-to-end lineage relationship detail data from a target database according to the query condition, converting the end-to-end lineage relationship detail data to obtain node information; wherein the node information is used to determine the metadata of the field. Further, a data lineage relationship graph may be drawn based on the end-to-end lineage relationship detail data and the node information, and the data lineage relationship graph may be used as the data lineage relationship information set.

[0049] In this embodiment, the existing structured field-level data kinship information can be queried according to the query conditions to obtain end-to-end kinship detail data. The end-to-end kinship detail data can also be converted to obtain node information. Furthermore, the metadata information of each field can be obtained based on the node information, and the end-to-end kinship detail data and node information can be returned to the front-end page. Through the JavaScript plug-in program, the end-to-end kinship detail data is recycled to generate an SVG vector diagram to draw a full-link data kinship diagram. Among them, JavaScript is a high-level, multi-paradigm, interpreted programming language. It is a prototype-based, function-first language that supports object-oriented programming, imperative programming, and functional programming.

[0050] In this embodiment, the data lineage relationship diagram can be as follows Figure 2 As shown in , where the meaning of each identifier in the figure can be as shown in Table 1.

[0051] Table 1

[0052] Main fields illustrate System Name The system name of the node Table name The table name of the node Field Name Field name of the node A Data quality test results, percentage value T Technical metadata, icons C Business metadata, icons P Profiling results, icons E Number retrieval rules, icons

[0053] In this embodiment, Figure 2 The data quality test result A in the data quality test result can be a historical data quality test result. If no data quality test has been performed in the history, the data corresponding to the data quality test result A can be empty.

[0054] In this embodiment, click Figure 2 Each icon in the figure can jump to the display page of the corresponding data. In some embodiments, it can also be displayed on the current page. The specific details can be determined according to actual conditions, and the embodiments of this specification do not limit this.

[0055] In one implementation, the query conditions may include: system name, table name, field name, version information, and keywords representing business meanings.

[0056] In one embodiment, after generating the data lineage information set based on the query conditions, the method may further include: obtaining a preset analysis strategy, and generating a structured query statement for digital data and a structured query statement for non-digital data according to the query conditions and the preset analysis strategy. Further, the structured query statement for digital data and the structured query statement for non-digital data may be executed to obtain a digital data analysis report and a non-digital data analysis report.

[0057] In this embodiment, the above-mentioned preset analysis strategy may include: strategies including data integrity analysis, data classification analysis, data distribution analysis, data attribute analysis, data format analysis, etc. Of course, it can be understood that the above-mentioned data analysis strategy may also include repeated data analysis, etc., which can be determined according to actual conditions, and the embodiments of this specification do not limit this.

[0058] In this embodiment, the data can be classified into digital data and non-digital data. Since there are large differences between the two types of data, the two types of data can be analyzed separately. Structured query statements for tables and fields corresponding to query conditions can be generated for digital data and non-digital data, so that structured query statements for digital data and structured query statements for non-digital data can be obtained.

[0059] In this embodiment, the structured query statements for digital data and the structured query statements for non-digital data may be executed periodically, thereby obtaining a digital data analysis report and a non-digital data analysis report.

[0060] In this embodiment, the analysis of digital data may include: Average, Min / Max, Sum, Count, Range, Type, Length, whether it is a primary key; the analysis of non-digital data may include: Type, Length, whether it is a primary key, List of values, Data Dictionary. The list of data field values ​​may display 10, 20, 50 or 100 values, which can be determined according to actual conditions, and this specification does not limit this.

[0061] In this embodiment, the digital data analysis report may be as shown in Table 2, and the non-digital data analysis report may be as shown in Table 3. Wherein, from indicates the source of the data, and to indicates the ownership of the data.

[0062] Table 2

[0063] from to Table name Field Name Length Average Max Min Sum Count Range

[0064] Table 3

[0065] from to Table name Field Name Type Length Is it a primary key? List of values

[0066] In one embodiment, using the data lineage information set to determine the data range that needs to be detected may include: determining the data range that needs to be detected based on the data lineage information set, a digital data analysis report, and a non-digital data analysis report.

[0067] In this embodiment, the data range that needs to be detected can be determined by combining the data lineage information set, the digital type data analysis report and the non-digital type data analysis report.

[0068] In this embodiment, if there is a quality problem with the data associated with other data, it will affect the other data. Therefore, the data associated with other data can be used as the data to be detected. In some embodiments, all data can also be used as the object of detection, or the data with high frequency of use can be used as the object of detection. In some embodiments, the data that may have potential quality problems shown in the analysis report can also be used as the data to be detected. The specific data can be determined according to the actual situation, and the embodiments of this specification are not limited to this.

[0069] In one embodiment, the quality detection parameters include: output fields, screening conditions, output sorting methods, and output grouping methods.

[0070] In this embodiment, the above-mentioned quality detection parameters may include: output field, screening condition, output sorting method, output grouping method, etc. Of course, it can be understood that the above-mentioned quality detection parameters may also include: detection frequency, field type, etc. The specific parameters may be determined according to actual conditions, and the embodiments of this specification do not limit this.

[0071] In this embodiment, a visual method can be used to set quality inspection parameters according to the navigation, and the output field can be selected in the navigation order -> screening conditions -> define sorting output -> define grouping output -> submit. This processing method can maintain the corresponding SQL script in the target rule library to solve the problem that business personnel cannot code.

[0072] In this embodiment, the setting of the quality detection parameters may include the following steps:

[0073] Step 1: Select the output field in the navigation foreground. The interface for selecting the output field can be as follows: Figure 3 As shown in , the selectable field names can be displayed based on the data range, and the corresponding interface can also include fields such as removing duplicate records and inputting aliases. Among them, the SQL statements corresponding to different main fields can be shown in Table 4.

[0074] Table 4

[0075] Main fields illustrate Example Remove duplicate records Select the deduplication query in the SQL statement. Select distinct aerio Enter an alias Select query field alias in SQL statement Select distinct aerio as A

[0076] Step 2: Select the filter conditions according to the navigation. The interface for selecting the filter conditions can be as follows: Figure 4 As shown in , the corresponding interface may include a condition editing area, and the screening conditions may include: conjunctions, operators, left value area and type, right value area, arithmetic function, etc. Among them, the SQL statements corresponding to each screening condition may be as shown in Table 5.

[0077] Table 5

[0078]

[0079]

[0080] Step 3: Sort the output according to the navigation definition. The interface for defining the sort output can be as follows: Figure 5 The sorting output may include ascending order and descending order, and the SQL statement of the sorting output may be as shown in Table 6.

[0081] Table 6

[0082]

[0083] Step 4: Define group output according to navigation. The interface for defining group output can be as follows: Figure 6 As shown in , grouping can be performed by selecting different field combinations. The SQL statement for grouping can be as shown in Table 7.

[0084] Table 7

[0085]

[0086] Step 5: Submit SQL according to the navigation. Users who know how to code can directly enter this step to write and modify rules.

[0087] In this embodiment, the data storage platform information to which the table belongs can be obtained from the data lineage interface, and the setting information of steps 1 to 4 can be obtained through the JavaScript plug-in, and then the statements are spliced ​​to generate SQL scripts according to the standards of the data storage platform. Among them, the interface for submitting SQL can be as follows Figure 7 As shown in , the generated target structured query statement can also be modified in the SQL submission interface, and the final target structured query statement can be submitted after confirmation.

[0088] In this implementation, the generated target structured query statement may be stored in a target database so that it can be obtained in a timely manner during subsequent quality inspection.

[0089] In one embodiment, using a target structured query statement to perform a quality test on data within a data range that needs to be tested to obtain a quality test result may include: displaying the target structured query statement in a preset interface, and setting a time frequency for executing the target structured query statement. When it is determined that the target structured query statement does not need to be modified, the target structured query statement is executed according to the time frequency to perform a quality test on the data within the data range that needs to be tested to obtain a quality test result.

[0090] In this embodiment, the target structured query statement can be displayed in a preset interface so that the target user can confirm whether to modify the target structured query statement. Figure 7 Submit SQL interface in.

[0091] In this embodiment, the time frequency of executing the target structured query statement can be set. The above time frequency can be executed once every 1 hour, once a day, or once a week, etc. The specific time frequency can be determined according to actual conditions, and the embodiments of this specification are not limited to this.

[0092] In this embodiment, when it is determined that the target structured query statement does not need to be modified, the target structured query statement can be executed according to the time frequency to perform quality inspection on the data within the data range that needs to be inspected, and the quality inspection results can be obtained. Each quality inspection result can be stored in the target database to ensure the traceability of the data.

[0093] In one embodiment, after obtaining the quality detection result, the method may further include: obtaining a preset threshold, and determining the number of data items that passed the quality detection and the total number of data items that participated in the quality detection according to the quality detection result. When the ratio of the number of data items that passed the quality detection to the total number of data items that participated in the quality detection is less than the preset threshold, generating quality detection prompt information, and sending the quality detection prompt information to the target processing object.

[0094] In this embodiment, a threshold value can be preset, and the preset threshold value can be used to determine whether the overall quality inspection passes. The preset threshold value can be a value greater than 0 and less than or equal to 1, for example: 0.98, 0.86, 0.79, etc., which can be determined according to actual conditions, and this specification embodiment does not limit this.

[0095] In this embodiment, the value of the number of data items that pass the quality test / the total number of data items involved in the quality test can be calculated, and compared with the preset threshold value to obtain pass / fail, and the input value type of the preset threshold value is a percentage digital type. Pass rate = number of data items that pass the quality test / total number of data items involved in the quality test. If the pass rate is less than the preset threshold value, it means that the quality test fails this time, and a quality test prompt message needs to be generated.

[0096] In this embodiment, the quality detection prompt information can be sent to the target processing object, wherein the target processing object can be the target user or the person in charge of the corresponding system table. The specific method can be determined according to the actual situation, and this embodiment of the specification does not limit this.

[0097] In this implementation, the quality inspection prompt information is registered and distributed to the person in charge of the corresponding system table. The distributed content may include 3 sample data, and the problem is closed after the governance is completed. In some embodiments, the problems detected by data quality can also be pushed to the target user via email. The target user can fill in the business personnel to be distributed, and transfer it to the business personnel to be distributed for problem governance and closed-loop problems. The specific can be determined according to the actual situation, and this specification embodiment does not limit this.

[0098] In this embodiment, data quality control and evaluation can be carried out based on the data quality detection results. Quality control and evaluation can be carried out for data with high usage frequency and high attention, and reports can be formed to drive problem rectification. Data quality reports can also be generated regularly and sent to relevant departments or responsible personnel.

[0099] In one scenario example, a data quality detection system may be provided, and the data quality detection system may include:

[0100] The data quality rule management module is used to obtain data lineage diagrams and analysis reports to determine the data tables that need to be managed; pass the generated data quality rule script information to the data quality storage module; receive the data quality rule script execution result information returned by the data quality storage module, view the data quality rules, subscribe to the data indicator report, and submit it to the data quality exception handling module for closed-loop management of problems.

[0101] The data quality rule execution module is used to receive the data quality rule script information transmitted by the data quality storage module, execute the data quality rule script, compare the data quality detection result with the preset threshold, and then return the comparison between the quality rule detection result and the preset threshold to the data quality storage module.

[0102] The data quality exception processing module is used to issue reminders and warnings when the comparison results of the indicator data quality rule detection results and the preset threshold indicate failure, and notify the department to which the data table belongs to take measures. It can also quickly generate data quality reports based on the inspection results.

[0103] The data quality storage module is used to store information such as data lineage relationship diagrams, general quality inspection rules, business data quality inspection rules, and business data quality inspection results.

[0104] In one embodiment, the data quality rule management module may further include: a data quality rule query unit and a data quality indicator report subscription unit, wherein the data quality rule query unit is used to receive the query conditions (including the quality rule name) input by the user, and transmit the query condition information to the data quality storage module, thereby obtaining the corresponding quality rule script information. The data quality indicator report subscription unit is used to utilize the instant analysis function of BI (business intelligence), and the user can subscribe to the indicator report in real time and quickly, and keep the professional governance monitoring report in time.

[0105] In one embodiment, the data quality rule execution module may include:

[0106] The quality rule execution unit is used to receive the general quality inspection rules and business data quality inspection rules transmitted by the data quality storage module. The general quality inspection rules are scheduled uniformly to perform full table scans in batches every night; the business data quality inspection rules perform full table scans on different data storage platforms according to the time frequency set by the user.

[0107] The threshold comparison unit is used to calculate the ratio of the number of data items that pass the quality test to the total number of data items participating in the quality test, and compare it with the preset threshold to determine whether it passes or fails. The input value type of the threshold is a percentage digital type.

[0108] Based on the same inventive concept, a data quality detection device is also provided in the embodiments of this specification, such as the following embodiments. Since the principle of solving the problem by the data quality detection device is similar to that of the data quality detection method, the implementation of the data quality detection device can refer to the implementation of the data quality detection method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived. Figure 8 is a structural block diagram of a data quality detection device according to an embodiment of this specification, such as Figure 8As shown, it may include: a first generating module 801, a determining module 802, a setting module 803, a second generating module 804, and a processing module 805. The structure is described below.

[0109] The first generation module 801 can be used to generate a data lineage relationship information set based on a query condition; wherein the data lineage relationship information set is used to represent the relationship between data;

[0110] The determination module 802 may be used to determine the data range that needs to be detected by using the data lineage relationship information set;

[0111] The setting module 803 can be used to set quality detection parameters based on the data range to be detected;

[0112] The second generating module 804 may be used to generate a target structured query statement according to the data range and quality detection parameters to be detected;

[0113] The processing module 805 can be used to perform quality inspection on the data within the data range to be inspected using the target structured query statement to obtain a quality inspection result; wherein the quality inspection result is used to indicate whether the data has passed the quality inspection.

[0114] The embodiment of this specification also provides an electronic device, which can be specifically referred to in Fig. 9 The electronic device composition structure diagram of the data quality detection method provided by the embodiment of this specification is shown, and the electronic device may specifically include an input device 91, a processor 92, and a memory 93. Among them, the input device 91 can be specifically used to input query conditions. The processor 92 can be specifically used to generate a data lineage information set based on the query conditions; wherein the data lineage information set is used to characterize the relationship between data; using the data lineage information set, determine the data range that needs to be detected; set quality detection parameters based on the data range that needs to be detected; generate a target structured query statement based on the data range and quality detection parameters that need to be detected; use the target structured query statement to perform quality detection on the data within the data range that needs to be detected, and obtain a quality detection result; wherein the quality detection result is used to indicate whether the data has passed the quality detection. The memory 93 can be specifically used to store data such as the target structured query statement and the quality detection result.

[0115] In this embodiment, the input device can specifically be one of the main devices for information exchange between the user and the computer system. The input device can include a keyboard, a mouse, a camera, a scanner, a light pen, a handwriting input board, a voice input device, etc.; the input device is used to input the original data and the program for processing these numbers into the computer. The input device can also obtain and receive data transmitted from other modules, units, and devices. The processor can be implemented in any appropriate manner. For example, the processor can take the form of a computer-readable medium, a logic gate, a switch, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable logic controller, and an embedded microcontroller, etc., such as a microprocessor or a processor and a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor. The memory can specifically be a memory device used to store information in modern information technology. The memory can include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function without a physical form is also called a memory, such as a RAM, a FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, a TF card, etc.

[0116] In this embodiment, the functions and effects specifically realized by the electronic device can be explained in comparison with other embodiments and will not be described in detail here.

[0117] In an implementation manner of the embodiments of the present specification, a computer storage medium based on a data quality detection method is also provided, and the computer storage medium stores computer program instructions, which can achieve the following when the computer program instructions are executed: generating a data lineage relationship information set based on query conditions; wherein the data lineage relationship information set is used to characterize the relationship between data; using the data lineage relationship information set, determining a data range that needs to be detected; setting quality detection parameters based on the data range that needs to be detected; generating a target structured query statement based on the data range that needs to be detected and the quality detection parameters; using the target structured query statement to perform quality detection on the data within the data range that needs to be detected, and obtaining a quality detection result; wherein the quality detection result is used to indicate whether the data has passed the quality detection.

[0118] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD), or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0119] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other embodiments and will not be repeated here.

[0120] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of this specification can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of this specification are not limited to any specific combination of hardware and software.

[0121] Although the embodiments of this specification provide method operation steps as described in the above embodiments or flowcharts, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of this specification. When the device or terminal product of the method described in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or drawings (for example, a parallel processor or a multi-threaded processing environment).

[0122] It should be understood that the above description is for illustration and not for limitation. By reading the above description, many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art. Therefore, the scope of the embodiments of this specification should not be determined with reference to the above description, but should be determined with reference to the full scope of the aforementioned claims and the equivalents possessed by these claims.

[0123] The above is only a preferred embodiment of the embodiment of this specification, and is not intended to limit the embodiment of this specification. For those skilled in the art, the embodiment of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of this specification shall be included in the protection scope of the embodiment of this specification.

Claims

1. A data quality detection method, characterized in that: include: Generate a data lineage relationship information set based on the query conditions; wherein the data lineage relationship information set is used to characterize the relationship between the data; Determine the data range that needs to be detected by using the data lineage relationship information set; Setting quality detection parameters based on the data range that needs to be detected; Generate a target structured query statement according to the data range to be detected and the quality detection parameters; Using the target structured query statement to perform quality inspection on the data within the data range to be inspected, and obtaining a quality inspection result; wherein the quality inspection result is used to indicate whether the data has passed the quality inspection; The step of generating a data blood relationship information set based on the query condition includes: Receive the query conditions input by the target user, and perform a query based on the query conditions to generate a data lineage relationship information set of the queried data; the query conditions are feature information of the data to be queried; the data lineage relationship information set is used to characterize the full-link relationship of end-to-end data generation and conversion and the processing logic of data processing; After generating the data kinship information set based on the query conditions, it also includes: Get the preset profiling strategy; Generate structured query statements for digital data and structured query statements for non-digital data according to the query conditions and the preset analysis strategy; Execute the structured query statement for the digital data and the structured query statement for the non-digital data to obtain a digital data analysis report and a non-digital data analysis report; The step of using the data lineage relationship information set to determine the data range to be detected includes: Determine the data range that needs to be detected based on the data lineage relationship information set, the digital type data analysis report, and the non-digital type data analysis report; The using the target structured query statement to perform quality detection on the data within the data range that needs to be detected includes: The target structured query statement is executed to perform quality detection on the data within the data range that needs to be detected, and a quality detection result is obtained.

2. The method according to claim 1, characterized in that Generate a data lineage relationship information set based on the query conditions, including: Acquire end-to-end blood relationship detail data from the target database according to the query condition; The end-to-end blood relationship detail data is converted to obtain node information; wherein the node information is used to determine the metadata of the field; Draw a data blood relationship diagram according to the end-to-end blood relationship detailed data and the node information; The data lineage relationship diagram is used as the data lineage relationship information set.

3. The method according to claim 1, characterized in that: The query conditions include: system name, table name, field name, version information, and keywords representing business meanings.

4. The method according to claim 1, characterized in that The quality detection parameters include: output fields, screening conditions, output sorting methods, and output grouping methods.

5. The method according to claim 1, characterized in that The target structured query statement is used to perform quality inspection on the data within the data range to be inspected, and a quality inspection result is obtained, including: Displaying the target structured query statement in a preset interface; Setting the time frequency of executing the target structured query statement; When it is determined that the target structured query statement does not need to be modified, the target structured query statement is executed according to the time frequency to perform quality detection on the data within the data range that needs to be detected, and a quality detection result is obtained.

6. The method according to claim 1, characterized in that After obtaining the quality test results, it also includes: Get the preset threshold value; According to the quality inspection result, determine the number of data items that passed the quality inspection and the total number of data items that participated in the quality inspection; When the ratio of the number of data items that pass the quality inspection to the total number of data items that participate in the quality inspection is less than the preset threshold, generating quality inspection prompt information; The quality detection prompt information is sent to the target processing object.

7. A data quality detection device, characterized in that: include: A first generating module is used to generate a data lineage relationship information set based on a query condition; wherein the data lineage relationship information set is used to characterize the relationship between data; A determination module, used to determine the data range that needs to be detected by using the data blood relationship information set; A setting module, used for setting quality detection parameters based on the data range to be detected; A second generating module is used to generate a target structured query statement according to the data range to be detected and the quality detection parameters; A processing module, used to perform quality inspection on the data within the data range to be inspected using the target structured query statement to obtain a quality inspection result; wherein the quality inspection result is used to indicate whether the data has passed the quality inspection; The first generation module is specifically used to: receive the query condition input by the target user, and perform a query based on the query condition to generate a data lineage relationship information set of the queried data; the query condition is the characteristic information of the data to be queried; the data lineage relationship information set is used to characterize the full-link relationship of end-to-end data generation and conversion and the processing logic of data processing; The device is also used to: obtain a preset analysis strategy; generate a structured query statement for digital data and a structured query statement for non-digital data according to the query condition and the preset analysis strategy; Execute the structured query statement for the digital data and the structured query statement for the non-digital data to obtain a digital data analysis report and a non-digital data analysis report; The determination module is specifically used to: determine the data range that needs to be detected according to the data blood relationship information set, the digital type data analysis report and the non-digital type data analysis report; The processing module is specifically used to: execute the target structured query statement to perform quality detection on the data within the data range that needs to be detected, and obtain a quality detection result.

8. A data quality detection device, characterized in that: The method comprises a processor and a memory for storing processor-executable instructions, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the instructions.

9. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed, the steps of the method according to any one of claims 1 to 6 are implemented.

Images