A method for automatically discovering data quality inspection rules

Abstract

The present application relates to a kind of automatic discovery data quality inspection rule method.The present method includes that for the data table that has been entered in the warehouse, system automatically executes rule discovery task, and forms single table rule;Management personnel according to system current standard and experience judge whether the single table rule formed in the previous step has secondary cascade relationship, if the single table rule has secondary cascade relationship, then manually confirm, then using the single table rule after confirmation to the data table for discovering the single table rule carries out quality inspection, and after quality inspection is completed, the single table rule changes into quasi rule;All quasi rules found in multiple data tables are subjected to similarity check, and the same or similar quasi rules of multiple data tables are combined to generate general rule, and the generated general rule is used for the quality inspection of all data tables.The present method can be based on the discovery of single table rule, and the combination of multi-table quality inspection rule is carried out by similarity test, so as to provide the basis for the discovered general data quality inspection rule applicable to all data tables.

Description

Technical Field

[0001] This invention belongs to the technical field of data quality inspection rule discovery methods, and particularly relates to a method and system for automatically discovering data quality inspection rules. Background Technology

[0002] Existing rule discovery methods mostly discover rules at the data content level, with very few abstracted and merged into field-level rules. Moreover, the rules discovered by such methods are only applicable to the data table where the rules were discovered, rather than being general data quality inspection rules, which greatly limits their application. Therefore, it is necessary to develop a method that can discover general data quality inspection rules. Summary of the Invention

[0003] To overcome the aforementioned shortcomings of existing rule discovery methods, this invention proposes a new method for automatically discovering data quality inspection rules, which can effectively solve the problem of discovering general data quality inspection rules.

[0004] The overall design architecture of this method is as follows: Figure 1 As shown, the implementation process of this method includes: automatically executing the rule discovery task after the data table is updated and entered into the database, exploring and discovering sub-rules of data content between different fields in the data table, combining multiple sub-rules to form a single-table rule, forming a quasi-rule after manual confirmation of the single-table rule, and merging the same or similar quasi-rules of multiple tables into a general rule.

[0005] Specifically, this invention provides a method for automatically discovering data quality inspection rules, the method comprising:

[0006] S1. Single-table rule discovery: For data tables that have been entered into the database, the system automatically performs rule discovery tasks and generates single-table rules;

[0007] S2. Quasi-rule conversion: Based on the current standards and experience of the system, the administrator judges whether the single table rule formed in the previous step has a two-level cascading relationship. If the single table rule has a two-level cascading relationship, it is manually confirmed. Then, the confirmed single table rule is used to perform quality inspection on the data table that found the single table rule. After the quality inspection is completed, the single table rule is converted into a quasi-rule.

[0008] S3. Generate general rules: Perform similarity checks on all quasi-rules found in multiple data tables, merge identical or similar quasi-rules from multiple data tables to generate general rules, and use the generated general rules for quality inspection of all data tables.

[0009] Furthermore, the single-table rule discovery in step S1 of the method for automatically discovering data quality inspection rules of the present invention includes:

[0010] S11. For data tables that have already been entered into the database, the single-table rule discovery model analyzes the data content based on the system's built-in knowledge base;

[0011] S12. If the confidence of field A and field B is ≥95% and the support is ≥1%, then a sub-rule between field A and field B is found;

[0012] S13. When one or more sub-rules between field A and field B are found in a data table, a two-level cascading rule between field A and field B is formed, and a single-table rule is established based on the two-level cascading rule between field A and field B.

[0013] Furthermore, the knowledge base built into the system in step S11 of the above-mentioned method for automatically discovering data quality inspection rules includes: a unified management dictionary standard and quality knowledge.

[0014] Furthermore, in step S12 of the above-mentioned method for automatically discovering data quality inspection rules, the confidence of field A and field B refers to the proportion of data that both field A and field B meet the conditions to the proportion of data that field A meets the conditions; the support of field A and field B refers to the proportion of data that field A meets the conditions to the total data.

[0015] Furthermore, in step S3 of the method for automatically discovering data quality control rules of the present invention, a similarity check is performed on all quasi-rules found in multiple data tables. The similarity check method includes:

[0016] (1) Generate an initial rule set: Incorporate all quasi-rules found in multiple data tables into the initial rule set that needs to be merged;

[0017] (2) Rule sorting: Sort the quasi-rules in the initial rule set according to the confidence and support indices of the quasi-rules;

[0018] (3) Rule matching: Compare each quasi-rule to see if they have similar conditions and conclusions. If two or more quasi-rules have similar conditions and conclusions, they can be merged to generate a general rule.

[0019] Secondly, this invention provides a system for automatically discovering data quality inspection rules, the system comprising:

[0020] Single-table rule discovery module: This module is used to automatically perform rule discovery tasks on data tables that have been entered into the database, and to generate single-table rules.

[0021] Quasi-rule conversion module: Used to determine whether a single table rule has a two-level cascading relationship and to confirm it. After the confirmation, the single table rule is used to perform quality inspection on the data table that has the single table rule. After the quality inspection is completed, the single table rule is converted into a quasi-rule.

[0022] General rule generation module: It is used to perform similarity checks on all quasi-rules found in multiple data tables. Identical or similar quasi-rules from multiple data tables are merged to generate general rules, which are then used for quality inspection of all data tables.

[0023] Each module operates according to the above-mentioned method for automatically discovering data quality inspection rules.

[0024] Furthermore, the general rule generation module in the system for automatically discovering data quality inspection rules of the present invention includes:

[0025] Initial rule set generation submodule: used to incorporate all quasi-rules found in multiple data tables into the initial rule set that needs to be merged;

[0026] Rule sorting submodule: Used to sort the quasi-rules in the initial rule set according to the confidence and support indicators of the quasi-rules;

[0027] The rule matching submodule is used to compare quasi-rules one by one to see if they have similar conditions and conclusions. If two or more quasi-rules have similar conditions and conclusions, they are merged to generate a general rule.

[0028] In addition, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for automatically discovering data quality inspection rules.

[0029] In summary, the method for automatically discovering data quality inspection rules in this invention can merge multiple table quality inspection rules through similarity testing based on the discovery of single-table rules, thereby providing a foundation for the discovery of general data quality inspection rules applicable to all data tables. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the design architecture of the method of the present invention.

[0032] Figure 2 This is a flowchart illustrating the overall implementation process of the method of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. This invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this invention.

[0034] At the same time, it should be understood that the scope of protection of the present invention is not limited to the specific embodiments described below; it should also be understood that the terminology used in the embodiments of the present invention is for describing specific embodiments and not for limiting the scope of protection of the present invention.

[0035] Example: A method for automatically discovering data quality control rules

[0036] This method primarily addresses the issue of discovering general data quality control rules. Once a data table is updated and stored, a rule discovery task is automatically executed to explore and discover sub-rules for the data content of different fields in the data table. Multiple sub-rules are combined to form a single-table rule. After manual confirmation, the single-table rule becomes a quasi-rule. The same or similar quasi-rules from multiple tables are merged into a general rule.

[0037] like Figure 1 and Figure 2 As shown, this method includes the following steps:

[0038] Step 1: If field A in the data table equals X, then field B equals Y. If the confidence level is greater than or equal to 95% and the support level is greater than or equal to 1%, then a single sub-rule "When A = X, then B = Y" is found (Support level: the proportion of data that satisfies X out of the total data; Confidence level: the proportion of data that satisfies both X and Y out of the data that satisfies X). For example, in Table 1 below, there are 3 records where the city is Nanjing and 4 records where the province is Jiangsu. According to the support and confidence calculation rules, the support is calculated as 3 / 4 = 0.75 > 0.01, and the confidence is calculated as 3 / 3 = 1 > 0.95. Therefore, according to the sub-rule discovery rules, a single sub-rule "When the city is Nanjing, then the province is Jiangsu" can be found in Table 1. Similarly, in Table 1, there is 1 record where the city is Suzhou and 4 records where the province is Jiangsu. According to the support and confidence calculation rules, the support is calculated as 1 / 4 = 0.25 > 0.01, and the confidence is calculated as 1 / 1 = 1 > 0.95. Therefore, according to the sub-rule discovery rules, a single sub-rule "When the city is Suzhou, then the province is Jiangsu" can be found in Table 1.

[0039] Table 1

[0040] Province city districts and counties Jiangsu Nanjing Xuanwu District Jiangsu Nanjing Yuhuatai District Jiangsu Nanjing Qinhuai District Jiangsu Suzhou Gusu District

[0041] Step 2: When one or more sub-rules are found between field A and field B in the data table, a two-level cascading rule is formed between field A and field B.

[0042] As shown in Table 1, two sub-rules are found between the province and city fields:

[0043] 1) "If the city is Nanjing, then the province is Jiangsu";

[0044] 2) "If the city is Suzhou, then the province is Jiangsu";

[0045] Then, establish a single-table rule 1 for "province and city two-level cascading rule" between the province and city fields.

[0046] Step 3: Based on experience, the manager judges whether the "Provincial and Municipal Level-2 Cascading Rules" single table rule 1 found in Table 1 has a level-2 cascading relationship. If the single table rule 1 found has a level-2 cascading relationship, it is manually confirmed. The confirmed single table rule 1 performs quality inspection on the Table 1 currently found with the rule. After the quality inspection is completed, the single table rule 1 becomes the quasi-rule 1.

[0047] Step 4: Now perform steps 1 to 3 above on Table 2 below. From Table 2, we find the single table rule 2 "Provincial and municipal level two-level linkage rule". After manual confirmation, it is changed from single table rule 2 to quasi-rule 2.

[0048] Table 2

[0049] provincial prefecture-level cities districts and counties Jiangsu Xuzhou Yunlong District Jiangsu Xuzhou Yunlong District Jiangsu Nanjing Qinhuai District Jiangsu Suzhou Gusu District

[0050] Step 5: Use a similarity algorithm to check if the quasi-rules found in multiple tables are the same. Check if the metadata corresponding to each quasi-rule in its respective table is similar or consistent. If they are similar or consistent, it means that the multiple tables where the quasi-rules were found have the same or similar quasi-rules. Multiple quasi-rules are merged into a general rule, which can be used for quality inspection of all data tables. For example, if quasi-rule one corresponds to the province and city metadata in table one, and quasi-rule two corresponds to the province and city metadata in table two, then quasi-rule one and quasi-rule two can be merged into a general rule for quality inspection of all data tables.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to obtain equivalent embodiments without departing from the scope of the technical solution of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the technical solution of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for automatically discovering data quality inspection rules, characterized in that, The method includes: S1. Single-table rule discovery: For data tables that have already been entered into the database, the system automatically performs rule discovery tasks and generates single-table rules, including: S11. For data tables that have already been entered into the database, the single-table rule discovery model analyzes the data content based on the system's built-in knowledge base; S12. If the confidence of field A and field B is ≥95% and the support is ≥1%, then a sub-rule between field A and field B is found; S13. When one or more sub-rules between field A and field B are found in a data table, a two-level cascading rule between field A and field B is formed, and a single-table rule is established based on the two-level cascading rule between field A and field B. S2. Quasi-rule conversion: Based on the current standards and experience of the system, the administrator judges whether the single table rule formed in the previous step has a two-level cascading relationship. If the single table rule has a two-level cascading relationship, it is manually confirmed. Then, the confirmed single table rule is used to perform quality inspection on the data table that found the single table rule. After the quality inspection is completed, the single table rule is converted into a quasi-rule. S3. Generate general rules: Perform similarity checks on all quasi-rules found in multiple data tables, merge identical or similar quasi-rules from multiple data tables to generate general rules, and use the generated general rules for quality inspection of all data tables.

2. The method for automatically discovering data quality inspection rules according to claim 1, characterized in that, The knowledge base built into the system in step S11 includes: a unified management dictionary standard and quality knowledge.

3. The method for automatically discovering data quality inspection rules according to claim 1, characterized in that, In step S12, the confidence level of field A and field B refers to the proportion of data where both field A and field B meet the conditions out of the total data where field A meets the conditions; the support level of field A and field B refers to the proportion of data where field A meets the conditions out of the total data.

4. The method for automatically discovering data quality inspection rules according to claim 1, characterized in that, Step S3 involves performing a similarity check on all quasi-rules found in multiple data tables. The similarity check method includes: (1) Generate an initial rule set: Incorporate all quasi-rules found in multiple data tables into the initial rule set that needs to be merged; (2) Rule sorting: Sort the quasi-rules in the initial rule set according to the confidence and support indices of the quasi-rules; (3) Rule matching: Compare each quasi-rule to see if they have similar conditions and conclusions. If two or more quasi-rules have similar conditions and conclusions, they can be merged to generate a general rule.

5. A system for automatically discovering data quality inspection rules, characterized in that, The system includes: Single-table rule discovery module: Used to automatically perform rule discovery tasks on data tables that have been entered into the database, and generate single-table rules, including: (1) For data tables that have been entered into the database, the single-table rule discovery model analyzes the data content based on the system's built-in knowledge base; (2) If the confidence of field A and field B is ≥95% and the support is ≥1%, then a sub-rule between field A and field B is found; (3) When a single or multiple sub-rules between field A and field B are found in a data table, a second-level cascading rule between field A and field B is formed, and a single-table rule is established based on the second-level cascading rule between field A and field B. Quasi-rule conversion module: Used to determine whether a single table rule has a two-level cascading relationship and to confirm it. After the confirmation, the single table rule is used to perform quality inspection on the data table that has the single table rule. After the quality inspection is completed, the single table rule is converted into a quasi-rule. General rule generation module: It is used to perform similarity checks on all quasi-rules found in multiple data tables. Identical or similar quasi-rules from multiple data tables are merged to generate general rules, which are then used for quality inspection of all data tables.

6. The system for automatically discovering data quality inspection rules according to claim 5, characterized in that, The general rule generation module includes: Initial rule set generation submodule: used to incorporate all quasi-rules found in multiple data tables into the initial rule set that needs to be merged; Rule sorting submodule: Used to sort the quasi-rules in the initial rule set according to the confidence and support indicators of the quasi-rules; The rule matching submodule is used to compare quasi-rules one by one to see if they have similar conditions and conclusions. If two or more quasi-rules have similar conditions and conclusions, they are merged to generate a general rule.

7. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for automatically discovering data quality control rules according to any one of claims 1-4.

Images