Data quality rule intelligent generation method, data quality intelligent auditing method and computer program product

Abstract

The invention discloses a data quality rule intelligent generation method, a data quality intelligent auditing method and a computer program product. The generation method comprises the following steps: acquiring a data standard specification document in an industry and / or an enterprise, and extracting a plurality of data terms in data required to be audited; a standard specification corresponding to each data term is extracted from the data standard specification document, and a data quality business rule of the data term is obtained according to the standard specification; according to each data quality business rule, determining a business object and a check function template corresponding to the business object; and for each data quality business rule, obtaining the table name of the table where the business object is located and the field corresponding to the business object from the metadata of the data needing to be audited, and filling the table name and the field into the corresponding checking function template to generate the data quality rule. According to the method and the device, the data quality rule can be automatically generated for the data needing to be subjected to quality auditing without defining feature description of field-level data in metadata of the data by technicians.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality auditing, and in particular to a method for intelligently generating data quality rules, a method for intelligently auditing data quality, and a computer program product. Background Art

[0002] The production and operation activities of an enterprise will generate a large amount of business data. In order to ensure the quality of business data, enterprises usually conduct quality management on business data and regularly check the quality of business data; if any quality problems are found in business data, targeted rectification measures will be given. The way to check the quality of business data in the industry is generally: business personnel sort out data quality business rules based on their business experience, and then technical personnel program the corresponding data quality rules (SQL scripts) based on the data quality business rules, and upload the data quality rules to the database system; when the inspection time point is reached, the database system will automatically execute the data quality rules to conduct quality audits on the business data. If any quality problems are found in the business data, these problem data will be output. The above method requires cooperation between business personnel and technical personnel, and requires manual sorting of data quality business rules and manual programming to form data quality rules, which has high labor costs and low efficiency. The industry has proposed some solutions for automatically generating data quality rules. For example, Chinese patent document CN114896234A discloses a method for generating data quality rules: obtaining the data quality check classification applicable to the data according to the type of data, constructing the characteristic attributes of the data and determining the rule template corresponding to the characteristic attributes according to the data quality check classification and the (manually defined) characteristic description of the data; placing the data into the above rule template to generate data quality rules that match the characteristic attributes of the data. Although the method of the above patent document realizes the automatic generation of data quality rules, it requires technicians to define the characteristic description of each field-level data in detail in the metadata of the data in advance, so that the system can automatically use the characteristic description in the metadata of the data to automatically construct the characteristic attributes of the data and its rule template, and then automatically generate the data quality rules of the data. Summary of the invention

[0003] The purpose of the present invention is to provide a method for intelligently generating data quality rules, a method for intelligently auditing data quality based on the method for intelligently generating data quality rules, and a computer program product that can implement any one of the methods for intelligently generating data quality rules and the method for intelligently auditing data quality when executed. The method for intelligently generating data quality rules can automatically generate data quality rules for data that need to be quality audited without the need for technicians to define feature descriptions of field-level data in the metadata of the data.

[0004] In order to achieve the above object, the present invention provides a method for intelligently generating data quality rules, comprising the following steps: S1. Obtain data standard specification documents within the industry and / or enterprise, and extract multiple data terms contained in the data to be audited; S2. For each extracted data term, extract the corresponding standard specification from the data standard specification document according to the data term, and obtain the corresponding data quality business rule according to the standard specification; S3. According to each data quality business rule, determine its business object and its corresponding preset check function template, wherein the table name of the check object and the check object field of the check function template are left blank; S4. For each data quality business rule, obtain the table name of the table where the business object of the data quality business rule is located and the field corresponding to the business object from the metadata of the data to be audited; S5. For each data quality business rule, take its business object as the inspection object, fill the table name of the business object in the data to be audited and the field corresponding to the business object into the inspection function template corresponding to the data quality business rule, and generate the corresponding data quality rule.

[0005] Further: In S1, the data standard specification document is vectorized; The S2 system extracts the standard specifications from the vectorized data standard document.

[0006] The present invention also provides an intelligent data quality audit method, which generates data quality rules for the data to be audited according to the intelligent data quality rule generation method as described above, and then executes the audit steps, wherein the audit steps include S6, automatically executing the above data quality rules to perform quality audit on the data to obtain details of the problem data.

[0007] Furthermore, in S6, the problem data details are output to the user.

[0008] Furthermore, the auditing step includes S7, for the problem data details, classifying and counting the problem data according to the preset data quality analysis dimensions, and filling the number of problem data obtained by classification and statistics under each dimension into the preset data quality report display template to generate a data quality report.

[0009] Furthermore, the auditing step includes S8, dividing each problem data in the problem data details into multiple problem data tables according to the departments to which they belong, putting each problem data table into a preset work order template to generate a corresponding rectification work order, and sending it to the person in charge of the department to which it belongs.

[0010] Furthermore, the auditing step includes S9, after sending the rectification work order for a preset period of time, executing the data quality rules involved in the problem data details to determine the rectification status of the problem data.

[0011] The present invention also provides a computer program product, including a computer program, which is executed by a processor to implement the above-mentioned data quality rule intelligent generation method or the above-mentioned data quality intelligent audit method.

[0012] The intelligent data quality rule generation method S2 provided by the present invention automatically extracts the standard specifications corresponding to each data term from the data standard specification document, and then obtains the corresponding data quality business rules according to the standard specifications corresponding to the data term, and then S3-S5 automatically generates the corresponding data quality rules (SQL scripts) by using the preset check function template corresponding to each data quality business rule and the fields corresponding to the business objects of the business rules. The present invention can automatically generate data quality rules for data that need to be quality audited without the need for technicians to define the feature description of field-level data in the metadata of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flowchart of the intelligent generation method of data quality rules.

[0014] Figure 2 It is a flowchart of the intelligent data quality audit method. DETAILED DESCRIPTION

[0015] The present invention is further described in detail below in conjunction with specific implementation methods.

[0016] In this embodiment, the data quality intelligent audit method is programmed by computer code to form a computer program product, and the computer program product is executed by a processor to achieve the following Figure 2 The data quality intelligent audit method shown in the figure. The following takes the execution of the data quality intelligent audit method by a computer processor as an example. The computer first performs the following steps according to Figure 2 S1-S5 (i.e. Figure 1 The data quality rules are generated by the computer according to the intelligent generation method of data quality rules shown in the figure. Figure 2 The data quality audit is performed in S6-S10. The execution process of this method is described in detail below.

[0017] S1. Obtain data standard specification documents within the industry and / or enterprise, and extract multiple data terms contained in the data to be audited.

[0018] Many industries will have some common data standard specification documents formulated by relevant industry organizations in the industry. Some enterprises will also formulate some data standard specification documents of their own enterprises based on the data standard specification documents in the industry according to their own business needs. Data standard specification documents are usually in word document format or excel document format. The user uploads the existing data standard specification documents in the industry and the existing data standard specification documents in the enterprise to the computer, and the computer obtains the various data standard specification documents uploaded by the user. In other embodiments, the computer can be changed to automatically obtain the existing data standard specification documents in the industry and the enterprise through the relevant API interface. In a non-preferred embodiment, the computer can be changed to only obtain the existing data standard specification documents in the industry or only obtain the existing data standard specification documents in the enterprise. After the computer obtains the data standard specification document, the data standard specification document is vectorized, and then the vectorized data standard document is stored in the vector database. The user selects the data to be audited in the computer, and the computer extracts the multiple data terms contained in the data to be audited. Data terms are, for example, identity card number, name, gender, age, enterprise noun, unified social credit code, etc.

[0019] S2. For each extracted data term, extract the corresponding standard specification from the data standard specification document according to the data term, and obtain the corresponding data quality business rules according to the standard specification.

[0020] For each extracted data term, the computer extracts the standard specification corresponding to the data term from each data standard specification document. After the computer extracts the standard specification corresponding to each data term, it obtains the corresponding data quality business rule based on the standard specification. In the data standard specification document, the standard specification of the data term is usually described by key-value value. For example, for the data term of ID card number, a data standard specification document uses "ID card number" as the key and the standard specification of ID card number "ID card number is 18 digits" as the value. Taking the standard specification of ID card number as an example, the computer uses ID card number as the extraction keyword and extracts the value "ID card number is 18 digits" corresponding to the key "ID card number" from the data standard specification document, thus extracting the standard specification of ID card number. The computer then uses natural language processing technology to obtain the data quality business rule of ID card number "ID card number length is equal to 18 digits" based on the standard specification. S3. According to each data quality business rule, determine its business object and its corresponding preset check function template, wherein the table name of the check object and the check object field of the check function template are left blank.

[0021] After the computer obtains the data quality business rules corresponding to each data term, it determines the business object of each data quality business rule and the corresponding preset check function template. For the data quality business rule of the ID card number "ID card number length is equal to 18 digits", the computer uses the entity "ID card number" in the data quality business rule as its business object, and then uses the keywords "ID card number", "length", "equal to", and "18 digits" in the data quality business rule to find the corresponding check function template "select * from xx table where long (xx) <> 18" from the check function database. In the check function template, xx table refers to the table where the check object is located. The table name is left blank and the table name is to be filled in; (xx) refers to the check object field. It is left blank and the check object field is to be filled in.

[0022] S4. For each data quality business rule, obtain the table name of the table where the business object of the data quality business rule is located and the field corresponding to the business object from the metadata of the data to be audited.

[0023] After obtaining the business objects and check function templates of each data quality business rule, the computer obtains the fields corresponding to the business objects of each data quality business rule from the metadata of the data to be audited. For example, for the data quality business rule "ID number length equals 18 digits", it is now necessary to audit the data in the database. The computer finds the table name "Employee" of the table where the business object "ID number" is located and its corresponding field "card_no" in the table from the metadata of the database.

[0024] S5. For each data quality business rule, take its business object as the inspection object, fill the table name of the business object in the data to be audited and the field corresponding to the business object into the inspection function template corresponding to the data quality business rule, and generate the corresponding data quality rule.

[0025] After the computer obtains the table name of the table where the business object of each data quality business rule is located and the field corresponding to the business object, it takes the business object of each data quality business rule as the inspection object, fills the table name where the business object is located in the data to be audited and the field corresponding to the business object into the check function template corresponding to the data quality business rule, and generates the corresponding data quality rule. For example, for the data quality business rule "The length of the ID card number is equal to 18 digits", the computer takes the ID card number as the inspection object, fills the table name "Employee" where the ID card number is located in the database and the field "card_no" corresponding to the table into the check function template "select * from xx table where long(xx)<>18" corresponding to the data quality business rule, and generates the corresponding data quality rule "select * from Employee table where long(card_no)<>18".

[0026] S6. Automatically execute the above data quality rules to audit the data quality and obtain the details of the problematic data.

[0027] The computer executes the generated data quality rules on the data to be audited in sequence, implements data quality audit, obtains the details of the problematic data, and then outputs the details of the problematic data to the user. For example, after executing the data quality rule "select * from Employee table where long(card_no)<>18" on 10,000 data, 100 data are obtained whose ID card number length is not 18 digits. These 100 data are problematic data, and the details of these 100 data are output to the user.

[0028] S7. For the problem data details, classify and count the problem data according to the preset data quality analysis dimensions, and fill the number of problem data obtained by classification and statistics under each dimension into the preset data quality report display template to generate a data quality report.

[0029] In this embodiment, the user can select multiple data quality analysis dimensions in advance in the computer. For example, for the problem data details of the ID card number not equal to 18 digits, the user can pre-set the company main body classification statistical analysis dimension and the city classification statistical analysis dimension. After the computer executes the data quality rule "select * from Employee table where long (card_no) <> 18" to perform quality audit on the data to obtain the problem data details, on the one hand, the number of problem data of each branch company is counted according to the company main body classification statistical analysis dimension set by the user, and on the other hand, the number of problem data of each city is counted according to the city classification statistical analysis dimension set by the user; then, on the one hand, the number of problem data of each branch company is filled into the display chart corresponding to the company main body classification statistical analysis dimension of the preset data quality report display template to generate the company main body classification statistical chart, and on the other hand, the number of problem data of each city is filled into the display chart corresponding to the city classification statistical analysis dimension of the preset data quality report display template to generate the city classification statistical chart; in this way, the data quality report is generated. Preferably, the data quality report display template is provided with summary and analysis statistical indicators, such as error rate, time trend of problem occurrence, the top three data quality rule types with the highest error rate, institutional departments with high error rates, data sources with the highest proportion of problematic data, and other statistical indicators. After generating the corresponding statistical charts, the computer also automatically executes the calculation scripts of these statistical indicators to calculate the corresponding statistical indicator values, and then fills the values ​​of each statistical indicator into the data quality report display template to obtain the final data quality report.

[0030] S8. Divide each problem data in the problem data details into multiple problem data tables according to the departments to which they belong, put each problem data table into a preset work order template to generate a corresponding rectification work order, and send it to the person in charge of the department to which it belongs.

[0031] After generating the data quality report, the computer also initiates the corresponding rectification work order according to the problem data. The specific process is as follows: the computer divides each problem data in the problem data details into multiple problem data tables according to the department to which it belongs. For example, among the 100 problem data whose ID card number is not equal to 18 digits, 30 problem data belong to department A, 20 problem data belong to department B, and 50 problem data belong to department C. The computer divides these 100 problem data into 3 problem data tables; then each problem data table is put into the preset work order template to generate the corresponding rectification work order, that is, 3 rectification work orders a, b, and c are generated for the 3 problem data tables respectively, rectification work order a is sent to the person in charge of department A, rectification work order b is sent to the person in charge of department B, and rectification work order c is sent to the person in charge of department C. The persons in charge of departments A, B, and C rectify the problem data according to the rectification work orders.

[0032] S9. After the preset time has passed since the rectification work order was sent, the data quality rules involved will be executed on the problem data details to determine the rectification status of the problem data.

[0033] One week after the rectification work order is issued to the person in charge of the corresponding department, the computer executes the data quality rule "select * from Employee table where long(card_no)<>18" for the problem data details, such as 100 problem data details whose ID card numbers are not equal to 18 digits. If 0 problem data are obtained after execution, it means that the problem data has been rectified. If the execution finds that there are still problem data, it means that the problem data has not been rectified. The computer can initiate a rectification urging work order for the remaining problem data. The work order initiation method refers to S8 above and will not be repeated here.

[0034] The above is only an implementation method of the invention, and does not limit the scope of patent protection. Those skilled in the art can make non-substantial changes or substitutions based on the invention, which still fall within the scope of patent protection.

Claims

1. A method for intelligently generating data quality rules, characterized in that: The following steps are involved: S1. Obtain data standard specification documents within the industry and / or enterprise, and extract multiple data terms contained in the data to be audited; S2. For each extracted data term, extract the corresponding standard specification from the data standard specification document according to the data term, and obtain the corresponding data quality business rule according to the standard specification; S3. According to each data quality business rule, determine its business object and its corresponding preset check function template, wherein the table name of the check object and the check object field of the check function template are left blank; S4. For each data quality business rule, obtain the table name of the table where the business object of the data quality business rule is located and the field corresponding to the business object from the metadata of the data to be audited; S5. For each data quality business rule, take its business object as the inspection object, fill the table name of the business object in the data to be audited and the field corresponding to the business object into the inspection function template corresponding to the data quality business rule, and generate the corresponding data quality rule.

2. The method for intelligently generating data quality rules according to claim 1, characterized in that: In S1, the data standard specification document is vectorized; The S2 system extracts the standard specifications from the vectorized data standard document.

3. A data quality intelligent audit method, characterized in that: Generate data quality rules for the data to be audited according to the intelligent data quality rule generation method as described in claim 1 or 2, and then execute the audit step, wherein the audit step includes S6, automatically executing the above data quality rules to perform quality audit on the data to obtain details of the problem data.

4. The data quality intelligent audit method according to claim 3, characterized in that: In S6, the problem data details are output to the user.

5. The data quality intelligent audit method according to claim 3, characterized in that: The auditing steps include S7, for the problem data details, classifying and counting the problem data according to the preset data quality analysis dimensions, and filling the number of problem data obtained by classification and statistics under each dimension into the preset data quality report display template to generate a data quality report.

6. The data quality intelligent audit method according to claim 3, characterized in that: The auditing steps include S8, dividing each problem data in the problem data details into multiple problem data tables according to the departments to which they belong, putting each problem data table into a preset work order template to generate a corresponding rectification work order, and sending it to the person in charge of the department to which it belongs.

7. The data quality intelligent audit method according to claim 6, characterized in that: The audit steps include S9, after the preset time period has passed since the rectification work order was sent, the data quality rules involved in the problem data details are executed to determine the rectification status of the problem data.

8. A computer program product comprising a computer program, characterized in that The computer program is executed by a processor to implement the data quality rule intelligent generation method as described in claim 1 or 2 or the data quality intelligent audit method as described in any one of claims 3 to 7.

Images