Data Quality Inspection Framework Method and System

Abstract

The present invention provides a data quality inspection framework system and method, including: extracting and traversing the data of its own system and data sources for two-way comparison, obtaining the uncollected data, classifying it by theme according to business rules, forming different problem categories under the theme, and classifying it according to the scope of influence to obtain the classified problem data; extracting and traversing the classified problem data, comparing it with the data of its own system and data sources, and regarding the problem data that conforms to the actual data situation as accurate. The present invention realizes the digitization of data governance itself and forms a complete closed-loop scheme design. The present invention performs accuracy verification after the first comparison to ensure the accuracy of the verification. The data that fails the verification cannot be included in the inspection list. The present invention uses two-way data comparison, classifies problems by theme, and associates data quality problems with relevant parties, which is convenient for clarifying the main body of the relevant responsible party to obtain information.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality inspection, and specifically, to a data quality inspection framework method and system. Background Art

[0002] With the rapid development of enterprise digital construction and the continuous growth of data themes and quantities, a single system has been difficult to meet the needs of enterprise operations, including not only application systems for themes such as people, finance, materials, and management, but also development tool chain software and office tool chain software related to informatization and digital transformation. In daily management operations and R & D production, the cooperation of several or dozens of systems is relied on. System interaction is essentially data interaction. In the interaction process, there is often a situation where one system depends on the data of another system, that is, the data source of this system is another system. If the data in the data source system is updated, there will be data differences between the two systems, resulting in data quality problems.

[0003] Modern enterprises should regard data business as an underlying business, which supports the people, finance, and management in the background, R & D and production in the middle platform, and marketing and channels in the front platform. Improving data quality is the foundation for strengthening data business. Only high-quality data can provide effective digital transformation support for the front, middle, and back-end businesses.

[0004] In addition, during the digital transformation process of enterprises, various data quality problems between different themes and different systems are often faced. Therefore, between the master data hosting system and the data application system, a set of data quality inspection mechanisms need to be established to ensure that the data maintains high consistency between different systems and different theme data. Therefore, in order to improve data quality, on the one hand, various data quality problems should be checked and verified in a patterned manner; on the other hand, the categories, quantities, and governance effects of the identified data quality problems should be observable; so that relevant parties can make reasonable understandings based on the understanding of the data quality level and promote digital transformation more smoothly.

[0005] The prior art is to, after obtaining the data to be governed, according to certain data governance standards, call data governance tools to govern the data to be governed and generate governance results.

[0006] Patent document CN112988720A discloses a data governance method, device, equipment, and storage medium. In the data governance method, before writing the data into the data model, it is verified whether the data conforms to the data verification rules adapted to the data governance standards; if not, the data can be first subjected to data governance operations and then the governed data is written into the data model.

[0007] Patent document CN110008208A discloses a data governance method and system, which calls corresponding data governance tools according to data governance tasks, and governs the data to be governed according to the data governance tools to generate governance results.

[0008] However, the prior art does not provide a patterned way for inspection and verification. Summary of the Invention

[0009] Aiming at the defects in the prior art, the purpose of the present invention is to provide a data quality inspection framework method and system.

[0010] A data quality inspection system provided by the present invention includes:

[0011] Data extraction module: Extract data from its own system and data sources;

[0012] Data comparison module: Traverse the data in its own system and compare it with the data in the data source to obtain the data not included in the data source; Traverse the data in the data source and compare it with the data in its own system to obtain the data not included in its own system;

[0013] Problem classification module: Classify the uncollected data according to business rules by theme, form different problem categories under the theme, and classify them according to the scope of influence to obtain the classified problem data;

[0014] Accuracy verification module: Extract and traverse the classified problem data, compare it with the data in its own system and the data source, and consider the problem data that conforms to the actual data situation as accurate.

[0015] Preferably, the data comparison module includes:

[0016] First comparison module: Traverse the data in its own system and compare it with the data in the data source. Take the theme as the index condition to screen out the data in its own system that conforms to the theme index; Take the data uniqueness ID as the condition to retrieve the record with the matching ID in the data source obtained by data extraction; If no match is found, record the theme and institutional data corresponding to the ID;

[0017] Second comparison module: Traverse the data in the data source and compare it with the data in its own system. Analyze the obtained data source and form records stored in the local server. Then, take the theme as the index condition and the data uniqueness ID as the condition to traverse the records with matching IDs in its own system; If no match is found, record the theme and institutional data corresponding to the ID;

[0018] Third comparison module: One-way comparison of data status to check the consistency of data status. Among them, with the theme as the indexing condition, the data in its own system that conforms to the theme index is screened out; by reading the fields to be checked configured in the JSON file, the fields to be checked in its own system are traversed and compared with the data source; if the status is inconsistent, the corresponding theme, institutional data, and inconsistent status are recorded; among them, the comparison function is implemented through Nodejs and Promise, and is temporarily stored in the local server cache, maintained in the form of key-value pairs, waiting to be processed.

[0019] Preferably, the accuracy verification module traverses the problem details by parsing the problem details in the classified problem records, and reconfirms whether each classified problem record in the problem details conforms to the actual data situation, including:

[0020] If the classified problem is the lack of data in its own system, all records in the problem details should simultaneously meet the conditions of not existing in its own system and existing in the data source;

[0021] If the classified problem is that there is data in its own system that is not included in the data source, all records in the problem details should simultaneously meet the conditions of not existing in the data source and existing in its own system;

[0022] If the classified problem is inconsistent status, all records in the problem details should simultaneously meet the conditions of existing in both its own system and the data source, and having different statuses;

[0023] If the accuracy verification fails, the problem record will not enter the database, and the record that fails the accuracy verification will be printed in the log.

[0024] Preferably, the accuracy verification module is only performed after the first data comparison to ensure the accuracy of the first data comparison and avoid continuously generating incorrect results during routine comparisons; when extracting data, the relevant fields of id, affiliated institution, and responsible person will be extracted and cached in the server; after the data comparison is completed, all problem data will be classified by theme first, and then indexed by the affiliated institution, and each affiliated institution will form a record, storing the relevant fields of the id and responsible person of the problem data under this institution.

[0025] A data quality inspection method provided by the present invention includes:

[0026] Data extraction step: Extract the data of its own system and the data of the data source;

[0027] Data comparison step: Traverse the data of its own system, compare it with the data of the data source to obtain the data not included in the data source; traverse the data of the data source, compare it with the data of its own system to obtain the data not included in its own system;

[0028] Problem classification steps: For the uncollected data, classify it by theme according to business rules, form different problem categories under the theme, and classify it according to the scope of influence to obtain the classified problem data;

[0029] Accuracy verification steps: Extract and traverse the classified problem data, compare it with the data of its own system and the data source, and consider the problem data that conforms to the actual data situation as accurate.

[0030] Preferably, the data comparison step includes:

[0031] The first comparison step: Traverse the data of its own system, compare it with the data of the data source, use the theme as the index condition to filter out the data of its own system that conforms to the theme index; use the data uniqueness ID as the condition to retrieve the record with the matching ID in the data source obtained by data extraction; if no match is found, record the theme and institutional data corresponding to the ID;

[0032] The second comparison step: Traverse the data of the data source, compare it with the data of its own system, parse the obtained data source, form a record and store it in the local server, and then use the theme as the index condition and the data uniqueness ID as the condition to traverse the records with matching IDs in its own system; if no match is found, record the theme and institutional data corresponding to the ID;

[0033] The third comparison step: One-way comparison of data status, check the consistency of data status, where use the theme as the index condition to filter out the data of its own system that conforms to the theme index; by reading the fields to be checked configured in the JSON file, traverse the fields to be checked in its own system and compare them with the data source; if the status is inconsistent, record the corresponding theme, institutional data and the inconsistent status; among them, the comparison function is implemented by Nodejs and Promise and temporarily stored in the local server cache in the form of key-value pairs waiting to be processed.

[0034] Preferably, the accuracy verification step is to parse the problem details in the classified problem records, traverse the problem details and reconfirm whether each classified problem record in the problem details conforms to the actual data situation, including:

[0035] If the classified problem is the lack of data in its own system, all records in the problem details should simultaneously meet the conditions of not existing in its own system and existing in the data source;

[0036] If the classified problem is that there is data in its own system that is not included in the data source, all records in the problem details should simultaneously meet the conditions of not existing in the data source and existing in its own system;

[0037] If the classified problem is inconsistent status, all records in the problem details should simultaneously meet the conditions of existing in both its own system and the data source and having different statuses;

[0038] If the accuracy verification fails, the problem record will not be entered into the database, and the record of failed accuracy verification will be printed in the log.

[0039] Preferably, the accuracy verification step is only performed after the first data comparison to ensure the accuracy of the first data comparison and avoid continuously generating incorrect results during routine comparisons; when extracting data, relevant fields such as id, affiliated institution, and responsible person will be extracted and cached in the server; after the data comparison is completed, all problem data will be classified by theme first, and then indexed by the affiliated institution, with each affiliated institution forming a record to store the relevant fields of the problem data id and responsible person under that institution.

[0040] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the data quality inspection method are implemented.

[0041] According to an electronic device provided by the present invention, it includes the data quality inspection system described above, or includes the computer-readable storage medium storing the computer program described above.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention realizes the digitization of data governance itself, covering scenarios such as extraction, classification, inspection, association, and monitoring of data quality problems, and forms a complete closed-loop scheme design.

[0044] 2. The present invention performs accuracy verification after the first comparison to ensure the accuracy of the verification. Data that fails the verification cannot be included in the inspection list.

[0045] 3. The present invention adopts two-way data comparison, classifies problems by theme, and associates data quality problems with relevant parties, facilitating the identification of relevant responsible parties to obtain information and providing a modular way for inspection and verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] By reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings, other features, objects, and advantages of the present invention will become more apparent:

[0047] Figure 1 It is a schematic diagram of the working cooperation principle between modules of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several changes and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0049] As Figure 1 shown, a data quality inspection system provided by the present invention includes:

[0050] A data extraction module: extracts data from the target database of its own system and the data source. The data extraction module includes: a database extraction module: if the data source system provides an interface, directly call the interface, and cache the response data of the target database in the local server in the form of key-value pairs once or in batches; an interface extraction module: if the data source system provides a readable and writable database table or exposes an accessible database table, call the interface module responsible for establishing the interface, send a request to the target database of the data source in the database table, and cache the response data of the target database in the local server in the form of key-value pairs.

[0051] A data comparison module: performs two-way comparison on the data. The data comparison module includes: a first comparison module: responsible for initiating a comparison with the data source based on the data of its own system; specifically, screening out the data that conforms to the theme index in its own system with the theme as the index condition; retrieving the records with matching IDs in the data source obtained by data extraction with the data uniqueness ID as the condition. If no match is found, record the theme and institutional data corresponding to the ID; a second comparison module: responsible for initiating a comparison with the data of its own system based on the data of the data source; specifically, parsing the obtained data source, forming records and storing them in the local server, and then traversing the records with matching IDs in its own system with the theme as the index condition and the data uniqueness ID as the condition. If no match is found, record the theme and institutional data corresponding to the ID; a third comparison module: one-way comparison of data status, checking the consistency of data status (running status, validity period, etc.): screening out the data that conforms to the theme index in its own system with the theme as the index condition; traversing the fields to be checked in its own system and comparing them with the data source by reading the fields to be checked configured in the JSON file. If the status is inconsistent, record the corresponding theme, institutional data and the inconsistent status. Among them, the comparison function is implemented by Nodejs and Promise, and is temporarily stored in the local server cache, maintained in the form of key-value pairs, and waiting for processing.

[0052] Problem Classification Module: Classify the data obtained by the Data Comparison Module according to business rules by theme, form different problem categories under the theme, classify them according to the scope of influence (organization, stage, etc.), and form a record for each classified problem according to the user's configuration, and insert it into the data governance database table, and build an index through the auto-incrementing id.

[0053] Accuracy Verification Module: Extract and traverse the classified problem data, and compare it with the data of its own system and the data source. The accuracy verification module is only performed after the first data comparison to ensure the accuracy of the first data comparison and avoid continuously generating incorrect results during routine comparisons. The accuracy verification module parses the problem details in the classified problem record, traverses the details and reconfirms that each record in the details conforms to the actual data situation. For example, if the problem is the lack of data in its own system, all records in the details should simultaneously meet the conditions of not existing in its own system and existing in the data source; another example, if the problem is that there is data in its own system that is not included in the data source, all records in the details should simultaneously meet the conditions of not existing in the data source and existing in its own system; yet another example, if the problem is inconsistent status, all records in the details should simultaneously meet the conditions of existing in both its own system and the data source and having different statuses. If the accuracy verification fails, the problem record will not enter the database, and the records that fail the accuracy verification will be printed in the log.

[0054] A data quality inspection method provided by the present invention includes:

[0055] Step 1: Data extraction step. Data extraction is mainly divided into two types, database extraction and interface extraction. If the data source provides a readable and writable database table or exposes an accessible database, the data source data can be obtained through database extraction; if the database cannot be provided, the data can also be obtained singly or in batches through interface extraction.

[0056] Step 2: First data comparison step. The said step 2 includes: Step 2.1: Traverse the data of its own system, compare with the data source data, and obtain the data not included in the data source. Using the theme as the indexing condition, screen out the data in its own system that conforms to the theme index; using the data uniqueness ID as the condition, retrieve the records with matching IDs in the data source obtained by data extraction. If no match is found, record the theme and institutional data corresponding to the ID; Step 2.2: Traverse the data of the data source, compare with its own system, and obtain the data missing in its own system. Analyze the data of the obtained data source and form records stored in the local server. Then, using the theme as the indexing condition and the data uniqueness ID as the condition, traverse the records with matching IDs in its own system. If no match is found, record the theme and institutional data corresponding to the ID; Step 2.3: One-way comparison of data status, check the consistency of data status (running status, validity period, etc.): Using the theme as the indexing condition, screen out the data in its own system that conforms to the theme index; by reading the fields to be checked configured in the JSON file, traverse the fields to be checked in its own system and compare with the data source. If the status is inconsistent, record the corresponding theme, institutional data and the inconsistent status.

[0057] Step 3: Comparison accuracy verification step. Extract and traverse the classified problem data, and compare it with the data of its own system and the data source. When comparing records, if the problem data exists in its own system and does not exist in the data source system, or the problem data does not exist in its own system and exists in the data source system, the accuracy verification is successful. Otherwise, the verification fails and the data will not be stored in the database.

[0058] Step 4: Routine data comparison. The detailed steps are the same as those of the first data comparison. By configuring a scheduled task, the function of the first data comparison is executed regularly. The execution cycle of the scheduled task is configured in the environment variable.

[0059] The present invention will be described in more detail below.

[0060] In the data comparison of the present invention, two-way data comparison is adopted, such as in Step 2.1 and Step 2.2. First, traverse the data of its own system, compare with the data source data, and obtain the data not included in the data source; then traverse the data of the data source, compare with its own system, and obtain the data missing in its own system. Since not all the data of the data source needs to be stored in its own system, this part of the data needs to be further processed logically.

[0061] In the present invention, problems are classified by theme, and data quality problems are associated with relevant parties. Among them, when extracting data, relevant fields such as id, affiliated institution, and responsible person are extracted and cached in the server. After the data comparison is completed, all problem data will be classified by theme first, and then indexed by the affiliated institution. Each affiliated institution forms a record, storing relevant fields such as the id and responsible person of the problem data under the institution.

[0062] The data quality inspection system provided by the present invention can be implemented by executing the process steps of the data quality inspection method. That is, those skilled in the art can understand the data quality inspection method as a preferred implementation manner of the data quality inspection system.

[0063] Those skilled in the art know that in addition to implementing the system, device, and their respective modules provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system, device, and their respective modules provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers to achieve the same program. Therefore, the system, device, and their respective modules provided by the present invention can be regarded as a hardware component, and the modules included therein for implementing various programs can also be regarded as the structure within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing the method or the structure within the hardware component.

[0064] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined arbitrarily.

Claims

1. A data quality inspection system, characterized in that, Including: Data extraction module: Extract data from its own system and data sources. Data comparison module: Traverse the data of its own system and compare it with the data of the data source to obtain the data not included in the data source. Traverse the data of the data source and compare it with the data of its own system to obtain the data not included in its own system. Problem classification module: Classify the uncollected data according to business rules by theme, form different problem categories under the theme, and classify them according to the scope of influence to obtain the classified problem data. Accuracy verification module: Extract and traverse the classified problem data, compare it with the data of its own system and the data source, and consider the problem data that conforms to the actual data situation as accurate. The data comparison module includes: First comparison module: Traverse the data of its own system and compare it with the data of the data source. Using the theme as the index condition, filter out the data in its own system that conforms to the theme index; using the data uniqueness ID as the condition, retrieve the record with a matching ID in the data source obtained by data extraction; if no match is found, record the theme and institutional data corresponding to the ID. Second comparison module: Traverse the data of the data source and compare it with the data of its own system. Parse the obtained data source and form records stored in the local server. Then, using the theme as the index condition and the data uniqueness ID as the condition, traverse the records with matching IDs in its own system; if no match is found, record the theme and institutional data corresponding to the ID. Third comparison module: One-way comparison of data status, check the consistency of data status. Among them, using the theme as the index condition, filter out the data in its own system that conforms to the theme index; by reading the fields to be checked configured in the JSON file, traverse the fields to be checked in its own system and compare them with the data source; if the status is inconsistent, record the corresponding theme, institutional data and the inconsistent status; among them, the comparison function is implemented by Nodejs and Promise and temporarily stored in the local server cache in the form of key-value pairs waiting to be processed. The accuracy verification module traverses the problem details by parsing the problem details in the classified problem records and reconfirms whether each classified problem record in the problem details conforms to the actual data situation, including: If the classified problem is the lack of data in its own system, all records in the problem details should simultaneously meet the conditions of not existing in its own system and existing in the data source. If the classified problem is that there is data in its own system that is not included in the data source, all records in the problem details should simultaneously meet the conditions of not existing in the data source and existing in its own system. If the classified problem is inconsistent status, all records in the problem details should simultaneously meet the conditions of existing in both its own system and the data source and having different statuses. If the accuracy verification fails, the problem record will not enter the database, and the record that fails the accuracy verification will be printed in the log.

2. The data quality inspection system according to claim 1, characterized in that The accuracy verification module is only performed after the first data comparison to ensure the accuracy of the first data comparison and avoid continuously generating incorrect results during routine comparisons. When extracting data, relevant fields such as the ID, the affiliated institution, and the responsible person will be extracted and cached in the server. After the data comparison is completed, all problem data will first be classified by theme, and then indexed by the affiliated institution. Each affiliated institution will form a record storing the ID of the problem data under the institution and the relevant fields of the responsible person.

3. A data quality inspection method, characterized in that, Including: Data extraction steps: Extract data from its own system and data sources. Data comparison steps: Traverse the data in its own system and compare it with the data in the data source to obtain the data not included in the data source. Traverse the data in the data source and compare it with the data in its own system to obtain the data not included in its own system. Problem classification steps: Classify the uncollected data according to business rules by theme, form different problem categories under the theme, and classify them according to the scope of influence to obtain the classified problem data. Accuracy verification steps: Extract and traverse the classified problem data, compare it with the data in its own system and the data source, and consider the problem data that conforms to the actual data situation as accurate. The data comparison steps include: The first comparison step: Traverse the data in its own system and compare it with the data in the data source. Using the theme as the indexing condition, filter out the data in its own system that conforms to the theme index; using the data uniqueness ID as the condition, retrieve the record with a matching ID in the data source obtained by data extraction; if no match is found, record the theme and institutional data corresponding to the ID. The second comparison step: Traverse the data in the data source and compare it with the data in its own system. Parse the obtained data source and form a record to be stored in the local server. Then, using the theme as the indexing condition and the data uniqueness ID as the condition, traverse the records with matching IDs in its own system; if no match is found, record the theme and institutional data corresponding to the ID. The third comparison step: One-way comparison of data status to check the consistency of data status. Among them, using the theme as the indexing condition, filter out the data in its own system that conforms to the theme index; by reading the fields to be checked configured in the JSON file, traverse the fields to be checked in its own system and compare them with the data source; if the status is inconsistent, record the corresponding theme, institutional data, and the inconsistent status; among them, the comparison function is implemented through Nodejs and Promise and is temporarily stored in the local server cache in the form of key-value pairs waiting to be processed. The accuracy verification steps are achieved by parsing the problem details in the classified problem records, traversing the problem details, and reconfirming whether each classified problem record in the problem details conforms to the actual data situation, including: If the classified problem is the lack of data in its own system, all records in the problem details should simultaneously meet the conditions of not existing in its own system and existing in the data source. If the classified problem is that there is data in its own system that is not included in the data source, all records in the problem details should simultaneously meet the conditions of not existing in the data source and existing in its own system. If the classified problem is inconsistent status, all records in the problem details should simultaneously meet the conditions of existing in both its own system and the data source and having different statuses. If the accuracy verification fails, the problem record will not be entered into the database, and the record of failed accuracy verification will be printed in the log.

4. The data quality inspection method according to claim 3, wherein The accuracy verification step is only performed after the first data comparison to ensure the accuracy of the first data comparison and avoid continuously generating incorrect results during routine comparisons; During data extraction, relevant fields of the id, affiliated institution, and responsible person will be extracted and cached in the server; after the data comparison is completed, all problem data will first be classified by theme, and then indexed by the affiliated institution, with each affiliated institution forming a record to store the relevant fields of the id and responsible person of the problem data under that institution.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data quality inspection method described in claim 3 or 4.

6. An electronic device, characterized in that, It includes the data quality inspection system described in claim 1 or 2, or includes the computer-readable storage medium storing the computer program described in claim 5.

Images