Data quality inspection method and system based on dynamic rules and computer device

Abstract

This invention relates to a data quality inspection method, system, and computer device based on dynamic rules. First, rule information and identification strategy information are acquired. The rule information is parsed to obtain historical data that matches the records in the rule information. The matched data categories are then converted into a first rule information code using a hash function. Finally, the first rule information code is linked to a metadata information table to obtain an initial condition script for quality inspection. Simultaneously, the identification strategy information is parsed to obtain the identification period. Based on the identification period and the first rule information code, an exemption script for historical duplicate data is generated. Finally, the two scripts are combined to generate a quality inspection script, completing a rapid data quality inspection. By comparing the first rule information code, system data and table data that do not meet the quality inspection conditions can be directly removed from the data to be inspected. Therefore, this invention achieves the effects of improving the efficiency of data quality inspection and reducing resource consumption.

Description

Technical Field

[0001] This invention relates to data quality inspection, and in particular to data quality inspection methods, systems, and computer equipment based on dynamic rules. Background Technology

[0002] Traditional data quality checks have certain limitations. Currently, the most common traditional data quality checks used by enterprises mainly rely on manual checks or automated checks using system rules. Manual checks involve manually verifying data line by line, which is inefficient and prone to errors. Alternatively, checks based on preset system rules generate fixed scripts based on pre-defined rules (such as null value checks, format checks, and uniqueness checks) to periodically check data from the data source to identify data quality issues. However, this approach suffers from fixed validation rules, making it difficult to adapt to changes in business requirements. When the checked object (data table) changes, it cannot automatically adjust, resulting in low resource utilization. The test results are usually presented in preset dimensions, and when the target table dimensions change, it cannot adapt adaptively, requiring rule adjustments and leading to inefficiency. Furthermore, preset dimensions may retain unnecessary data, resulting in redundant data. This data redundancy not only increases the difficulty of analysis but also consumes significant database server memory resources. Summary of the Invention

[0003] Therefore, the present invention aims to provide a data quality inspection method, system, and computer device based on dynamic rules. This method isolates the rule-writing process and dynamically generates quality inspection scripts using a rule compiler based on rule configuration information, runtime environment information during the data quality inspection process, and generated historical data, thus forming dynamic rules. This allows for adaptive data identification when the inspection object (data table) changes, retaining only necessary data, thereby improving the efficiency of data quality inspection and reducing resource consumption.

[0004] This invention is achieved through the following technical solution:

[0005] On the one hand, the present invention provides a data quality inspection method based on dynamic rules, which includes: a configuration initialization stage, a dynamic rule generation stage, and a data quality inspection stage;

[0006] Execute during the configuration initialization phase:

[0007] S10: Obtain rule information and identification strategy information;

[0008] The dynamic rule generation phase includes: the initial condition generation phase for quality inspection and the script generation phase for exempting historical duplicate data.

[0009] The initial condition generation phase for quality inspection is executed as follows:

[0010] S211: Parse the rule information to generate a metadata information table, which contains the rule code, system name, table name, field name, and verification rule information of the target data;

[0011] S212: Match historical data according to the metadata information table, and generate a first rule information code by using the MD5 hash function for the matched rule code, system name, table name, and field name;

[0012] S213: Generate a quality check initial condition script based on the metadata information table and the first rule information encoding;

[0013] Exempting the historical duplicate data generation script from execution:

[0014] S22: Parse the identification strategy information to obtain the identification period, and encode the exemption historical duplicate data script according to the identification period and the first rule information;

[0015] Performed during the data quality check phase:

[0016] Step S30: Generate a quality inspection script based on the initial condition script for quality inspection and the script for exempting historical duplicate data;

[0017] Step S40: Extract valid target data from historical data according to the quality inspection script, and perform quality inspection according to the verification rule information to obtain problem record information.

[0018] Further, step S22 includes:

[0019] S221: Obtain the recognition period based on the recognition strategy information;

[0020] S222: Based on the first rule information code, extract the data in the historical data that are consistent with the rule information code and the first rule information code to form a result table;

[0021] S223: Generate an exemption history duplicate data script based on the identification period and the result table.

[0022] Furthermore, the configuration initialization phase also includes obtaining information on the responsible party and the responsibility strategy; the quality inspection initial condition generation phase also includes: a responsible person association script generation phase;

[0023] Execution step S23: Parse the accountability strategy information, associate each quality problem with the corresponding responsible person through LEFT JOIN syntax, and form a responsible person association script;

[0024] In the data quality inspection stage, step S30 is: generating a quality inspection script based on the initial condition script for quality inspection, the script for exempting historical duplicate data, and the script for associating responsible persons.

[0025] Furthermore, after step S40, the following is also included:

[0026] S50: Send each relevant issue to the corresponding responsible person based on the issue record information.

[0027] On the other hand, the present invention also provides a data quality inspection system based on dynamic rules, comprising:

[0028] Configure the initialization module, the dynamic rule generation module, and the data quality check module;

[0029] Configuration initialization module: used to obtain rule information and recognition strategy information;

[0030] The dynamic rule generation module includes a quality inspection initialization script generation submodule: used to parse the rule information, match historical data according to the parsing results, and generate a quality inspection initialization script;

[0031] Exemption from Historical Duplicate Data Script Submodule: Used to parse the identification strategy information to obtain the identification period, and form an exemption from historical duplicate data script based on the identification period;

[0032] The data quality inspection module includes a quality inspection script generation submodule: used to generate a quality inspection script based on the quality inspection initial condition script and the script for exempting historical duplicate data.

[0033] Data Inspection Submodule: Used to extract valid target data from historical data according to the quality inspection script, and to perform quality inspection according to the verification rules to obtain problem record information.

[0034] Furthermore, the quality inspection initialization script generation submodule includes:

[0035] Rule parsing unit: used to parse rule information to generate metadata information table, which contains the rule code, system name, table name, field name and verification rule information of the target data;

[0036] First rule information encoding acquisition unit: Based on the metadata information table, it matches historical data and generates a first rule information encoding by using the MD5 hash function for the matched rule encoding, system name, table name, and field name;

[0037] Check script initialization unit: used to generate quality check initial condition script based on metadata information table and the first rule information encoding.

[0038] Furthermore, the exemption historical duplicate data script submodule includes:

[0039] Identification period acquisition unit: used to acquire the identification period based on the identification strategy information;

[0040] Historical data filtering unit: used to extract data from historical data that are consistent with the rule information code and the first rule information code according to the first rule information code, and form a result table;

[0041] Exemption data script generation unit: Generates an exemption history duplicate data script based on the identification period and the result table.

[0042] Furthermore, the data quality inspection system based on dynamic rules also includes:

[0043] The accountability script generation submodule is used to parse the accountability strategy information after the configuration initialization module obtains the accountability subject information and accountability strategy information, and associate each quality problem with the corresponding responsible person through LEFT JOIN syntax to form a responsible person association script.

[0044] On the other hand, this application also provides a computer device, including:

[0045] At least one memory and at least one processor;

[0046] The memory is used to store one or more programs;

[0047] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a dynamic rule-based data quality inspection method as described in any of the above-mentioned claims.

[0048] This invention proposes a data quality inspection method based on dynamic rules. First, it acquires rule information and identification strategy information. By parsing the rule information, it obtains historical data that matches the records in the rule information. The matched data categories are then converted into a first rule information code using a hash function. Finally, the first rule information code is linked with a metadata information table to obtain an initial condition script for quality inspection. Simultaneously, it parses the identification strategy information to obtain the identification period. Based on the identification period and the first rule information code, it generates a script to exempt historical duplicate data. Finally, it combines the two scripts to generate a quality inspection script, completing a rapid data quality inspection. By comparing the first rule information code, it can directly remove system data and table data that do not meet the quality inspection conditions. Therefore, this invention improves the efficiency of data quality inspection and reduces resource consumption. Furthermore, a responsible person association script is added to the quality inspection script. Using the LEFT JOIN syntax, each quality issue is associated with a corresponding responsible person, allowing problematic data information to be linked to the relevant person in charge during data quality inspection, thus enabling rapid accountability after the inspection.

[0049] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0050] Figure 1 A structural block diagram of a data quality inspection system based on dynamic rules provided in one embodiment of the present invention;

[0051] Figure 2 To execute Figure 1 The flowchart shown is a data quality inspection method based on dynamic rules for the inspection system.

[0052] Figure 3 A flowchart for the script generation stage of the exemption from historical duplicate data provided by the present invention;

[0053] Figure 4 A structural block diagram of a data quality inspection system based on dynamic rules, provided in another embodiment of the present invention;

[0054] Figure 5 This is a structural block diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0056] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0057] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0058] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0059] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0060] Existing data quality inspection methods typically use pre-defined, fixed rules to periodically inspect certain fixed data tables in order to discover data problems. However, when the object of inspection (data table) changes, the inspection results are usually presented in a pre-defined dimension. Most importantly, when the dimensions of the target table change, the pre-defined data quality inspection method will fail.

[0061] Based on the aforementioned shortcomings, research has revealed that data quality inspection essentially involves checking existing data. Therefore, by matching the existing data to inspection categories before the inspection, the design dimensions required for the data inspection process can be determined, thus enabling dynamic setting of quality inspection rules. Furthermore, this invention introduces the MD5 hash function to calculate a unique rule information code for each table data to be inspected. By comparing the unique rule information codes during data matching and data inspection, target data from historical data can be directly filtered out, thereby reducing the utilization of computer resources based on dynamic rule-based data quality inspection.

[0062] To address this, the present invention provides a data quality inspection system based on dynamic rules, comprising a configuration initialization module 10, a dynamic rule generation module, and a data quality inspection module. The configuration initialization module primarily acquires configured rule information and identification strategy information. Subsequently, the quality inspection initialization script generation submodule within the dynamic rule generation module performs historical data matching, and converts the matched target data file into the corresponding first rule information encoding using an MD5 hash function. The historical duplicate data exemption script submodule obtains the identification period based on the identification strategy information, enabling it to filter out data outside the identification period in the subsequent data quality inspection process, reducing the amount of invalid data matching. Finally, the two scripts are combined to generate a corresponding quality inspection script, which completes the expected data quality inspection task and obtains problem record information. The execution flow of each component of this data quality inspection system during the data inspection task is as follows: Figure 2 As shown, it includes:

[0063] The process includes configuration initialization, dynamic rule generation, and data quality check.

[0064] During the configuration initialization phase, the configuration initialization module is used to execute step S10: obtaining rule information and identification strategy information.

[0065] The rule information includes multiple verification rules composed of system name, table name, field name and verification method. The identification strategy information specifies that the same data problem is checked only once within the identification period. In this invention, a seven-day identification period is used. If the identification period is exceeded, the data problem will be redistributed and the identification period will be recalculated.

[0066] The dynamic rule generation phase includes: the initial condition generation phase for quality inspection and the script generation phase for exempting historical duplicate data.

[0067] In the initial condition generation stage of quality inspection, the rule parsing unit 211, the first rule information encoding acquisition unit 212, and the inspection script initialization unit 213 of the quality inspection initialization script generation submodule generate the quality inspection initialization script. Specifically,

[0068] The rule parsing unit 211 is used to perform step S211: parsing rule information to generate a metadata information table, wherein the metadata information table contains the rule code, system name, table name, field name and verification rule information of the target data.

[0069] In a data quality inspection task, data from multiple data sources can be inspected simultaneously. Therefore, it is necessary to set separate verification rules for each data source to be inspected. This requires assigning a fixed code to each verification rule, known as a rule code. For example, the rule code for the first data inspection rule is 1. The corresponding metadata information table generated from this parsed rule is as follows:

[0070] Data inspection rule 1: User registration system, age table, 'userage', validation rule is age_min <usergae<age_max。

[0071] The first rule information encoding acquisition unit 2212 is used to perform step S212: matching historical data according to the metadata information table, and generating the first rule information encoding by using the MD5 hash function for the matched rule encoding, system name, table name, and field name.

[0072] Historical data refers to all data stored in the database or storage medium before data inspection. However, each data quality inspection does not require checking all data. Therefore, not all data is needed. By matching the data to be retrieved in the metadata information table, it is possible to find whether there is corresponding target data to be inspected in the historical data. If so, the file corresponding to the target data can be found by system name or table name. The unique rule information code of the file is found by using the MD5 hash function and used as the first rule information code for subsequent data verification.

[0073] The script initialization unit 213 is used to execute step S213: generate a quality check initial condition script based on the metadata information table and the first rule information encoding;

[0074] By linking the first rule information code with the rule code, the generated initial condition script for quality inspection can directly identify the rule code and find the corresponding first rule information code. Based on the first rule information code, it matches the corresponding file in the historical data, thereby reducing the resource consumption during secondary matching. Based on the matched file, the preset system name, table name, and field name can be found. Finally, the data quality inspection can be completed by verifying the rules.

[0075] During the stage of generating the historical duplicate data script for exemption, the historical duplicate data script submodule 22 executes step S22: parsing the identification strategy information to obtain the identification period, and encoding the historical duplicate data script for exemption according to the identification period and the first rule information.

[0076] Before performing data checks, in order to avoid repeatedly checking data that has already been checked, an identification cycle is bound to the first rule information encoding to avoid receiving the same data error repeatedly.

[0077] Please see Figure 3 Step S22 specifically includes:

[0078] S221: Obtain the recognition period based on the recognition strategy information;

[0079] S222: Based on the first rule information code, extract the data in the historical data that are consistent with the rule information code and the first rule information code to form a result table;

[0080] S223: Generate an exemption history duplicate data script based on the identification period and the result table.

[0081] Obtain the MD5 hash of all tuples in the result table for the recognition period where the current check time minus the historical check time is less than or equal to S205. Use NOT EXISTS (SELECT 1 FROM result table T WHERE T.check time - current check time >= recognition period AND T.MD5(full code) = current result.MD5(full code)) to exempt historical duplicate data.

[0082] Performed during the data quality check phase:

[0083] Step S30: Generate a quality inspection script based on the initial condition script for quality inspection and the script for exempting historical duplicate data.

[0084] The two scripts are combined to form the final quality inspection script.

[0085] Step S40: Extract valid target data from historical data according to the quality inspection script, and perform quality inspection according to the verification rule information to obtain problem record information.

[0086] The target data is filtered from historical data according to the quality inspection script, and then duplicate data that has already been identified is removed through the identification cycle. Finally, each data is checked according to the verification rules to obtain the final problem record information.

[0087] In another embodiment, to ensure automatic accountability after data inspection, this invention also adds an accountability script generation submodule to the dynamic rule-based data quality inspection system. This submodule, after the configuration initialization module obtains the accountability subject information and accountability strategy information, parses the accountability strategy information and uses LEFT JOIN syntax to associate each quality issue with its corresponding responsible person, forming a responsible person association script. Please refer to... Figure 4 and Figure 5 In this embodiment, the configuration initialization phase further includes obtaining information on the responsible party and the responsible strategy.

[0088] The accountability entity primarily records the relationship between accountability keywords and responsible persons. Multiple sets of accountability entities are maintained based on different keyword information. For example, a project accountability entity assigns responsibility to a project leader based on the project.

[0089] The initial conditions generation stage for quality inspection also includes: the generation stage of the responsible person association script; the accountability strategy information mainly specifies the problem identification cycle, such as not identifying data problems repeatedly within 1 week when they are first discovered.

[0090] The responsibility assignment script generation submodule 23 is used to execute step S23: parse the responsibility assignment strategy information, associate each quality problem with the corresponding responsible person through LEFTJOIN syntax, and form a responsible person association script.

[0091] For example, the LEFT JOIN syntax associates the corresponding responsible parties, and nested CASE WHEN Responsible Party 1 ISNOT NULL THEM Responsible Party 1 ELSE CASE WHERE Responsible Party 2 IS NOT NULL…END checks to match the responsible party for the data issue. This generates a script for associating responsible parties.

[0092] In the data quality inspection stage, step S30 is changed to: generating a quality inspection script based on the initial condition script for quality inspection, the script for exempting historical duplicate data, and the script for associating responsible persons.

[0093] Preferably, the data quality inspection system based on dynamic rules also includes an automatic accountability submodule for performing step S50: sending each relevant problem to the corresponding responsible person according to the problem record information.

[0094] Based on the aforementioned responsibility assignment script generation submodule, after generating the problem record information, it can directly locate the person responsible for the location data quality problem, thereby improving the efficiency of problem rectification.

[0095] This invention proposes a data quality inspection method based on dynamic rules. First, it acquires rule information and identification strategy information. By parsing the rule information, it obtains historical data that matches the records in the rule information. The matched data categories are then converted into a first rule information code using a hash function. Finally, the first rule information code is linked with a metadata information table to obtain an initial condition script for quality inspection. Simultaneously, it parses the identification strategy information to obtain the identification period. Based on the identification period and the first rule information code, it generates a script to exempt historical duplicate data. Finally, it combines the two scripts to generate a quality inspection script, completing a rapid data quality inspection. By comparing the first rule information code, it can directly remove system data and table data that do not meet the quality inspection conditions. Therefore, this invention improves the efficiency of data quality inspection and reduces resource consumption. Furthermore, a responsible person association script is added to the quality inspection script. Using the LEFT JOIN syntax, each quality issue is associated with a corresponding responsible person, allowing problematic data information to be linked to the relevant person in charge during data quality inspection, thus enabling rapid accountability after the inspection.

[0096] This invention can take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0097] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.

Claims

1. A data quality inspection method based on dynamic rules, characterized in that, include: The process includes configuration initialization, dynamic rule generation, and data quality check. Execute during the configuration initialization phase: S10: Obtain rule information and identification strategy information; The dynamic rule generation phase includes: the initial condition generation phase for quality inspection and the script generation phase for exempting historical duplicate data. The initial condition generation phase for quality inspection is executed as follows: S211: Parse the rule information to generate a metadata information table, which contains the rule code, system name, table name, field name, and verification rule information of the target data; S212: Match historical data according to the metadata information table, and generate a first rule information code by using the MD5 hash function for the matched rule code, system name, table name, and field name; S213: Generate a quality check initial condition script based on the metadata information table and the first rule information encoding; Exempting the historical duplicate data generation script from execution: S22: Parse the identification strategy information to obtain the identification period, and encode the exemption historical duplicate data script according to the identification period and the first rule information; Performed during the data quality check phase: Step S30: Generate a quality inspection script based on the initial condition script for quality inspection and the script for exempting historical duplicate data; Step S40: Extract valid target data from historical data according to the quality inspection script, and perform quality inspection according to the verification rule information to obtain problem record information.

2. The data quality inspection method based on dynamic rules according to claim 1, characterized in that, Step S22 includes: S221: Obtain the recognition period based on the recognition strategy information; S222: Based on the first rule information code, extract the data in the historical data that is consistent with the rule information code and the first rule information code to form a result table; S223: Generate an exemption history duplicate data script based on the identification period and the result table.

3. The data quality inspection method based on dynamic rules according to any one of claims 1-2, characterized in that: The configuration initialization phase also includes obtaining information on the responsible party and the responsibility strategy. The initial conditions generation stage for quality inspection also includes: the generation stage of the script associated with the responsible person; Execution step S23: Parse the accountability strategy information, associate each quality problem with the corresponding responsible person through LEFT JOIN syntax, and form a responsible person association script; In the data quality inspection stage, step S30 is: generating a quality inspection script based on the initial condition script for quality inspection, the script for exempting historical duplicate data, and the script for associating responsible persons.

4. The data quality inspection method based on dynamic rules according to claim 3, characterized in that, The process after step S40 also includes: S50: Send each relevant issue to the corresponding responsible person based on the issue record information.

5. A data quality inspection system based on dynamic rules, characterized in that, include: Configure the initialization module, the dynamic rule generation module, and the data quality check module; Configuration initialization module: used to obtain rule information and recognition strategy information; The dynamic rule generation module includes a quality check initialization script generation submodule and an exemption historical duplicate data script submodule; The quality inspection initialization script generation submodule includes: Rule parsing unit: used to parse rule information to generate metadata information table, which contains the rule code, system name, table name, field name and verification rule information of the target data; First rule information encoding acquisition unit: used to match historical data according to the metadata information table, and generate first rule information encoding by using MD5 hash function for the matched rule encoding, system name, table name and field name; The script initialization unit is used to generate a quality inspection initial condition script based on the metadata information table and the first rule information encoding. Exemption from Historical Duplicate Data Script Submodule: Used to parse the identification strategy information to obtain the identification period, and form an exemption from historical duplicate data script based on the identification period; The data quality inspection module includes a quality inspection script generation submodule: used to generate a quality inspection script based on the quality inspection initial condition script and the script for exempting historical duplicate data. Data Inspection Submodule: Used to extract valid target data from historical data according to the quality inspection script, and to perform quality inspection according to the verification rules to obtain problem record information.

6. The data quality inspection system based on dynamic rules according to claim 5, characterized in that, The exemption historical duplicate data script submodule includes: Identification period acquisition unit: used to acquire the identification period based on the identification strategy information; Historical data filtering unit: used to extract data from historical data that are consistent with the rule information code and the first rule information code according to the first rule information code, and form a result table; Exemption data script generation unit: Generates an exemption history duplicate data script based on the identification period and the result table.

7. The data quality inspection system based on dynamic rules according to any one of claims 5-6, characterized in that, Also includes: The accountability script generation submodule is used to parse the accountability strategy information after the configuration initialization module obtains the accountability subject information and accountability strategy information, and associate each quality problem with the corresponding responsible person through LEFT JOIN syntax to form a responsible person association script.

8. The data quality inspection system based on dynamic rules according to claim 7, characterized in that, Also includes: Automatic accountability submodule: Used to send each relevant issue to the corresponding responsible person based on the issue record information.

9. A computer device, characterized in that, include: At least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor performs the steps of the data quality inspection method based on dynamic rules as described in any one of claims 1-4.

Images