Data quality inspection method capable of being dynamically expanded

Abstract

The invention relates to a data quality inspection method capable of dynamically expanding, which can dynamically add, modify and delete a data quality inspection rule, dynamically store database connection information and the quality inspection rule in a database, and obtain the database connection information and the quality inspection rule from the database in real time when the quality inspection is executed; the execution process of quality inspection is dynamically scheduled according to a frequency defined by a user, and comprises the following steps: realizing a quality inspection rule management system, performing the quality inspection execution process: performing timed task scheduling, acquiring a quality inspection rule, connecting a database, executing data query, executing quality inspection logic and storing a quality inspection result, and performing the quality inspection result processing process: performing anomaly detection, generating a quality inspection report and sending the report. The problem that a traditional data quality inspection system is insufficient in rule expansibility, complex rule support, database type adaptability and exception handling efficiency is solved, and quality monitoring and verification of large-scale and multi-source heterogeneous data are achieved by dynamically registering database links and quality inspection rules and combining complex expression analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to a data management and quality control technology, in particular to a data quality inspection method for multi-type databases. BACKGROUND

[0002] With the development of informatization, various industries have generated massive amounts of data, and the timeliness, consistency and compliance of data are crucial to the reliability of business. Currently, the main data quality inspection implementation methods mainly include the following:

[0003] 1. Fixed rule-based quality inspection method:

[0004] Implementation: A set of fixed quality inspection rules (such as field format inspection, data range verification, etc.) are defined in advance, and data is inspected through batch processing or real-time execution.

[0005] Advantages: Simple implementation, suitable for structured and single rule data inspection scenarios.

[0006] Disadvantages: Lack of flexibility, unable to dynamically adjust rules, difficult to meet complex business requirements such as cross-table association inspection or multi-field condition verification.

[0007] 2. Custom script-based quality inspection method:

[0008] Implementation: Specific data inspection logic is implemented by writing scripts (such as SQL queries, Python scripts).

[0009] Advantages: Highly customizable rules can handle complex data inspection requirements.

[0010] Disadvantages: Requires high technical threshold, complex rule management and maintenance, high cost of repeated development, and not suitable for rapidly changing scenarios.

[0011] 3. Quality inspection method based on special tools or platforms:

[0012] Implementation: Use commercial or open source quality inspection tools (such as Talend Data Quality, Informatica Data Quality) for data quality inspection.

[0013] Advantages: Comprehensive functions, support for certain degree of rule configuration and cross-table inspection.

[0014] Disadvantages: Complex tools, high learning cost, unable to flexibly meet specific industry customization requirements, limited support for multiple database types.

[0015] Although the above technologies have solved the data quality inspection problem to a certain extent, there are still problems such as insufficient rule scalability, insufficient support for complex rules, poor database type adaptability, and delayed exception handling.

[0016] The traditional data quality inspection system still has the following shortcomings in practical applications:

[0017] 1. Insufficient rule scalability:

[0018] Traditional data quality inspection systems typically use static, predefined rule sets, preventing users from flexibly adding new quality inspection rules based on their needs. This is especially difficult to update and maintain when faced with ever-changing business needs, making them unable to adapt to rapidly changing environments.

[0019] 2. Insufficient support for complex rules:

[0020] Existing systems only support simple logical judgments or basic operations in quality inspection rules and are unable to handle complex expressions across tables and fields. When complex calculations, joins, and aggregations are required, these technologies often require additional development work, reducing the system's applicability and flexibility.

[0021] 3. Poor database type adaptability:

[0022] Currently, most data quality inspection systems can only support a specific type of database, and their support for multiple database types (such as relational databases, NoSQL databases, etc.) is relatively weak. Enterprises often need to perform complex database adaptation or redevelopment during use, which increases the complexity and maintenance costs of the system.

[0023] 4. Exception handling lags:

[0024] When traditional systems discover data quality issues, they lack a timely feedback mechanism and are unable to notify users in a timely manner for processing. As a result, potential quality issues cannot be resolved in a timely manner, affecting the efficiency and effectiveness of data governance. Summary of the Invention

[0025] To address the shortcomings of traditional data quality inspection systems, including insufficient rule extensibility, support for complex rules, database type adaptability, and exception handling efficiency, we propose a dynamically scalable data quality inspection method. By dynamically registering database links and quality inspection rules and combining them with complex expression parsing, this method enables quality monitoring and verification of large-scale, multi-source, heterogeneous data. This method is suitable for data governance and quality assurance in a variety of scenarios, including finance, healthcare, the internet, and manufacturing.

[0026] The technical solution of the present invention is:

[0027] A dynamically scalable data quality inspection method that can dynamically add, modify, and delete data quality inspection rules. This is achieved by storing database connection information and quality inspection rules in the database, and then retrieving them from the database in real time during quality inspection execution. The quality inspection execution process is dynamically scheduled based on a user-defined frequency and is divided into the following steps:

[0028] Step 1. Quality Inspection Initialization Process

[0029] Step 1.1 Implement the quality inspection rule management system

[0030] Step 1.1.1 The quality inspection rule management system includes an API or front-end interface, a quality inspection scheduling module, a quality inspection rule table, a database, a quality inspection result table, and a notification module, which implements the functions of adding, deleting, modifying, and checking database connection information, as well as adding, deleting, modifying, and checking quality inspection rules;

[0031] Step 1.1.2 The system provides an API or front-end interface to support users in managing database connection information and quality inspection rules;

[0032] Step 1.2 User registration database connection information

[0033] Step 1.2.1 The user enters the database type in the system interface;

[0034] Step 1.2.2 provides database connection information;

[0035] Step 1.2.3 provides the database name corresponding to the database connection;

[0036] After step 1.2.4 is submitted, the system verifies the validity of the connection. If the connection is successful, it is stored in the database connection management table;

[0037] Step 1.3 User Registration Quality Inspection Rules

[0038] Step 1.3.1 The user selects a registered database connection;

[0039] Step 1.3.2 User defines quality inspection rules;

[0040] Step 1.3.3 User sets execution cycle

[0041] Step 1.3.4: Store the rules in the quality inspection rule table and bind the database connection;

[0042] Step 2. Quality Inspection Execution Process

[0043] Step 2.1 Scheduled Tasks

[0044] Step 2.1.1 The quality inspection scheduling module triggers quality inspection tasks regularly according to the execution cycle set by the rules;

[0045] Step 2.1.2: The quality inspection rules to be executed are stored in the task queue;

[0046] Step 2.2 Obtain quality inspection rules

[0047] Step 2.2.1 Read the quality inspection rule table to obtain the rules that need to be executed currently;

[0048] Step 2.2.2 Obtain the database connection information corresponding to each rule;

[0049] Step 2.3 Database Connection

[0050] Step 2.3.1 Dynamically select the appropriate database connection driver based on the database type;

[0051] Step 2.3.2 Establish a connection based on the database connection information;

[0052] Step 2.3.3 If the connection fails, log the error and send an exception notification.

[0053] Step 2.4 Execute data query

[0054] Step 2.4.1 parse the SQL statement or query expression according to the quality inspection rules;

[0055] Step 2.4.2: Connect to the database, execute the query, and obtain the data;

[0056] If the query fails in step 2.4.3, an error is logged and a warning is triggered;

[0057] Step 2.5: Execute quality control logic

[0058] Data quality includes the quality standards of compliance, timeliness, integrity, uniqueness, and consistency. Compliance includes the validity of formats, types, domain values, and business rules. Timeliness includes the timeliness and speed of data updates, modifications, and retrieval operations. Integrity includes the absence of missing entities, attributes, records, and field values. Uniqueness includes the uniqueness of primary keys and candidate keys. Consistency includes unified data sources, unified data storage, and unified data caliber.

[0059] Step 2.6 Store quality inspection results

[0060] Step 2.6.1 Generate quality inspection results;

[0061] The results of step 2.6.2 are stored in the quality inspection result table;

[0062] Step 3. Quality inspection result processing flow

[0063] Step 3.1 Anomaly Detection

[0064] Step 3.1.1 If abnormal data is detected, an alarm is triggered;

[0065] Step 3.1.2 records the abnormal information and sends it to the notification module;

[0066] Step 3.2 Generate quality inspection report

[0067] Step 3.2.1 Generate a report based on the quality inspection results;

[0068] Step 3.2.2 Statistical analysis of abnormal ratio, qualified rate, and trend;

[0069] Step 3.3 Send report

[0070] Step 3.3.1 Push the report via communication tools;

[0071] Step 3.3.2 records the report sending log.

[0072] Furthermore, the timeliness of executing the quality inspection logic in step 2.5 is:

[0073] Step 2.5.1 Timely quality inspection

[0074] Step 2.5.1.1 Analyze the quality inspection rules and obtain the data update frequency;

[0075] Step 2.5.1.2: Construct a query statement based on the database type to obtain the timestamp field of the latest data;

[0076] Step 2.5.1.3: Get the current system time and calculate the corresponding time range;

[0077] Step 2.5.1.4 determines whether the latest data timestamp is within the set update cycle: if the latest data time is within the update cycle, the data timeliness is normal; if the latest data time exceeds the update cycle, the data timeliness is abnormal, the abnormal data is recorded and an alarm is triggered.

[0078] Furthermore, the compliance of the quality control logic in step 2.5 is:

[0079] Step 2.5.2 Compliance Inspection

[0080] Step 2.5.2.1 Parse the expression of the quality inspection rule and support the following calculation logic: Form 1: supports mathematical operations between fields; Form 2: supports logical operations; Form 3: supports predefined functions; Form 4: supports nested expressions combining the above operations;

[0081] Step 2.5.2.2 Select data based on the sampling method set by the rules: If set to full verification, the data requiring quality inspection is obtained based on the quality inspection rules and database connection information, and all data are traversed for calculation; if set to sampling verification, the data requiring quality inspection is obtained based on the quality inspection rules and database connection information, and a portion of the data is randomly sampled according to the set ratio for calculation;

[0082] Step 2.5.2.3 performs expression operations to determine whether the data complies with the rules: if all sampled / traversed data complies with the rules, the data compliance is normal; if there is data that does not comply with the rules, the data compliance is abnormal, the abnormal data is recorded, and an alarm is triggered.

[0083] The beneficial effects of the present invention are:

[0084] The present invention provides a full-process, efficient and flexible data quality inspection system and method. Through dynamic rule management, complex expression parsing, multi-type database compatibility and real-time exception reminders, it achieves high-quality inspection and real-time feedback of multi-source data, and has wide applicability and significant technical advantages.

[0085] It has the following advantages:

[0086] 1. Dynamic rule management: Supports dynamic addition, modification, and deletion of quality inspection rules without restarting or redeploying the system, greatly improving the flexibility of rule management.

[0087] 2. Complex expression parsing: Built-in support for complex expression parsing and execution can easily handle complex inspection scenarios such as cross-table verification, multi-conditional logic nesting, and dynamic variable replacement.

[0088] 3. Compatible with multiple database types: Through standardized interface design, it supports the connection and inspection of multiple types of databases (such as relational databases and NoSQL databases).

[0089] 4. Real-time exception reminders: Based on the execution results of quality inspection tasks, the system can automatically generate exception reports and send reminders in a timely manner to facilitate rapid processing of data problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 This is a data quality inspection flow chart of the present invention. DETAILED DESCRIPTION

[0091] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0092] A method and system for dynamic data quality inspection, the core of which is to dynamically add, modify and delete data quality inspection rules, and the dynamic implementation is to save the database connection information and quality inspection rules in the database, and then obtain the database connection information and quality inspection rules from the database in real time during quality inspection execution. The execution process of quality inspection is dynamically scheduled according to the frequency defined by the user, which includes the following steps:

[0093] Step 1. Quality inspection initialization process

[0094] Step 1.1 Implement quality inspection rule management system

[0095] Step 1.1.1 The core of the system is to realize the functions of adding, deleting, modifying and querying the database connection information, and the functions of adding, deleting, modifying and querying the quality inspection rules. The quality inspection rule management system includes API or front-end interface, quality inspection scheduling module, quality inspection rule table, database, quality inspection result table, and notification module.

[0096] Step 1.1.2 The system provides API or front-end interface to support user management of database connection information and quality inspection rules.

[0097] Step 1.2 User registers database connection information

[0098] Step 1.2.1 The user inputs the database type (such as MySQL, MongoDB, ES) in the system interface.

[0099] Step 1.2.2 Provide database connection address, port, username, password and other information.

[0100] Step 1.2.3 Provide the database name corresponding to the database connection.

[0101] Step 1.2.4 After submission, the system verifies the connection validity, and if the connection is successful, it is stored in the database connection management table.

[0102] Step 1.3 User registers quality inspection rules

[0103] Step 1.3.1 The user selects the registered database connection.

[0104] Step 1.3.2 The user defines quality inspection rules, including timeliness rules, consistency rules, compliance rules, etc.

[0105] Step 1.3.3 The user sets the execution period, such as every 1 hour or every day.

[0106] Step 1.3.4 The rules are stored in the quality inspection rule table and bound to the database connection.

[0107] Step 2. Quality inspection execution process

[0108] Step 2.1 Timed task scheduling

[0109] Step 2.1.1. The quality inspection scheduling module triggers the quality inspection task periodically according to the execution period set by the rules.

[0110] Step 2.1.2. The task queue stores the quality inspection rules to be executed.

[0111] Step 2.2. Obtain quality inspection rules

[0112] Step 2.2.1. Read the quality inspection rule table to obtain the rules to be executed currently.

[0113] Step 2.2.2. Obtain the database connection information corresponding to each rule.

[0114] Step 2.3. Database connection

[0115] Step 2.3.1. Dynamically select the appropriate database connection driver according to the database type.

[0116] Step 2.3.2. Establish a connection according to the database connection information.

[0117] Step 2.3.3. If the connection fails, record the log and send an exception notification.

[0118] Step 2.4. Execute data query

[0119] Step 2.4.1. Parse the SQL statement or query expression according to the quality inspection rules.

[0120] Step 2.4.2. Connect to the database, execute the query, and obtain the data.

[0121] Step 2.4.3. If the query fails, record the error and trigger a warning.

[0122] Step 2.5. Execute quality inspection logic

[0123] Data quality includes, but is not limited to, compliance (mainly including format, type, domain value, and validity of business rules), timeliness (referring to the timeliness and speed of data updating, modifying, and extracting operations), completeness (mainly including entity absence, attribute absence, record absence, and field value absence), uniqueness (referring to primary key uniqueness and candidate key uniqueness), consistency (referring to unified data source, unified data storage, and unified data caliber), and other quality standards. The following takes timeliness and compliance as examples to illustrate how to execute quality inspection logic, and the same applies to others.

[0124] Step 2.5.1. Timeliness quality inspection

[0125] Step 2.5.1.1. Parse the quality inspection rules to obtain the update frequency of the data (such as daily, weekly, and monthly).

[0126] Step 2.5.1.2 Depending on the database type, construct the query statement to obtain the timestamp field of the latest data.

[0127] Step 2.5.1.3 Obtain the current system time and calculate the corresponding time range (e.g., one month before the current time).

[0128] Step 2.5.1.4 Determine whether the latest data timestamp is within the set update period: if the latest data time is within the update period, the data timeliness is normal; if the latest data time exceeds the update period, the data timeliness is abnormal, record the abnormal data and trigger an alarm.

[0129] Step 2.5.2 Compliance quality inspection

[0130] Step 2.5.2.1 Analyze the expression of the quality inspection rule, supporting the following calculation logic:

[0131] (Form 1) Support mathematical operations between fields (such as field A + field B).

[0132] (Form 2) Support logical operations (such as AND, OR, NOT).

[0133] (Form 3) Support predefined functions (fixed logic implemented by code, such as determining whether the data range is within the defined range).

[0134] (Form 4) Support combined nested expressions of the above various operations.

[0135] Step 2.5.2.2 Select data according to the sampling method set by the rule: if set to full verification, obtain the data to be inspected through the quality inspection rule and database connection information, and traverse all data for calculation; if set to sampling verification, obtain the data to be inspected through the quality inspection rule and database connection information, and randomly select a portion of the data according to the set proportion for calculation.

[0136] Step 2.5.2.3 Execute expression operations to determine whether the data meets the rules: if all sampled / traversed data meet the rules, the data compliance is normal; if there is data that does not meet the rules, the data compliance is abnormal, record the abnormal data and trigger an alarm.

[0137] Step 2.6 Store quality inspection results

[0138] Step 2.6.1 Generate quality inspection results, including: rule ID, quality inspection time, whether passed, and abnormal data details

[0139] Step 2.6.2 Store the results in the quality inspection result table.

[0140] Step 3. Quality inspection result processing flow

[0141] Step 3.1 Abnormality detection

[0142] Step 3.1.1 If abnormal data is detected, an alarm is triggered.

[0143] Step 3.1.2 Abnormal information is recorded and sent to a notification module.

[0144] Step 3.2 Generation of quality inspection report

[0145] Step 3.2.1 According to the quality inspection result, daily report, weekly report and other reports are generated.

[0146] Step 3.2.2 Abnormality ratio, qualification rate, trend analysis and the like are counted.

[0147] Step 3.3 Sending of report

[0148] Step 3.3.1 The report is pushed through email, short message, instant communication tool (such as WeChat and DingDing).

[0149] Step 3.3.2 Report sending log is recorded.

[0150] The above-described embodiments only express one embodiment of the present application, which is described in a more specific and detailed manner, but should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A dynamically scalable data quality inspection method, characterized in that: Data quality inspection rules can be added, modified, and deleted dynamically. This is achieved by saving database connection information and quality inspection rules in the database, and then obtaining the database connection information and quality inspection rules from the database in real time during quality inspection execution. The quality inspection execution process is dynamically scheduled according to the user-defined frequency and is divided into the following steps: Step 1. Quality Inspection Initialization Process Step 1.1 Implement the quality inspection rule management system Step 1.1.1 The quality inspection rule management system includes an API or front-end interface, a quality inspection scheduling module, a quality inspection rule table, a database, a quality inspection result table, and a notification module, which implements the functions of adding, deleting, modifying, and checking database connection information, as well as adding, deleting, modifying, and checking quality inspection rules; Step 1.1.2 The system provides an API or front-end interface to support users in managing database connection information and quality inspection rules; Step 1.2 User registration database connection information Step 1.2.1 The user enters the database type in the system interface; Step 1.2.2 provides database connection information; Step 1.2.3 provides the database name corresponding to the database connection; After step 1.2.4 is submitted, the system verifies the validity of the connection. If the connection is successful, it is stored in the database connection management table; Step 1.3 User Registration Quality Inspection Rules Step 1.3.1 The user selects a registered database connection; Step 1.3.2 User defines quality inspection rules; Step 1.3.3 User sets execution cycle Step 1.3.4: Store the rules in the quality inspection rule table and bind the database connection; Step 2. Quality Inspection Execution Process Step 2.1 Scheduled Tasks Step 2.1.1 The quality inspection scheduling module triggers quality inspection tasks regularly according to the execution cycle set by the rules; Step 2.1.2: The quality inspection rules to be executed are stored in the task queue; Step 2.2 Obtain quality inspection rules Step 2.2.1 Read the quality inspection rule table to obtain the rules that need to be executed currently; Step 2.2.2 Obtain the database connection information corresponding to each rule; Step 2.3 Database Connection Step 2.3.1 Dynamically select the appropriate database connection driver based on the database type; Step 2.3.2 Establish a connection based on the database connection information; Step 2.3.3 If the connection fails, log the error and send an exception notification. Step 2.4 Execute data query Step 2.4.1 Parse the SQL statement or query expression according to the quality inspection rules; Step 2.4.2: Connect to the database, execute the query, and obtain the data; If the query fails in step 2.4.3, an error is logged and a warning is triggered; Step 2.5: Execute quality control logic Data quality includes the quality standards of compliance, timeliness, integrity, uniqueness, and consistency. Compliance includes the validity of formats, types, domain values, and business rules. Timeliness includes the timeliness and speed of data updates, modifications, and retrieval operations. Integrity includes the absence of missing entities, attributes, records, and field values. Uniqueness includes the uniqueness of primary keys and candidate keys. Consistency includes unified data sources, unified data storage, and unified data caliber. Step 2.6 Store quality inspection results Step 2.6.1 Generate quality inspection results; The results of step 2.6.2 are stored in the quality inspection result table; Step 3. Quality inspection result processing flow Step 3.1 Anomaly Detection Step 3.1.1 If abnormal data is detected, an alarm is triggered; Step 3.1.2 records the abnormal information and sends it to the notification module; Step 3.2 Generate quality inspection report Step 3.2.1 Generate a report based on the quality inspection results; Step 3.2.2 Statistical analysis of abnormal ratio, qualified rate, and trend; Step 3.3 Send report Step 3.3.1 Push the report via communication tools; Step 3.3.2 records the report sending log.

2. The dynamically scalable data quality inspection method according to claim 1, characterized in that: The timeliness of executing the quality inspection logic in step 2.5 is: Step 2.5.1 Timely quality inspection Step 2.5.1.1 Analyze the quality inspection rules and obtain the data update frequency; Step 2.5.1.2: Construct a query statement based on the database type to obtain the timestamp field of the latest data; Step 2.5.1.3: Get the current system time and calculate the corresponding time range; Step 2.5.1.4: Determine whether the latest data timestamp is within the set update period. If the latest data time is within the update period, the data timeliness is normal. If the latest data time exceeds the update cycle, the data timeliness is abnormal, the abnormal data is recorded and an alarm is triggered.

3. The dynamically scalable data quality inspection method according to claim 1, characterized in that: Step 2.5 performs the quality control logic compliance as follows: Step 2.5.2 Compliance Inspection Step 2.5.2.1 Parse the expression of the quality inspection rule and support the following calculation logic: Form 1: supports mathematical operations between fields; Form 2: supports logical operations; Form 3: supports predefined functions; Form 4: supports nested expressions combining the above operations; Step 2.5.2.2 Select data based on the sampling method set by the rules: If set to full verification, the data requiring quality inspection is obtained based on the quality inspection rules and database connection information, and all data are traversed for calculation; if set to sampling verification, the data requiring quality inspection is obtained based on the quality inspection rules and database connection information, and a portion of the data is randomly sampled according to the set ratio for calculation; Step 2.5.2.3 performs expression operations to determine whether the data complies with the rules: if all sampled / traversed data complies with the rules, the data compliance is normal; if there is data that does not comply with the rules, the data compliance is abnormal, the abnormal data is recorded, and an alarm is triggered.

Images