Multi-source database data quality monitoring method and system based on configurable rules

Abstract

The invention discloses a multi-source database data quality monitoring method and system based on configurable rules. The method comprises the following steps: registering connection information of a multi-source database; customizing and configuring a rule for verifying data quality; automatically extracting data from a data source according to a rule, and evaluating the data quality to form a set of timing operation mechanism; and summarizing evaluation results, generating a quality report, adjusting rules or taking corrective measures. The system comprises a quality definition layer, a data quality operation layer, a data quality analysis layer and a data scheduling server. The method is suitable for a big data environment, and aims to find and correct data problems in time by monitoring and evaluating the data quality from a plurality of heterogeneous data sources so as to guarantee the accuracy and efficiency of a service system.

Description

Technical Field

[0001] The present invention relates to the field of data quality management, and in particular to a multi-source database data quality monitoring method and system based on configurable rules. Background Art

[0002] In the era of big data, in order to ensure the accuracy and efficiency of business systems, it is necessary to monitor and evaluate the data quality from multiple heterogeneous data sources in real time so as to promptly discover and correct data problems. However, the existing data quality management system has certain limitations: the flexibility of rule configuration is insufficient, and it often relies on regular manual review. This lag leads to a long cycle of problem discovery and resolution. Especially when processing large amounts of data, manual data quality management is not only costly, but also relatively inefficient, making it difficult to achieve a comprehensive check of all data. To this end, there is an urgent need for a system and implementation method for multi-source database data quality monitoring based on configurable rules to overcome the above problems and improve the efficiency and effectiveness of data governance. Summary of the invention

[0003] Purpose of the invention: The present invention provides a multi-source database data quality monitoring method and system based on configurable rules to effectively overcome existing problems and significantly improve the efficiency of data governance.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] In a first aspect, a method for monitoring data quality of a multi-source database based on configurable rules is provided. The method is based on a multi-source database data quality monitoring system and comprises the following steps:

[0006] Registering in the multi-source database quality monitoring system according to database connection information from different data sources;

[0007] Customizing the configuration of data quality verification rules in the multi-source database quality monitoring system, including determining rule templates based on data source type and rule type, or customizing rules through SQL query statements. The multi-source database quality monitoring system provides built-in rules as references, and provides SQL statement references for custom rules, supporting real-time dynamic loading and modification of rules;

[0008] The multi-source database quality monitoring system automatically extracts data from the data source according to the configured rules, and determines the data quality evaluation mechanism based on the specified rule template or the custom rule; based on the data quality evaluation mechanism, generates a scheduled task in the task scheduling system DS-SERVER of the multi-source database quality monitoring system according to the given timing type, performs polling verification, and generates a data quality evaluation result;

[0009] The multi-source database quality monitoring system summarizes the data quality assessment results and generates a quality report, and adjusts the configured rules and / or provides corrective action suggestions based on the quality report to improve data quality.

[0010] Further, registering in the multi-source database quality monitoring system according to database connection information from different data sources includes:

[0011] In the multi-source database quality monitoring system, a multi-source heterogeneous database is selected, and the database connection information is filled in, wherein the connection information includes one or more of the database IP address, port number, database name, user name, and password; and the data call system DS-SERVER is used to test whether the connection to the database can be successfully made based on the connection information, and an error message is provided for connection failure.

[0012] Furthermore, when customizing rules through SQL query statements, SQL query statements are defined according to rule types and rule dimensions; the rule types include table-level rules and field-level rules; the table-level rules focus on the data quality of the entire table, and the field-level rules focus on the data quality of specified fields; the rule dimensions include one or more of uniqueness, completeness, timeliness, validity, accuracy, and consistency; the uniqueness means that each record in a data set is unique in a specific field and there are no duplicate values; the completeness means that all necessary fields in the data set have values ​​and there is no missing or undefined data; the timeliness means that for time-limited data, the data is the latest and is not overwritten by old data; the validity means that according to business rules or logic, the data is legal and valid; the accuracy means that the value of the data is consistent with the actual situation and there are no errors; the consistency means that the data is consistent between different systems, databases or data sources.

[0013] Furthermore, a data quality assessment mechanism is determined based on a specified rule template or a custom rule, including:

[0014] A rule template that meets the current data quality monitoring requirements is selected from the rule templates provided by the multi-source database quality monitoring system. When the rule template provided by the system does not meet the monitoring requirements, a custom rule is performed through an SQL statement, including setting a specific comparison value and a comparison method for the custom rule to clarify the specific execution conditions of the rule, wherein the comparison value includes a maximum value and a minimum value; and the comparison method includes greater than, less than, equal to, not equal to, greater than or equal to, and less than or equal to.

[0015] Further, a timing type is selected, and a timing task is generated in the task scheduling system DS-SERVER of the multi-source database quality monitoring system according to the given timing type, and polling verification is performed, including:

[0016] Select the type of scheduled task, including visually selecting minute intervals, hour intervals, daily, weekly, and monthly schedules, or setting the schedule through a time-based task schedule expression cron expression; or do not set the schedule and call it manually;

[0017] After the scheduled task is generated, the database is queried through the task scheduling system DS-SERVER according to the set scheduled time and rules. After the return value is obtained, it is compared with the comparison value to obtain the quality report result.

[0018] Further, the data quality assessment results are summarized and a quality report is generated, and the configured rules are adjusted and / or corrective action suggestions are provided according to the quality report to improve the data quality, including:

[0019] Summarize the data quality assessment results, fill in the data quality report framework according to the data quality assessment results, generate a quality report, adjust the configured rules and / or provide corrective action suggestions according to the quality report; the multi-source database quality monitoring system provides a graphical tool through an interface, which is convenient for users to create, edit and manage complex validation rules on the interface according to the quality report, and supports multiple rule expressions and data source types.

[0020] Further, adjusting configured rules and / or providing corrective action suggestions based on quality reports include:

[0021] View quality report: View the generated data quality report through the interface and analyze the specific situation of data quality issues;

[0022] Adjust rules: If the report shows that the existing rules are not accurate enough or need to be improved, use graphical tools to adjust rule parameters or define new rules;

[0023] Provide corrective measures: Provide suggestions for corrective measures that can be taken for data quality issues found in the report;

[0024] Rule testing: After adjusting the rules, test the new rules to ensure they are executed correctly;

[0025] Rule deployment: After confirming that the rule adjustment is correct, deploy the new rule to the system for use in the next data quality assessment.

[0026] In the second aspect, a multi-source database data quality monitoring system based on configurable rules includes a quality definition layer, a data quality operation layer, a data quality analysis layer, and a data scheduling system DSServer;

[0027] Data quality definition layer: responsible for defining and configuring data quality monitoring rules. The data quality definition layer provides the following functions: registering database connection information from different data sources; and customizing and configuring rules for verifying data quality, including determining rule templates based on data source type and rule type, or customizing rules through SQL query statements. The multi-source database quality monitoring system provides built-in rules as references, and provides SQL statement references for custom rules, supporting real-time dynamic loading and modification of rules;

[0028] Data quality operation layer: responsible for automatically executing data extraction and evaluation tasks according to the configured rules. The data quality operation layer provides the following functions: automatically extracting data from the data source according to the configured rules, determining the data quality evaluation mechanism based on the specified rule template or custom rules; based on the data quality evaluation mechanism, generating a scheduled task in the task scheduling system DS-SERVER according to the given timing type, performing polling verification, and generating data quality evaluation results;

[0029] Data Scheduling Server DSServer: It is an auxiliary component of the data quality monitoring system. It mainly acts as a task scheduling system, responsible for generating and executing scheduled tasks according to the rules configured in the data quality operation layer;

[0030] Data quality analysis layer: responsible for analyzing the results of data quality assessment and generating relevant quality reports, as well as adjusting configured rules and / or providing corrective action suggestions based on the quality reports to improve data quality.

[0031] In a third aspect, a computer device is provided, comprising one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the multi-source database data quality monitoring method based on configurable rules as described in the first aspect of the present invention are implemented.

[0032] In a fourth aspect, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the multi-source database data quality monitoring method based on configurable rules as described in the first aspect of the present invention are implemented.

[0033] Compared with the prior art, the invention has the following beneficial effects:

[0034] (1) Ensure continuous monitoring and management of data quality through an automated and continuous data quality monitoring mechanism. The system can automatically execute tasks and form a set of scheduled operation mechanisms to effectively improve the stability and reliability of data quality. In addition, the system supports custom configuration of rules for verifying data quality, and can flexibly adjust rules according to specific business needs to meet data quality monitoring needs in different scenarios. By summarizing evaluation results and generating quality reports, users can understand the data quality status in real time, and adjust rules or take corrective measures based on report results to achieve continuous improvement of data.

[0035] (2) It is flexible and scalable, supports multi-source database connections, and can connect to multiple different types of data sources to achieve data quality monitoring of heterogeneous data sources. At the same time, it provides graphical tools to lower the technical threshold, allowing non-technical users to intuitively create, edit and manage complex validation rules. In addition, the real-time dynamic adjustment function of the rules can take effect without restarting the system, which improves the availability and flexibility of the system. As a scheduling system, DS-SERVER realizes the management and monitoring of the data quality monitoring process and improves management efficiency. Through data quality reports and analysis results, users can get a comprehensive overview of data quality and make more informed decisions based on the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of an implementation of a multi-source database data quality monitoring method based on configurable rules provided by the present invention.

[0037] Figure 2 This is a schematic diagram of the multi-source database connection information registration process provided by the present invention.

[0038] Figure 3 A schematic diagram of a rule flow for custom configuration verification of data quality provided by the present invention.

[0039] Figure 4 The present invention provides a flow chart of a mechanism for automatically extracting data from a data source according to rules, evaluating the data quality, and forming a set of scheduled operations. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings.

[0041] Reference Figure 1The present invention provides a multi-source database data quality monitoring method based on configurable rules. The method is implemented based on a multi-source database data quality monitoring system. The multi-source database data quality monitoring system (hereinafter referred to as the system) includes a data quality definition layer, a data quality operation layer, a data quality analysis layer, and a data scheduling system DS-SERVER. Different layers assume different functions and roles.

[0042] Data quality definition layer: This layer is the foundation of the entire data quality management system. It involves the process of defining and configuring data quality rules. This includes registering database connection information (S101) and customizing data quality validation rules (S102). The data quality definition layer ensures that the system understands which data needs to be monitored and how to monitor this data. It provides standards and basis for subsequent data quality assessments.

[0043] Data quality operation layer: This layer is responsible for automatically executing data extraction and evaluation tasks according to the defined data quality rules. It includes setting up scheduled tasks to periodically check data quality (S103). The data quality operation layer is where data quality monitoring is actually performed. It ensures that the rules can be automatically run according to the predetermined schedule, thereby continuously monitoring data quality.

[0044] DS-SERVER: DS-SERVER (also known as data scheduling server) is an auxiliary component of the data quality monitoring system. It mainly serves as a task scheduling system, responsible for generating and executing scheduled tasks according to the rules configured in the data quality operation layer.

[0045] Data quality analysis layer: This layer is responsible for analyzing the results of data quality assessment and generating relevant reports. It also involves adjusting data quality rules or taking corrective measures based on the analysis results (S104). The data quality analysis layer is the "intelligent brain" of data quality management. It identifies problems, makes improvement suggestions, and helps users understand the overall status of data quality by analyzing the assessment results.

[0046] In general, these levels together constitute a complete data quality monitoring system. Each level has its specific responsibilities and functions, and they work together to ensure the quality and reliability of the data.

[0047] According to the multi-source database data quality monitoring system, the process and logic of implementing data quality monitoring are as follows: the system establishes a connection with the database, which is a prerequisite for all subsequent operations, because data cannot be accessed without a connection. After the connection is established, the user needs to define the standards and rules for data quality. These rules are based on business requirements and data quality goals, and they will be used to evaluate the data in the database. After the rules are defined, the system will automatically extract data from the database according to these rules and evaluate its quality. This step is performed periodically and may be a scheduled task to ensure continuous monitoring of the data. The data quality operation layer directly depends on the rules provided by the data quality definition layer. Without clear rules, the system cannot perform effective data quality assessment. After the operation layer completes the data quality assessment, the analysis layer collects the assessment results, generates reports, and analyzes the results. Based on the analysis results, the user may need to adjust the data quality rules to improve the accuracy of monitoring, or take corrective measures for the problems found. The data quality analysis layer uses the data quality assessment results provided by the operation layer to conduct in-depth analysis and provide feedback to the definition layer, forming a closed-loop data quality improvement process.

[0048] Specifically, continue to refer to Figure 1 The multi-source database data quality monitoring method based on configurable rules provided by the present invention comprises the following steps:

[0049] Step S101, registering multi-source database connection information;

[0050] Databases are where enterprise data is stored. Before data quality monitoring can be performed, it is necessary to ensure that the system can access these databases. This step is a prerequisite for the system to establish connections with multiple data sources. It involves collecting and storing connection details such as database type, address, port, user authentication information, etc. In this way, the system can access and extract the required data in subsequent steps. The purpose is to ensure that the system can connect to different databases securely and stably, providing a data access basis for data quality monitoring.

[0051] Step S102, customizing and configuring rules for verifying data quality;

[0052] In this step, users or system administrators define rules based on specific business requirements and data quality standards. These rules may include verification conditions for data completeness, accuracy, consistency, timeliness, and uniqueness. The purpose is to ensure that the monitoring system can check data according to predefined quality standards and improve the accuracy and relevance of data monitoring.

[0053] Step S103, automatically extracting data from the data source according to the rules, evaluating the data quality, and forming a set of regular operation mechanisms;

[0054] In this step, the system automatically performs the data extraction and evaluation process according to the configured rules. This process is usually periodic, which can be hourly, daily, weekly, etc., to ensure continuous monitoring of data. The purpose is to detect data quality problems in a timely manner through the automated evaluation process, and ensure that problems are discovered and handled in a timely manner to maintain the overall quality of the data.

[0055] Step S104, summarize the evaluation results and generate a quality report, adjust the rules or take corrective measures.

[0056] In this step, the data quality assessment results collected in the previous steps are summarized and generated into an easy-to-understand report. Based on the report content, data quality rules can be adjusted to more accurately reflect the data status, or specific corrective measures can be taken for the problems found. The purpose is to provide decision support for data managers, help them understand the overall status of data quality, and optimize data management strategies and processes based on the assessment results.

[0057] Through the above system and step description, a comprehensive and automated data quality monitoring solution is provided to ensure that the data within the organization maintains a high standard of quality and supports the data-driven decision-making process.

[0058] Reference Figure 2 In the present invention, in step S101, multi-source database connection information registration is implemented, and the process includes:

[0059] Step S201, selecting a multi-source heterogeneous database;

[0060] In this step, users need to select databases whose data quality needs to be monitored from different database types. These databases may be heterogeneous, that is, they may use different database management systems (such as MySQL, Oracle, SQL Server, PostgreSQL, etc.). The purpose is to ensure that the system can adapt to different types of data sources and meet diverse and complex data monitoring needs.

[0061] Step S202, fill in the connection information;

[0062] The user needs to provide the details required to connect to the selected database, including but not limited to the database's IP address, port number, database name, user name, password, etc. This information is necessary for database connection. Ensure that the system can establish a stable connection to each database using the correct credentials and parameters.

[0063] Step S203, testing whether the database can be successfully connected according to the filled connection information.

[0064] The present invention uses the data source adaptation framework in the SG-ETL system to test whether the provided connection information is correct and whether the connection with the database can be successfully established according to the SQL92 standard (a universal standard for database query languages). SG-ETL is a set of data conversion and integration tools based on the kettle open source tool, and DS-SERVER is its calling subsystem. Through the database connection information filled in the previous steps, DS-SERVER is called, and the corresponding driver is loaded according to different database types to test whether the filled-in IP address, port number, user name and password and other information can correctly connect to the database. The purpose is to verify the accuracy of the connection information and ensure that the system can access the database during the actual data quality monitoring process. If the test fails, the system provides an error message to help users diagnose and correct the connection problem.

[0065] Overall, the sub-process of step S101 ensures that the system can correctly identify and connect to various databases, which is a prerequisite for effective data quality monitoring. By testing the connection, the system can detect and solve potential connection problems in advance, ensuring the smooth progress of subsequent data quality assessment.

[0066] In the present invention, in step S102, a custom configuration is implemented to verify the data quality rules, and the process includes: distinguishing the rule template according to the data source type and whether the rule type is table level or field level. Supporting the rule expression method of SQL query, providing built-in rules as reference, and the built-in rules include verifying the maximum value, verifying the number of table rows, etc. Providing SQL statement reference for custom rules, the rules can be dynamically loaded and modified in real time, and can take effect without restarting the system.

[0067] The system provides different rule templates according to the type of data source (such as relational database, NoSQL database, etc.) and the level at which the rule needs to be applied (table level or field level). Table-level rules focus on the data quality of the entire table, such as the number of rows in the data table, the uniqueness of the data table, etc.; field-level rules focus on the data quality of specific fields, such as the validity and integrity of field values. The purpose is to simplify the rule configuration process and ensure that users can choose appropriate rule templates according to different needs and scenarios. The system allows users to define data quality rules using SQL queries. This means that users can use the power of SQL to write complex rule expressions to detect specific quality issues in the data. Provide flexibility and scalability so that rule definitions can adapt to various complex data quality inspection requirements. In addition, the system has built-in a series of commonly used data quality rules, such as validating the maximum value, validating the number of table rows, etc., for users to refer to and use. These built-in rules can be directly applied to data quality monitoring or used as a starting point for custom rules. This can reduce the user's configuration workload and speed up the rule definition process by providing common rule templates. When users define custom rules, the system provides reference examples of SQL statements to help users understand how to write effective SQL queries to meet specific rule requirements. This reduces the difficulty for users to write SQL rules and improves user experience. Users can add new rules or modify existing rules in real time while the system is running, and these changes will take effect immediately without restarting the system. This improves the availability and flexibility of the system, allowing users to quickly respond to changes in data quality issues without affecting the continued operation of the system.

[0068] Reference Figure 3 In the step of customizing the rules for verifying data quality, a large number of built-in rules will be preset in the system. The built-in rules cannot be modified. If you cannot find a quality rule that meets your needs in the built-in rules, you can customize the rules according to your business needs. First, you need to clarify which type of database the quality rules are used for. Next, you will determine the level at which these rules apply, such as table level or field level. After that, build the corresponding SQL query statements based on these rules and dimensions. The rule dimensions include: uniqueness, completeness, timeliness, validity, accuracy, and consistency. When these SQL statements are executed, they will interact with the database to verify the data quality. You can create different types of rule instances according to your needs and generate corresponding SQL query statements for each rule. Then, check the quality of the data by executing these SQL statements. For example, you need to define a rule for a MySQL-type database to check whether the data in a certain mailbox field meets the mailbox format requirements to verify the validity of the data. You only need to create an SQL statement: select count(*)as value from${table}where${conditionField}

[0069] ~"^[A-Z0-9._%-]+@[A-Z0-9.-]+\.[AZ]{2,4}$" is enough.

[0070] In data quality monitoring, rule dimensions refer to the specific aspects that data quality rules focus on. The following is an explanation of each rule dimension:

[0071] Uniqueness: Ensure that each record in the data set is unique on a specific field, with no duplicate values. The purpose is to prevent data redundancy and ensure unique identification of data.

[0072] Completeness: Ensure that all necessary fields in the data set have values ​​and there is no missing or undefined data. Ensure that the data is complete and no key information is missing.

[0073] Timeliness: For data with time limits, ensure that the data is up-to-date and not overwritten by old data. Ensure that the data reflects the latest business status or facts.

[0074] Validity: Ensure the legitimacy and validity of data based on business rules or logic. Prevent invalid or non-compliant data from entering the database.

[0075] Accuracy: Ensure that the data value is consistent with the actual situation and there are no errors. Improve the reliability and accuracy of the data.

[0076] Consistency: Ensure data consistency across different systems, databases, or data sources. Avoid data conflicts or mismatches between different systems.

[0077] These rule dimensions together form the basis for data quality monitoring. Through them, we can comprehensively evaluate the quality of data and take corresponding measures to improve data quality. In different scenarios, the focus is different, and different rule dimensions can be selected.

[0078] In the present invention, in step S103, data is automatically extracted from the data source according to the rules, and the data quality is evaluated to form a set of timed operation mechanisms. Figure 4 The specific process of this step includes:

[0079] Step S401, selecting a suitable rule template or custom rule to form a mechanism for evaluating data quality.

[0080] Selecting a rule template means that the user selects the template that best suits the current data quality monitoring needs from the rule templates provided by the system. This can simplify the configuration process and quickly apply predefined data quality checks. If there is no suitable rule template, that is, the template provided by the system does not meet specific needs, the user can write a custom SQL statement to define the rule. Provide more flexible configuration options to meet data quality monitoring needs in specific scenarios. Users set specific comparison values ​​(such as maximum value, minimum value, etc.) and comparison methods (greater than, less than, equal to, not equal to, greater than or equal to, less than or equal to) for custom rules. This clarifies the specific execution conditions of the rules and ensures the accuracy of the monitoring results. Users use SQL statements to specify specific tables and fields that need to be monitored. For example, to verify whether all the fields of a mailbox in a table conform to the mailbox format, create the SQL statement select count(*)as value from${table}where

[0081] After ${conditionField}~”^[A-Z0-9._%-]+@[A-Z0-9.-]+\.[AZ]{2,4}$”, you only need to set the corresponding table name, field name, and the comparison method to <=, and the comparison value to 0. In this way, during the scheduled operation of the verification task, as long as the operation result does not meet the set comparison method and comparison value, an alarm will be issued to prompt the user.

[0082] Step S402, select the timing type, generate a timing task for the corresponding table or field in the SG-ETL task scheduling system (DS-SERVER) according to the configured rules, and perform polling verification.

[0083] Selecting the timing type means that the user selects the type of scheduled task, including visually selecting minute intervals, hour intervals, daily timing, weekly timing, monthly timing, and more professional timing settings through time-based task planning expressions cron expressions. You can also not set the timing and call it manually. In this way, you can set the appropriate monitoring frequency according to the frequency of data updates and business needs.

[0084] After generating a scheduled task (creating a scheduled task in the SG-ETL task scheduling system DS-SERVER, which will perform data quality monitoring according to the scheduled type set by the user), the task scheduling system will query the database according to the set scheduled time and rules, that is, the task scheduling system periodically executes SQL query operations according to the definition of the scheduled task to monitor data quality. Automatically execute data quality monitoring tasks to maintain continuous data monitoring. After obtaining the return value, compare it with the comparison value. For example, compare the result of the query operation with the comparison value set by the user to obtain the quality report result. The quality report provides the output of data quality assessment to help users understand the status of data quality and take corresponding measures.

[0085] The process of steps S401 and S402 above ensures that the system can automatically and periodically monitor the quality of the data source according to the rules and timing settings defined by the user, and generate corresponding quality reports, thereby achieving efficient and automated data quality management.

[0086] Step S104, summarize the evaluation results and generate a quality report, adjust the rules or take corrective measures.

[0087] Summarize the evaluation results and generate quality reports, adjust rules or take corrective measures, etc. The interface provides graphical tools that allow users to intuitively create, edit and manage complex validation rules, and supports multiple rule expressions and data source types.

[0088] Specifically, the system summarizes the data quality assessment results performed in the previous steps and generates easy-to-understand reports based on these results. The report may include detailed information, statistics, trend analysis, etc. of data quality issues. It provides users with a clear overview of data quality, helping them quickly identify problems and make corresponding decisions. The system provides a graphical user interface that allows users to create, edit and manage complex validation rules through intuitive operations such as dragging and dropping and clicking. It lowers the technical threshold and improves the user experience, allowing non-technical users to easily configure and manage data quality rules. In addition, users can use graphical tools to define and modify complex validation rules without writing code. This simplifies the rule configuration process and improves configuration efficiency.

[0089] The system supports multiple rule expressions, such as SQL queries, regular expressions, etc., and can adapt to different data source types, such as relational databases, NoSQL databases, etc. It provides flexibility and compatibility to ensure that the system can meet the data quality monitoring needs in different scenarios. Based on the results of the quality report, users can adjust the existing data quality rules to more accurately reflect the data status, or take specific corrective measures for the problems found. It ensures that the data quality monitoring system can continue to adapt to changes in the data environment and can effectively solve the identified data quality problems.

[0090] The specific process may include the following steps:

[0091] View quality report: Users can view the generated data quality report through the interface and analyze the specific circumstances of data quality issues.

[0092] Adjust rules: If reports show that existing rules are not accurate enough or need improvement, users can use graphical tools to adjust rule parameters or define new rules.

[0093] Implement corrective measures: For data quality issues found in the report, users can take corresponding corrective measures, such as data cleaning, data correction, etc.

[0094] Rule Testing: After adjusting the rules, users can test the new rules to ensure that they are executed correctly.

[0095] Rule deployment: After confirming that the rule adjustment is correct, deploy the new rule to the system for use in the next data quality assessment.

[0096] Through these steps, the S104 process ensures the continuous optimization and improvement of the data quality monitoring system, while providing a user-friendly operation interface to make data quality management more efficient and convenient.

[0097] The present invention also provides a computer device, comprising one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the multi-source database data quality monitoring method based on configurable rules as described above are implemented.

[0098] The present invention also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-source database data quality monitoring method based on configurable rules as described above are implemented.

[0099] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices (systems), computer equipment or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0100] The present invention is described with reference to a flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and a combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A device that specifies functions in a process or multiple processes.

[0101] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

Claims

1. A multi-source database data quality monitoring method based on configurable rules, characterized in that: The method is based on a multi-source database data quality monitoring system and includes the following steps: Registering in the multi-source database quality monitoring system according to database connection information from different data sources; Customizing the configuration of data quality verification rules in the multi-source database quality monitoring system, including determining rule templates based on data source type and rule type, or customizing rules through SQL query statements. The multi-source database quality monitoring system provides built-in rules as references, and provides SQL statement references for custom rules, supporting real-time dynamic loading and modification of rules; The multi-source database quality monitoring system automatically extracts data from the data source according to the configured rules, and determines the data quality evaluation mechanism based on the specified rule template or the custom rule; based on the data quality evaluation mechanism, generates a scheduled task in the task scheduling system DS-SERVER of the multi-source database quality monitoring system according to the given timing type, performs polling verification, and generates a data quality evaluation result; The multi-source database quality monitoring system summarizes the data quality assessment results and generates a quality report, and adjusts the configured rules and / or provides corrective action suggestions based on the quality report to improve data quality.

2. The method according to claim 1, characterized in that Registering in the multi-source database quality monitoring system according to database connection information from different data sources includes: In the multi-source database quality monitoring system, a multi-source heterogeneous database is selected, and the database connection information is filled in, wherein the connection information includes one or more of the database IP address, port number, database name, user name, and password; and the data call system DS-SERVER is used to test whether the connection to the database can be successfully made based on the connection information, and an error message is provided for connection failure.

3. The method according to claim 1, characterized in that When customizing rules through SQL query statements, SQL query statements are defined according to rule types and rule dimensions; the rule types include table-level rules and field-level rules; the table-level rules focus on the data quality of the entire table, and the field-level rules focus on the data quality of specified fields; the rule dimensions include one or more of uniqueness, completeness, timeliness, validity, accuracy, and consistency; the uniqueness means that each record in a data set is unique in a specific field and there are no duplicate values; the completeness means that all necessary fields in the data set have values ​​and there is no missing or undefined data; the timeliness means that for time-limited data, the data is the latest and is not overwritten by old data; the validity means that according to business rules or logic, the data is legal and valid; the accuracy means that the value of the data is consistent with the actual situation and there are no errors; the consistency means that the data is consistent between different systems, databases or data sources.

4. The method according to claim 1, characterized in that Determine the data quality assessment mechanism based on specified rule templates or custom rules, including: A rule template that meets the current data quality monitoring requirements is selected from the rule templates provided by the multi-source database quality monitoring system. When the rule template provided by the system does not meet the monitoring requirements, a custom rule is performed through an SQL statement, including setting a specific comparison value and a comparison method for the custom rule to clarify the specific execution conditions of the rule, wherein the comparison value includes a maximum value and a minimum value; and the comparison method includes greater than, less than, equal to, not equal to, greater than or equal to, and less than or equal to.

5. The method according to claim 1, characterized in that Select a timing type, generate a timing task in the task scheduling system DS-SERVER of the multi-source database quality monitoring system according to the given timing type, and perform polling verification, including: Select the type of scheduled task, including visually selecting minute intervals, hour intervals, daily, weekly, and monthly schedules, or setting the schedule through a time-based task schedule expression cron expression; or do not set the schedule and call it manually; After the scheduled task is generated, the database is queried through the task scheduling system DS-SERVER according to the set scheduled time and rules. After the return value is obtained, it is compared with the comparison value to obtain the quality report result.

6. The method according to claim 1, characterized in that Summarize the data quality assessment results and generate a quality report, adjust the configured rules and / or provide corrective action suggestions to improve data quality based on the quality report, including: Summarize the data quality assessment results, fill in the data quality report framework according to the data quality assessment results, generate a quality report, adjust the configured rules and / or provide corrective action suggestions according to the quality report; the multi-source database quality monitoring system provides a graphical tool through an interface, which is convenient for users to create, edit and manage complex validation rules on the interface according to the quality report, and supports multiple rule expressions and data source types.

7. The method according to claim 6, characterized in that Rules for adjusting configurations and / or providing corrective action recommendations based on quality reports include: View quality report: View the generated data quality report through the interface and analyze the specific situation of data quality issues; Adjust rules: If the report shows that the existing rules are not accurate enough or need to be improved, use graphical tools to adjust rule parameters or define new rules; Provide corrective measures: Provide suggestions for corrective measures that can be taken for data quality issues found in the report; Rule testing: After adjusting the rules, test the new rules to ensure they are executed correctly; Rule deployment: After confirming that the rule adjustment is correct, deploy the new rule to the system for use in the next data quality assessment.

8. A multi-source database data quality monitoring system based on configurable rules, characterized in that: The system includes: quality definition layer, data quality operation layer, data quality analysis layer, data scheduling system DSServer, Data quality definition layer: responsible for defining and configuring data quality monitoring rules. The data quality definition layer provides the following functions: registering database connection information from different data sources; and customizing and configuring rules for verifying data quality, including determining rule templates based on data source type and rule type, or customizing rules through SQL query statements. The multi-source database quality monitoring system provides built-in rules as references, and provides SQL statement references for custom rules, supporting real-time dynamic loading and modification of rules; Data quality operation layer: responsible for automatically executing data extraction and evaluation tasks according to the configured rules. The data quality operation layer provides the following functions: automatically extracting data from the data source according to the configured rules, determining the data quality evaluation mechanism based on the specified rule template or custom rules; based on the data quality evaluation mechanism, generating a scheduled task in the task scheduling system DS-SERVER according to the given timing type, performing polling verification, and generating data quality evaluation results; Data Scheduling Server DSServer: It is an auxiliary component of the data quality monitoring system. It mainly acts as a task scheduling system, responsible for generating and executing scheduled tasks according to the rules configured in the data quality operation layer; Data quality analysis layer: responsible for analyzing the results of data quality assessment and generating relevant quality reports, as well as adjusting configured rules and / or providing corrective action suggestions based on the quality reports to improve data quality.

9. A computer device, characterized in that: The device includes one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the multi-source database data quality monitoring method based on configurable rules as described in any one of claims 1-7 are implemented.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-source database data quality monitoring method based on configurable rules are implemented as described in any one of claims 1 to 7.

Images