A data quality monitoring method, device, equipment, storage medium and product

Abstract

The embodiments of this specification relate to the field of data analysis technology, and provide a data quality monitoring method, apparatus, device, storage medium, and product, including: obtaining table structures and corresponding data from multiple source databases; matching data models from a data model library according to the table structure; verifying the table structure using the data model to obtain a verified table structure; matching monitoring rules for each field in the verified table structure from a pre-established monitoring rule pool; and performing quality monitoring on the data corresponding to each field in the verified table structure according to the monitoring rules. Through the embodiments of this specification, it is possible to automatically monitor the quality of data from multiple data sources and improve the efficiency of data quality monitoring from multiple data sources.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of data analysis technology, and in particular to a data quality monitoring method, device, equipment, storage medium and product. Background Art

[0002] Data quality monitoring is an important means to ensure data accuracy and reliability. It involves the inspection and evaluation of data integrity, accuracy, consistency, timeliness and credibility. In practical applications, many companies and organizations have established data quality monitoring systems to ensure data quality. For example, the financial industry uses data quality monitoring to ensure the accuracy and integrity of customer information, thereby improving risk management and customer service levels; the medical industry uses data quality monitoring to ensure the accuracy and consistency of medical data, thereby improving medical quality and safety. Data quality is not only the basis for the validity and accuracy of data analysis and data mining conclusions, but also the premise of data-driven decision-making. In order to ensure data quality and ensure data availability, data quality needs to be monitored.

[0003] Most of the existing data quality monitoring methods are targeted at a single data source, and lack quality monitoring for data from multiple data sources. Due to the significant differences in data structure, storage method, access mechanism, etc. between different data source databases, the existing quality monitoring methods for a single data source are difficult to apply directly. Moreover, in the multi-data source data quality monitoring scenario, the existing methods form a set of monitoring methods for each data source, and then perform quality monitoring on the data of each data source. When there are many data sources, the monitoring efficiency is low. Therefore, there is an urgent need for a data quality monitoring method that can monitor the quality of data from multiple data sources and improve the efficiency of data quality monitoring from multiple data sources. Summary of the invention

[0004] In view of the above problems in the prior art, the purpose of the embodiments of this specification is to provide a data quality monitoring method, device, equipment, storage medium and product, so as to monitor the quality of data from multiple data sources and improve the efficiency of data quality monitoring from multiple data sources.

[0005] In order to solve the above technical problems, the specific technical solutions of the embodiments of this specification are as follows:

[0006] On the one hand, an embodiment of this specification provides a data quality monitoring method, the method comprising:

[0007] Obtain table structures and corresponding data from multiple source databases;

[0008] Matching a data model from a data model library according to the table structure;

[0009] Using the data model to check the table structure, to obtain a checked table structure;

[0010] Matching a monitoring rule for each field in the checked table structure from a pre-established monitoring rule pool;

[0011] The quality of the data corresponding to each field in the checked table structure is monitored according to the monitoring rules.

[0012] Furthermore, obtaining table structures and corresponding data from multiple source databases includes:

[0013] Configure connection parameters for each source database based on the source database type;

[0014] Connecting multiple source databases to the query interface according to the connection parameters;

[0015] According to the query statement or query script sent by the query interface, the table structure and corresponding data are obtained.

[0016] Furthermore, matching the data model from the data model library according to the table structure includes:

[0017] Obtaining basic information of fields in each of the table structures;

[0018] Calculating the similarity between the basic information of the field and the basic information of the corresponding field of each data model in the data model library;

[0019] A data model of each of the table structures is determined according to the similarity.

[0020] Furthermore, the data model includes constraints for each field;

[0021] The checking the table structure by using the data model includes:

[0022] S1: Compare the fields in the table structure with the fields in the corresponding data model one by one to determine whether each field in the table structure satisfies the constraint condition;

[0023] S2: If yes, then the verification is completed and the verified table structure is obtained;

[0024] S3: If not, adjusting the fields in the table structure that do not meet the constraint conditions according to the comparison result and the constraint conditions to obtain an adjusted table structure;

[0025] S4: Determine whether each field in the adjusted table structure satisfies the constraint condition;

[0026] S5: If not, repeat step S3 until each field in the table structure satisfies the constraint condition.

[0027] Furthermore, matching a monitoring rule for each field in the checked table structure from a pre-established monitoring rule pool includes:

[0028] Obtaining the field attributes of each field in the verified table structure;

[0029] Matching a monitoring rule for each field from the monitoring rule pool according to the field attributes and a preset matching rule;

[0030] Determine whether the match is successful;

[0031] If yes, the matched monitoring rule is used as the monitoring rule for each field;

[0032] If not, configuring a new monitoring rule according to the field attribute, and using the new monitoring rule as the monitoring rule for each field;

[0033] The new monitoring rule is stored in the monitoring rule pool.

[0034] Furthermore, the method further comprises:

[0035] According to the monitoring rules corresponding to each field in each of the table structures, a monitoring scenario corresponding to the table structure is constructed;

[0036] The monitoring scenario is stored in a monitoring scenario case library.

[0037] Furthermore, the method further comprises:

[0038] Obtain data quality monitoring results;

[0039] A monitoring report is generated according to the data quality monitoring result and sent to the target terminal.

[0040] On the other hand, an embodiment of the present specification provides a data quality monitoring device, the device comprising:

[0041] The acquisition module is used to obtain table structures and corresponding data from multiple source databases;

[0042] A first matching module, used for matching a data model from a data model library according to the table structure;

[0043] A checking module, used to check the table structure using the data model to obtain a checked table structure;

[0044] A second matching module, used for matching monitoring rules for each field in the checked table structure from a pre-established monitoring rule pool;

[0045] The monitoring module is used to perform quality monitoring on the data corresponding to each field in the checked table structure according to the monitoring rules.

[0046] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of any one of the above methods are executed.

[0047] On the other hand, an embodiment of the present specification further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of a computer device, the computer program executes instructions of any one of the above-described methods.

[0048] On the other hand, the embodiments of this specification further provide a computer program product, which, when executed by a processor of a computer device, executes instructions of any one of the above methods.

[0049] By adopting the above technical solution, the data quality monitoring method provided in the embodiment of this specification first obtains the table structure and the corresponding data from multiple source databases, matches the corresponding data model from the data model library according to the table structure, and then uses the data model to verify the table structure to obtain the verified table structure, matches the monitoring rules for each field in the verified table structure from the pre-established monitoring rule pool, and performs quality monitoring on the data corresponding to each field in the verified table structure according to the monitoring rules. In this way, by obtaining data from multiple data sources, it is possible to process data from multiple data sources at the same time, thereby improving the monitoring efficiency. Through the data model library and the monitoring rule pool, unified management and standardization of the table structure and data quality of multiple data sources are achieved, thereby eliminating data differences and conflicts between different source databases and improving data consistency and comparability.

[0050] The above description is only an overview of the technical solutions of some embodiments of this specification. In order to more clearly understand the technical means of some embodiments of this specification, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of this specification more obvious and easy to understand, the following specifically cites the preferred embodiments and describes them in detail in conjunction with the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A schematic diagram showing a flow chart of a data quality monitoring method in some embodiments of this specification;

[0053] Figure 2 A schematic diagram of a process for obtaining table structures and corresponding data from multiple source databases in some embodiments of this specification is shown;

[0054] Figure 3 A schematic diagram of a process of matching a data model from a data model library according to a table structure in some embodiments of this specification is shown;

[0055] Figure 4 A schematic diagram of a process of checking a table structure using a data model in some embodiments of this specification is shown;

[0056] Figure 5 A schematic diagram of a process of matching monitoring rules for each field in a table structure after verification from a pre-established monitoring rule pool in some embodiments of this specification is shown;

[0057] Figure 6 Another schematic diagram of a data quality monitoring method in some embodiments of this specification is shown;

[0058] Figure 7 Another schematic diagram of a data quality monitoring method in some embodiments of this specification is shown;

[0059] Figure 8 A schematic diagram of the module structure of a data quality monitoring device in some embodiments of this specification is shown;

[0060] Fig. 9 A schematic diagram of the structure of a computer device in this specification is shown.

[0061] Description of the accompanying symbols:

[0062] 801, obtaining module;

[0063] 802. A first matching module;

[0064] 803, verification module;

[0065] 804. A second matching module;

[0066] 805, monitoring module;

[0067] 902. Computer equipment;

[0068] 904, processor;

[0069] 906. Memory;

[0070] 908, driving mechanism;

[0071] 910, input / output module;

[0072] 912. Input device;

[0073] 914. Output device;

[0074] 916. Presentation equipment;

[0075] 918. Graphical user interface;

[0076] 920, network interface;

[0077] 922, communication link;

[0078] 924. Communication bus. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0080] In order to solve the above problems, an embodiment of this specification provides a data quality monitoring method. Figure 1 It is a flow chart of a data quality monitoring method provided in the embodiment of this specification. This specification provides the method operation steps described in the embodiment or flow chart, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiment or the accompanying drawings or in parallel.

[0081] It should be noted that the terms "first", "second", etc. in this specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0082] Reference Figure 1As shown, an embodiment of this specification provides a data quality monitoring method, the method comprising:

[0083] S101: Obtain table structures and corresponding data from multiple source databases.

[0084] In this embodiment, the source database may include relational databases and non-relational databases, such as Oracle, MySQL, Hive, GoldenDB, EverDB and other databases. In other embodiments, a custom database may be added according to the application scenario or business needs. The table structure in different databases is different. The table structure is used to store data. In a relational database, the table structure usually contains information such as table name, field name, data type and constraint conditions.

[0085] S102: Matching a data model from a data model library according to the table structure.

[0086] In this embodiment, the data model can be used as a unified data standard to standardize field definitions, types and other information. Through the data model, the consistency of field-level standards can be ensured when initializing table creation, thereby improving data quality and availability. The data model library contains multiple predefined data models, each of which contains constraints on the table structure.

[0087] S103: Using the data model to check the table structure, to obtain a checked table structure.

[0088] In this embodiment, using the data model to check the table structure refers to comparing the table structure of the source database with the definition in the data model, checking whether the field name, data type, constraint conditions, etc. are consistent, ensuring that the table structure of the source database meets the requirements of the data model, thereby ensuring the accuracy and consistency of the data. Through verification, errors or inconsistencies in the table structure can be discovered and corrected to avoid subsequent data quality issues.

[0089] S104: Matching a monitoring rule for each field in the checked table structure from a pre-established monitoring rule pool.

[0090] In this embodiment, the monitoring rule pool contains several pre-set monitoring rules, wherein the monitoring rules are set according to the data quality monitoring standard, that is, when setting the monitoring rules, it is necessary to comprehensively consider the four aspects of data integrity, consistency, accuracy and timeliness. Integrity refers to whether there is a missing state of data information. The missing data may be the missing of the entire data record or the missing of a record of a field information in the data. The value of incomplete data will be greatly reduced, and it is also the most basic evaluation standard for data quality. The integrity of data quality can be evaluated by the record value and unique value in the data statistics. Consistency refers to whether the data follows a unified specification and whether the data set maintains a unified format. The consistency of data quality is mainly reflected in the specification of data records and whether the data is logical. When monitoring data consistency, it is necessary to ensure that the format of the data field is consistent with expectations, such as date format, number format, etc.; it is also necessary to check whether the data follows business rules or logical constraints, such as whether the logical relationship between the data is reasonable; and for data across multiple systems or data sources, it is necessary to ensure that the data between them is consistent. Accuracy refers to whether there are abnormalities or errors in the information recorded in the data, that is, it is necessary to ensure that the data comes from a reliable and accurate information source. For calculated or derived fields, you need to ensure that the calculation logic is correct and verify the accuracy of the data by comparing it with other reliable data sources. Timeliness refers to the time interval from the generation of data to the time it can be viewed, also known as the data delay. When monitoring timeliness, you need to consider the following factors: data update frequency, ensuring that data is updated at the predetermined frequency; data processing delay, monitoring delays in data processing to ensure that data is available in a timely manner; data availability time, for real-time or near real-time data, ensuring that data can be obtained in a timely manner when needed.

[0091] In some embodiments, the monitoring frequency can be determined according to the characteristics and needs of the monitored object, such as real-time, daily, weekly, monthly, etc., or a long-term monitoring plan can be formulated, including arrangements for regular inspections and temporary inspections. By setting reasonable monitoring rules, data quality can be effectively monitored and managed, and the reliability and value of data can be improved. At the same time, it is also necessary to regularly evaluate and adjust monitoring rules to adapt to business changes and data quality improvement needs.

[0092] S105: Performing quality monitoring on the data corresponding to each field in the checked table structure according to the monitoring rules.

[0093] By adopting the above technical solution, the data quality monitoring method provided in the embodiment of this specification first obtains the table structure and the corresponding data from multiple source databases, matches the corresponding data model from the data model library according to the table structure, and then uses the data model to verify the table structure to obtain the verified table structure, matches the monitoring rules for each field in the verified table structure from the pre-established monitoring rule pool, and performs quality monitoring on the data corresponding to each field in the verified table structure according to the monitoring rules. In this way, by obtaining data from multiple data sources, it is possible to process data from multiple data sources at the same time, thereby improving the monitoring efficiency. Through the data model library and the monitoring rule pool, unified management and standardization of the table structure and data quality of multiple data sources are achieved, thereby eliminating data differences and conflicts between different source databases and improving data consistency and comparability.

[0094] In the embodiments of this specification, refer to Figure 2 , the obtaining of table structures and corresponding data from multiple source databases includes:

[0095] S201: configuring connection parameters for each source database according to the source database type;

[0096] S202: Connecting multiple source databases to the query interface according to the connection parameters;

[0097] S203: Acquire the table structure and corresponding data according to the query statement or query script sent by the query interface.

[0098] It can be understood that, since the connection mode and required parameters of each database may be different, the source database type to be connected is first determined, such as MySQL, PostgreSQL, Oracle, etc. According to the database type, the corresponding connection parameters can be configured, and the connection parameters include the database address (IP or domain name), port number, database name, user name, password, etc. In other embodiments, the connection parameters also include SSL configuration, connection timeout, etc. After obtaining the connection parameters, a connection to each source database is established through programming or configuration tools (such as a database connection pool), and the established database connection is integrated into a unified query interface. This query interface can be an API, Web service or other form of interface for receiving and executing query statements or scripts. The query statements or scripts can be SQL statements, specific query languages ​​or other forms of scripts. In this way, by providing a unified query entry, users or systems can query and obtain data from multiple source databases through this interface. After receiving the query statements or scripts from the user or system, the query interface forwards the received query statements or scripts to the corresponding source database, executes these queries, obtains the query results from the source database, including table structure information (such as column names, data types, etc.) and corresponding data records, and then returns the query results to the user or system.

[0099] In the embodiments of this specification, refer to Figure 3 , matching the data model from the data model library according to the table structure includes:

[0100] S301: Obtaining basic information of fields in each of the table structures;

[0101] S302: Calculate the similarity between the basic information of the field and the basic information of the corresponding field of each data model in the data model library;

[0102] S303: Determine a data model of each of the table structures according to the similarity.

[0103] Specifically, the basic information of the fields in the table structure includes the Chinese name of the field, the English name of the field, whether it is a primary key, whether it is a business primary key, whether it is required, the field type, length, precision, display format, default value, etc. For each field, it is necessary to compare its basic field information with the basic field information of each data model in the data model library, and calculate the similarity between them. In some embodiments, a string matching algorithm (such as Levenshtein distance) can be used to calculate the similarity between field names, and type compatibility rules can be used to determine whether the data type matches. For the matching of multiple fields, a weighted average or other aggregation method can be used to calculate the similarity between the entire table structure and the data model. For each table structure, record its similarity with each data model in the data model library, and select the data model with the highest similarity as the matching result of the table structure. If there are multiple data models with the same similarity, other rules (such as the frequency of use of the data model, the creation time, or the user-specified requirements, etc.) can be further used to select. After selecting the data model of each table structure, the matching structure is recorded, and the record content includes the corresponding relationship between the table structure and the data model and the similarity score for subsequent reuse. In this way, according to the basic information of the fields in the table structure, a more appropriate data model can be effectively and accurately matched for a given surface structure from the data model library, thereby facilitating subsequent verification of the table structure.

[0104] In the embodiments of this specification, refer to Figure 4 , the checking the table structure by using the data model includes:

[0105] S1: Compare the fields in the table structure with the fields in the corresponding data model one by one to determine whether each field in the table structure satisfies the constraint condition;

[0106] S2: If yes, then the verification is completed and the verified table structure is obtained;

[0107] S3: If not, adjusting the fields in the table structure that do not meet the constraint conditions according to the comparison result and the constraint conditions to obtain an adjusted table structure;

[0108] S4: Determine whether each field in the adjusted table structure satisfies the constraint condition;

[0109] S5: If not, repeat step S3 until each field in the table structure satisfies the constraint condition.

[0110] It can be understood that the data model is used as the verification standard for the fields in the table structure. Before data quality monitoring, the obtained table structure needs to be compared with the data model until the two are consistent, and then the data quality monitoring is carried out under the unified data standard. Each data model includes the constraints of each field, such as data type, field length, uniqueness, primary key / foreign key relationship, etc. For each table structure, traverse each field, find the corresponding field constraints in the data model, and check whether the fields in the table structure meet the constraints in the data model. For example, if the data model requires the field to be an integer type, check whether the field is an integer type; for a string type field, verify whether its length does not exceed the maximum length defined in the data model; if the data model defines a default value for the field, verify whether the default value will be automatically used when inserting a new record if no value is provided for the field; for fields that require uniqueness (such as primary keys or fields with unique constraints), verify whether there are duplicate values ​​in the database table. If all fields in the table structure meet the constraints in the data model, the verification process is completed and a table structure that meets the requirements of the data model is obtained.

[0111] If there are fields in the table structure that do not meet the constraints in the data model, these fields need to be adjusted according to the comparison results and the constraints. For example, according to the constraints, modify the field definition, field name, data type, length, whether to allow empty and other attributes in the table structure. In some embodiments, fields can also be added or deleted to meet the requirements of the data model. After adjusting the fields that do not meet the constraints, it is necessary to check again whether the adjusted table structure meets all the constraints in the data model. If there are still fields in the table structure after one adjustment that do not meet the constraints, it is necessary to continue to adjust and recheck until all fields meet the requirements of the data model. In some embodiments, before checking the table structure, the original table structure can be backed up to prevent the need to restore the adjusted table structure to its original state later.

[0112] In the embodiments of this specification, refer to Figure 5 , matching a monitoring rule for each field in the checked table structure from a pre-established monitoring rule pool includes:

[0113] S501: Acquire the field attribute of each field in the checked table structure;

[0114] S502: Matching a monitoring rule for each field from the monitoring rule pool according to the field attribute and a preset matching rule;

[0115] S503: Determine whether the match is successful;

[0116] S504: If yes, the matched monitoring rule is used as the monitoring rule for each field;

[0117] S505: If not, configure a new monitoring rule according to the field attribute, and use the new monitoring rule as the monitoring rule for each field;

[0118] S506: Storing the new monitoring rule in the monitoring rule pool.

[0119] Specifically, the rule name, rule type, expression, variable and status are configured according to the data quality monitoring standard. After saving, a monitoring rule will be generated. After the monitoring rule is created, it will be classified and stored in the monitoring rule pool. The monitoring rules in the monitoring rule pool can be called by different projects to improve the reusability of the monitoring rules. In addition, the monitoring rules can be prefabricated for users to call, which improves usability.

[0120] In this embodiment, the following factors are mainly considered for setting data integrity monitoring rules: (1) field integrity, ensuring that all necessary fields in the data set are filled without omissions; (2) record integrity, checking whether all relevant records are included in the data set without missing records; (3) time range integrity, for time series data, ensuring that the data covers the required time range without data breakpoints. Therefore, the corresponding monitoring rules that can be set include: setting field non-empty checks, marking or filtering records with missing fields; implementing record count checks, comparing the expected number of records with the actual number of records; monitoring the continuity of time series data, identifying and processing data breakpoints.

[0121] When setting data consistency monitoring rules, the following factors are mainly considered: (1) format consistency, ensuring that the format of the data field is consistent with the expectation, such as date format, number format, etc.; (2) logical consistency, checking whether the data follows business rules or logical constraints, such as whether the logical relationship between data is reasonable; (3) inter-system consistency, for data across multiple systems or data sources, ensuring that the data between them is consistent. Therefore, the corresponding monitoring rules that can be set include: using regular expressions or data validation tools to check field formats; implementing logical checks, such as checking dependencies or constraints between data; regularly comparing data from different systems or data sources to identify and handle inconsistencies.

[0122] When setting data accuracy monitoring rules, the following factors are mainly considered: (1) ensuring that the data comes from a reliable and accurate information source; (2) ensuring that the calculation logic is correct for calculated or derived fields; and (3) verifying the accuracy of the data by comparing it with other reliable data sources. Therefore, the corresponding monitoring rules that can be set include: verifying the reliability and accuracy of the data source; performing unit tests or integration tests on calculated or derived fields to ensure their accuracy; setting data comparison rules, regularly comparing with other data sources, identifying and handling inaccurate data.

[0123] For setting data timeliness monitoring rules, the following factors are mainly considered: (1) ensuring that data is updated at a predetermined frequency; (2) monitoring delays in the data processing process to ensure that data is available in a timely manner; (3) for real-time or near real-time data, ensuring that data can be obtained in a timely manner when needed. Therefore, the corresponding monitoring rules that can be set include: setting data update frequency check rules to monitor whether the data is updated according to the predetermined time; implementing data processing delay monitoring to identify and handle factors that cause delays; setting data availability time check rules to ensure that data can be obtained in a timely manner when needed. In other embodiments, the frequency of monitoring can also be determined according to the characteristics and needs of the monitored object, such as real-time, daily, weekly, monthly, etc., or a long-term monitoring plan can be formulated, including arrangements for regular inspections and temporary inspections.

[0124] When matching monitoring rules, it is necessary to obtain the detailed attributes of each field in the table structure that has been verified and confirmed to meet the requirements of the data model, including field name, data type, length, whether it is allowed to be empty, default value, whether it is a primary key or a foreign key, etc. Then for each field, according to the field attributes and the preset matching rules, the matching monitoring rules are searched in the monitoring rule pool, where the matching rules are set based on the field attributes, for example, the monitoring rules applicable to the data type can be selected from the monitoring rule pool according to the data type of the field (such as integer, floating point number, string, date, etc.). In some embodiments, multiple field attributes can also be considered to match the monitoring rules. According to the matching results, it is confirmed whether a suitable monitoring rule has been found. If multiple matching monitoring rules are found, it may be necessary to further select the optimal rule, and the final rule can be determined according to the frequency of use and update time of the rule. If a suitable monitoring rule cannot be matched from the monitoring rule pool, it is necessary to configure a new monitoring rule according to the field attributes and monitoring requirements. In this way, by matching the monitoring rules for each field, the quality of the data corresponding to each field in the table structure after verification is monitored according to the monitoring rules, so that the data of each field has targeted monitoring rules, thereby improving the accuracy of data quality monitoring.

[0125] In the embodiments of this specification, refer to Figure 6 , the method further comprises:

[0126] S601: constructing a monitoring scenario corresponding to each table structure according to a monitoring rule corresponding to each field in each table structure;

[0127] S602: Storing the monitoring scenario in a monitoring scenario case library.

[0128] It can be understood that the monitoring rules configured for each field in the table structure, such as the field is not empty, the length is 10, etc., are arranged according to the table dimension to form a monitoring scenario corresponding to the table structure. By converting the monitoring rules in the table structure into specific monitoring scenarios and storing these scenarios, when data quality monitoring of the same type of table structure is required in the future, the monitoring scenario can be directly selected from the monitoring scenario case library for monitoring, thereby improving the efficiency and accuracy of data quality monitoring, and also providing strong support for data governance.

[0129] In the embodiments of this specification, refer to Figure 7 , the method further comprises:

[0130] S701: Obtain data quality monitoring results;

[0131] S702: Generate a monitoring report according to the data quality monitoring result, and send it to the target terminal.

[0132] It can be understood that the collected monitoring data is summarized and organized to form a comprehensive data quality monitoring result. The monitoring result may include statistical information such as the monitoring status of each field, the number of errors, and the distribution of error types. According to the summarized monitoring results, a detailed monitoring report is automatically generated. The monitoring report may include the report title, monitoring time, monitoring scope, monitoring result overview, detailed error list, error cause analysis, improvement suggestions, etc. The report format can be text, table, chart, PDF file, etc., so that users can read and understand it. In this way, by obtaining the data quality monitoring results and generating a detailed monitoring report to send to the target terminal, managers can promptly understand the data quality status, find potential problems, and take corresponding improvement measures.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the acquisition, storage, use, and processing of data in the technical solutions described in the embodiments of this application are in compliance with relevant regulations.

[0134] Based on the above-described data quality monitoring method, the embodiments of this specification also provide a data quality monitoring device. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiments of this specification and a device in combination with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiments of this specification is as described in the following embodiments. Since the implementation scheme and method for solving the problem of the device are similar, the implementation of the specific device of the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0135] Specifically, Figure 8 This is a schematic diagram of the module structure of an embodiment of a data quality monitoring device provided in the embodiments of this specification, referring to Figure 8 As shown, a data quality monitoring device provided in an embodiment of this specification includes:

[0136] The acquisition module 801 is used to acquire table structures and corresponding data from multiple source databases;

[0137] A first matching module 802, used to match a data model from a data model library according to the table structure;

[0138] A checking module 803 is used to check the table structure using the data model to obtain a checked table structure;

[0139] A second matching module 804, configured to match a monitoring rule for each field in the checked table structure from a pre-established monitoring rule pool;

[0140] The monitoring module 805 is used to perform quality monitoring on the data corresponding to each field in the checked table structure according to the monitoring rules.

[0141] The beneficial effects obtained by the device provided in the embodiments of this specification are consistent with the beneficial effects obtained by the above method and will not be repeated here.

[0142] Reference Fig. 9As shown, based on the above-mentioned data quality monitoring method, a computer device 902 is also provided in an embodiment of this specification, wherein the above-mentioned method runs on the computer device 902. The computer device 902 may include one or more processors 904, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 902 may also include any memory 906, which is used to store any kind of information such as code, settings, data, etc. Non-limiting, for example, the memory 906 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 902. In one case, when the processor 904 executes an associated instruction stored in any memory or a combination of memories, the computer device 902 can perform any operation of the associated instruction. The computer device 902 also includes one or more drive mechanisms 908 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0143] The computer device 902 may also include an input / output module 910 (I / O) for receiving various inputs (via input devices 912) and for providing various outputs (via output devices 914). A specific output mechanism may include a presentation device 916 and an associated graphical user interface (GUI) 918. In other embodiments, the input / output module 910 (I / O), the input device 912, and the output device 914 may not be included, and the computer device 902 may be used as a computer device in a network. The computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the components described above together.

[0144] The communication link 922 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0145] Corresponding to Figures 1 to 7 In addition to the method shown, an embodiment of the specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0146] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 7 The method shown.

[0147] The embodiments of the present specification also provide a computer program product, including at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the following Figures 1 to 7 The method shown.

[0148] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0149] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0150] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0154] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0156] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A data quality monitoring method, characterized in that: The method comprises: Obtain table structures and corresponding data from multiple source databases; Matching a data model from a data model library according to the table structure; Using the data model to check the table structure, to obtain a checked table structure; Matching a monitoring rule for each field in the checked table structure from a pre-established monitoring rule pool; The quality of the data corresponding to each field in the checked table structure is monitored according to the monitoring rules.

2. The method according to claim 1, characterized in that The obtaining of table structures and corresponding data from multiple source databases includes: Configure connection parameters for each source database based on the source database type; Connecting multiple source databases to the query interface according to the connection parameters; According to the query statement or query script sent by the query interface, the table structure and corresponding data are obtained.

3. The method according to claim 1, characterized in that The matching of the data model from the data model library according to the table structure includes: Obtaining basic information of fields in each of the table structures; Calculating the similarity between the basic information of the field and the basic information of the corresponding field of each data model in the data model library; A data model of each of the table structures is determined according to the similarity.

4. The method according to claim 1, characterized in that: The data model includes constraints for each field; The checking the table structure by using the data model includes: S1: Compare the fields in the table structure with the fields in the corresponding data model one by one to determine whether each field in the table structure satisfies the constraint condition; S2: If yes, then the verification is completed and the verified table structure is obtained; S3: If not, adjusting the fields in the table structure that do not meet the constraint conditions according to the comparison result and the constraint conditions to obtain an adjusted table structure; S4: Determine whether each field in the adjusted table structure satisfies the constraint condition; S5: If not, repeat step S3 until each field in the table structure satisfies the constraint condition.

5. The method according to claim 1, characterized in that The matching of monitoring rules for each field in the checked table structure from a pre-established monitoring rule pool includes: Obtaining the field attributes of each field in the verified table structure; Matching a monitoring rule for each field from the monitoring rule pool according to the field attributes and a preset matching rule; Determine whether the match is successful; If yes, the matched monitoring rule is used as the monitoring rule for each field; If not, configuring a new monitoring rule according to the field attribute, and using the new monitoring rule as the monitoring rule for each field; The new monitoring rule is stored in the monitoring rule pool.

6. The method according to claim 1, characterized in that The method further comprises: According to the monitoring rules corresponding to each field in each of the table structures, a monitoring scenario corresponding to the table structure is constructed; The monitoring scenario is stored in a monitoring scenario case library.

7. The method according to claim 1, characterized in that The method further comprises: Obtain data quality monitoring results; A monitoring report is generated according to the data quality monitoring result and sent to the target terminal.

8. A data quality monitoring device, characterized in that: The device comprises: The acquisition module is used to obtain table structures and corresponding data from multiple source databases; A first matching module, used for matching a data model from a data model library according to the table structure; A checking module, used to check the table structure using the data model to obtain a checked table structure; A second matching module, used for matching monitoring rules for each field in the checked table structure from a pre-established monitoring rule pool; The monitoring module is used to perform quality monitoring on the data corresponding to each field in the checked table structure according to the monitoring rules.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The method comprises at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1 to 7.

Images