A method and device for monitoring data quality

Abstract

The present application provides a data quality monitoring method and device, which belongs to the computer field and can be used in the financial field. The method includes: screening metadata in a data warehouse according to pre-set data quality monitoring rules to obtain a data quality monitoring table-level list; using a pre-built machine learning model to screen the metadata to obtain a data quality monitoring field-level list; determining a data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list; performing data quality monitoring on the monitoring information in the data quality monitoring output list; the monitoring information at least includes a database name, a data table name, a data field name, and unsatisfied monitoring rules.

Description

Technical Field

[0001] The present application relates to the computer field and can be used in the financial field, specifically a data quality monitoring method and device. Background Art

[0002] An important part of the big data construction system is data quality monitoring. Usually, the data quality monitoring system will perform batch management on the data it monitors at regular intervals. All data tables that have not been subject to basic monitoring rules (such as primary key uniqueness check and table not empty check) are pushed to the responsible person for confirmation, and the monitoring rules are manually configured. Although the above process can serve as a data quality feedback channel to help business personnel monitor data, it is relatively cumbersome. Moreover, data users often notify the responsible person to troubleshoot the problem and configure the corresponding rules only when they find problems with the data, which also lags in time.

[0003] To sum up, the defects of the existing technology are as follows: First, manually configuring data monitoring rules will increase the operation time of the entire data link; second, there are many data quality issues other than basic monitoring rules that need to be actively monitored, such as fluctuations in indicator data of certain fields, which are not reflected in the existing technology. Summary of the invention

[0004] In response to the problems in the prior art, the present application provides a data quality monitoring method and device, which can perform data quality monitoring based on data quality monitoring rules using a pre-built machine learning model.

[0005] In order to solve the above technical problems, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides a data quality monitoring method, comprising:

[0007] Screen the metadata in the data warehouse according to the pre-set data quality monitoring rules to obtain a data quality monitoring table-level list;

[0008] Using a pre-built machine learning model to screen the metadata, obtaining a field-level list of data quality monitoring;

[0009] Determine the data quality monitoring output list based on the data quality monitoring table-level list and the data quality monitoring field-level list;

[0010] Data quality monitoring is performed on the monitoring information in the data quality monitoring output list; the monitoring information at least includes a database name, a data table name, a data field name, and unsatisfied monitoring rules.

[0011] Furthermore, before determining the data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list, it also includes:

[0012] Obtaining table information, field information, field dependency and monitoring rule information in the data warehouse;

[0013] The metadata is constructed according to the table information, field information, field dependency and monitoring rule information.

[0014] Furthermore, the metadata in the data warehouse is screened according to the preset data quality monitoring rules to obtain a data quality monitoring table-level list, including:

[0015] Screening whether each interface table in the data warehouse is empty;

[0016] If it is not empty, then check whether the primary key of the interface table is unique;

[0017] If the table primary key is unique, screen each interface table in the data warehouse to see if there are abnormal fields;

[0018] Determine the data quality monitoring table level list based on the screening results.

[0019] Furthermore, the metadata is screened using a pre-built machine learning model to obtain a data quality monitoring field-level list, including:

[0020] Construct a field node directed graph according to the table information, field information, and field dependency;

[0021] Determine the importance value of each field node in the field node directed graph using a pre-built machine learning model;

[0022] The data quality monitoring field level list is determined according to a preset importance threshold and the importance value of each field node.

[0023] Furthermore, the step of determining the data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list includes:

[0024] Respectively obtain the fields in the data quality monitoring table level list and the fields in the data quality monitoring field level list;

[0025] Take the union of the acquired fields to get the data quality monitoring output list.

[0026] Furthermore, the data quality monitoring of the monitoring information in the data quality monitoring output list includes:

[0027] Generate a corresponding database description language according to the data quality monitoring rules;

[0028] Data quality monitoring is performed on the monitoring information in the data quality monitoring output list according to the database description language.

[0029] In a second aspect, the present application provides a data quality monitoring device, comprising:

[0030] A table-level list generating unit, used to screen metadata in the data warehouse according to preset data quality monitoring rules to obtain a data quality monitoring table-level list;

[0031] A field-level list generating unit, configured to screen the metadata using a pre-built machine learning model to obtain a field-level list for data quality monitoring;

[0032] An output list generating unit, used for determining a data quality monitoring output list according to a data quality monitoring table-level list and a data quality monitoring field-level list;

[0033] The data quality monitoring unit is used to perform data quality monitoring on the monitoring information in the data quality monitoring output list; the monitoring information at least includes a database name, a data table name, a data field name and unsatisfied monitoring rules.

[0034] Furthermore, the data quality monitoring device further includes:

[0035] An information acquisition unit, used to acquire table information, field information, field dependency and monitoring rule information in the data warehouse;

[0036] The metadata construction unit is used to construct the metadata according to the table information, field information, field dependency and monitoring rule information.

[0037] Furthermore, the table-level list generating unit includes:

[0038] An empty table screening module is used to screen whether each interface table in the data warehouse is empty;

[0039] A primary key screening module, used for screening whether the primary key of the interface table is unique if it is not empty;

[0040] An abnormal field screening module is used to screen whether there are abnormal fields in each interface table in the data warehouse if the table primary key is unique;

[0041] The table-level list determination module is used to determine the data quality monitoring table-level list according to the screening results.

[0042] Furthermore, the field level list generating unit includes:

[0043] A directed graph construction module, used to construct a field node directed graph according to the table information, field information, and field dependency;

[0044] An importance value determination module, used to determine the importance value of each field node in the field node directed graph using a pre-built machine learning model;

[0045] The field-level list determination module is used to determine the data quality monitoring field-level list based on a preset importance threshold and the importance value of each field node.

[0046] Furthermore, the output list generating unit includes:

[0047] A field acquisition module is used to respectively acquire fields in a data quality monitoring table-level list and fields in a data quality monitoring field-level list;

[0048] The output list determination module is used to take the union of the acquired fields to obtain the data quality monitoring output list.

[0049] Furthermore, the data quality monitoring unit includes:

[0050] A description language generation module, used to generate a corresponding database description language according to the data quality monitoring rules;

[0051] The data quality monitoring module is used to perform data quality monitoring on the monitoring information in the data quality monitoring output list according to the database description language.

[0052] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data quality monitoring method when executing the program.

[0053] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the data quality monitoring method when executed by a processor.

[0054] In response to the problems in the prior art, the data quality monitoring method and device provided in this application can utilize the input layer, rule strategy layer, algorithm layer and output layer of the data quality monitoring system to achieve full coverage and more accurate data quality monitoring at the table and field levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 This is one of the flow charts of the data quality monitoring method in the embodiment of the present application;

[0057] Figure 2 This is the second flow chart of the data quality monitoring method in the embodiment of the present application;

[0058] Figure 3 A flowchart of obtaining a data quality monitoring table level list in an embodiment of the present application;

[0059] Figure 4 A flowchart of obtaining a field-level list of data quality monitoring in an embodiment of the present application;

[0060] Figure 5 A flowchart for determining a data quality monitoring output list in an embodiment of the present application;

[0061] Figure 6 This is a flow chart for performing data quality monitoring in an embodiment of the present application;

[0062] Figure 7 This is one of the structural diagrams of the data quality monitoring device in the embodiment of the present application;

[0063] Figure 8 This is the second structural diagram of the data quality monitoring device in the embodiment of the present application;

[0064] Fig. 9 This is a structural diagram of a table-level list generation unit in an embodiment of the present application;

[0065] Fig.10 This is a structural diagram of a field-level list generation unit in an embodiment of the present application;

[0066] Fig.11 This is a structural diagram of an output list generation unit in an embodiment of the present application;

[0067] Fig.12 This is a structural diagram of a data quality monitoring unit in an embodiment of the present application;

[0068] Fig.13 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0069] Fig.14 This is one of the schematic diagrams of the data quality monitoring method in the embodiment of the present application;

[0070] Fig.15 This is the second schematic diagram of the data quality monitoring method in the embodiment of the present application;

[0071] Fig.16 This is the third schematic diagram of the data quality monitoring method in the embodiment of the present application;

[0072] Fig.17 This is the fourth schematic diagram of the data quality monitoring method in the embodiment of the present application. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] It should be noted that the data quality monitoring method and device provided in the present application can be used in the financial field, and can also be used in any field other than the financial field. The application field of the data quality monitoring method and device provided in the present application is not limited.

[0075] See also Figure 1 In order to monitor data quality using a pre-built machine learning model according to data quality monitoring rules, the present application provides a data quality monitoring method, including:

[0076] S101: screening metadata in the data warehouse according to pre-set data quality monitoring rules to obtain a data quality monitoring table-level list;

[0077] S102: Screen the metadata using a pre-built machine learning model to obtain a data quality monitoring field-level list;

[0078] It is understandable that the data quality monitoring method provided by the present application can be applied to data storage systems in multiple fields such as financial data centers and e-commerce platforms to achieve table-level monitoring and field-level monitoring of data quality. The so-called table-level monitoring refers to the general monitoring of each data table according to the pre-set data quality table-level monitoring rules for each data table stored in the database. The monitoring granularity of table-level monitoring is relatively coarse, and it is impossible to monitor each data field in the table in depth. The so-called field-level monitoring refers to the use of a pre-built machine learning model to deeply screen each data field in each data table on the basis of table-level monitoring, and complete data quality monitoring with relatively fine granularity. The data quality monitoring method provided by the present application is implemented based on a graph algorithm, and this implementation method can improve the comprehensiveness and accuracy of the configuration of data quality monitoring rules. The above-mentioned data quality monitoring can be implemented by relying on a data quality monitoring system. In other words, the data quality monitoring system can serve as the executor of the data quality monitoring method provided by the present application.

[0079] S103: Determine a data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list;

[0080] S104: Perform data quality monitoring on the monitoring information in the data quality monitoring output list.

[0081] It is understandable that, based on the generation of the data quality monitoring table-level list and the data quality monitoring field-level list, the data quality monitoring system can read the monitoring information in the data quality monitoring table-level list and the data quality monitoring field-level list, and obtain the data quality monitoring output list after processing and integration; finally, the monitoring information in the data quality monitoring output list is subjected to data quality monitoring. It should be noted that the monitoring information at least includes the database name, data table name, data field name and unsatisfied monitoring rules, and in a preferred embodiment, it may also include the person in charge of data quality monitoring, etc.

[0082] Among them, the data quality monitoring table-level list and data quality monitoring field-level list can be shown in the following tables respectively:

[0083]

[0084]

[0085] From the above description, it can be seen that the data quality monitoring method provided in this application can utilize the input layer, rule strategy layer, algorithm layer and output layer of the data quality monitoring system to achieve full coverage and more accurate data quality monitoring at the table and field levels.

[0086] See also Figure 2 , before determining the data quality monitoring output list based on the data quality monitoring table-level list and the data quality monitoring field-level list, it also includes:

[0087] S201: Acquire table information, field information, field dependency and monitoring rule information in the data warehouse;

[0088] S202: Build metadata based on table information, field information, field dependency, and monitoring rule information.

[0089] Understandably, see Fig.17 In the embodiment of the present application, the data quality monitoring system can be divided into four layers, namely, input layer, rule strategy layer, algorithm layer and output layer. Among them, the process of constructing metadata can be performed in the input layer.

[0090] Specifically, the data quality monitoring system can access the metadata of the data warehouse or data mart at the input layer. These metadata include but are not limited to table information, field information, field dependency and monitoring rule information. Metadata can be used to describe the basic information, dependency information and call information of the data, such as table information can include the maintainer of the table, the life cycle of the table, the name of the table, the name of the library to which the table belongs, the number of visits to the table and the storage size of the table; field information can include the Chinese description of the field, the name of the table to which the field belongs, the maintainer of the field, the null value rate of the field and the type of the field; field dependency can include the three elements of the parent node field (i.e., table name, library name, field name), the three elements of the child node (i.e., table name, library name, field name), the maintainer of the parent and child nodes, the strong and weak dependency type, etc.; monitoring rule information can include monitoring objects, specific monitoring rules, monitoring thresholds, the strength of monitoring rules, etc. These metadata can be obtained through automatic database collection and manual registration.

[0091] For example, a table information can be as follows:

[0092]

[0093] The monitoring rule information can be as follows:

[0094] Serial number Library Name Table name Field Name Rule Level Rule Description ... 1 A TABLE 1 / Table Level Primary key unique ...

[0095] By associating the two metadata according to the library name + table name, more information can be obtained, such as whether the table is configured with data monitoring rules and what specific monitoring rules are configured, etc., which provides data support for the rule operation of the rule strategy layer. This process can also be understood as the process of building metadata. Of course, metadata can also be directly obtained from the data warehouse, and this application is not limited to this.

[0096] From the above description, it can be seen that the data quality monitoring method provided by this application can obtain and construct metadata.

[0097] See also Figure 3 , according to the pre-set data quality monitoring rules, the metadata in the data warehouse is screened to obtain a data quality monitoring table-level list, including:

[0098] S301: Screening whether each interface table in the data warehouse is empty;

[0099] S302: If it is not empty, then check whether the primary key of the interface table is unique;

[0100] S303: When the table primary key is unique, screen each interface table in the data warehouse to see if there are abnormal fields;

[0101] S304: Determine a data quality monitoring table-level list based on the screening results.

[0102] S305: The background records the interface table with abnormalities.

[0103] It is understandable that the data quality monitoring system in the embodiment of the present application can complete table-level monitoring at the rule strategy layer, and pull the data quality monitoring table-level list by configuring the table-level data quality monitoring rules. At the rule strategy layer, in the embodiment, relatively simple rules can generally be configured to perform preliminary and coarse-grained monitoring of the data table (interface table). These rules can screen out problematic data tables (interface tables) that do not meet the table-level data quality monitoring rules by performing simple statistics on the metadata.

[0104] For example, the present application embodiment recommends configuring the following rules:

[0105] ①The interface table must be configured with the rule that the table is not empty

[0106] ②The interface table must be configured with a primary key unique rule

[0107] ③The interface table must be configured with field-level monitoring rules

[0108] ④ The character fields of the interface table need to be configured with vacancy rate fluctuation monitoring rules (for example, if the vacancy rate of the field "user's permanent residence" was 10% 7 days ago and the vacancy rate on the day is 50%, it is considered that the fluctuation is too large and there is a data quality problem).

[0109] ⑤ The numerical fields in the interface table need to be configured with total fluctuation monitoring rules (for example, for the field "Transaction amount in the last year", if the sum of all records in the table before one month is 100 million, and the sum of all records in the table on that day is 800 million, it is considered that the fluctuation is too large and there is a quality problem).

[0110] It should be noted that the temporary tables and intermediate tables generated in the business are not output outside the business system. Therefore, compared with the interface tables output outside the business system, the temporary tables and intermediate tables are relatively less important. In order to avoid delaying the output time of the data link, in the embodiment of the present application, the data in the temporary tables and intermediate tables are not subject to data quality monitoring according to the above rules, and the above data quality monitoring rules are only implemented on the interface tables. In a more preferred embodiment, the above data quality monitoring rules applied to the interface tables can also be applied to the temporary tables and intermediate tables.

[0111] Among them, although ④ and ⑤ involve fields, they need to be monitored as a whole at the table level, so they need to be monitored according to the table-level data quality monitoring rules.

[0112] From the above description, it can be seen that the data quality monitoring method provided in the present application can screen the metadata in the data warehouse according to the pre-set data quality monitoring rules to obtain a data quality monitoring table-level list.

[0113] See also Figure 4, using pre-built machine learning models to screen metadata and obtain a field-level checklist for data quality monitoring, including:

[0114] S401: construct a field node directed graph according to table information, field information, and field dependency;

[0115] S402: Determine the importance value of each field node in the field node directed graph using a pre-built machine learning model;

[0116] S403: Determine a data quality monitoring field level list according to a preset importance threshold and the importance value of each field node.

[0117] It is understandable that the data quality monitoring system in the embodiment of the present application can complete field-level monitoring at the rule strategy layer, extract field features using graph algorithms, and then use machine learning models to predict whether the field needs to be monitored. The specific process is as follows:

[0118] ① Graph construction: The dependency relationship between fields can be stored in a binary relationship table (Table 1). Using the idea of ​​graph, the binary relationship is converted into a graph relationship. The database name + table name + field name are used as nodes, and the dependency relationship is used as the edge to construct the graph. The graph is a directed graph, and the direction is from the parent node with a dependency relationship to the child node.

[0119] For example, the binary relationship described in Table 1 can be converted to Fig.14 .

[0120] Table 1 Field lineage relationship diagram

[0121] Parent library name Parent table name Parent field name Sub-library name Subtable name Subfield name A Table 1 var1 A Table 2 Var1 A Table 1 Var1 B Table 3 Var3

[0122] ②The importance of computing nodes

[0123] The importance of a node is often calculated based on two assumptions: first, the more downstream nodes a node has, the more important the node may be; second, for an important node, its upstream nodes are often relatively important. The embodiment of the present application adopts an improved PageRank algorithm (which can also be understood as a reconstructed machine learning model) and uses the PR value calculated for each node as the importance value of the node.

[0124] After improving the applicability of the scenario, the embodiment of the present application uses the following formula to measure the importance of the node:

[0125]

[0126] p i is the current node, p j For p i One of the downstream nodes of the node, Mp i For pi All downstream sets of nodes, L(p j ) is p j The in-degree of the node.

[0127] The PR values ​​of all nodes will be iteratively calculated until the results converge, and then the PR values ​​will be normalized to obtain the final value. This application does not specifically limit the convergence threshold, which can be selected according to actual conditions.

[0128] Fig.15 A simple example is used to describe the calculation process of the above algorithm. Assume that there are Fig.15 Dependencies shown.

[0129] Then, from the in-degree case, we can build the matrix Dividing by the sum of the numbers in that row gives After transposing, we can get

[0130] Assuming the initial PR value of each node is 1, we can get but

[0131]

[0132] Continue iterating until X' converges, that is:

[0133] In one iteration calculation,

[0134] In another iterative calculation,

[0135] It can be found that X' has converged at this time, and then the PR value is normalized to obtain That is, the PR values ​​of the three nodes a, b and c are 0.4, 0.2 and 0.4 respectively.

[0136] Let's take another example: For example, the dependency of a field can be transformed into Fig.16 The directed graph shown.

[0137] The formula can be calculated as follows:

[0138] 'B':0.027777907029846994

[0139] 'A':0.2777758071818005

[0140] 'C':0.06250017033612078

[0141] 'D':0.07638886092778685

[0142] 'E':0.06944431583032772

[0143] 'F':0.04166659762151306

[0144] 'G':0.027777907029846994

[0145] 'X':0.027777907029846994

[0146] 'Y':0.027777907029846994

[0147] 'Z':0.027777907029846994

[0148] 'M':0.16666735647660766

[0149] 'H':0.16666735647660766

[0150] The embodiments of the present application may use the above-mentioned algorithm for calculation, or may select other algorithms for calculation, and the present application is not limited thereto.

[0151] ③ Determine the threshold: For nodes that exceed a certain threshold, it can be considered that monitoring rules need to be configured and enter the data quality monitoring field level list. If the threshold is set to 0.07, for the above example, the four nodes A, M, H, and D are considered to be important fields and monitoring rules need to be configured.

[0152] From the above description, it can be seen that the data quality monitoring method provided in this application can use a pre-built machine learning model to screen metadata and obtain a data quality monitoring field-level list.

[0153] See also Figure 5 , determine the data quality monitoring output list based on the data quality monitoring table-level list and the data quality monitoring field-level list, including:

[0154] S501: respectively obtaining fields in the table-level list and the field-level list according to data quality monitoring;

[0155] S502: Taking a union of the acquired fields to obtain a data quality monitoring output list.

[0156] It is understandable that the data quality monitoring system can obtain which table needs to be monitored from the data quality monitoring table-level list. For the table that needs to be monitored, it can be understood that all the fields therein also need to be monitored. At the same time, the data quality monitoring system can also read the data quality monitoring field-level list to obtain the fields that need to be quality monitored. Finally, all fields are taken together to obtain a data quality monitoring output list. The data quality monitoring output list includes at least but is not limited to monitoring information such as database name, data table name, data field name, and unmet monitoring rules.

[0157] From the above description, it can be seen that the data quality monitoring method provided by the present application can determine the data quality monitoring output list based on the data quality monitoring table-level list and the data quality monitoring field-level list.

[0158] See also Figure 6 , perform data quality monitoring on the monitoring information in the data quality monitoring output list, including:

[0159] S601: Generate a corresponding database description language according to data quality monitoring rules;

[0160] S602: Perform data quality monitoring on the monitoring information in the data quality monitoring output list according to the database description language.

[0161] It is understandable that the data quality monitoring system can generate a corresponding database description language according to the data quality monitoring rules. For example, when configuring table-level data quality monitoring rules, the screening of primary key uniqueness can correspond to the following database description language. Among them, the more commonly used is Structured Query Language (SQL).

[0162] Select t1.database name, t1.table name, t1.field name, t1.maintainer, 'primary key unique' as missing rule description

[0163] From

[0164] (Select * from table metadata where table type = 'interface table') t1

[0165] Left outer join

[0166] (Select*from monitoring rule metadata where field name = ' / ' and rule description =

[0167] 'Primary key unique')t2

[0168] On t1.library name = t2.library name and t1.table name = t2.table name

[0169] where t2.table name is null

[0170] The above database description language can be used to screen out data tables with non-unique primary keys, and other data quality monitoring rules can be deduced in the same way. Finally, data quality monitoring can be performed on the monitoring information in the data quality monitoring output list according to the database description language.

[0171] It can be seen from the above description that the data quality monitoring method provided in the present application can perform data quality monitoring on the monitoring information in the data quality monitoring output list.

[0172] Based on the same inventive concept, the embodiments of the present application also provide a data quality monitoring device, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of solving the problem by the data quality monitoring device is similar to that of the data quality monitoring method, the implementation of the data quality monitoring device can refer to the implementation of the method based on the software performance benchmark determination, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0173] See also Figure 7 In order to monitor data quality using a pre-built machine learning model according to data quality monitoring rules, the present application provides a data quality monitoring device, including:

[0174] The table-level list generating unit 701 is used to screen the metadata in the data warehouse according to the preset data quality monitoring rules to obtain the data quality monitoring table-level list;

[0175] A field-level list generating unit 702 is used to screen the metadata using a pre-built machine learning model to obtain a data quality monitoring field-level list;

[0176] An output list generating unit 703 is used to determine a data quality monitoring output list according to a data quality monitoring table-level list and a data quality monitoring field-level list;

[0177] The data quality monitoring unit 704 is used to perform data quality monitoring on the monitoring information in the data quality monitoring output list; the monitoring information at least includes a database name, a data table name, a data field name and unsatisfied monitoring rules.

[0178] See also Figure 8 , the data quality monitoring device further comprises:

[0179] The information acquisition unit 801 is used to acquire table information, field information, field dependency and monitoring rule information in the data warehouse;

[0180] The metadata construction unit 802 is used to construct the metadata according to the table information, field information, field dependency and monitoring rule information.

[0181] See also Fig. 9 , the table-level list generating unit 701 includes:

[0182] Empty table screening module 901, used to screen whether each interface table in the data warehouse is empty;

[0183] A primary key screening module 902 is used to screen whether the table primary key of the interface table is unique if it is not empty;

[0184] An abnormal field screening module 903 is used to screen whether there are abnormal fields in each interface table in the data warehouse if the table primary key is unique;

[0185] The table-level list determination module 904 is used to determine the data quality monitoring table-level list according to the screening results.

[0186] See also Fig.10 , the field level list generating unit 702 comprises:

[0187] A directed graph construction module 1001 is used to construct a field node directed graph according to the table information, field information, and field dependency relationship;

[0188] An importance value determination module 1002, used to determine the importance value of each field node in the field node directed graph using a pre-built machine learning model;

[0189] The field-level list determination module 1003 is used to determine the data quality monitoring field-level list according to a preset importance threshold and the importance value of each field node.

[0190] See also Fig.11 , the output list generating unit 703 includes:

[0191] A field acquisition module 1101 is used to respectively acquire fields in a data quality monitoring table-level list and fields in a data quality monitoring field-level list;

[0192] The output list determination module 1102 is used to obtain the union of the acquired fields to obtain the data quality monitoring output list.

[0193] See also Fig.12 , the data quality monitoring unit 704 includes:

[0194] Description language generation module 1201, used to generate corresponding database description language according to the data quality monitoring rules;

[0195] The data quality monitoring module 1202 is used to perform data quality monitoring on the monitoring information in the data quality monitoring output list according to the database description language.

[0196] From the hardware level, in order to be able to perform data quality monitoring using a pre-built machine learning model according to data quality monitoring rules, the present application provides an embodiment of an electronic device for implementing all or part of the content of the data quality monitoring method, and the electronic device specifically includes the following content:

[0197] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the data quality monitoring device and related devices such as core business systems, user terminals and related databases; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the data quality monitoring method and the embodiment of the data quality monitoring device in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0198] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0199] In practical applications, part of the data quality monitoring method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0200] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0201] Fig.13 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig.13 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig.13 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0202] In one embodiment, the data quality monitoring method function may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control:

[0203] S101: screening metadata in the data warehouse according to pre-set data quality monitoring rules to obtain a data quality monitoring table-level list;

[0204] S102: Screen the metadata using a pre-built machine learning model to obtain a data quality monitoring field-level list;

[0205] S103: Determine a data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list;

[0206] S104: Perform data quality monitoring on the monitoring information in the data quality monitoring output list; the monitoring information at least includes a database name, a data table name, a data field name, and unsatisfied monitoring rules.

[0207] From the above description, it can be seen that the data quality monitoring method provided in this application can utilize the input layer, rule strategy layer, algorithm layer and output layer of the data quality monitoring system to achieve full coverage and more accurate data quality monitoring at the table and field levels.

[0208] In another embodiment, the data quality monitoring device can be configured separately from the central processing unit 9100. For example, the data quality monitoring device of the data composite transmission device can be configured as a chip connected to the central processing unit 9100, and the function of the data quality monitoring method is realized through the control of the central processing unit.

[0209] like Fig.13 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig.13 In addition, the electronic device 9600 may also include Fig.13 For components not shown, reference may be made to the prior art.

[0210] like Fig.13 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0211] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0212] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0213] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0214] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0215] The communication module 9110 is a transmitter / receiver 9110 that sends and receives signals via an antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0216] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0217] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the data quality monitoring method in the above embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the data quality monitoring method in the above embodiment, where the execution subject is a server or a client, are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0218] S101: screening metadata in the data warehouse according to pre-set data quality monitoring rules to obtain a data quality monitoring table-level list;

[0219] S102: Screen the metadata using a pre-built machine learning model to obtain a data quality monitoring field-level list;

[0220] S103: Determine a data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list;

[0221] S104: Perform data quality monitoring on the monitoring information in the data quality monitoring output list; the monitoring information at least includes a database name, a data table name, a data field name, and unsatisfied monitoring rules.

[0222] From the above description, it can be seen that the data quality monitoring method provided in this application can utilize the input layer, rule strategy layer, algorithm layer and output layer of the data quality monitoring system to achieve full coverage and more accurate data quality monitoring at the table and field levels.

[0223] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0224] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0225] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0226] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0227] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data quality monitoring method, characterized in that: include: Screen the metadata in the data warehouse according to the pre-set data quality monitoring rules to obtain a data quality monitoring table-level list; The metadata is screened using a pre-built machine learning model to obtain a data quality monitoring field-level list; wherein the method of screening the metadata using a pre-built machine learning model to obtain a data quality monitoring field-level list includes: constructing a field node directed graph based on table information, field information, and field dependencies; determining the importance value of each field node in the field node directed graph using a pre-built machine learning model; and determining the data quality monitoring field-level list based on a pre-set importance threshold and the importance value of each field node; Determine the data quality monitoring output list based on the data quality monitoring table-level list and the data quality monitoring field-level list; Data quality monitoring is performed on the monitoring information in the data quality monitoring output list; the monitoring information at least includes a database name, a data table name, a data field name, and unsatisfied monitoring rules.

2. The data quality monitoring method according to claim 1, characterized in that: Before determining the data quality monitoring output list based on the data quality monitoring table-level list and the data quality monitoring field-level list, it also includes: Obtaining table information, field information, field dependency and monitoring rule information in the data warehouse; The metadata is constructed according to the table information, field information, field dependency and monitoring rule information.

3. The data quality monitoring method according to claim 2, characterized in that: The metadata in the data warehouse is screened according to the preset data quality monitoring rules to obtain a data quality monitoring table-level list, including: Screening whether each interface table in the data warehouse is empty; If it is not empty, then check whether the primary key of the interface table is unique; If the table primary key is unique, screen each interface table in the data warehouse to see if there are abnormal fields; Determine the data quality monitoring table level list based on the screening results.

4. The data quality monitoring method according to claim 1, characterized in that: Determining the data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list includes: Respectively obtain the fields in the data quality monitoring table level list and the fields in the data quality monitoring field level list; Take the union of the acquired fields to get the data quality monitoring output list.

5. The data quality monitoring method according to claim 1, characterized in that: The performing data quality monitoring on the monitoring information in the data quality monitoring output list includes: Generate a corresponding database description language according to the data quality monitoring rules; Data quality monitoring is performed on the monitoring information in the data quality monitoring output list according to the database description language.

6. A data quality monitoring device, characterized in that: include: A table-level list generating unit, used to screen metadata in the data warehouse according to preset data quality monitoring rules to obtain a data quality monitoring table-level list; A field-level list generation unit is used to screen the metadata using a pre-built machine learning model to obtain a data quality monitoring field-level list; wherein the field-level list generation unit includes: a directed graph construction module, which is used to construct a field node directed graph according to table information, field information, and field dependencies; an importance value determination module, which is used to determine the importance value of each field node in the field node directed graph using a pre-built machine learning model; a field-level list determination module, which is used to determine the data quality monitoring field-level list according to a pre-set importance threshold and the importance value of each field node; An output list generating unit, used for determining a data quality monitoring output list according to a data quality monitoring table-level list and a data quality monitoring field-level list; The data quality monitoring unit is used to perform data quality monitoring on the monitoring information in the data quality monitoring output list; the monitoring information at least includes a database name, a data table name, a data field name and unsatisfied monitoring rules.

7. The data quality monitoring device according to claim 6, characterized in that: Also includes: An information acquisition unit, used to acquire table information, field information, field dependency and monitoring rule information in the data warehouse; The metadata construction unit is used to construct the metadata according to the table information, field information, field dependency and monitoring rule information.

8. The data quality monitoring device according to claim 7, characterized in that: The table-level list generating unit includes: An empty table screening module is used to screen whether each interface table in the data warehouse is empty; A primary key screening module, used for screening whether the primary key of the interface table is unique if it is not empty; An abnormal field screening module is used to screen whether there are abnormal fields in each interface table in the data warehouse if the table primary key is unique; The table-level list determination module is used to determine the data quality monitoring table-level list according to the screening results.

9. The data quality monitoring device according to claim 6, characterized in that: The output list generating unit comprises: A field acquisition module is used to respectively acquire fields in a data quality monitoring table-level list and fields in a data quality monitoring field-level list; The output list determination module is used to take the union of the acquired fields to obtain the data quality monitoring output list.

10. The data quality monitoring device according to claim 6, characterized in that: The data quality monitoring unit comprises: A description language generation module, used to generate a corresponding database description language according to the data quality monitoring rules; The data quality monitoring module is used to perform data quality monitoring on the monitoring information in the data quality monitoring output list according to the database description language.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data quality monitoring method according to any one of claims 1 to 5 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data quality monitoring method according to any one of claims 1 to 5 are implemented.

Images