Data quality assessment method based on data consanguinity

Abstract

The invention provides a data quality evaluation method based on data consanguinity. The method comprises the following steps of 1, obtaining data information needing data quality evaluation; 2, constructing a data consanguinity map of a table and a field of a data warehouse based on an AST tree through the data information obtained in the step 1; 3, establishing a data quality detection rule and an evaluation index system of the data warehouse, and evaluating the data quality of the data warehouse; and step 4, performing automatic data quality detection and notification, and performing quality detection on the influenced data entity and the related whole data link according to various detection rules set in the step 3 when the automatic data quality detection changes through data consanguinity or reaches a preset time period. According to the invention, the source of the data quality problem can be quickly and accurately positioned.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and specifically provides a data quality assessment method based on data lineage. Background Art

[0002] There are many ways to obtain data lineage. For example, Apache Atlas is an open-source data governance and metadata management platform widely used in the big data ecosystem. It provides rich management and query functions for data metadata, including data lineage. Another example is ANTLR (ANother Tool for Language Recognition), which is a powerful parser generator that can support the generation of parsers for multiple languages. Secondly, it can generate an abstract syntax tree (AST) according to SQL syntax, providing a basis for subsequent data lineage analysis.

[0003] Internationally common data quality standards are often measured from dimensions such as accuracy, integrity, consistency, timeliness, validity, and uniqueness. Accuracy means that the content recorded in the data should match the actual situation. For example, the recorded customer age cannot be incorrect. Integrity refers to that the data should cover all necessary information. When filling in user information, required fields cannot be missing. Consistency emphasizes that in different systems or different data recording links, the expression of the same data should be unified. For example, the contact information of the same customer in different data tables should be consistent. Timeliness means that the data should be able to reflect the current real situation in a timely manner. For example, the inventory data of goods should be updated in real time. Validity requires that the data conforms to established specifications such as format and value range. For example, the date format should be correct. Uniqueness is to ensure that key data has no duplicates. For example, the employee number of each employee is unique.

[0004] Traditional data quality assessment methods mainly focus on the static attributes of the data itself. For example, they check whether the format of the data in a single data table is correct and whether it meets basic specifications such as established value ranges. They often view data entities in isolation, ignoring the complex flow process of data throughout its life cycle and the mutual relationships between various links. This makes it difficult to comprehensively and accurately grasp the data quality situation when facing a complex data ecosystem and unable to effectively trace the root cause of quality problems. Summary of the Invention

[0005] In view of the above technical problems, the present invention provides a data quality assessment method based on data lineage, which can quickly and accurately locate the source of data quality problems.

[0006] To achieve the above object, the present invention provides the following technical solution: A data quality assessment method based on data lineage, comprising the following steps:

[0007] Step 1: Obtain the data information that needs to be evaluated for data quality;

[0008] Step 2: Construct a data lineage graph of the tables and fields in the data warehouse based on the AST tree using the data information obtained in Step 1;

[0009] Step 3: Establish a data quality detection rule and evaluation index system for the data warehouse, evaluate the data quality of the data warehouse, and the data quality evaluation is as follows:

[0010] First step: Assign weights to each data quality dimension (accuracy, integrity, consistency, validity, uniqueness, traceability, impact), the sum of the weights is 100%, and the scores for each dimension are w 准确性 、w 完整性 、w 一致性 、w 有效性 、w 唯一性 、w 可追溯性 、w 影响度 ;

[0011] Second step: Obtain the scores of each dimension in the first step and the assigned weights to calculate the total data quality score Q, and the calculation formula is as follows:

[0012]

[0013] Third step: Through normalization processing, map the data quality score to the interval of 0-1, and the calculation formula is as follows:

[0014]

[0015] Among them, Q min is the minimum quality score that may occur, and Q max is the maximum quality score that may occur;

[0016] Step 4: Automated data quality detection and notification. Automated data quality detection is triggered by changes in data lineage or reaching a preset time period. According to various detection rules set in Step 3, quality detection is performed on the affected data entities and their related entire data link.

[0017] Preferably, the calculation of each dimension in the first step of Step 3 is as follows:

[0018] Among them, n is a certain number of randomly selected data samples, and m is the number of accurate data in n;

[0019] Among them, k is the key information elements included in all data records, j is the number of key information elements actually included in each data record, and P is all data records;

[0020] where q is the number of times data inconsistencies are counted, and r is the total number of pairs of comparison data elements;

[0021] where u is the amount of data that does not meet the specifications, and v is the total amount of data;

[0022] where w is the number of duplicate data elements counted, and x is the total number of key data elements;

[0023] S 可追溯性 , score according to the clarity of the data lineage relationship and the ability to trace the source, transfer path, etc. of the data smoothly, and divide it into level 1 (poor), score range (0 - 30); level 2 (average), score range (31 - 60); level 3 (good), score range (61 - 80); level 4 (excellent), score range (81 - 100);

[0024] S 影响度 , the possible scope of influence and severity along the data lineage chain can also be graded and scored, and divided into level 1 (low impact), score range (0 - 30); level 2 (medium impact), score range (31 - 60); level 3 (high impact), score range (61 - 80); level 4 (extremely severe impact), score range (81 - 100).

[0025] Preferably, the automated data quality detection is implemented according to the data lineage change in step four as follows:

[0026] (1) Trigger the detection task

[0027] When it is detected that the data lineage has changed or according to a preset time period, automatically trigger the data quality detection task, and perform quality detection on the affected data entities and the entire related data link according to various previously set detection rules.

[0028] (2) Dynamically adjust the detection strategy in combination with the lineage

[0029] As the data is continuously updated and the business process changes dynamically, the data lineage will also change accordingly. The automated detection operation adjusts the detection strategy in real - time or periodically according to these changing lineage relationships.

[0030] Preferably, the data lineage graph of the tables and fields of the data warehouse based on the AST tree is constructed in step two as follows:

[0031] Step S1: Obtain the abstract syntax tree AST0 corresponding to the target SQL statement. By using a suitable SQL parsing tool, perform lexical analysis and syntax analysis on the target SQL statement, and present the SQL statement in a tree - like structure to form AST0;

[0032] Step S2: Starting from the root node of AST0, traverse each node of AST0 layer by layer, and enter different branches according to the node type, comprehensively and systematically check all nodes in AST0, and decide the subsequent specific blood relationship construction operation based on whether the node represents a table or a field;

[0033] Step S3: Establishing a blood relationship. After determining that the current node is the node corresponding to the table, establish the blood relationship between them by checking the nodes corresponding to other tables in its descendant nodes and brother nodes;

[0034] Step S4: Establishing the blood relationship of the fields. For the nodes corresponding to the fields, considering the node conditions corresponding to the fields in their descendant nodes, sibling nodes, and associated nodes, the blood relationship between the fields is constructed.

[0035] Step S5: obtaining a final table blood relationship set, sorting and screening the table blood relationships constructed in the above steps, and ensuring that the first table of each table blood relationship in the final table blood relationship set is not the last table of other table blood relationships;

[0036] Step S6: Obtain the final field blood relationship set.

[0037] Preferably, the operation steps of step S5 are as follows:

[0038] Step S5-1: Collect all table kinship relationships previously constructed based on the abstract syntax tree AST0 through the operation of step S3, and aggregate them into a set trea;

[0039] Step S5-2: Check each table blood relationship treax in the set trea one by one;

[0040] Step S5-3: When it is found that the first table tabx,1 of treax is the tail table of other table blood relations, it is necessary to find all table blood relations with tabx,1 as the tail table and aggregate them into a set tfrox;

[0041] Step S5-4: Based on the tfrox set obtained in step S5-3, a new table kinship relationship ntrex,c(x) is constructed for each tfrox,c(x) and appended to the first table kinship relationship set;

[0042] Step S5-5: Check the first table blood relationship set again to determine whether the first table of each table blood relationship is no longer the last table of other table blood relationships. If the conditions are met, it is determined as the final table blood relationship set; if the conditions are not met, the first table blood relationship set is continued to be updated according to the source table blood relationship corresponding to the table blood relationship, and the checking and updating process is repeated until the conditions are met.

[0043] Preferably, the operation steps of step S6 are as follows:

[0044] Step S6-1: Aggregate all the field blood relationships constructed through the operation of step S4 based on the abstract syntax tree AST0, and put them together in the set drea;

[0045] Step S6-2: Check each field blood relationship dreaf in the set drea one by one, and judge whether the first field fief,1 in dreaf is the last field of any other field blood relationship in the set drea. If not, add dreaf to the first field blood relationship set; if the first field fief,1 is the last field of other field blood relationships, it is necessary to enter the next step S6-3 for further processing;

[0046] Step S6-3: Find all the field blood relationships with fief,1 as the last field, and converge them into a set dfrof. Each element dfrof,u(f) in the set is a field blood relationship, and they all point to the first field fief,1 of the currently processed dreaf;

[0047] Step S6-4: Based on the obtained dfrof set, construct a new field blood relationship ndref,u(f) for each dfrof,u(f), and add it to the first field blood relationship set;

[0048] Step S6-5: Check the first field blood relationship set again to see if the first field of each field blood relationship is no longer the last field of other field blood relationships. If the condition is met, it is determined as the final field blood relationship set; if the condition is not met, continue to update the first field blood relationship set according to the source field blood relationship corresponding to the field blood relationship, and continuously repeat the process of checking and updating until the condition is met.

[0049] Advantages of the present invention: Incorporating the evaluation of data quality into data lineage extends the evaluation perspective from simply the data appearance to the entire life cycle of the data. It can trace the source of the data and the changes experienced in each link along the data flow path, realizing the transformation from static analysis to dynamic tracing. With the data lineage relationship, it is convenient to quickly and accurately locate the source of data quality problems. When data anomalies are found, instead of only being able to troubleshoot at the current level where the problem appears as in the past, it is possible to trace back along the lineage link to possible errors in the original data entry, logical deviations in the data conversion process, or conflicts generated during the integration of different data sources, etc., greatly narrowing the scope of problem troubleshooting, improving the efficiency of problem-solving, providing a more targeted direction for optimizing data quality, and quickly and accurately locating the source of data quality problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. In the accompanying drawings:

[0051] Figure 1 It is a schematic structural diagram of a simple data quality evaluation method based on data lineage proposed by the present invention.

[0052] Figure 2 It is a schematic structural diagram of the data lineage map of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] In order to make the technical means, creative features, achieved purposes and effects of the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments and the accompanying drawings. However, the following embodiments are only the preferred embodiments of the present invention, not all of them. Based on the embodiments in the embodiments, other embodiments obtained by those skilled in the art without creative work all belong to the protection scope of the present invention.

[0054] Please refer to Figure 1-2 , a data quality evaluation method based on data lineage, comprising the following steps:

[0055] Step 1: Obtain data information that requires data quality evaluation;

[0056] Step 2: Construct a data lineage map of the tables and fields of the data warehouse based on the AST tree through the data information obtained in Step 1;

[0057] The construction of the data lineage map of the tables and fields of the data warehouse based on the AST tree in Step 2 is as follows:

[0058] Step S1: By using a suitable SQL parsing tool (such as a parser implemented based on related technologies like ANTLR), perform lexical analysis and syntactic analysis on the target SQL statement, present the SQL statement in a tree structure, and form AST0. Each node of this tree corresponds to different syntactic elements in the SQL statement, such as keywords (SELECT, FROM, WHERE, etc.), table names, column names, and various operators, providing a basic structural framework for subsequent tracing of data lineage;

[0059] Example: For the SQL statement "SELECT column1,column2*2 AS new_column2 FROM table1 WHERE column3>10", in the AST0 generated after parsing, the root node may correspond to the entire SELECT statement, and the next-level branch nodes respectively correspond to the SELECT clause, FROM clause, WHERE clause, etc. Further subdivision will have nodes corresponding to the table name "table1", column names "column1", "column2", etc., clearly showing the syntactic composition of the statement.

[0060] Step S2: Starting from the root node of AST0, traverse each node of AST0 layer by layer and enter different branches according to the node type; comprehensively and systematically check all nodes in AST0, and determine subsequent specific lineage construction operations based on whether the node represents a table or a field, ensuring that no possible data association information is missed. This way of hierarchical traversal can process each element in the tree structure orderly and analyze them in turn according to the level of the SQL statement syntactic structure;

[0061] Example: Continuing with the AST0 corresponding to the above SQL statement as an example, starting from the root node and traversing downwards, when encountering the node corresponding to the table "table1", step S3 will be entered; when encountering nodes corresponding to fields such as "column1" and "column2", step S4 will be entered.

[0062] Step S3: Establish lineage. When it is determined that the current node is a node corresponding to a table, establish the lineage between them by checking the nodes corresponding to other tables among its descendant nodes and sibling nodes; the lineage here reflects the flow and association of data between different tables. For example, in scenarios such as multi-table joins (JOIN), subqueries, etc., data flows from one table to another or multiple tables jointly participate in data generation, etc.;

[0063] Example: Suppose there is a more complex SQL statement "SELECT t1.column1, t2.column2 FROM table1 t1 JOIN table2 t2 ON t1.key = t2.key WHERE t1.column3 > 10". In its AST0, for the node "table1 t1", its sibling node is "table2 t2". Through step S3, the table lineage relationship between "table1" and "table2" based on this JOIN operation will be established, indicating that data is retrieved from these two tables and associated and integrated during the query process.

[0064] Step S4: Establish field lineage relationships. For the nodes corresponding to fields, consider the nodes corresponding to fields in their descendant nodes, sibling nodes, and associated nodes (descendant nodes of sibling nodes), and construct the lineage relationships between fields. This helps to refine to the specific field level of data, trace the data source of a certain field, and which operations it has gone through to become other fields. For example, in the scenario of operations such as function operations and alias settings on fields in the SELECT clause;

[0065] Example: For the SQL statement "SELECT column1, column2 * 2 AS new_column2 FROM table1 WHERE column3 > 10", in the AST0, for the node corresponding to "column2", its descendant nodes (nodes corresponding to newly generated fields after operations, etc.) include the node corresponding to "new_column2". According to step S4, the field lineage relationship between "column2" and "new_column2" will be established, reflecting the data change association that the field has gone through the operation of "*2" and alias setting.

[0066] Step S5: Obtain the final set of table lineage relationships. Organize and filter the table lineage relationships constructed in the previous steps to ensure that the first table of each table lineage relationship in the final set of table lineage relationships is not the last table of other table lineage relationships, avoiding circular references or logical contradictions, so that the set of table lineage relationships accurately and reasonably reflects the actual data table flow relationship in the SQL statement.

[0067] Example: Suppose multiple table lineage relationships are constructed through step S3. After screening and organizing, those relationships that do not meet the requirements (such as the unreasonable circular relationship from table A to table B and then from table B to table A) are removed, and a final set of table lineage relationships that clearly and accurately reflects the one-way flow of data between tables is obtained, facilitating subsequent work such as data tracing and impact analysis at the table level.

[0068] The sub-steps of step S5 are as follows:

[0069] Step S5-1: Obtain the table blood relationship set trea corresponding to AST0;

[0070] First, all the table lineage relationships constructed based on the abstract syntax tree AST0 through step S3 and other operations are collected and summarized into a set trea. Here, the structure of each element treax in the set is clarified. It is a two-tuple representing a table lineage relationship, where tabx,1 represents the first table in this lineage relationship, that is, the source table of the data, and tabx,2 represents the tail table, which means the target table of the data flow. The overall data association from tabx,1 to tabx,2 is reflected.

[0071] Example: Assume that after analyzing the AST0 corresponding to a complex SQL statement, the following table relationships are constructed:

[0072] trea1=(table1,table2), indicating that data flows from table1 to table2;

[0073] trea2=(table3,table4), indicating that data flows from table3 to table4;

[0074] trea3=(table2,table5), indicating that data flows from table2 to table5;

[0075] At this time, trea={trea1, trea2, trea3}.

[0076] Step S5-2: traverse the trea for preliminary screening;

[0077] Each table lineage relationship treax in the set trea is checked one by one. Determine whether the first table tabx,1 in treax is the tail table of any other table lineage relationship in the set trea. If not, then directly add treax to the first table lineage relationship set (initial empty set), indicating that this table lineage relationship does not have obvious circular references and other problems at the current stage; if the first table tabx,1 is the tail table of other table lineage relationships, it means that there may be complex data flow intersections, and it is necessary to enter the next step S5-3 for further processing.

[0078] Example: Continuing with the trea collection above, start traversing:

[0079] For trea1 = (table1, table2), it is found that table1 is not the tail table of the other table blood relationship, so trea1 is added to the first table blood relationship set;

[0080] For trea2=(table3, table4), similarly, since table3 is not the end table of the blood relationship of other tables, trea2 is also appended to the first set of table blood relationships;

[0081] For trea3=(table2, table5), at this time, table2 is the end table of trea1, so it is necessary to enter S5-3 for processing.

[0082] Step S5-3: Obtain the set tfrox of the previous table blood relationships corresponding to treax;

[0083] When it is found that the first table tabx,1 of treax is the end table of the blood relationship of other tables, it is necessary to find all those table blood relationships with tabx,1 as the end table, and summarize them into a set tfrox. Each element tfrox,c(x) in this set is a table blood relationship, and they all point to the first table tabx,1 of the currently processed treax, reflecting the possible complex data inflow into tabx,1.

[0084] Example: For trea3=(table2, table5), its first table tabx,1 (that is, table2) is the end table of trea1, so tfrox={trea1}, where e(x)=1 because there is only one table blood relationship with the end table being table2.

[0085] Step S5-4: Construct and append new table blood relationships to the first set of table blood relationships;

[0086] Based on the tfrox set obtained in the previous step, a new table blood relationship ntrex,c(x) is constructed for each tfrox,c(x) and appended to the first set of table blood relationships. The newly constructed table blood relationship ntrex,c(x) reflects the complete data flow from the first table of tfrox,c(x) (i.e., tstax,c(x)) through tabx,1 to tabx,2. In this way, the complex cross-data relationships are further sorted out and integrated to more comprehensively and accurately reflect the data flow between tables.

[0087] Example: For the previous example, tfrox={trea1}, trea1=(table1, table2), for trea3=(table2, table5), construct ntrex,1=(table1, table2, table5), and then append ntrex,1 to the first set of table blood relationships.

[0088] Step S5-5: Determine the final set of table blood relationships;

[0089] After the previous processing, check the first table blood relationship set again to see if the head table of each table blood relationship in it is no longer the tail table of other table blood relationships. If this condition is met, it means that this set has been sorted out, and there is no unreasonable data flow relationship such as circular reference, and it can be directly determined as the final table blood relationship set; if it is not met, then the first table blood relationship set needs to be continuously updated according to the source table blood relationship corresponding to the table blood relationship, and this checking and updating process is repeated until the condition is met. Finally, a final table blood relationship set with clear logic and accurately reflecting the reasonable flow of data between tables is obtained.

[0090] Example: Suppose after the previous step of appending, the first table blood relationship set contains table blood relationships such as (table1, table2, table5) and (table3, table4). After checking, it is found that the condition that the head table is not the tail table of other table blood relationships is met. Then this set is determined as the final table blood relationship set, which accurately presents the actual and non-circular contradictory flow situation of data between tables in the SQL statement and can be used for subsequent data traceability and analyzing the influence range of data at the table level and other work.

[0091] Step S6: Obtain the final field blood relationship set;

[0092] The operation steps of step S6 are as follows:

[0093] Step S6-1: Obtain the field blood relationship set drea corresponding to AST0;

[0094] First, summarize all the field blood relationships constructed previously according to the abstract syntax tree AST0 through operations such as step S4, and put them in the set drea uniformly. The composition form of each element dreaf in the set is clarified. It is a binary tuple, where fief,1 represents the head field in this field blood relationship, that is, the data source field of the target field, and fief,2 represents the tail field, that is, the target field finally obtained after a series of operations, which overall reflects the field data association situation from fief,1 to fief,2.

[0095] Example: Suppose after analyzing AST0 corresponding to a certain SQL statement, the following field blood relationships are constructed:

[0096] drea1 = (column1, column2), which means that the data of column2 is obtained from column1 through certain operations;

[0097] drea2 = (column3, column4), indicating that the data of column4 comes from column3;

[0098] drea3 = (column2, column5), indicating that the data in column5 is generated based on column2;

[0099] At this time, drea = {drea1, drea2, drea3}.

[0100] Step S6-2: Traverse drea for preliminary screening;

[0101] Check each field blood relationship dreaf in the set drea one by one. Determine whether the first field fief,1 in dreaf is the tail field of any other field blood relationship in the set drea. If not, directly add dreaf to the first field blood relationship set (initially an empty set), indicating that there are no obvious unreasonable situations such as circular references in this field blood relationship for the time being; if the first field fief,1 is the tail field of other field blood relationships, it means that there may be complex cross-problems in the field data flow direction, and it is necessary to enter the next step S6-3 for further processing.

[0102] Example: Continuing with the above drea set as an example for traversal operations:

[0103] For drea1 = (column1, column2), it is checked that column1 is not the tail field of other field blood relationships, so drea1 is appended to the first field blood relationship set;

[0104] For drea2 = (column3, column4), similarly column3 is not the tail field of other field blood relationships, and drea2 is also appended to the first field blood relationship set;

[0105] For drea3 = (column2, column5), at this time column2 is the tail field of drea1, so it is necessary to enter S6-3 for subsequent processing.

[0106] Step S6-3: Obtain the set dfrof of the previous field blood relationships corresponding to dreaf;

[0107] When it is found that the first field fief,1 of dreaf is the tail field of other field blood relationships, find all those field blood relationships with fief,1 as the tail field, and then converge them into a set dfrof. Each element dfrof,u(f) in the set is a field blood relationship, and they all point to the first field fief,1 of the currently processed dreaf, reflecting the possible complex situation of data flowing into fief,1.

[0108] Example: For drea3 = (column2, column5), its first field fief,1 (which is column2) is the last field of drea1. So dfrof = {drea1}, where h(f) = 1 because there is only one field blood relationship whose last field is column2.

[0109] Step S6-4: Construct and append a new field blood relationship to the first field blood relationship set;

[0110] Based on the dfrof set obtained in the previous step, construct a new field blood relationship ndref,u(f) for each dfrof,u(f), and add it to the first field blood relationship set. The newly constructed field blood relationship ndref,u(f) reflects the complete field data flow from the first field (i.e., sfief,u(f)) of dfrof,u(f) through fief,1 to fief,2. In this way, the complex and cross-field data relationships are further sorted and integrated to more comprehensively and accurately reflect the flow and change of data between fields.

[0111] Example: For the previous example, dfrof = {drea1}, drea1 = (column1, column2), for drea3 = (column2, column5), construct ndref,1 = (column1, column2, column5), and then append ndref,1 to the first field blood relationship set.

[0112] Step S6-5: Determine the final field blood relationship set;

[0113] After a series of operations above, check the first field blood relationship set again to see if the first field of each field blood relationship in it is no longer the last field of other field blood relationships. If this condition is met, it means that this set has been sorted out and there are no unreasonable data flow relationships such as circular references at the field level, and it can be directly determined as the final field blood relationship set; if the requirement is not met, the first field blood relationship set needs to be continuously updated according to the source field blood relationship corresponding to the field blood relationship, and the process of checking and updating is repeated until the condition is met. Finally, a final field blood relationship set with clear logic and accurately reflecting the reasonable flow of data between fields is obtained.

[0114] Example: Assume that after the addition in the previous steps, the set of field lineage relationships in the first field includes field lineage relationships such as (column1, column2, column5) and (column3, column4). After inspection, it is found that the condition that the first field is not the tail field of other field lineage relationships is met. Then this set is determined as the final set of field lineage relationships, which clearly and accurately presents the actual and logically consistent flow of data among fields in the SQL statement, facilitating subsequent work such as data tracing and data change analysis at the field level.

[0115] The formed data lineage graph is as Figure 2 shown.

[0116] Step 3: Establish data quality detection rules and an evaluation index system for the data warehouse to evaluate the data quality of the data warehouse, and the data quality evaluation is as follows:

[0117] First step: Assign weights to each data quality dimension (accuracy, integrity, consistency, validity, uniqueness, traceability, impact), with the total weight being 100%, and the scores for each dimension being w 准确性 、w 完整性 、w 一致性 、w 有效性 、w 唯一性 、w 可追溯性 、w 影响度 ;

[0118] The calculation for each dimension is as follows:

[0119] where n is a certain number of randomly selected data samples, and m is the number of accurate data in n;

[0120] where k is the key information elements included in all data records, j is the number of key information elements actually included in each data record, and P is all data records;

[0121] where q is the number of times data inconsistency occurs, and r is the total number of pairs of compared data elements;

[0122] where u is the amount of data that does not meet the specifications, and v is the total amount of data;

[0123] where w is the number of duplicate data elements counted, and x is the total number of key data elements;

[0124] S 可追溯性 , score according to the clarity of the data lineage relationship and the ability to trace the source and transfer path of data smoothly, as follows:

[0125] Level 1 (Poor): The data lineage records are vague, and it is difficult to trace the key transfer links of most data. The score range is 0 - 30.

[0126] Level 2 (Fair): The sources of some major data and some core transfer steps can be traced, but there are some omissions or unclear situations. The score range is 31 - 60.

[0127] Level 3 (Good): The data lineage is relatively clear and complete, and the detailed transfer process of most data can basically be traced. The score range is 61 - 80.

[0128] Level 4 (Excellent): The data lineage is clear and complete, and all links of any data from the source to the current state can be traced easily and accurately. The score range is 81 - 100.

[0129] The specific score is determined according to the actual evaluation level. For example, if the evaluation is Level 3, then S 可追溯性 = 70 (In actual operation, the scoring criteria can be more refined. Here is a simplified illustration).

[0130] S 影响度 , to evaluate the possible influence scope and severity of data quality problems along the data lineage chain, it can also be graded and scored. For example:

[0131] Level 1 (Low impact): Even if there are quality problems, it only has a slight impact on local and unimportant business applications. The score range is 0 - 30.

[0132] Level 2 (Medium impact): Quality problems will affect some important business links, but can be quickly alleviated through certain measures. The score range is 31 - 60.

[0133] Level 3 (High impact): Once data quality problems occur, they will seriously interfere with multiple key business processes, resulting in the inability to carry out business normally or causing major losses. The score range is 61 - 80.

[0134] Level 4 (Extremely severe impact): The influence scope of quality problems is extensive, involving core business systems, and may even cause catastrophic consequences such as system paralysis. The score range is 81 - 100.

[0135] The score is determined according to the actual evaluation impact level. For example, if it is determined as Level 2, then S 影响度 = 50 (The scoring rules can be further refined as well).

[0136] Step 2: Obtain the scores of each dimension in the first step and the assigned weights to calculate the total data quality score Q. The calculation formula is as follows:

[0137]

[0138] For example, according to the weights set above and assuming the scoring situation of each dimension:

[0139] Q = 0.25 * 90 + 0.2 * 80 + 0.15 * 90 + 0.1 * 85 + 0.1 * 95 + 0.1 * 80 + 0.1 * 60 = 84

[0140] Step 3: Through normalization processing, map the data quality score to the interval of 0 - 1. The calculation formula is as follows:

[0141]

[0142] where \(Q\) is the original data quality score calculated above min \(Q_{min}\) is the minimum quality score that may occur (the minimum value calculated according to the weights when the lowest score of each dimension is 0 theoretically, as in the above weight setting) max \(Q_{max}\) is the maximum quality score that may occur (the maximum value calculated according to the weights when each dimension is scored 100 points, which is 100 here).

[0143] Perform normalization processing on the \(Q\) calculated above:

[0144]

[0145] In this way, the finally obtained normalized data quality score that comprehensively considers the characteristics related to data lineage and traditional data quality dimensions can intuitively measure the data quality situation and is convenient for comparison and analysis in different environments.

[0146] Step 4: Automated data quality detection and notification. Automated data quality detection is triggered by changes in data lineage or reaching a preset time period. According to various detection rules set in Step 3, quality detection is performed on the affected data entities and their related entire data link;

[0147] Automated data quality detection is as follows:

[0148] (1) Trigger the detection task

[0149] When it is detected that there are changes in data lineage or according to the preset time period, the data quality detection task is automatically triggered. According to various detection rules set before, quality detection is performed on the affected data entities and their related entire data link. For example, once it is found that the operation logic in the data cleaning link has changed, the quality detection process for all data passing through this cleaning link and subsequent associated data is immediately started.

[0150] (2) Dynamically adjust the detection strategy in combination with lineage

[0151] As data is continuously updated and business processes dynamically change, data lineage will also change accordingly. Automated detection operations need to adjust detection strategies in real time or periodically according to these changing lineage relationships. For example, when a new data source is added and integrated into an existing data link, the detection operation should correspondingly increase the data quality detection content for the new data source and its subsequent associated parts; or when the operation method of a certain data processing link changes, based on the possible quality impact on subsequent links through lineage analysis, the focus and frequency of detection should be changed accordingly to ensure that the detection always fits the actual situation of the data and accurately reflects the data quality status.

[0152] In the present invention, (1) when constructing data lineage, in addition to using the AST tree to analyze whether the tables / fields in a single SQL are nodes in the data lineage; the ancestor nodes / descendant nodes of the nodes can also be found by traversing the set formed by all the current nodes, truly achieving the search for the data flow of the entire data warehouse.

[0153] (2) After evaluating data quality and integrating it into data lineage, the evaluation perspective extends from the simple data appearance to the entire life cycle of the data, enabling tracing of its source and the changes experienced in each link along the data transfer path, realizing the transformation from static analysis to dynamic tracing; relying on the data lineage relationship, the source of data quality problems can be quickly and accurately located. When data anomalies are found, instead of only being able to conduct investigations at the current level where the problem appears as in the past, it is possible to trace back along the lineage link to possible errors in the original data entry link, logical deviations in the data conversion process, or conflicts generated during the integration of different data sources, greatly narrowing the scope of problem investigation, improving the efficiency of problem-solving, and providing a more targeted direction for optimizing data quality.

[0154] Compared with the prior art, traditional data quality assessment methods mainly focus on the static attributes of the data itself. For example, in a single data table, they check whether the data format is correct and whether it meets basic specifications such as established value ranges. They often view data entities in isolation, ignoring the complex data flow process throughout the life cycle and the interrelationships between various links. This makes it difficult to comprehensively and accurately grasp the data quality situation and effectively trace the root causes of quality problems when facing a complex data ecosystem. In the present invention, considering that traditional data quality assessment methods are difficult to comprehensively consider the quality changes of data during the flow process, accurately trace the source of quality problems and determine the scope of influence, cannot well adapt to the dynamic changes of the data environment and the needs of diverse business scenarios, and have deficiencies in integrating with existing systems and ensuring assessment efficiency, etc., by leveraging the correlation context of data generation and flow presented by data lineage, it is possible to more accurately quantify data quality, quickly locate the root causes of quality problems and their scope of influence, dynamically adapt to data increases and decreases and process changes, flexibly conduct assessments according to different business scenarios, and at the same time ensure efficient integration and cooperation with existing systems, thereby comprehensively improving the accuracy, effectiveness and practicality of data quality assessment.

[0155] The present invention has the following advantages compared with the prior art:

[0156] 1. Comprehensive assessment

[0157] Multi-dimensional consideration: This method covers multiple important data quality dimensions. It includes both traditional data quality measurement indicators, such as accuracy to ensure that the data reflects the real business situation, integrity to ensure that there is no missing data, and consistency to make data from different sources match, and also incorporates traceability and impact degree related to data lineage. Traceability can reflect the traceability ability of data during the flow process, and the impact degree reflects the scope and severity of the impact of data quality problems on the business, thus achieving a full-range assessment from the state of the data itself to its impact in the business, avoiding missing key information due to single-dimensional assessment, and more comprehensively and accurately grasping the overall data quality level. Adapt to complex data environment: In today's complex data ecosystem of enterprises, data sources are diverse, flow links are complex, and application scenarios are rich. By comprehensively evaluating these dimensions, whether dealing with simple business data tables or complex data architectures involving multi-system interactions and multi-level data processing, it can effectively measure data quality and better meet the data management needs in actual business.

[0158] 2. Precise positioning and problem analysis

[0159] Trace the root cause of problems through lineage: After considering the traceability dimension of data lineage, when data quality problems are discovered, the root cause of the problems can be quickly located along the data flow path based on clear lineage relationships. For example, if data anomalies are found in a data analysis report, by tracing the data lineage, it can be determined whether it is an error in the original data source entry or a deviation in intermediate links such as data extraction, transformation, and loading. This helps to accurately troubleshoot problems, reduce troubleshooting time and costs, and improve the efficiency of problem-solving.

[0160] Control business risks based on impact: The impact dimension can intuitively present the scope and severity of the impact of data quality problems on business processes, decision-making, and downstream applications. This enables data managers and business personnel to prioritize the handling of data quality problems that have a significant impact on the business according to the magnitude of the impact, allocate resources reasonably, and take corresponding risk prevention and control measures in advance to avoid serious impacts on the business due to poor data quality and ensure the stable operation of the business.

[0161] 3. Optimize decision-making driven by data

[0162] Provide a reliable basis for decision-making: The data quality score calculated through comprehensive calculation has specificity and comparability after normalization, and can intuitively reflect the data quality status. Enterprise managers can clearly understand the data quality advantages and disadvantages of different business processes and different data sets based on this quantitative result, so as to have a more scientific and accurate reference basis when making data-related decisions (such as whether to continue using a certain data source, which data processing processes to optimize first, etc.), prompting the decision-making to be more in line with the actual data situation and improving the decision-making quality.

[0163] Facilitate continuous improvement: Regularly conduct data quality assessments according to this method, and compare the data quality scores and the score changes of each dimension in different periods. The changing trend of data quality can be analyzed, and the weak links and recurring problem points in data management can be discovered. Furthermore, the data governance strategy, data processing process, quality control measures, etc. can be continuously optimized in a targeted manner to form a virtuous cycle of continuous improvement of data quality and better play the value of data in enterprise operation and development.

[0164] 4. Unified standards and communication and collaboration

[0165] Establish unified evaluation standards: Among different departments and business teams within an enterprise, there are often differences in the understanding and measurement standards of data quality, which can easily lead to poor communication and collaboration difficulties. Adopting this clear and comprehensive calculation method provides a unified data quality evaluation standard for the entire enterprise, enabling each department to communicate based on the same "language" when discussing data quality problems, formulating data management plans, etc., reducing understanding deviations, and enhancing the efficiency and effectiveness of cross-departmental collaboration.

[0166] Enhancing data awareness and shared responsibility: This quantitative and comprehensive data quality assessment method enables different roles within an enterprise (from data producers, data processors to data users, etc.) to more clearly recognize the importance of data quality and their own responsibilities in the process of ensuring data quality. For example, data entry personnel will pay more attention to the accuracy and integrity of data, and developers will consider the impact on data lineage traceability when designing data processing flows, etc., promoting the full participation of all employees in data quality management and forming a good data culture atmosphere.

[0167] The above has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art of this industry should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have various changes and improvements, and all these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A data quality assessment method based on data lineage, characterized in that: The following steps are involved: Step 1: Obtain data information that requires data quality assessment; Step 2: Use the data information obtained in step 1 to build a data lineage map of the data warehouse table and fields based on the AST tree; Step 3: Establish data quality detection rules and evaluation indicator system for data warehouse, evaluate the data quality of data warehouse, and the data quality evaluation is as follows: Step 1: Assign weights to each data quality dimension (accuracy, completeness, consistency, validity, uniqueness, traceability, and impact), with the total weight being 100%, and the scores for each dimension being w 准确性 、w 完整性 、w 一致性 、w 有效性 、w 唯一性 、w 可追溯性 、w 影响度 ; Step 2: Obtain the scores of each dimension in the first step and the assigned weights to calculate the total data quality score Q. The calculation formula is as follows: Q=w 准确性 ×S 准确性 +w 完整性 ×S 完整性 +w 一致性 ×S 一致性 +w 有效性 ×S 有效性 +w 唯一性 ×S 唯一性 + In 可追溯性 ×S 可追溯性 +in 影响度 ×S 影响度 ; Step 3: Through normalization, the data quality score is mapped to the range of 0-1. The calculation formula is as follows: Among them, Q min is the minimum possible mass score, Q max is the maximum possible mass score; Step 4: Automated data quality detection and notification. Automated data quality detection is performed on the affected data entity and its related entire data link according to the various detection rules set in step 3 when the data lineage changes or reaches a preset time period.

2. The data quality assessment method based on data lineage according to claim 1 is characterized by: The calculation of each dimension in the first step of step three is as follows: Where n is a certain number of data samples randomly selected, and m is the number of accurate data in n; Where k is the key information element contained in all data records, j is the number of key information elements actually contained in each data record, and P is all data records; Where q is the number of statistically inconsistent data occurrences, and r is the total number of comparison data element pairs; Where u is the amount of data that does not meet the statistical standards, and v is the total amount of data; Where w is the number of statistically repeated data elements, and x is the total number of key data elements; S 可追溯性 , according to the clarity of the data lineage and whether the source and flow path of the data can be smoothly traced, the scores are divided into level 1 (poor), score range (0-30); level 2 (average), score range (31-60); level 3 (good), score range (61-80); level 4 (excellent), score range (81-100); S 影响度 The scope and severity of the impact that may occur along the data lineage chain can also be divided into levels and scored, and are divided into Level 1 (low impact), score range (0-30); Level 2 (medium impact), score range (31-60); Level 3 (high impact), score range (61-80); Level 4 (extremely serious impact), score range (81-100).

3. The data quality assessment method based on data lineage according to claim 1 is characterized by: In step 4, automatic data quality detection is implemented according to data lineage changes as follows: (1) Triggering detection tasks When a change in data lineage is detected or a preset time period is followed, the data quality detection task is automatically triggered. According to the various detection rules set previously, quality detection is performed on the affected data entity and its entire related data link. (2) Adjusting the detection strategy based on blood relationship dynamics As data is constantly updated and business processes change dynamically, data lineage will also change accordingly. Automated detection operations adjust detection strategies in real time or regularly based on these changing lineage relationships.

4. The data quality assessment method based on data lineage according to claim 1 is characterized by: The data lineage map of the table and field of the data warehouse based on the AST tree constructed in step 2 is as follows: Step S1: Obtain an abstract syntax tree AST0 corresponding to a target SQL statement, perform lexical analysis and syntax analysis on the target SQL statement by using a suitable SQL parsing tool, and present the SQL statement in a tree structure to form AST0; Step S2: Starting from the root node of AST0, traverse each node of AST0 layer by layer, and enter different branches according to the node type, comprehensively and systematically check all nodes in AST0, and decide the subsequent specific blood relationship construction operation based on whether the node represents a table or a field; Step S3: Establishing a blood relationship. After determining that the current node is the node corresponding to the table, establish the blood relationship between them by checking the nodes corresponding to other tables in its descendant nodes and brother nodes; Step S4: Establishing the blood relationship of the fields. For the nodes corresponding to the fields, considering the node conditions corresponding to the fields in their descendant nodes, sibling nodes, and associated nodes, the blood relationship between the fields is constructed. Step S5: obtaining a final table blood relationship set, sorting and screening the table blood relationships constructed in the above steps, and ensuring that the first table of each table blood relationship in the final table blood relationship set is not the last table of other table blood relationships; Step S6: Obtain the final field blood relationship set.

5. The data quality assessment method based on data lineage according to claim 4 is characterized by: The operation steps of step S5 are as follows: Step S5-1: Collect all table kinship relationships previously constructed based on the abstract syntax tree AST0 through the operation of step S3, and aggregate them into a set trea; Step S5-2: Check each table blood relationship treax in the set trea one by one; Step S5-3: When it is found that the first table tabx,1 of treax is the tail table of other table blood relations, it is necessary to find all table blood relations with tabx,1 as the tail table and aggregate them into a set tfrox; Step S5-4: Based on the tfrox set obtained in step S5-3, a new table kinship relationship ntrex,c(x) is constructed for each tfrox,c(x) and appended to the first table kinship relationship set; Step S5-5: Check the first table blood relationship set again to determine whether the first table of each table blood relationship is no longer the last table of other table blood relationships. If the condition is met, it is determined as the final table blood relationship set; If the condition is not met, the first table blood relationship set is continuously updated according to the source table blood relationship corresponding to the table blood relationship, and the checking and updating process is continuously repeated until the condition is met.

6. The data quality assessment method based on data lineage according to claim 4 is characterized by: The operation steps of step S6 are as follows: Step S6-1: Summarize all the field relationships constructed by the operation in step S4 according to the abstract syntax tree AST0, and put them into the set drea; Step S6-2: Check each field kinship relationship dreaf in the set drea one by one to determine whether the first field fief,1 in dreaf is the tail field of any other field kinship relationship in the set drea. If not, add dreaf to the first field kinship relationship set; if the first field fief,1 is the tail field of other field kinship relationships, proceed to the next step S6-3 for further processing; Step S6-3: Find all the field lineage relations with fief,1 as the tail field, and aggregate them into a set dfrof. Each element dfrof,u(f) in the set is a field lineage relation, and they all point to the first field fief,1 of the dreaf currently being processed; Step S6-4: Based on the obtained dfrof set, construct a new field kinship relationship ndref,u(f) for each dfrof,u(f), and add it to the first field kinship relationship set; Step S6-5: Check the first field kinship set again to see whether the first field of each field kinship is no longer the last field of other field kinships. If the condition is met, it is determined as the final field kinship set; If the condition is not met, the first field kinship set is continuously updated according to the source field kinship corresponding to the field kinship, and the checking and updating process is continuously repeated until the condition is met.

Images