A data quality monitoring method and system based on knowledge graph

Abstract

The present invention discloses a data quality monitoring method and system based on a knowledge graph. The method constructs data tables and fields in multi-source heterogeneous data as graph nodes, constructs graph edges through primary and foreign key constraint relationships and field value overlap analysis, and forms a data kinship knowledge graph; encodes business rules into the graph to perform data constraint checks, and tracks the impact range of abnormal data through directed graph traversal; and implements dynamic monitoring based on the time changes of abnormal score sequences. The present invention also provides a system implementation including data collection preprocessing, knowledge graph construction, quality monitoring, problem diagnosis, and quality assessment. The present invention solves the problem that traditional methods are difficult to track the impact range of abnormal data and lack of systematic assessment, and improves the automation level and accuracy of data quality management.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality management, and in particular to a data quality monitoring method and system based on a knowledge graph. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, enterprises are faced with the challenge of managing and processing multi-source heterogeneous data. These data come from a wide range of sources, including structured data in relational databases, semi-structured data in XML and JSON formats, and unstructured data such as text documents and log files. In the era of data-driven decision-making, data quality directly affects the operational efficiency and decision-making accuracy of enterprises. Ensuring data quality and consistency has become the core task of enterprise data governance.

[0003] At present, there are many technical solutions in the field of data quality management, mainly including rule-based data quality inspection methods, statistical analysis methods, machine learning methods, and metadata management technologies, etc. These methods can discover and deal with data quality problems to a certain extent, but there are still significant shortcomings.

[0004] The existing technologies mainly have the following problems: traditional static rule checking methods are difficult to effectively identify and process complex dependencies between data; lack of systematic management and analysis capabilities for data lineage relationships, unable to quickly track the source of abnormal data and its scope of influence; lack of analysis and evaluation mechanisms for the propagation path of data quality problems, difficult to accurately assess the severity of the problem; lack of dynamic monitoring and early warning mechanisms, unable to timely discover the changing trend of data quality. At the same time, there is a lack of effective technical means for the unified management and quality monitoring of multi-source heterogeneous data.

[0005] Therefore, there is an urgent need for a data quality management method and system that can systematically manage data lineage, intelligently monitor data quality, automatically track the scope of abnormal impact, and have dynamic early warning capabilities. This method should be able to organically combine knowledge graph technology to achieve unified management and intelligent quality control of multi-source heterogeneous data, and provide more effective technical support for enterprise data governance. Summary of the invention

[0006] In order to solve the above technical problems, the present invention discloses an intelligent quality monitoring and consistency verification method and system based on data lineage knowledge graph, aiming to achieve unified management and intelligent quality control of multi-source heterogeneous data.

[0007] The technical solution adopted by the present invention is: an intelligent quality monitoring and consistency verification method based on data lineage knowledge graph, which is characterized by comprising the following steps:

[0008] S1, construct the data tables and fields in the collected multi-source heterogeneous data into graph nodes, construct graph edges through primary and foreign key constraint relationships and field value overlap analysis, and form a data lineage relationship knowledge graph;

[0009] S2, encodes business rules and constraints into the node and edge attributes of the knowledge graph, performs data constraint checks through graph query and graph traversal algorithms, and records abnormal information to achieve intelligent monitoring of data quality;

[0010] S3, for the detected abnormal data, build a directed graph traversal path to track its upstream and downstream impacts, and build an abnormal propagation graph based on the correlation strength and abnormal severity for quantitative evaluation;

[0011] S4, based on the data lineage knowledge graph, tracks the time changes of abnormal score sequences, and issues early warnings by setting change rate thresholds to achieve dynamic monitoring of the data quality improvement process.

[0012] Furthermore, the multi-source heterogeneous data in S1 includes structured data in relational databases, semi-structured data in XML and JSON formats, and unstructured data in text documents and log files.

[0013] Furthermore, the data kinship knowledge graph constructed in S1 adopts the entity-relationship-attribute triple modeling method for unified graph processing, where the triple is represented as , e represents the entity node set, R represents the relationship set, and a represents the attribute set, including:

[0014] Mapping the data tables in the structured data of the relational database into entity type nodes , map the fields in the table to attribute nodes ;

[0015] Mapping objects in the semi-structured data in XML and JSON format to entity nodes , map the object's attribute fields to attribute nodes ;

[0016] The unstructured data of the text document and log file are segmented by a set of predefined delimiters, the text content is matched based on a predefined business object template, and the matched business objects are constructed as entity nodes. , construct the business object's identification, category, time and other information as attribute nodes ;

[0017] By scanning the attribute values ​​V(a) of all nodes, the table structure definition is analyzed to extract explicit primary and foreign key constraints. , compare field naming rules to identify potential reference relationships , the overlapping coefficient of the field values ​​is statistically analyzed to determine the association strength w. When w is greater than the preset threshold θ, the identified relationship is constructed as an edge in the knowledge graph to complete the knowledge graph construction.

[0018] Furthermore, S2 includes: encoding business rules into the knowledge graph, defining a set of data value range constraint rules based on the constructed entity node e and attribute node a , which is used to limit the field length, value range and data format; define the set of constraint rules between nodes according to the edge relationship r in the knowledge graph , used to ensure primary and foreign key consistency, field value dependencies and data update synchronization;

[0019] The rule set The corresponding nodes and edges are stored in the form of attribute graphs, where the node attribute P(e) stores a collection of constraint rules related to a single field. , the edge attribute P(r) stores a set of cross-field and cross-table constraint rules ;

[0020] Use graph query language to query node attribute values Make a judgment to perform a single-field constraint check , use the graph traversal algorithm to compare the attribute values ​​of the associated nodes along the direction r of the edge to perform the association constraint check The detection results include the position pos, type type and severity level of the constraint violation, and the detection results Record it in the status attributes of the corresponding node to achieve automatic detection and verification of data consistency and integrity.

[0021] Furthermore, when performing backtracking analysis in S3, the abnormal data node detected in S2 And its test results , construct a directed graph traversal path set , where each path Expressed as , through the graph traversal algorithm to identify abnormal data nodes Upstream impact path , and downstream propagation paths Conduct tracking;

[0022] The influence degree on each path is calculated using the formula:

[0023]

[0024] Where w(r) is the edge association strength defined in S1, level is the severity of the anomaly, and the impact is quantified by accumulating the product of the association strength of each edge on the path and the severity of the anomaly;

[0025] Based on the path influence degree I(p) and node attributes P(e), an anomaly propagation graph G(V, E) is constructed, where the vertex set V represents the affected nodes. , the edge set E represents the anomaly propagation path, and the comprehensive score of the anomaly is calculated. The formula is:

[0026]

[0027] Among them, α and β are weight coefficients, I(p) is the path impact degree, and level is the severity of the anomaly, which realizes the quantitative evaluation of the impact scope and severity of the anomaly.

[0028] Furthermore, in S4, a time series analysis model is constructed to track data quality changes, and the abnormal comprehensive score Φ within the continuous time window is collected for each entity node e to form a score sequence , where Φ is the abnormal comprehensive score defined in S3, and the formula for calculating the score change rate at adjacent time points is:

[0029]

[0030] in, is the score at the current time point, Set the score change rate threshold for the previous time point ,when The early warning signal is triggered in real time to realize dynamic monitoring of the data quality improvement process.

[0031] The present invention also discloses an intelligent quality monitoring and consistency verification system based on data lineage knowledge graph, comprising:

[0032] Data acquisition and preprocessing module: responsible for the acquisition and preprocessing of multi-source heterogeneous data, including structured data in relational databases, semi-structured data in XML and JSON formats, and unstructured data in text documents and log files;

[0033] Knowledge graph construction module: Build a data kinship knowledge graph based on the entity-relationship-attribute triple modeling method, build data tables and fields as graph nodes, and build graph edges through primary and foreign key constraint relationships and field value overlap analysis;

[0034] Quality monitoring module: responsible for encoding business rules and constraint relationships into the node and edge attributes of the knowledge graph, executing checks on data range constraint rules and inter-node constraint rules, and recording abnormal data;

[0035] Problem diagnosis module: constructs a directed graph traversal path to track the upstream and downstream impacts of abnormal data, builds an abnormal propagation graph based on the correlation strength and abnormal severity, and calculates the comprehensive score of the abnormality;

[0036] Quality assessment module: Track the time changes of abnormal score sequences, calculate the score change rate, and issue early warnings by setting thresholds to achieve dynamic monitoring of the data quality improvement process.

[0037] Furthermore, the knowledge graph construction module adopts a graph database storage engine, supports graph data storage and query, and implements graph analysis functions through a graph computing engine.

[0038] Furthermore, the quality monitoring module performs data constraint checks through graph query and graph traversal algorithms to detect and warn of data quality issues.

[0039] Furthermore, the problem diagnosis module supports upstream and downstream impact analysis of abnormal data, and the quality assessment module monitors data quality through the score change rate.

[0040] The beneficial effects of the present invention are as follows:

[0041] (1) The present invention encodes business rules into the node and edge attributes of the knowledge graph, and combines graph query and graph traversal algorithms to perform constraint checking. It can simultaneously support the verification of single-field constraints and cross-field association constraints, solving the problem that traditional static rule checking is difficult to handle complex data dependencies, and improving the accuracy and comprehensiveness of data quality detection.

[0042] (2) The present invention constructs an anomaly propagation assessment mechanism based on a directed graph. By tracing the upstream and downstream impact paths of abnormal data and quantitatively assessing the degree of impact, it solves the problem that traditional methods lack the ability to analyze anomaly impacts and improves the traceability and controllability of data quality issues.

[0043] (3) The present invention establishes a dynamic quality monitoring mechanism based on time series. By tracking the changing trend of abnormal score sequences and issuing early warnings, it solves the problem that traditional methods lack dynamic monitoring and early warning capabilities, and improves the real-time and proactive nature of data quality management. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of data quality monitoring based on knowledge graph.

[0045] Figure 2 Build a flowchart for the knowledge graph.

[0046] Figure 3 Coding and checking flow chart for rules.

[0047] Figure 4 It is a dynamic monitoring flow chart. DETAILED DESCRIPTION

[0048] The above detailed description is a specific description of a feasible embodiment of the present invention. The embodiment is not intended to limit the patent scope of the present invention. Any equivalent implementation or modification that does not deviate from the present invention should be included in the patent scope of this case.

[0049] Reference Figure 1 The main steps of the method provided by the present invention are as follows:

[0050] (1) The data tables and fields in the collected multi-source heterogeneous data are constructed as graph nodes, and graph edges are constructed through primary and foreign key constraint relationships and field value overlap analysis to form a data lineage relationship knowledge graph;

[0051] (2) Encode business rules and constraints into the node and edge attributes of the knowledge graph, perform data constraint checks through graph query and graph traversal algorithms, and record abnormal data to achieve intelligent monitoring of data quality;

[0052] (3) For the detected abnormal data, a directed graph traversal path is constructed to track its upstream and downstream impacts, and an anomaly propagation graph is constructed based on the correlation strength and anomaly severity for quantitative evaluation;

[0053] (4) Based on the data lineage knowledge graph, the temporal changes of abnormal score sequences are tracked, and early warnings are issued by setting change rate thresholds to achieve dynamic monitoring of the data quality improvement process.

[0054] As a preferred embodiment of the present invention, refer to Figure 2 In step (1), the entity-relationship-attribute triple modeling method is used for unified graph processing, where the triple is represented as , e represents the entity node set, R represents the relationship set, and a represents the attribute set. The specific process of constructing the data lineage relationship knowledge graph is as follows:

[0055] (1-1) Entity node mapping:

[0056] For structured data in a relational database, map the data table to an entity type node , map the fields in the table to attribute nodes For example, when processing an order management system, the "order table" is constructed as an entity node, and fields such as "order number" and "order amount" are constructed as attribute nodes;

[0057] For semi-structured data in XML and JSON format, map objects to entity nodes , map the object's attribute fields to attribute nodes For example, when processing system configuration files, the configuration objects are constructed as entity nodes, and the configuration items are constructed as attribute nodes;

[0058] For unstructured data in text documents and log files, the system first segments the text using a predefined set of delimiters (such as line breaks, semicolons, etc.), and then matches the content based on predefined business object templates. The matched business objects are constructed as entity nodes. , construct the business object's identification, category, time and other information as attribute nodes .

[0059] (1-2) Relationship edge construction:

[0060] The system scans the attribute values ​​V(a) of all nodes, first analyzes the foreign key declarations in the table structure definition, and extracts explicit primary and foreign key constraints. ;

[0061] For tables that do not declare foreign key constraints, identify potential reference relationships by comparing field naming conventions (such as "user_id" to "id") ;

[0062] For each pair of fields that may be associated, the overlap coefficient of the field values ​​is statistically analyzed to determine the association strength w. The calculation of the overlap coefficient takes into account multiple factors such as the number of intersections of field values, data type matching, and value range;

[0063] When the association strength w is greater than the preset threshold θ (usually θ>0.7), the identified relationship is constructed as an edge in the knowledge graph. The threshold θ here can be adjusted according to the actual business scenario. A higher threshold can reduce misjudgment, while a lower threshold can discover more potential associations.

[0064] In addition, the source data is cleaned and standardized before building the map, including: data type unification, ensuring that the same type of fields are represented by consistent data types; null value processing, reasonably filling or marking missing values; outlier processing, identifying and processing outliers and outliers in the data; coding standardization, unifying character encoding, date format, measurement units, etc.; naming standardization, unifying field naming rules, and handling synonyms and ambiguous words.

[0065] As a preferred embodiment of the present invention, refer to Figure 3 In step (2), the process of encoding business rules into the knowledge graph mainly includes three key links: rule definition, rule storage, and rule execution:

[0066] (2-1) Based on the constructed entity node e and attribute node a, the system defines two types of core constraint rules: data value range constraint rule set and a set of constraints between nodes Among them, the data value domain constraint rules are mainly used to limit the basic characteristics of the field, such as the field length must not exceed the specified value, the value must be within the specified range, the string must conform to a specific format, etc. The node constraint rules focus on maintaining the integrity of the association between data, ensuring the consistency of primary and foreign keys, ensuring the dependency of field values, and maintaining the synchronization of data updates.

[0067] (2-2) In the rule storage stage, the system uses attribute graphs to store rule sets. Stored in the corresponding nodes and edges. Specifically, the constraints related to the single field Stored in the node attribute P(e), for example, for the order amount field, you can set the constraint condition "the amount must be greater than 0 and less than 1 million"; set cross-field and cross-table constraints Stored in the edge attribute P(r), such as the referential integrity constraint of "the customer ID in the order table must exist in the customer table". In this way, the rules and data structures are tightly bound, which facilitates subsequent rule checking and maintenance.

[0068] (2-3) In the rule execution phase, the system implements single-field constraint checking and association constraint checking respectively. For single-field constraint checking, the graph query language is used to query the node attribute value. Make judgments and execute functions For example, check string length, verify value range, check date format, etc. For association constraint checking, the system uses a graph traversal algorithm to compare the attribute values ​​of the associated nodes along the direction r of the edge and executes the function , for example, verifying the integrity of foreign key references, checking the consistency of summary data, etc.

[0069] The result of each constraint check contains three key dimensions: the position of the constraint violation (precisely located at the specific table, field or record), the type of problem (such as "value range out of bounds", "reference violation", etc.), and the severity level (which can be divided into "warning", "error", "serious", etc.). The system will check the results The problem is recorded in the status attribute of the corresponding node, and a problem index is built to support subsequent query and analysis.

[0070] As a preferred embodiment of the present invention, in step (3), the system performs a comprehensive impact analysis and propagation evaluation on the detected abnormal data. And its test results ,The system analyzes the propagation path and impact scope of the anomaly by constructing a directed graph.

[0071] (3-1) The system constructs a set of directed graph traversal paths , where each path Expressed as , through the graph traversal algorithm to identify abnormal data nodes Upstream impact path , and downstream propagation paths Tracking

[0072] For example, in a sales data analysis scenario, if an exception occurs in the sales amount field in the order table, the system will track up the discount rules, product prices and other related data that affect the amount, while also tracking down derivative data such as sales statistics and financial statements that depend on the amount.

[0073] For each propagation path identified, the system calculates its impact using the following formula:

[0074]

[0075] Among them, w(r) is the edge association strength defined in step (1), which reflects the closeness of the association between data; level is the severity of the anomaly, which can be assigned different weights according to the degree of business impact. By accumulating the product of the association strength of each edge on the path and the severity of the anomaly, the system quantitatively evaluates the impact of anomaly propagation.

[0076] Based on the path influence degree I(p) and node attributes P(e), the system constructs an anomaly propagation graph G(V, E).

[0077] The vertex set V represents the affected nodes , the edge set E represents the anomaly propagation path, and the comprehensive score of the anomaly is calculated. The formula is:

[0078]

[0079] Among them, I(p) is the degree of path impact, level is the severity of the anomaly, and the weight coefficients α and β can be adjusted according to the specific business scenario to control the proportion of propagation impact and anomaly severity in the score respectively. For example, for core business data, the β value can be increased to pay more attention to the severity of the anomaly itself; for derived data, the α value can be increased to pay more attention to the propagation impact.

[0080] As a preferred embodiment of the present invention, refer to Figure 4 In step (4), the system constructs a time series analysis model to achieve dynamic monitoring of the data quality improvement process. Specifically, the system collects the abnormal comprehensive score Φ within the continuous time window for each entity node e to form a score sequence These scores are calculated based on the comprehensive scoring method defined in step (3) and truly reflect the changing trend of data quality over time.

[0081] The system calculates the rate of change of scores at adjacent time points using the following formula:

[0082]

[0083] in, is the score at the current time point, For the score at the previous time point, set the score change rate threshold , (the recommended initial value is 0.2, which can be adjusted according to actual conditions), when For example, if the data quality score of a core business table drops by more than 20% in a short period of time, the system will immediately issue a warning notification to remind relevant personnel to intervene in time.

Claims

1. A data quality monitoring method based on knowledge graph, characterized in that: The steps include: S1, construct the data tables and fields in the collected multi-source heterogeneous data into graph nodes, construct graph edges through primary and foreign key constraint relationships and field value overlap analysis, and form a data lineage relationship knowledge graph; S2, encodes business rules and constraints into the node and edge attributes of the knowledge graph, performs data constraint checks through graph query and graph traversal algorithms, and records abnormal data to achieve intelligent monitoring of data quality, including: The node attribute value in the knowledge graph is represented as V(a), the set of edge relationships is represented as R, and a single edge relationship in the set R is represented as r. Based on the entity node e and the attribute node a in the knowledge graph, a set of data value range constraint rules is defined. , used to limit field length, value range and data format; Define a set of constraint rules between nodes according to the edge relationship r in the knowledge graph , used to ensure primary and foreign key consistency, field value dependencies and data update synchronization; The rule set The property graph is used to store the corresponding nodes and edges, where the node attribute P(e) stores a set of constraint rules related to a single field. , the edge attribute P(r) stores a set of cross-field and cross-table constraint rules ; Use graph query language to query node attribute values Make a judgment to perform a single-field constraint check , use the graph traversal algorithm to compare the attribute values ​​of the associated nodes along the direction r of the edge to perform the association constraint check The detection results include the position pos, type type and severity level of the constraint violation, and the detection results Record it in the status attributes of the corresponding node to realize automatic detection and verification of data consistency and integrity; S3, for the detected abnormal data, build a directed graph traversal path to track its upstream and downstream impacts, and build an abnormal propagation graph based on the correlation strength and abnormal severity for quantitative evaluation; S4, based on the data lineage knowledge graph, tracks the time changes of abnormal score sequences, and issues early warnings by setting change rate thresholds to achieve dynamic monitoring of the data quality improvement process.

2. According to the data quality monitoring method based on knowledge graph in claim 1, it is characterized in that: The multi-source heterogeneous data in S1 includes structured data in relational databases, semi-structured data in XML and JSON formats, and unstructured data in text documents and log files.

3. According to claim 2, a data quality monitoring method based on knowledge graph is characterized in that: The data kinship knowledge graph constructed in S1 adopts the entity-relationship-attribute triple modeling method for unified graph processing, where the triple is represented as , e represents an entity node set, and a represents an attribute set, specifically including: Mapping the data tables in the structured data of the relational database into entity type nodes , map the fields in the table to attribute nodes ; Mapping objects in the semi-structured data in XML and JSON format to entity nodes , map the object's attribute fields to attribute nodes ; The unstructured data of the text document and log file are segmented by a set of predefined delimiters, the text content is matched based on a predefined business object template, and the matched business objects are constructed as entity nodes. , construct the business object's identification, category, and time information into attribute nodes ; By scanning the node attribute value V(a) of all nodes, the table structure definition is analyzed to extract the explicit primary and foreign key constraint relationships. , compare field naming rules to identify potential reference relationships , the overlapping coefficient of the field values ​​is statistically analyzed to determine the association strength w. When w is greater than the preset threshold θ, the identified relationship is constructed as an edge in the knowledge graph to complete the knowledge graph construction.

4. According to the data quality monitoring method based on knowledge graph in claim 1, it is characterized in that: When performing backtracking analysis in S3, detect abnormal data nodes in S2 And its test results , construct a directed graph traversal path set , where each path Expressed as , through the graph traversal algorithm to identify abnormal data nodes Upstream impact path , and downstream propagation paths Conduct tracking; The influence degree on each path is calculated using the formula: Where w(r) is the edge association strength defined in S1, level is the severity of the anomaly, and the impact is quantified by accumulating the product of the association strength of each edge on the path and the severity of the anomaly; Based on the path influence degree I(p) and node attributes P(e), an anomaly propagation graph G(V, E) is constructed, where the vertex set V represents the affected nodes. , the edge set E represents the anomaly propagation path, and the comprehensive score of the anomaly is calculated. The formula is: Among them, α and β are weight coefficients, I(p) is the path impact degree, and level is the severity of the anomaly, which realizes the quantitative evaluation of the impact scope and severity of the anomaly.

5. According to the data quality monitoring method based on knowledge graph in claim 1, it is characterized in that: In S4, a time series analysis model is constructed to track data quality changes, and the abnormal comprehensive score Φ within the continuous time window is collected for each entity node e to form a score sequence , where Φ is the abnormal comprehensive score defined in S3, and the formula for calculating the score change rate at adjacent time points is: in, is the score at the current time point, Set the score change rate threshold for the previous time point ,when The early warning signal is triggered in real time to realize dynamic monitoring of the data quality improvement process.

6. A data quality monitoring system based on knowledge graph, characterized in that: The method for monitoring data quality based on a knowledge graph according to claim 1 is used, and the data quality monitoring system comprises: Data acquisition and preprocessing module: responsible for the acquisition and preprocessing of multi-source heterogeneous data, including structured data in relational databases, semi-structured data in XML and JSON formats, and unstructured data in text documents and log files; Knowledge graph construction module: Build a data kinship knowledge graph based on the entity-relationship-attribute triple modeling method, build data tables and fields as graph nodes, and build graph edges through primary and foreign key constraint relationships and field value overlap analysis; Quality monitoring module: responsible for encoding business rules and constraint relationships into the node and edge attributes of the knowledge graph, executing checks on data range constraint rules and inter-node constraint rules, and recording the detection results into the status attributes of the corresponding nodes, and recording abnormal data; Problem diagnosis module: Constructs a directed graph traversal path to track the upstream and downstream impacts of abnormal data. The upstream impact path is used to track the source data that causes the abnormality, and the downstream propagation path is used to evaluate the impact of the abnormality on other data. An abnormal propagation graph is constructed based on the correlation strength and abnormality severity, and the comprehensive score of the abnormality is calculated; Quality assessment module: Track the time changes of abnormal score sequences, calculate the score change rate, and issue early warnings by setting thresholds to achieve dynamic monitoring of the data quality improvement process.

7. A data quality monitoring system based on knowledge graph according to claim 6, characterized in that: The knowledge graph construction module adopts a graph database storage engine, supports graph data storage and query, and implements graph analysis functions through a graph computing engine.

8. A data quality monitoring system based on knowledge graph according to claim 6, characterized in that: The quality monitoring module performs data constraint checks through graph query and graph traversal algorithms to detect and warn of data quality issues.

9. A data quality monitoring system based on knowledge graph according to claim 6, characterized in that: The problem diagnosis module supports upstream and downstream impact analysis of abnormal data, and the quality assessment module monitors data quality through the score change rate.

Images