Data quality problem deep analysis and traceability system and method

Abstract

The invention discloses a data quality problem deep analysis and traceability system and method, and relates to the technical field of data processing, and the system comprises a data collection module, a data monitoring module, a knowledge graph construction module, a machine learning analysis module and a traceability analysis module. According to the system, the data is closely connected with the business knowledge, the source and the processing flow through the data association network constructed based on the knowledge graph technology, when a data quality problem occurs, the source and the propagation path of problem data can be rapidly tracked in the complex knowledge graph, and meanwhile, the data quality is improved. According to the abnormal data association analysis driven by machine learning, the internal relation between the seemingly isolated abnormal data can be mined, the whole view of the data quality problem is presented from the global perspective, the defect that the potential systematic problem is ignored due to the fact that only single-point anomaly is processed is avoided, and the data quality problem can be fundamentally solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a system and method for in-depth analysis and tracing of data quality issues. Background Art

[0002] In today's digital age, all industries are accelerating their digital transformation. The amount of data is growing explosively and has become one of the most valuable assets of enterprises. Whether it is transaction data in the financial field, patient diagnosis and treatment data in the medical industry, or user behavior data of Internet companies, they all play a key role in the company's decision-making, business operation optimization, risk assessment and management. High-quality data can provide companies with accurate market insights, efficient business process support and good customer experience guarantees, and is a core element for companies to maintain their competitiveness.

[0003] However, in terms of data traceability, existing data quality analysis and traceability technologies mostly rely on simple log records or limited metadata information, making it difficult to build a comprehensive and in-depth network of association relationships. When faced with complex data processing processes and multi-source data fusion scenarios, they are unable to effectively integrate the relationship between data and business knowledge, data sources, and processing processes, resulting in vague and incomplete traceability paths, making it difficult to quickly and accurately locate the root cause of the problem. For abnormal data association analysis, they only look at single abnormal data points in isolation and cannot deeply explore the potential association patterns between abnormal data. This means that when dealing with data quality issues, they can only solve superficial phenomena and cannot fundamentally prevent the recurrence of similar problems. It is also difficult to discover the systemic risks hidden behind the data.

[0004] To sum up, existing data quality analysis and traceability technologies can no longer meet the increasingly complex data management needs. The present invention aims to provide an innovative data quality problem in-depth analysis and traceability system and method to achieve accurate positioning, in-depth analysis and comprehensive traceability of data quality problems. Summary of the Invention

[0005] The purpose of this invention is to make up for the shortcomings of the existing technology and provide a deep analysis and tracing system and method for data quality problems. It can associate data with relevant business knowledge, data sources, and processing flow information by introducing knowledge graph technology, and build a large and complex knowledge network. When data quality problems occur, the system can quickly locate the source and propagation path of the problem through deep search and reasoning in the knowledge graph, helping users to understand the overall picture of the data quality problem more comprehensively.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: On the one hand, a data quality problem in-depth analysis and traceability system, the system comprises: a data acquisition module, a data monitoring module, a knowledge graph construction module, a machine learning analysis module and a traceability analysis module;

[0007] The data acquisition module is responsible for collecting data from various data sources and performing preliminary sorting and formatting of the data. The data sources include databases, file systems, and network interfaces.

[0008] The data monitoring module monitors the data collected by the data acquisition module and the data flowing in the system in real time, and sets data quality monitoring rules. The data quality monitoring rules are formulated based on multi-dimensional quality indicators such as data integrity, accuracy, consistency, timeliness and uniqueness;

[0009] When monitored data violates established rules, it is determined that a data quality anomaly has occurred. The data monitoring module automatically generates a data quality problem report, which includes the identifier of the abnormal data, the time when the anomaly occurred, the type of anomaly, and contextual information related to the abnormal data. The report is sent to the traceability analysis module for traceability tracking;

[0010] The knowledge graph construction module obtains data from the data acquisition module, and combines business knowledge and data processing flow information to map the data to the knowledge graph, and constructs a knowledge network containing data nodes, business knowledge nodes, data source nodes, data processing flow nodes, and the relationships between them. The relationships include data generation relationships, data conversion relationships, and data dependency relationships. The construction process of the knowledge graph is as follows:

[0011] Data ontology design: Define data ontology O, including data class C, data attribute P and association relationship R, where data class C covers data record class C r 、Business knowledge C k , Data source class C s , Data processing process class C p , data attribute P has data record name attribute P n , data source identification attribute P i , data processing flow attribute P t , the association relationship R includes the generation relationship R between data records and data sources fs , the conversion relationship between data records and data processing flow R fp ;

[0012] Knowledge graph initialization: construct an empty knowledge graph KG = (N, E), where N is the node set and E is the edge set;

[0013] Data node mapping: Get data D from the data acquisition module, and create a data node n for each data record d∈D d ∈N, set attribute values ​​for it according to the data ontology O, that is, when d is product sales data, including product name pn, sales quantity qn, and sales time tm fields, then P n (n d )=pn,P qn (n d )=qn,P tm (n d ) = tm;

[0014] Business knowledge node construction: Create business knowledge nodes n with the knowledge base KB k ∈N, create the corresponding business knowledge node n in the knowledge graph k , and establish data node n d and business knowledge node n k The relationship between fb ∈E;

[0015] Data source node association: For each data record d, determine its data source s and create a corresponding data source node n in the knowledge graph s ∈N, and establish data node n d With data source node n s The generative relationship R fs ∈E, that is (n d ,n s ,R fs );

[0016] Data processing process node mapping: Take the data processing process information F, and create the corresponding processing process node n in the knowledge graph for the data d that has undergone data processing f∈F f ∈N, and establish data node n d and process node n f The conversion relationship R fp ∈E, that is (n d ,n f ,R fp );

[0017] Relationship strengthening: Using data dependency judgment algorithm to strengthen the knowledge graph relationship, that is, for data node n d1 and n d2 , when the attribute value P of data d1 v (n d1 ) and the attribute value P of data d2 u (n d2 ) has a dependency relationship Dep(P v (n d1 ),P u(n d2 )), then add n in the knowledge graph d1 to n d2 Data dependency edge R dd , that is, R dd ∈E, the data dependency judgment algorithm is: When this function makes the equation true, there is data dependency, otherwise there is no data dependency;

[0018] The machine learning analysis module stores historical data and abnormal data samples marked by the data monitoring module, uses a machine learning algorithm for training and learning, and establishes an abnormal data association model, which can identify the association pattern between different abnormal data features;

[0019] The traceability analysis module receives data quality problem reports and uses the knowledge graph and abnormal data association module to perform a global traceability analysis on the data quality problems. The problem report includes abnormal information detected by the data monitoring module and data quality problems manually submitted by users.

[0020] Furthermore, the data quality monitoring rule based on integrity in the data monitoring module is: according to the data set D, the data record r i , data field set F, calculation integrity index I comp for: Where n is the total number of data records, m is the total number of key data fields, and c ij is the indicator function, that is, when the data record r i The data field f in j If there is a value, c ij =1, otherwise c ij =0, when I comp Below the integrity threshold T comp When , it is determined that the integrity is abnormal.

[0021] Furthermore, the data quality monitoring rule based on accuracy in the data monitoring module is: for the numerical data field f k , whose true value is v trus,k , the monitoring value is v mon,k , then the accuracy index I acc for: Where p is the total number of numeric data fields. acc Below the accuracy threshold T acc When the accuracy is abnormal, for text data fields, similarity matching is performed with the standard text set S, that is, the text field t is matched with the most similar text s in the standard text set. max The similarity is sim(t,s max ), when sim(t,smax ) is lower than the text similarity threshold T sim , the judgment accuracy is abnormal.

[0022] Furthermore, the data quality monitoring rule based on consistency in the data monitoring module is: for q data processing stages, the field f of the data in the lth data processing stage l The value set is V l , then the consistency index I con for: Where V m For all fields f of the data in the lth data processing stage m The value set of con Below the preset consistency threshold T con When , it is determined that the consistency is abnormal.

[0023] Furthermore, the data quality monitoring rule based on timeliness in the data monitoring module is: set the last update time of data d to t last , the current time is t now , set the effective time interval to t valid , then the timeliness index I tim for: When I tim =0, the timeliness is determined to be abnormal.

[0024] Furthermore, the data quality monitoring rule based on uniqueness in the data monitoring module is: for the field f to be checked for uniqueness in the data set D, uni , calculate its uniqueness index I uni for: where |{f uni}| is the field f uni The number of different values, |D| is the number of records in the data set D, when I uni When ≠1, it is determined that the uniqueness is abnormal.

[0025] Furthermore, the specific process of the machine learning analysis module to build the abnormal data management model is as follows:

[0026] Data preprocessing and feature extraction: From historical data D hist and abnormal data samples D marked by the data monitoring module anom Extract data features and record the data and For each data record, extract t features to form a feature vector and

[0027] Model training: Using the association rule mining algorithm in machine learning, set the minimum support threshold as minSup and the minimum confidence threshold as minConf. For the feature item set I, calculate its support Sup(I) and confidence Conf(I→J), where J is another feature item set, that is, When Sup(I)≥minSup and Conf(I→J)≥minConf, the association rule I→J is determined to be a valid association rule, and the abnormal data association model M is constructed. assoc .

[0028] Furthermore, when the source tracing analysis module traces the problem data in the report, the specific steps are as follows:

[0029] Conduct in-depth searches in the knowledge graph to locate the source and propagation path of problematic data based on the associated paths of the data in the knowledge graph. This means tracing the connections between data nodes and data source nodes to determine the original source of the data.

[0030] By analyzing the relationship between data nodes and data processing flow nodes, we can understand the processing steps that the data goes through and the links where problems are introduced;

[0031] The abnormal data association model established by the machine learning analysis module is used to perform association analysis on the problem data. When anomalies are found in data fields, the model is used to find problems in other related data fields and data processing links, achieving in-depth tracing from local anomalies to global problems.

[0032] On the other hand, a method for in-depth analysis and tracing of data quality issues is provided. The specific steps of this method are as follows:

[0033] S1. Collect data from different data sources and perform preliminary sorting and formatting to make all data formats uniform;

[0034] S2. Monitor the collected data in real time and set data quality monitoring rules. If data violates the established rules, it will be judged as a data quality anomaly and a data quality problem report will be automatically generated.

[0035] S3. Design the data ontology, determine how the data is represented in the knowledge graph and define the relationships, extract key information from the data, and map the data and related information to the knowledge graph in combination with business knowledge and process information to build a complete knowledge network.

[0036] S4. Collect historical data and abnormal data samples, perform data cleaning and feature extraction, and use machine learning algorithms to train abnormal data association models;

[0037] S5. Receive data quality problem reports, conduct in-depth searches in the knowledge graph for the problem data in the reports, and use the abnormal data association model to perform association analysis on the problem data.

[0038] Compared with the existing technology, this data quality problem in-depth analysis and traceability system and method has the following beneficial effects:

[0039] 1. The system of the present invention closely connects data with business knowledge, sources and processing flows through a data association network constructed based on knowledge graph technology. When data quality problems occur, the source and propagation path of the problem data can be quickly traced in the complex knowledge graph. At the same time, the abnormal data association analysis driven by machine learning can dig out the intrinsic connections between seemingly isolated abnormal data, presenting the full picture of data quality problems from a global perspective, avoiding the drawback of only processing single-point anomalies and ignoring potential systemic problems, so that data quality problems can be fundamentally solved, the recurrence rate of the problem can be reduced, and a large amount of manpower, material resources and time costs can be saved for enterprises.

[0040] 2. The present invention uses machine learning algorithms to study and analyze large amounts of historical data and abnormal data, and establishes a correlation model between abnormal data. This machine learning-driven abnormal data correlation analysis can automatically identify potential correlations between seemingly isolated abnormal data, thereby gaining a more comprehensive understanding of the overall picture of data quality issues. When abnormal values ​​are found in data fields, correlation analysis can be used to identify problems in other related data fields and data processing links, achieving in-depth tracing from point to surface, and significantly improving the level of intelligent processing of data quality issues.

[0041] Other advantages, objects and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be learned from the practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0043] Figure 1 This is an operational flow chart for a system for in-depth analysis and traceability of data quality issues;

[0044] Figure 2This is a diagram of the knowledge graph construction process in a deep analysis and traceability system for data quality issues. DETAILED DESCRIPTION

[0045] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] Example 1

[0047] This embodiment elaborates on the data quality problem tracing process based on the knowledge graph. By constructing a knowledge graph, the data is associated with business knowledge, sources and processing procedures, and combined with the multi-dimensional quality indicator monitoring of the data monitoring module, a comprehensive tracing analysis of data quality problems is achieved.

[0048] First, the data collection module starts working and collects data from various data sources (databases, file systems, network interfaces). The collected data covers various types of information in the business operations of the enterprise, including sales data, customer information, and financial data. These raw data have differences in format and content. The data collection module will perform preliminary sorting and formatting operations on them, converting the data into a format that the system can uniformly process. The knowledge graph construction module designs the data ontology, defines the data ontology O, and clearly includes the data class C, data attributes P and association relationship R. The data class covers the data record class C. r 、Business knowledge C k , Data source class C s , Data processing process class C p ; Data attributes include data record name attribute P n , data source identification attribute P i , data processing process step attributes P t ; The association relationship includes the generation relationship R between data records and data sources fs , the conversion relationship between data records and data processing flow R fp , initialize the knowledge graph and construct an empty knowledge graph KG = (N, E), where N is the node set and E is the edge set, to prepare for the subsequent mapping of data nodes, business knowledge nodes, data source nodes and data processing flow nodes and the establishment of association relationships.

[0049] Then, the data collection module transmits the collected data D to the knowledge graph construction module. For each data record d∈D, which contains the fields of product name pn, sales quantity qn, and sales time tm, a data node n is created in the knowledge graph. d∈N, and set attribute values ​​for it according to the data ontology O, that is, P n (n d )=pn,P qn (n d )=qn,P t m(n d )=tm, each data record has a corresponding node representation in the knowledge graph, and the node carries the attribute information of the data. The knowledge graph construction module interacts with the knowledge base KB to create business knowledge nodes n k ∈N, and create the corresponding business knowledge node n in the knowledge graph k , and establish data node n d and business knowledge node n k The relationship between R fb ∈E; For each data record d, determine its data source s, and find or create the corresponding data source node n in the knowledge graph s ∈N, establish data node n d With data source node n s The generative relationship R fs ∈E, that is (n d ,n s ,R fs ), which makes the data establish a clear connection with the source of its generation in the knowledge graph, which helps to trace the origin of the data during tracing; using the data dependency judgment algorithm to strengthen the association relationship of the knowledge graph, for data node n d1 and n d2 , when data n d1 The attribute value P v (n d1 ) and data n d2 The attribute value P u (n d2 ) has a dependency relationship Dep(P v (n d1 ),P u (n d2 )) When adding n in the knowledge graph d1 to n d2 The "data dependence" edge R dd ∈E, the data dependency judgment algorithm is When this relationship is satisfied, a dependency edge is established between the two data nodes in the knowledge graph, thereby more comprehensively reflecting the intrinsic connection between the data.

[0050] Subsequently, the data monitoring module monitors the data collected by the data acquisition module and the data flowing in the system in real time, and formulates monitoring rules based on the multi-dimensional quality indicators of data integrity, accuracy, consistency, timeliness and uniqueness. For integrity monitoring, according to the data set D, data record r i , the data field set F calculates the integrity index Where n is the total number of data records, m is the total number of key data fields, and c ij is the indicator function, when the data record r i The data field f in j If there is a value, c ij =1, otherwise c ij =0, when I comp Below the integrity threshold T comp When the integrity is abnormal, it is determined that the integrity is abnormal; for accuracy monitoring, set the numerical data field f k , whose true value is v true,k , the monitoring value is v mon,k , accurate indicators Where p is the total number of numeric data fields. acc Below the accuracy threshold T acc When , the accuracy is judged to be abnormal; for text data fields, by similarity matching with the standard text set S, that is, the text field t and the most similar text s in the standard text set are matched. max The similarity is sim(t,s max ), when sim(t,s max ) is lower than the text similarity threshold T sim , determine the accuracy abnormality; for consistency monitoring, for q data processing stages, the data in the field f of the ιth data processing stage l The value set is V l , consistency index When I con Below the preset consistency threshold T con When , the consistency is determined to be abnormal; for timeliness monitoring, set the last update time of data d to t last , the current time is t now , set the effective time interval to t valid , timeliness index When I tim = 0, the timeliness is determined to be abnormal; for uniqueness monitoring, for the field f to be checked for uniqueness in the data set D uni , calculate its uniqueness index where |{f uni}| is the field f uni The number of different values, |D| is the number of records in the data set D, when I uniWhen the value is ≠1, a uniqueness anomaly is determined. When monitored data violates established rules, it is determined that a data quality anomaly has occurred. The data monitoring module automatically generates a data quality problem report containing the identifier of the abnormal data, the time when the anomaly occurred, the anomaly type, and contextual information related to the abnormal data, such as the data table where the abnormal data is located and the relevant data processing flow. The report is then sent to the traceability analysis module.

[0051] Finally, after receiving the data quality problem report, the traceability analysis module conducts a deep search in the knowledge graph, locates the source and propagation path of the problem data based on the association path of the data in the knowledge graph, and determines the original source of the data by tracing the connection between the data node and the data source node. For example, if a problem is found in the sales data, the problem can be found through the generation relationship R in the knowledge graph. fs , find its corresponding data source node (database table or external data interface), and at the same time, by analyzing the relationship between the data node and the data processing flow node, understand the processing steps of the data and the link that introduces the problem, and according to the conversion relationship R fp , determine the flow of data in each processing flow node, and analyze at which processing step data quality problems occur.

[0052] To sum up, this embodiment demonstrates the entire process of tracing the source of data quality problems based on the knowledge graph. Through the collaborative work of various modules of the system, from data collection, knowledge graph construction, data monitoring to traceability analysis, a complete in-depth analysis and traceability system for data quality problems is formed, providing a comprehensive information foundation for traceability.

[0053] Example 2

[0054] This example focuses on the specific application of machine learning-driven abnormal data association analysis in data quality problem handling. It elaborates in detail how to use machine learning algorithms to process historical data and abnormal data samples, build association models, and apply them to actual data quality problem analysis to achieve in-depth tracing from local anomalies to global problems.

[0055] In the specific implementation, first, the data acquisition module starts to collect data from multiple data sources (databases, file systems, network interfaces). These data sources contain various key data in enterprise operations, including production data, sales data, and customer data. After entering the system, the collected data will be preliminarily formatted and organized by the data acquisition module to remove obvious errors and incomplete data to ensure the basic availability of the data. The collected data will be stored in a historical database and a data cache area for real-time monitoring. The historical database is used to store massive data accumulated over a long period of time. These data will serve as learning resources for the machine learning analysis module. The data cache area stores real-time collected data and data flowing in the system, which facilitates real-time monitoring and analysis by the data monitoring module.

[0056] The machine learning analysis module obtains historical data D from the historical database hist And obtain the marked abnormal data sample D from the data monitoring module anom For each data record, whether it is from historical data or abnormal data samples, feature extraction operation is performed. For sales data records, the extracted features include sales amount, sales time, product category, and sales region. These features are combined into a feature vector. Its eigenvector is expressed as Where t is the number of features extracted; for the data records in the abnormal data sample Its eigenvector is The association rule mining algorithm is used for model training. The minimum support threshold minSup and the minimum confidence threshold minConf are set. These two thresholds will be used to screen effective association rules. For the feature item set I, its support is calculated. The support reflects the frequency of feature item set I in the data set, and the confidence is calculated at the same time (where J is another feature item set), the confidence level indicates the probability of feature item set J appearing when feature item set I appears. When Sup(I)≥minSup and Conf(I→J)≥minConf, the association rule I→J is determined to be a valid association rule. By analyzing a large amount of historical data and abnormal data samples, we continuously mine and filter out effective association rules and gradually build an abnormal data association model M. assoc ,This model is able to capture the potential correlation patterns between different abnormal data features.

[0057] The data monitoring module continuously monitors the data in the data cache in real time, and detects it based on the monitoring rules of the multi-dimensional quality indicators of integrity, accuracy, consistency, timeliness and uniqueness. For integrity monitoring, according to the data set D (real-time data in the data cache), data record r iAnd the data field set F, according to the integrity index formula Calculation Integrity Index I cornp , where n is the total number of data records, m is the total number of key data fields, and c ij is the indicator function, when the data record r i The data field f in j If there is a value, c ij =1, otherwise c ij =0 if I comp Below the integrity threshold T comp , then it is determined that an integrity anomaly occurs; in terms of accuracy monitoring, for numeric data fields f k , according to its true value v true,k and monitoring value v mon,k , through the accurate indicator formula Calculate the accuracy index (p is the total number of numerical data fields), when I acc Below the accuracy threshold T acc When the accuracy is abnormal, for text data fields, by performing similarity matching with the standard text set S, the most similar text s between the text field t and the standard text set is calculated. max The similarity sim(t,s max ), if it is lower than the text similarity threshold T sim , determine the accuracy abnormality; in consistency monitoring, for q data processing stages, according to the field f of the data in the lth data processing stage l The value set V l , using the consistency index formula Calculate the consistency index, when I con Below the preset consistency threshold T con When , the consistency is judged to be abnormal; the timeliness monitoring is based on the last update time t of the set data d last , current time t now and effective time interval t valid , through the timeliness index formula Judging timeliness, when I tim = 0, the timeliness is determined to be abnormal; uniqueness monitoring is for the field f to be checked for uniqueness in the data set D uni , calculate the uniqueness index (where |{f uni}| is the field f uni The number of different values, |D| is the number of records in the data set D), when I uniWhen the value is ≠1, a uniqueness anomaly is determined. Once the data monitoring module detects that the data violates any of the established rules, it determines that a data quality anomaly has occurred and automatically generates a detailed data quality problem report. The report content includes the identification of the abnormal data (data record number, field name), the time when the anomaly occurred, the anomaly type (completeness anomaly, accuracy anomaly, consistency anomaly, timeliness anomaly, and uniqueness anomaly), and the contextual information related to the abnormal data (the business process to which it belongs, the relevant data processing links). The report is then sent to the traceability analysis module.

[0058] After receiving the data quality problem report, the traceability analysis module preliminarily identifies and classifies the problem data, determines the data table and field where the abnormal data is located, and its approximate position in the data processing flow, and inputs the characteristics of the problem data into the constructed abnormal data association model. Through the association rules learned in the model, it searches for problems in other data fields and data processing links related to the problem data characteristics. If the sales amount is found to be abnormal, the model prompts to check whether there are abnormalities in the product unit price, sales quantity, promotion activity data fields related to the sales business, and price calculation, data aggregation, and data transmission and processing links. This helps to expand from local sales amount anomalies to a comprehensive analysis of the entire sales business-related data and processes, and achieve in-depth traceability from point to surface.

[0059] Finally, the provenance analysis module integrates provenance information obtained from the knowledge graph and the abnormal data association model. It summarizes and organizes information on the source, propagation path, associated data fields, and processing steps of the problematic data to generate an intuitive data provenance report. This report, presented in charts and text, clearly illustrates the full picture of data quality issues, providing decision support for data managers and business personnel, allowing them to take targeted measures to fix data quality issues, optimize data processing processes, and prevent similar problems from recurring.

[0060] To sum up, this embodiment demonstrates in detail the specific application process of machine learning-driven abnormal data association analysis in the deep analysis and traceability system of data quality problems. From data collection and storage in the system preparation stage, to the construction of machine learning models, to data monitoring, anomaly detection and final association analysis and traceability, each link is closely connected and works together. Through the learning and analysis of large amounts of data by machine learning algorithms, the constructed abnormal data association model can effectively explore the potential correlations between abnormal data. Combined with the multi-dimensional detection of the data monitoring module and the comprehensive processing of the traceability analysis module, a more comprehensive and in-depth understanding and positioning of data quality problems is achieved.

[0061] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A data quality problem in-depth analysis and traceability system, characterized by: The system consists of: data collection module, data monitoring module, knowledge graph construction module, machine learning analysis module and traceability analysis module; The data acquisition module is responsible for collecting data from various data sources and preliminarily arranging and formatting the data. The data sources include databases, file systems, and network interfaces. The data monitoring module monitors the data collected by the data acquisition module and the data circulating in the system in real time, and sets data quality monitoring rules, which are formulated based on multi-dimensional quality indicators such as data integrity, accuracy, consistency, timeliness and uniqueness; When the monitored data violates the established rules, it is determined that data quality abnormalities occur. The data monitoring module automatically generates a data quality problem report, which contains the identification of the abnormal data, the time when the abnormality occurred, the abnormality type, and the context information related to the abnormal data. The report is sent to the traceability analysis module for traceability tracking; The knowledge graph construction module obtains data from the data acquisition module, and maps the data to the knowledge graph in combination with business knowledge and data processing flow information, and constructs a knowledge network including data nodes, business knowledge nodes, data source nodes, data processing flow nodes and the relationship between them. The relationship includes data generation relationship, data conversion relationship and data dependency relationship. The construction process of the knowledge graph is as follows: Data ontology design: define data ontology O, including data class C, data attribute P and association relationship R, where data class C covers data record class C r , Business Knowledge C k 、Data source category C s , Data processing flow class C p , data attribute P has data record name attribute P n , data source identification attribute P i , data processing flow attributes P t , the association relationship R includes the generation relationship R between data records and data sources fs , the conversion relationship between data record and data processing flow R fp ; Knowledge graph initialization: construct an empty knowledge graph KG = (N, E), where N is the node set and E is the edge set; Data node mapping: Get data D from the data acquisition module, and create a data node n for each data record d∈D d ∈N, set attribute values ​​for it according to the data entity O, that is, when d is product sales data, including product name pn, sales quantity qn, and sales time tm fields, then P n (n d )=pn,P qn (n d )=qn,P tm (n d ) = tm; Business knowledge node construction: Create business knowledge nodes n with the knowledge base KB k ∈N, create the corresponding business knowledge node n in the knowledge graph k , and establish data node n d With business knowledge node n k The relationship between fb ∈E; Data source node association: For each data record d, determine its data source s and create a corresponding data source node n in the knowledge graph s ∈N, and establish data node n d With data source node n s The generative relationship R fs ∈E, that is (n d ,n s ,R fs ); Data processing flow node mapping: Take the data processing flow information F, and create the corresponding processing flow node n in the knowledge graph for the data d that has undergone data processing f∈F f ∈N, and establish data node n d and process node n f The conversion relationship between fp ∈E, that is (n d ,n f ,R fp ); Relationship strengthening: Use data dependency judgment algorithm to strengthen the knowledge graph relationship, that is, for data node n d1 and n d2 , when the attribute value P of data d1 v (n d1 ) and the attribute value P of data d2 u (n d2 ) has a dependency relationship Dep(P v (n d1 ),P u (n d2 )) When n is added to the knowledge graph d1 to n d2 The data dependency edge R dd , that is, R dd ∈E, the data dependency judgment algorithm is: When this function makes the equation true, there is data dependency, otherwise there is no data dependency; The machine learning analysis module stores historical data and abnormal data samples marked by the data monitoring module, uses a machine learning algorithm for training and learning, and establishes an abnormal data association model, which can identify association patterns between different abnormal data features; The traceability analysis module receives a data quality problem report and uses the knowledge graph and abnormal data association module to perform a global traceability analysis on the data quality problem. The problem report includes abnormal information detected by the data monitoring module and data quality problems manually submitted by users.

2. A data quality problem in-depth analysis and traceability system according to claim 1, characterized in that: The data quality monitoring rule based on integrity in the data monitoring module is: according to the data set D, the data record r i , data field set F, calculation integrity index I comp for: Where n is the total number of data records, m is the total number of key data fields, and c is the total number of ij is the indicator function, that is, when the data record r i The data field f in j If there is a value, c ij =1, otherwise c ij =0, when I comp Below the integrity threshold T comp When , the integrity is judged to be abnormal.

3. A data quality problem in-depth analysis and traceability system according to claim 1, characterized in that: The data quality monitoring rule based on accuracy in the data monitoring module is: for the numerical data field f k , whose true value is v true,k , the monitoring value is v mon,k , then the accuracy index I acc for: Where p is the total number of numeric data fields. acc Below the accuracy threshold T acc When the accuracy is abnormal, for text data fields, similarity matching is performed with the standard text set S, that is, the text field t is matched with the most similar text s in the standard text set. max The similarity is sim(t,s max ), when sim(t,s max ) is lower than the text similarity threshold T sim , the judgment accuracy is abnormal.

4. A data quality problem in-depth analysis and traceability system according to claim 1, characterized in that: The consistency-based data quality monitoring rule in the data monitoring module is: for q data processing stages, the data in field f of the lth data processing stage l The value set is V l , then the consistency index I con for: Where V m is all the fields f of the data in the lth data processing stage m The value set of I con Below the preset consistency threshold T con When , the consistency is judged to be abnormal.

5. A data quality problem in-depth analysis and traceability system according to claim 1, characterized in that: The data quality monitoring rule based on timeliness in the data monitoring module is: set the last update time of data d to t last , the current time is t now , set the effective time interval to t valid , then the timeliness index I tim for: When I tim =0, the timeliness is judged to be abnormal.

6. A data quality problem in-depth analysis and tracing system according to claim 1, characterized in that: The data quality monitoring rule based on uniqueness in the data monitoring module is: for the field f to be checked for uniqueness in the data set D, uni , calculate its uniqueness index I uni for: where |{f uni }| is the field f uni The number of different values, |D| is the number of records in the data set D, when I uni When ≠1, the uniqueness is judged to be abnormal.

7. A data quality problem in-depth analysis and tracing system according to claim 1, characterized in that: The specific process of constructing the abnormal data management model by the machine learning analysis module is as follows: Data preprocessing and feature extraction: From historical data D hist and the abnormal data samples D marked by the data monitoring module anom Extract data features and record the data and For each data record, extract t features to form a feature vector and Model training: Using the association rule mining algorithm in machine learning, set the minimum support threshold as minSup and the minimum confidence threshold as minConf. For the feature item set I, calculate its support Sup(I) and confidence Conf(I→J), where J is another feature item set, that is, When Sup(I)≥minSup and Conf(I→J)≥minConf, the association rule I→J is determined to be a valid association rule, and the abnormal data association model M is constructed. assoc .

8. A data quality problem in-depth analysis and tracing system according to claim 1, characterized in that: When the source tracing analysis module traces the problem data in the report, the specific steps are as follows: Conduct in-depth searches in the knowledge graph to locate the source and propagation path of problematic data based on the association paths of data in the knowledge graph. That is, by tracing the connection between data nodes and data source nodes, determine the original source of data. By analyzing the relationship between data nodes and data processing flow nodes, we can understand the processing steps that the data goes through and the links where problems are introduced; The abnormal data association model established by the machine learning analysis module is used to perform association analysis on the problem data. When anomalies are found in the data fields, the model is used to find problems in other related data fields and data processing links, thereby achieving deep tracing from local anomalies to global problems.

9. A method for in-depth analysis and tracing of data quality problems, applicable to a system for in-depth analysis and tracing of data quality problems as claimed in any one of claims 1 to 8, characterized in that: The specific steps of this method are: S1. Collect data from different data sources, and perform preliminary sorting and formatting to make all data formats uniform; S2. Monitor the collected data in real time and set data quality monitoring rules. For data that violates the established rules, it is determined that data quality abnormalities have occurred and a data quality problem report is automatically generated; S3. Design the data ontology, determine the representation of data in the knowledge graph and the definition of association relationships, extract key information from the data, and map the data and related information to the knowledge graph in combination with business knowledge and processing flow information to build a complete knowledge network; S4, collect historical data and abnormal data samples, perform data cleaning and feature extraction, and use machine learning algorithms to train abnormal data association models; S5. Receive data quality problem reports, conduct in-depth searches in the knowledge graph for the problem data in the reports, and use the abnormal data association model to perform association analysis on the problem data.

Images