Data tracing method and device, electronic equipment and computer program product

Abstract

The invention discloses a data tracing method and device, electronic equipment and a computer program product. The method comprises the following steps: monitoring the data quality of a database to obtain to-be-traced data representing that the data quality is abnormal; querying the to-be-traced data in a plurality of preset dimensions through a preset knowledge graph to obtain a plurality of to-be-analyzed objects; based on a plurality of preset dimensions, a weighted score corresponding to each to-be-analyzed object is determined, and the weighted score is a weighted result of a comprehensive score obtained by the same to-be-analyzed object on the plurality of preset dimensions and a dimension weight corresponding to each preset dimension; and sorting the plurality of to-be-analyzed objects according to the weighted scores from high to low to obtain a root cause list of the to-be-traced data. According to the invention, the technical problem that the root cause obtained by tracing the data quality through the existing tracing technology is inaccurate is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data tracing method, device, electronic device and computer program product. Background Art

[0002] In the field of data quality management and traceability, existing technologies have some limitations.

[0003] 1. Insufficient coverage: Some methods only focus on the changing process of data quality and fail to fully cover the multi-dimensional information of data quality, such as space, entities, and events, and thus trace its root causes, making it impossible to take efficient measures to repair and improve it.

[0004] 2. Insufficient Explanation: While some technologies involve traceability, conventional methods (such as labeling and metadata tracking) are insufficiently explanatory for complex system anomalies and lack sufficient knowledge representation and reasoning mechanisms. The resulting models may not clearly represent data quality-related entities and relationships, resulting in inaccurate root cause identification and failure to meet practical application requirements.

[0005] Currently, no effective solution has been proposed to the problem that the root causes of data quality traceability obtained by the above-mentioned existing traceability technologies are inaccurate. Summary of the Invention

[0006] The embodiments of the present invention provide a data tracing method, device, electronic device and computer program product to at least solve the technical problem that the root cause obtained by tracing data quality using existing tracing technologies is inaccurate.

[0007] According to one aspect of an embodiment of the present invention, a data tracing method is provided, comprising: monitoring data quality of a database to obtain data to be traced indicating abnormal data quality; querying the data to be traced in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed, wherein nodes in the preset knowledge graph are determined based on entities that affect the data quality of the database, node attributes of the preset knowledge graph include: dimension attributes corresponding to the entities in multiple preset dimensions, edges in the preset knowledge graph are determined based on associations between the entities, and the entities include at least: the data to be traced, each object to be analyzed is the entity that has the association relationship with the data to be traced in the preset dimension; determining a weighted score corresponding to each object to be analyzed based on the multiple preset dimensions, wherein the weighted score is a comprehensive score obtained for the same object to be analyzed in the multiple preset dimensions, and a weighted result of the dimension weight corresponding to each preset dimension; and sorting the multiple objects to be analyzed in descending order according to the weighted scores to obtain a root cause list of the data to be traced.

[0008] Optionally, monitoring the data quality of the database and obtaining data to be traced indicating that the data quality is abnormal includes: collecting quality data for evaluating the data quality; performing quality evaluation on the quality data to obtain an evaluation result, wherein the evaluation result is used to indicate whether the quality data has an abnormal situation; and determining the data to be traced based on the quality data with the abnormal situation.

[0009] Optionally, before querying the data to be traced in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed, the method also includes: collecting quality data for evaluating the data quality; based on the quality data, determining the entities that affect the data quality of the database and the association relationships between the entities; and constructing the preset knowledge graph based on multiple entities and the association relationships.

[0010] Optionally, the data to be traced is queried in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed, including: calling the dimension indexer corresponding to each of the preset dimensions, wherein the dimension indexer is used to query based on the dimension attributes corresponding to the preset dimension; using the dimension indexer, in the preset knowledge graph, querying the node that has the association relationship with the data to be traced as the target node; and determining the entity indicated by the target node as the object to be analyzed.

[0011] Optionally, based on multiple preset dimensions, determining the weighted score corresponding to each object to be analyzed includes: calculating the comprehensive score corresponding to each preset dimension based on the explanatory power and surprise of each object to be analyzed in the same preset dimension, wherein the explanatory power measures the ability of the preset dimension to explain the abnormal situation by comparing the difference between the observed value of the abnormal situation and the expected value of the normal situation, and the surprise is used to indicate the degree of deviation of the abnormal situation from the normal situation in the preset dimension, and the comprehensive score is the weighted result of the explanatory power and the surprise of the same preset dimension; according to the dimension weights corresponding to the multiple preset dimensions, the comprehensive scores corresponding to the same object to be analyzed in the multiple preset dimensions are weightedly summed to obtain the weighted score corresponding to each object to be analyzed.

[0012] Optionally, based on the explanatory power and surprise of each object to be analyzed in the same preset dimension, calculating the comprehensive score corresponding to each preset dimension includes: calculating the explanatory power corresponding to each preset dimension based on the observed value of the abnormal situation, the expected value of the normal situation, and the edge weight of each object to be analyzed in the same preset dimension, wherein the edge weight is determined in advance according to the degree of association between the preset dimension and the abnormal situation; calculating the surprise corresponding to each preset dimension based on the observed value of the abnormal situation, and the mean and standard deviation of the normal situation of each object to be analyzed in the same preset dimension; assigning a first weight to the explanatory power, assigning a second weight to the surprise, and performing weighted summation of the explanatory power and the surprise of the same preset dimension to obtain the comprehensive score corresponding to each preset dimension, wherein, under the same preset dimension, the sum of the first weight and the second weight is 1.

[0013] Optionally, before the observed value of the abnormal situation, the expected value of the normal situation, and the edge weight of each object to be analyzed in the same preset dimension, the method also includes: identifying the target abnormality type to which the object to be analyzed belongs among multiple preset abnormality types, wherein each preset abnormality type is pre-configured with a corresponding edge weight calculation method; and determining the edge weight according to the edge weight calculation method corresponding to the target abnormality type.

[0014] According to another aspect of an embodiment of the present invention, a data tracing device is provided, comprising: a monitoring module for monitoring the data quality of a database, and obtaining data to be traced indicating that the data quality is abnormal; a query module for querying the data to be traced in multiple preset dimensions through a preset knowledge graph, and obtaining multiple objects to be analyzed, wherein the nodes in the preset knowledge graph are determined based on entities that affect the data quality of the database, and the node attributes of the preset knowledge graph include: the entity corresponds to dimensional attributes in multiple preset dimensions, and the edges in the preset knowledge graph are based on the relationships between the entities. The association relationship is determined, and the entity at least includes: the data to be traced, each of the objects to be analyzed is the entity that has the association relationship with the data to be traced on the preset dimension; a determination module, used to determine the weighted score corresponding to each object to be analyzed based on multiple preset dimensions, wherein the weighted score is a comprehensive score obtained for the same object to be analyzed on multiple preset dimensions, and a weighted result of the dimension weight corresponding to each preset dimension; a sorting module, used to sort the multiple objects to be analyzed in descending order according to the weighted scores, to obtain a root cause list of the data to be traced.

[0015] According to another aspect of an embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned data tracing method through the computer program.

[0016] According to another aspect of an embodiment of the present invention, a computer program product is provided, comprising computer instructions, which implement the steps of the above-mentioned data tracing method when executed by a processor.

[0017] In the above-mentioned embodiment of the present application, based on a preset knowledge graph, multiple objects to be analyzed that affect the data quality of the data to be traced can be determined from multiple preset dimensions, thereby achieving the purpose of data tracing based on multiple dimensions. Each object to be analyzed is evaluated based on multiple preset dimensions to obtain a weighted score for each object to be analyzed. The high or low weighted score can indicate the probability that the object to be analyzed is the root cause of the data to be traced, thereby achieving the technical effect of accurately determining the root cause affecting the data quality, thereby solving the technical problem of inaccurate root causes obtained by existing traceability technologies for tracing data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1This is a flow chart of a data tracing method according to an embodiment of the present invention;

[0020] Figure 2 2 is a schematic diagram of the architecture of a data quality tracing system using a four-dimensional traceability model based on a knowledge graph according to an embodiment of the present invention;

[0021] Figure 3 Schematic diagram of an execution process of a graph-modified Adtributor root cause engine according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of a data tracing device according to an embodiment of the present invention;

[0023] Figure 5 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0027] A knowledge graph is a semantic network used to represent entities and their relationships. It stores knowledge through a graph structure, where nodes represent entities and edges represent relationships between entities. In this application, it is used to build a traceability model for data quality, leveraging its powerful knowledge representation and reasoning capabilities to analyze data quality-related information.

[0028] Four-dimensional traceability model: A traceability model built for data quality that includes four dimensions: time, space, layer, and data distribution.

[0029] Time dimension: records the time points and time periods when data quality-related data is generated and changed, such as the creation time and modification time of the data.

[0030] Spatial dimension: involves the storage location information of data related to data quality, such as which server the data is stored in, and where in which data center.

[0031] Layer dimension: reflects the position and relationship of data in different hierarchical structures, such as data collection layer, data processing layer, data storage layer, etc., and is used to analyze the manifestation of data quality issues at different processing stages.

[0032] Data distribution dimension: Focus on the flow and distribution of data in the system, including information such as the input source, output destination, transmission path, and traffic volume of the data, in order to track the causes of data quality problems during the data flow process.

[0033] The adtributor algorithm, developed by Microsoft Research, can be applied in a variety of scenarios, including monitoring system operations and maintenance, advertising anomaly detection, and network traffic monitoring and anomaly analysis. In this system, this algorithm is used for root cause analysis. It uses the ARMA model for prediction and generates alerts when the predicted and actual results deviate by a certain percentage. Explanatory power and surprise are used to quantify the definition of root causes.

[0034] According to an embodiment of the present invention, an embodiment of a data traceability method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] Figure 1 is a flow chart of a data tracing method according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0036] Step S102: monitoring the data quality of the database to obtain data to be traced indicating that the data quality is abnormal;

[0037] Step S104: Query the data to be traced across multiple preset dimensions using a preset knowledge graph to obtain multiple objects to be analyzed, wherein the nodes in the preset knowledge graph are determined based on entities that affect the data quality of the database, and the node attributes of the preset knowledge graph include: entities corresponding to dimension attributes in multiple preset dimensions, and edges in the preset knowledge graph are determined based on associations between entities, and the entities include at least the data to be traced, and each object to be analyzed is an entity that has an association relationship with the data to be traced in the preset dimension;

[0038] Step S106: Determine a weighted score corresponding to each object to be analyzed based on multiple preset dimensions, where the weighted score is a comprehensive score obtained for the same object to be analyzed based on multiple preset dimensions, weighted by the dimensional weights corresponding to each preset dimension.

[0039] Step S108 : sorting the multiple objects to be analyzed in descending order according to the weighted scores to obtain a root cause list of the data to be traced.

[0040] In the above-mentioned embodiment of the present application, based on a preset knowledge graph, multiple objects to be analyzed that affect the data quality of the data to be traced can be determined from multiple preset dimensions, thereby achieving the purpose of data tracing based on multiple dimensions. Each object to be analyzed is evaluated based on multiple preset dimensions to obtain a weighted score for each object to be analyzed. The high or low weighted score can indicate the probability that the object to be analyzed is the root cause of the data to be traced, thereby achieving the technical effect of accurately determining the root cause affecting the data quality, thereby solving the technical problem of inaccurate root causes obtained by existing traceability technologies for tracing data quality.

[0041] In the above step S102, the database performs data management through the data storage system, the scheduling system and the monitoring system, and further the data quality monitoring of the database can be realized by connecting the data storage system, the scheduling system and the monitoring system of the database.

[0042] In the above step S102 , the data to be traced may be quality data with abnormal data quality, or an event indicating that the data quality is abnormal.

[0043] In the above step S104, the preset knowledge graph represents the entities that affect the data quality through nodes. The entity can be determined based on data such as users, fields, tables, databases (including schemas), data sources, data tasks, etc., or it can be determined based on the storage location of the data, or it can be determined based on the triggering event of data traceability. The triggering event can be the data to be traced that indicates that the quality of the data library is abnormal.

[0044] In the above step S104, the attributes of each entity in multiple preset dimensions can be represented by node attributes. For example, field nodes, table nodes, and data task nodes have node attributes in four dimensions: creation time, space (such as container, server, cluster), layer (such as ODS\EDW\DWD\AWS, etc., which can be customized according to business conditions), and distribution (flow direction, traffic).

[0045] In the above step S104, the preset knowledge graph represents the node relationship between nodes through edges, and the node relationship can be determined based on the association relationship between the entities represented by the nodes.

[0046] In the above step S104, the association relationship between entities includes at least: data flow relationship, such as information that can be used to describe the input, output, transmission path and traffic size of data between different entities; subordinate relationship, such as information that can be used to describe that a data field belongs to a data table; dependency relationship, such as information that can be used to describe that a task depends on a table; management relationship, such as information that can be used to describe that a user operates a data task online; time sequence relationship, such as information that can be used to describe that the creation time of a data table is earlier than the update time; spatial location relationship, such as information that can be used to describe that a storage location is associated with a server; event causal relationship, such as information that can be used to describe that a verification failure event causes data re-verification, etc.

[0047] In the above step S104, the preset dimensions include at least: a time dimension, a space dimension, a layer dimension, and a data distribution dimension.

[0048] Optionally, the above step S104 can be implemented by using a four-dimensional indexer to perform queries in multiple preset dimensions.

[0049] Optionally, the four-dimensional indexer includes at least: a time indexer, a space indexer, a hierarchical indexer, a distribution indexer, and a cross-indexer; wherein, the time indexer can index entities within a certain time period; the space indexer can index entities within a specified space; the hierarchical indexer can index entities within the same level; the distribution indexer can index all entities on the same relationship (such as a dependency link); and the cross-indexer can index according to any combination of the above four preset dimensions.

[0050] In the above step S106, for each object to be analyzed, it can be evaluated separately from multiple preset dimensions to obtain a comprehensive score corresponding to each preset dimension, and then the multiple comprehensive scores corresponding to the multiple preset dimensions are weightedly calculated with the dimension weights preset for each preset dimension to obtain a weighted score based on multiple preset dimensions.

[0051] In the above step S108, after determining the weighted scores corresponding to the multiple objects to be analyzed, the multiple objects to be analyzed can be sorted in descending order of the weighted scores. The higher the ranking of the object to be analyzed, the greater the probability that the object to be analyzed is the fundamental factor (i.e., the root cause) of the data to be traced.

[0052] As an optional embodiment, monitoring the data quality of a database and obtaining data to be traced indicating that the data quality is abnormal includes: collecting quality data for evaluating the data quality; performing quality evaluation on the quality data to obtain an evaluation result, wherein the evaluation result is used to indicate whether the quality data has an abnormal situation; and determining the data to be traced based on the quality data showing the abnormal situation.

[0053] In the above embodiments of the present application, the data to be traced is an object that needs to be traced when the data quality of the database is abnormal. Therefore, when determining the data to be traced, quality data for evaluating the data quality of the database can be collected, and the quality data can be evaluated to determine whether the database has abnormal data quality. If an abnormality occurs, the data to be traced is determined based on the quality data of the abnormality, thereby realizing the determination of the data to be traced.

[0054] Optionally, quality data includes at least: scheduling information collection, personnel entity information collection, operation log collection, technical metadata collection, operating system information collection, and data storage indicator collection

[0055] Optionally, quality data can be collected through the database's data storage system, scheduling system and monitoring system, wherein, based on the data storage system, information such as the creation time, modification time, storage location, etc. of the data is obtained from the database's log files; based on the monitoring system, information is obtained about which user has operated on the data and the type of operation (addition, deletion, modification, etc.); based on the scheduling system, event information such as data verification pass or fail is obtained from the event record, and data characteristic values ​​such as average value, maximum value, enumeration number, etc. are recorded at the same time.

[0056] Optionally, the quality data may be subjected to a quality assessment, and the quality data may be compared with a preset assessment indicator corresponding to the quality data. If the quality data does not meet the preset assessment indicator, the assessment result may be determined to be that the quality data is abnormal.

[0057] As an optional embodiment, before querying the traceability data in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed, the method also includes: collecting quality data for evaluating data quality; based on the quality data, determining the entities that affect the data quality of the database and the association relationships between entities; and constructing a preset knowledge graph based on multiple entities and association relationships.

[0058] In the above embodiments of this application, the relationships between entities can be data flows, data dependencies, data processing flows, etc. By constructing a knowledge graph, the relationships between entities and the attributes of entities in different dimensions can be intuitively displayed, thereby providing strong support for data quality monitoring and abnormal data tracing. By constructing a knowledge graph, this technical solution realizes the visual management of entities and their relationships in the database, solves the problem of complex and difficult to clarify entity relationships in traditional data governance, and improves the efficiency and accuracy of data governance.

[0059] As an optional embodiment, querying the data to be traced in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed includes: calling the dimension indexer corresponding to each preset dimension, wherein the dimension indexer is used to query based on the dimension attributes corresponding to the preset dimension; using the dimension indexer, in the preset knowledge graph, querying the node that has an association relationship with the data to be traced as the target node; and determining the entity indicated by the target node as the object to be analyzed.

[0060] In the above-mentioned embodiments of the present application, the dimension indexer can be an indexer based on the time dimension, an indexer based on the business dimension, etc. By calling the dimension indexer, entities related to the data to be traced can be quickly located, thereby improving the efficiency and accuracy of data traceability. This technical solution, through the use of the dimension indexer, realizes rapid query of the knowledge graph, solves the problems of low query efficiency and difficulty in quickly locating the root cause of problems in traditional data traceability, and improves the efficiency and accuracy of data traceability.

[0061] As an optional embodiment, based on multiple preset dimensions, determining the weighted score corresponding to each object to be analyzed includes: calculating the comprehensive score corresponding to each preset dimension based on the explanatory power and surprise of each object to be analyzed in the same preset dimension, wherein the explanatory power measures the ability of the preset dimension to explain the abnormal situation by comparing the difference between the observed value of the abnormal situation and the expected value of the normal situation, and the surprise is used to indicate the degree of deviation of the abnormal situation of the preset dimension from the normal situation, and the comprehensive score is the weighted result of the explanatory power and surprise of the same preset dimension; according to the dimension weights corresponding to the multiple preset dimensions, the comprehensive scores corresponding to the same object to be analyzed in the multiple preset dimensions are weighted and summed to obtain the weighted score corresponding to each object to be analyzed.

[0062] In the above-mentioned embodiments of the present application, explanatory power can be calculated by comparing the difference between the value of the abnormal data at a specific time point and the average value of the normal data at that time point, and surprisingness can be measured by calculating the ratio between the value of the abnormal data at a specific time point and the standard deviation of the normal data at that time point. This technical solution achieves a comprehensive evaluation of the root causes of data anomalies through a weighted scoring mechanism, solving the problem of incomplete and inaccurate single-dimensional evaluation in traditional data traceability, and improving the accuracy and comprehensiveness of data traceability.

[0063] Optionally, the formula for calculating the release force is:

[0064]

[0065] Among them, EP(d) is the explanatory power, Ai is the observed value of the i-th feature under abnormal conditions, Ei is the expected value of the i-th feature under normal conditions, and Wi is the weight of the edge corresponding to the i-th feature in the knowledge graph.

[0066] Alternatively, the formula for calculating surprise is:

[0067] S(d) = |Ad-μd| / σd, where S(d) is the surprise, Ad is the observed value of dimension d under abnormal conditions, μd is the mean of dimension d under normal conditions, and σd is the standard deviation of dimension d under normal conditions.

[0068] Optionally, the comprehensive score is calculated as:

[0069] Score(d) = α·EP(d) + (1-α)·S(d), where Score(d) is the comprehensive score and α is the weight coefficient, which is usually 0.5 and can be dynamically adjusted according to the configuration.

[0070] Optionally, the calculation formula for the weighted score of multi-dimensional feature fusion is:

[0071] WeightedScore(d)=∑wi·Score(di), where WeightedScore(d) is the weighted score of multi-dimensional feature fusion, and wi is the dimension weight of each preset dimension, which can be configured and adjusted.

[0072] As an optional embodiment, based on the explanatory power and surprise of each object to be analyzed in the same preset dimension, calculating the comprehensive score corresponding to each preset dimension includes: calculating the explanatory power corresponding to each preset dimension based on the observed value of the abnormal situation, the expected value of the normal situation, and the edge weight of each object to be analyzed in the same preset dimension, wherein the edge weight is determined in advance according to the degree of correlation between the preset dimension and the abnormal situation; calculating the surprise corresponding to each preset dimension based on the observed value of the abnormal situation, and the mean and standard deviation of the normal situation of each object to be analyzed in the same preset dimension; assigning a first weight to the explanatory power and a second weight to the surprise, and performing weighted summation on the explanatory power and the surprise of the same preset dimension to obtain the comprehensive score corresponding to each preset dimension, wherein, under the same preset dimension, the sum of the first weight and the second weight is 1.

[0073] In the above-mentioned embodiments of this application, edge weights can be pre-set based on the strength and frequency of associations between entities. The weights for explanatory power and surprise can be adjusted based on specific application scenarios and requirements to achieve optimal traceability. This technical solution achieves flexible evaluation of the root causes of data anomalies by dynamically adjusting edge weights and the weights of the comprehensive score. This solves the problem of fixed weights and poor adaptability in traditional data traceability, and improves the flexibility and adaptability of data traceability.

[0074] As an optional embodiment, before the observed values ​​of abnormal conditions, expected values ​​of normal conditions, and edge weights of each object to be analyzed in the same preset dimension, the method also includes: identifying the target abnormality type to which the object to be analyzed belongs among multiple preset abnormality types, wherein each preset abnormality type is pre-configured with a corresponding edge weight calculation method; and determining the edge weight according to the edge weight calculation method corresponding to the target abnormality type.

[0075] In the above-mentioned embodiments of the present application, the edge weight calculation method can be determined according to the target anomaly type to be analyzed. By identifying the target anomaly type to which the object to be analyzed belongs, a suitable edge weight calculation method can be selected based on the target anomaly type to determine the edge weight, so that the calculated edge weight is more accurate. In addition, the edge weight in the preset knowledge graph is an important component of the explanatory power. Through more accurate edge weights, the explanatory power can be better adjusted, and the modeling ability of complex relationships can be enhanced.

[0076] The present invention also provides an optional embodiment, which provides a data quality tracing method of a four-dimensional traceability model based on a knowledge graph. By integrating indicators of four dimensions of time, space, layer and data distribution, and then utilizing the powerful knowledge representation and reasoning capabilities of the knowledge graph, it achieves comprehensive tracing of data quality and improves the accuracy and comprehensiveness of data traceability; and provides an interface for tracing the source of data quality problems, which facilitates users to perform data quality tracing operations, and provides effective data quality assessment and improvement suggestions to help users improve data quality.

[0077] Optionally, a system that applies a four-dimensional traceability model data quality tracing method based on a knowledge graph is a four-dimensional traceability model data quality tracing system based on a knowledge graph.

[0078] Figure 2 is a schematic diagram of the architecture of a data quality tracing system using a four-dimensional traceability model based on a knowledge graph according to an embodiment of the present invention. Figure 2 As shown, it at least includes: data access layer, graph construction layer, analysis service layer and application layer.

[0079] Alternatively, as Figure 2 As shown, the data access layer is used to collect scheduling information, personnel entity information, operation logs, technical metadata, operating system information, and data storage indicators for the scheduling system, monitoring system, and data storage system.

[0080] As an optional embodiment, data quality related data can be collected from data sources of various data quality related systems (such as data storage system, scheduling system, monitoring system) through the data access layer.

[0081] For example, obtain data creation time, modification time, storage location and other information from the database log file; obtain which user performed the data operation and the type of operation (addition, deletion, modification, etc.) from the monitoring system; obtain event information such as data verification pass or failure from the event record of the scheduling system, and record data characteristic values ​​such as average value, maximum value, enumeration number, etc.

[0082] Optionally, for the layer dimension, the state changes of data at different levels can be recorded, such as the data conversion from the data acquisition layer to the data processing layer; for the data distribution dimension, the input source address, output target address, node information in the transmission path (i.e., data flow relationship) and data flow size of the data can be recorded.

[0083] Alternatively, as Figure 2 As shown, the graph construction layer includes at least: an entity node construction module, an indicator builder, a relationship builder, a data monitoring module and a quality assessment module.

[0084] As an optional embodiment, by constructing a graph through a graph construction layer, the collected data can be preprocessed to extract entities and relationships.

[0085] As an alternative example, Figure 2 As shown, the data monitoring module at least includes: a database feature extraction module and a data distribution monitoring module, which are used to extract data features and data distribution features.

[0086] Optionally, the data feature extraction module can periodically calculate and record data feature values, such as average value, maximum value, enumeration number, etc., through a big data computing engine (spark / flink, etc.).

[0087] Optionally, the data distribution monitoring module can record the data input table, output table, intermediate result table, data flow size, etc.

[0088] As an alternative example, Figure 2 As shown, the quality assessment module can perform quality assessment on the collected data based on quality configuration and quality tasks.

[0089] For example, check whether the table structure is complete (completeness assessment), whether the data complies with pre-set format specifications (accuracy assessment), whether the data from different data sources are consistent (consistency assessment), whether the data is updated in a timely manner (timeliness assessment), etc.

[0090] Optionally, based on the assessment results, it can be determined whether data traceability is necessary. If there are quality issues with the data table (such as completeness, accuracy, consistency, or timeliness), the traceability process is triggered and the focus of traceability is determined.

[0091] For example, if a data accuracy issue is found, the traceability focus may be on the data generation and modification links; if it is a consistency issue, the traceability focus may be on the data integration and transmission links.

[0092] Optionally, quality configuration is used to configure data quality rules and perform quality assessment on the collected data, including completeness assessment (checking whether the table structure is complete), accuracy assessment (whether the data conforms to pre-set format specifications), consistency assessment (whether the data from different data sources are consistent), timeliness assessment (whether the data is updated in a timely manner), etc.

[0093] Optionally, a quality task is used to schedule quality tasks in real time or periodically, and uses a calculation engine to calculate corresponding data quality indicator values ​​according to rules.

[0094] As an alternative example, Figure 2As shown, the entity node construction module is used to use entity objects that affect data quality as node objects of the graph, and use knowledge graph construction tools (such as Neo4j, NebulaGraph, etc.) to build a knowledge graph based on entities and relationships.

[0095] Alternatively, as Figure 2 As shown in the figure, in the four-dimensional traceability model of data quality, the nodes include: user nodes, field nodes, table nodes, database nodes (including schema), data source nodes, and data task nodes. Among them, field nodes, table nodes, and data task nodes have attributes of four dimensions: creation time, space (such as container, server, cluster), layer (such as ODS\EDW\DWD\AWS, etc., which can be customized according to business conditions), and distribution (flow direction, traffic); edges represent the data flow relationship between these nodes (such as describing the input, output, transmission path and traffic size of data between different entities), subordinate relationships (such as data fields belong to data tables) or dependent relationships (such as tasks depend on tables), and management relationships (such as users operating data tasks online); time sequence relationships (such as the creation time of the data table is earlier than the update time), spatial location relationships (such as the storage location is associated with a certain server), and event causal relationships (such as verification failure events leading to data re-verification, etc.).

[0096] Each relationship includes creation time and termination time.

[0097] For example, taking a data table as an entity, its related time information (such as creation time, update time), spatial location information, and hierarchy information as attributes, and the storage location as another entity, the relationship between the data table and the storage location can be "a data table is stored in a certain location."

[0098] For example, a data field is regarded as an entity and its data table is regarded as another entity. The relationship between the data field and its data table can be "a certain data field belongs to a certain data table".

[0099] For example, taking user as one entity and operation type as another entity, the relationship between user and operation type can be "a certain user performed a certain operation."

[0100] For example, taking the data validation task as one entity and the related user or system as another entity, the relationship between the data validation task and the related user or system can be "a user or system performs a data validation task."

[0101] Optionally, data lineage relationships can be imported through external systems or interfaces, or data lineage information can be obtained using message middleware based on the agreed data structure to connect related entities.

[0102] For example, in a knowledge graph, a data table node is connected to related data field nodes through a data lineage relationship, indicating that the data field comes from a specific data table.

[0103] As an alternative example, Figure 2 As shown, the indicator builder includes at least: an indicator generator and an event rule configuration module, which can generate indicators for tracing based on event triggering logic and according to the rules provided by the rule engine.

[0104] Optionally, the event rule configuration module is used to construct composite indicators for traceability based on event triggering logic using the basic indicators produced by the data monitoring module and the quality assessment module. The basic indicators, influencing factors, and calculation formulas can be configured, and are mainly used to configure the rules for indicator calculation.

[0105] Optionally, an indicator generator is used to use a graph computing engine (such as GraphScoop, GraphX, Flink-Gelly, etc.) according to the rules provided by the rule engine to access or periodically scan event indicators occurring on entities (such as sudden changes in average values, sudden changes in traffic, surges in null value rates, slow release of storage space, etc.) in real time and generate corresponding event entities. Events are written into the graph as a special entity for extracting graph entities and tracing calculations.

[0106] As an alternative example, Figure 2 As shown, the relationship builder is used to build entity-entity relationships and entity-event relationships. The entity-entity relationship is used to build relationships between different entities, such as the relationship between fields, tables, tasks, personnel and other entities; the entity-event relationship is used to build the relationship between triggering events and related entities.

[0107] Optionally, the method of constructing a knowledge graph in the graph construction layer can be implemented by the following pseudo code:

[0108]

[0109]

[0110] Alternatively, as Figure 2 As shown, the analysis service layer includes at least: a four-dimensional construction engine and a graph-improved Adtributor root cause engine. The four-dimensional indexing engine is used to build the context of the four-dimensional traceability model about data quality based on the knowledge graph and provide basic rule-based traceability capabilities; the graph-improved Adtributor root cause engine is used to use the Adtributor algorithm to perform root cause analysis and determine the root causes of data quality problems.

[0111] Alternatively, as Figure 2As shown, the four-dimensional construction engine includes at least: a four-dimensional indexer, a context builder and a joint analyzer.

[0112] As an alternative example, Figure 2 As shown, the four-dimensional indexer is used to build the fast indexing capability of the graph in four different dimensions of quality objects. It can accurately and quickly locate the nodes of entity association relationships with events occurring (indicator changes) in the same dimension. If no event occurs, the node will not be extracted.

[0113] Alternatively, as Figure 2 As shown, the four-dimensional indexer includes at least: a time indexer, a space indexer, a hierarchical indexer, a distribution indexer, and a cross-indexer; among them, the time indexer can index entities within a certain time period; the space indexer can index entities within a specified space; the hierarchical indexer can index entities within the same level; the distribution indexer can index all entities on the same relationship (such as a dependency link); and the cross-indexer can index according to any combination of the four dimensions.

[0114] As an alternative example, Figure 2 As shown, the context builder identifies nodes with abnormal data quality through data quality assessment tasks, and uses a four-dimensional indexer to inspect all related entities, events, and indicators as the context for quality traceability.

[0115] As an alternative example, Figure 2 As shown in the figure, the joint analyzer is a rule-based analyzer. Compared with the more general traceability analyzers or those based on operation and maintenance experience, it can directly derive the causes of quality problems quickly and accurately without using the root cause engine.

[0116] Alternatively, as Figure 2 As shown, the joint analyzer includes at least: an expert rule engine and a custom rule engine.

[0117] In the above embodiment of the present application, a four-dimensional construction engine is used to establish a four-dimensional traceability model for data quality based on the knowledge graph. Taking a modification of data as an example, from the time dimension, the time changes such as the start time and end time of the data modification can be traced back. From the spatial dimension, the changes in the storage location involved in the modification process of the data can be traced back. From the layer dimension, the flow process of data quality-related data in different hierarchical structures can be traced back, such as the changes and processing of data quality-related data in the process from the data acquisition layer to the data processing layer and then to the data storage layer. From the data distribution dimension, information such as the input, output, transmission path and traffic size of data quality-related data can be traced back.

[0118] Alternatively, as Figure 2As shown in the figure, the main steps of the graph-improved Adtributor root cause engine include:

[0119] Explanatory power calculation: Calculate the confidence of causal relationships by combining the edge weights of the knowledge graph.

[0120] Surprise calculation: the deviation between the predicted value and the actual value based on the ARMA model.

[0121] Comprehensive score calculation: Dynamically adjust the weight coefficient according to the anomaly type and calculate the comprehensive score for each dimension.

[0122] Output root cause list: sorted by comprehensive score, providing the most likely root cause.

[0123] Figure 3 Schematic diagram of the execution process of a graph-modified Adtributor root cause engine according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0124] Step S301: data preprocessing.

[0125] Step S302: calculation of explanatory power.

[0126] Step S303: surprise calculation.

[0127] Step S304: Calculate comprehensive score.

[0128] Step S305: multi-dimensional feature fusion.

[0129] Optionally, after the above step S305, step S306 may be further executed to output a root cause list.

[0130] Optionally, in step S301, the multidimensional data indicators of each node (including event nodes) are normalized, and the statistical characteristics (mean, standard deviation, etc.) of the event node in each dimension are calculated. For basic KPIs, the normal S value (i.e., surprise) and EP value (i.e., explanatory power) calculation formulas are directly used. For rate-based indicators (such as success rate and failure rate), it is necessary to combine partial derivatives and finite differences to derive the corresponding rate-based formula.

[0131] In the above step S302, the explanatory power EP(d) of the dimension of each node (including the event node) is calculated, where the explanatory power is the ability of a certain dimension or characteristic value to explain the anomaly, which can indicate to what extent the dimension can explain the occurrence of the anomaly.

[0132] Alternatively, the explanatory power EP(d) is measured by comparing the distribution of the dimension when an anomaly occurs with the distribution under normal circumstances.

[0133] Optionally, the explanatory power is calculated as:

[0134]

[0135] Among them, EP(d) is the explanatory power, Ai is the observed value of the i-th feature under abnormal conditions, Ei is the expected value of the i-th feature under normal conditions, and Wi is the weight of the edge corresponding to the i-th feature in the knowledge graph.

[0136] Optionally, the observation value extraction includes: extracting indicator data of entities corresponding to all nodes (including event nodes) related to the abnormal quality node through a context builder of the four-dimensional construction engine.

[0137] Optionally, the expected value calculation includes: using an ARMA model to predict the characteristic values ​​of these entities under normal circumstances.

[0138] It should be noted that the edge weights in the knowledge graph are regarded as an important component of the explanatory power. The calculation of the explanatory power is adjusted by the edge weights to enhance the modeling ability of complex relationships.

[0139] For example, by counting co-occurrence frequencies and introducing attenuation factors (such as temporal distance attenuation, spatial distance attenuation, and hop count attenuation), we can calculate objective quantities of relationships between entities (including event nodes) (such as the strength of data lineage relationships and traffic dependency). This method is the core approach for tracing data quality issues through graphs. It improves the Adtributor algorithm based on data quality issue tracing scenarios to enhance the logical correctness and explainability of the root cause.

[0140] Optionally, edge weights can be calculated based on statistical co-occurrence frequencies. The applicable scenario is: objective quantification of relationships between entities (such as the strength of data blood relationships). The specific implementation method can be: by calculating the number of associations between two entities in historical events, achieving statistical co-occurrence frequencies, and then calculating edge weights based on the co-occurrence frequencies.

[0141] For example, if the co-occurrence frequency is: the number of associations between data table A and storage node S is 100 times, and the number of associations between data table B and storage node S is 50 times, then the edge weight is: W A→S =100 / (100+50)=0.67, W B→S =0.33.

[0142] Optionally, edge weights can be determined based on data lineage analysis, and the applicable scenario is: causal strength evaluation of data flow paths, which can be implemented by assigning weights based on the hierarchical depth of the data processing link through lineage depth analysis.

[0143] For example, “mermaid;

[0144] graph LR;

[0145] A[original table]-->B[cleaned table];

[0146] B-->C[aggregate table];

[0147] C-->D[Application Table]", where the edge weight rule is that the weight decays by 20% after each layer of processing, then W A→B =1.0,W B→C =0.8,W C→D =0.64.

[0148] Optionally, edge weights can be calculated based on traffic dependency to assign weights based on the proportion of data traffic.

[0149] For example, if 60% of the input traffic to table X comes from table Y and 40% comes from table Z, the edge weight is: W Y→X =0.6,W Z→X =0.4.

[0150] Optionally, edge weights can be determined based on expert experience, which is applicable to the subjective priority setting of key business rules.

[0151] Table 1 is a schematic table of weight distribution according to an embodiment of the present invention. As shown in Table 1, corresponding edge weights are set according to relationship types.

[0152] Table 1

[0153]

[0154] Optionally, edge weights can be dynamic weights based on machine learning models, which are applicable to adaptive weight adjustments in complex systems.

[0155] As an optional example, the implementation process of determining edge weights based on the dynamic weights of a machine learning model includes the following steps:

[0156] Step 1.1, feature engineering, specifically including: time features, spatial features, business features and traffic features.

[0157] Optionally, the time feature is determined based on the time interval of the associated events; the spatial feature is determined based on the physical distance of the storage node; the business feature is determined based on the importance level (P0 / P1 / P2) of the data table; and the traffic feature is determined based on the data transmission rate and throughput.

[0158] Step S1.2, model training, specifically includes: using logistic regression or GNN (graph neural network) to predict edge weights.

[0159] The sample code is:

[0160] "python

[0161] from stellargraph import StellarGraph

[0162] from tensorflow import keras

[0163] #Build graph structure

[0164] graph=StellarGraph(nodes,edges)

[0165] #Define GNN model

[0166] generator=FullBatchNodeGenerator(graph, method="gcn")

[0167] gcn=GCN(layer_sizes=[32,32],generator=generator)

[0168] x_in,x_out=gcn.in_out_tensors()

[0169] predictions=keras.layers.Dense(1,activation="sigmoid")(x_out)

[0170] model=keras.Model(inputs=x_in,outputs=predictions)

[0171] #Train model

[0172] model.compile(optimizer="adam",loss="binary_crossentropy")

[0173] model.fit(train_flow,epochs=50)".

[0174] Step S1.3, weight output, the edge weight value predicted by the model is between 0 and 1, indicating the strength of the relationship.

[0175] Optionally, edge weights can be determined based on a dynamic weight adjustment strategy, which is applicable to weight updates in real-time systems.

[0176] Table 2 is a schematic diagram of a dynamic weight adjustment strategy according to an embodiment of the present invention. As shown in Table 2, different formulas are used to calculate edge weights for different strategies.

[0177] Table 2

[0178]

[0179]

[0180] In the above step S303, the surprise S(d) of each dimension of the event node is calculated, where the surprise is the degree of deviation of the dimensional characteristics of a certain graph node (including the event node) under abnormal conditions compared with the normal conditions.

[0181] Optionally, the surprise is calculated by using a statistical method (such as Z-score) to calculate the degree of deviation of the feature value.

[0182] Alternatively, the formula for calculating surprise is:

[0183] S(d) = |Ad-μd| / σd, where S(d) is the surprise, Ad is the observed value of dimension d under abnormal conditions, μd is the mean of dimension d under normal conditions, and σd is the standard deviation of dimension d under normal conditions.

[0184] In the above step S304, the comprehensive score Score(d) of each dimension of the graph node (including event node) entity is calculated, wherein the comprehensive score of each dimension is calculated by combining the explanatory power and the surprise for ranking and root cause location.

[0185] Optionally, the comprehensive score is calculated as:

[0186] Score(d) = α·EP(d) + (1-α)·S(d), where Score(d) is the comprehensive score and α is the weight coefficient, which is usually 0.5 and can be dynamically adjusted according to the configuration.

[0187] In the above step S305, the weighted score of the multi-dimensional feature fusion can be obtained by calculating the weighted comprehensive score of the four-dimensional (time, space, layer, data distribution) features of each graph node (including event node) entity.

[0188] Optionally, the calculation formula for the weighted score of multi-dimensional feature fusion is:

[0189] WeightedScore(d)=∑wi·Score(di), where WeightedScore(d) is the weighted score of multi-dimensional feature fusion, and wi is the weight of each dimension, which can be configured and adjusted.

[0190] In the above step S306, the event nodes are sorted from high to low according to their weighted scores. If the node is an event node, it is directly output; if it is not an event node, it is sorted according to the weighted scores of its corresponding event nodes. The data entity nodes (fields, tables, databases) have a higher sorting priority than the event nodes.

[0191] As an optional embodiment, the Adtributor algorithm is used to perform root cause analysis to determine the root cause of the data quality problem. For the case of missing e-commerce order data, the analysis process is as follows.

[0192] Step S2.1, anomaly detection.

[0193] Specifically, it was detected that the order table was missing data during the period from 00:00 to 02:00 on 2023-08-01; and the prediction deviation using the ARMA model exceeded the threshold (>20%).

[0194] Step S2.2, calculation of explanatory power.

[0195] Optionally, for the time dimension, it is determined that: the observed value Ai is an 80% drop in data traffic; the expected value Ei is normal traffic; the edge weight Wi is: 0.9 (the time dimension is highly correlated with anomalies); and the explanatory power EP(d) is 0.72.

[0196] Optionally, for the spatial dimension, determine: the observed value Ai is the space utilization rate of storage node Disk3 100%; the expected value Ei is the normal utilization rate 70%; the edge weight Wi: 0.8 (the spatial dimension is moderately correlated with anomalies); the explanatory power EP(d) is 0.64.

[0197] Step S2.3, surprise calculation.

[0198] Optionally, the surprise of the time dimension is calculated to be 2.5 (Z-score>2); the surprise of the space dimension is 3.2 (Z-score>3).

[0199] Step S2.4, calculation of comprehensive score.

[0200] Optionally, the comprehensive score of the time dimension is calculated to be 0.7; the comprehensive score of the space dimension is calculated to be 0.9.

[0201] Step S2.5, weighted scoring of multi-dimensional feature fusion.

[0202] Step S2.6: locating the root cause.

[0203] Optionally, output the root cause: Storage node Disk3 space usage exceeds 90% -> Database write restriction is enabled -> Order table data writing fails.

[0204] Optionally, the pseudo code for implementing root cause analysis using the Adtributor algorithm is:

[0205] "python

[0206] def improved_adtributor(data,anomaly_type,graph_weights):

[0207] #Step 1: Preprocess data

[0208] normalized_data=normalize(data)

[0209] #Step 2:Detect anomaly

[0210] is_anomaly,deviation=detect_anomaly(normalized_data)

[0211]

[0212]

[0213] Alternatively, as Figure 2 As shown in the figure, the application layer includes: traceability query module, quality report module and alarm center; through the application layer, users can receive relevant alarms of quality problems, and can simultaneously input or filter relevant information of the data quality task to be traced, and view the traceability results and evaluation reports.

[0214] Optionally, at the application layer, users can enter information related to the data quality to be traced and view the traceability results and evaluation reports.

[0215] For example, a user can enter the name or ID of a data table to view the four-dimensional traceability results and data quality assessment report of the data table.

[0216] The above-mentioned embodiments of the present application, through the construction and application of knowledge graphs, can use knowledge graphs to perform structured representation and knowledge reasoning on data quality-related data, build a four-dimensional traceability model for data quality, and be able to conduct in-depth analysis of complex data quality-related relationships, thereby improving the accuracy and comprehensiveness of data quality traceability.

[0217] The above-mentioned embodiments of the present application, through the establishment and application of a four-dimensional traceability model, can integrate information in four dimensions of time, space, layer and data distribution relationship, establish a four-dimensional traceability model for data quality, and comprehensively trace the data related to data quality. It can fully understand the source and change process of data related to data quality, and when quality problems occur in the data, it can accurately locate the root cause of the problem.

[0218] The above embodiments of the present application adopt a traceability logic based on rules and root cause analysis, and comprehensively use multiple methods to trace the data quality issues. In particular, the edge weight of the graph is used to improve the Adtributor algorithm for root cause analysis, which can improve the accuracy and comprehensiveness of traceability and enhance the logical correctness and explainability of the root cause.

[0219] According to an embodiment of the present invention, a data tracing device embodiment is also provided. It should be noted that the data tracing device can be used to execute the data tracing method in the embodiment of the present invention, and the data tracing method in the embodiment of the present invention can be executed in the data tracing device.

[0220] Figure 4 is a schematic diagram of a data tracing device according to an embodiment of the present invention. Figure 4 As shown, the device may include: a monitoring module 42, which is used to monitor the data quality of the database and obtain data to be traced indicating that the data quality is abnormal; a query module 44, which is used to query the data to be traced in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed, wherein the nodes in the preset knowledge graph are determined based on entities that affect the data quality of the database, and the node attributes of the preset knowledge graph include: entities corresponding to dimension attributes in multiple preset dimensions, and edges in the preset knowledge graph are determined based on the association relationship between entities. The entities at least include: data to be traced, and each object to be analyzed is an entity that has an association relationship with the data to be traced in the preset dimension; a determination module 46, which is used to determine the weighted score corresponding to each object to be analyzed based on multiple preset dimensions, wherein the weighted score is a comprehensive score obtained for the same object to be analyzed in multiple preset dimensions, and a weighted result of the dimension weight corresponding to each preset dimension; a sorting module 48, which is used to sort the multiple objects to be analyzed in descending order according to the weighted score to obtain a list of root causes of the data to be traced.

[0221] It should be noted that the monitoring module 42 in this embodiment can be used to execute step S102 in the embodiment of the present application, the query module 44 in this embodiment can be used to execute step S104 in the embodiment of the present application, the determination module 46 in this embodiment can be used to execute step S106 in the embodiment of the present application, and the sorting module 48 in this embodiment can be used to execute step S108 in the embodiment of the present application. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0222] In the above-mentioned embodiment of the present application, based on a preset knowledge graph, multiple objects to be analyzed that affect the data quality of the data to be traced can be determined from multiple preset dimensions, thereby achieving the purpose of data tracing based on multiple dimensions. Each object to be analyzed is evaluated based on multiple preset dimensions to obtain a weighted score for each object to be analyzed. The high or low weighted score can indicate the probability that the object to be analyzed is the root cause of the data to be traced, thereby achieving the technical effect of accurately determining the root cause affecting the data quality, thereby solving the technical problem of inaccurate root causes obtained by existing traceability technologies for tracing data quality.

[0223] As an optional embodiment, the monitoring module includes: a first acquisition unit, used to collect quality data for evaluating data quality; an evaluation unit, used to perform quality evaluation on the quality data and obtain an evaluation result, wherein the evaluation result is used to indicate whether there is any abnormality in the quality data; a first determination unit, used to determine the data to be traced based on the quality data with abnormality.

[0224] As an optional embodiment, the device also includes: a second collection unit, which is used to collect quality data for evaluating data quality before querying the traceability data in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed; a second determination unit, which is used to determine the entities and the association relationships between entities that affect the data quality of the database based on the quality data; and a component unit, which is used to construct a preset knowledge graph based on multiple entities and association relationships.

[0225] As an optional embodiment, the query module includes: a calling unit, used to call the dimension indexer corresponding to each preset dimension, wherein the dimension indexer is used to query based on the dimension attributes corresponding to the preset dimension; a query unit, used to use the dimension indexer to query the node that has an association relationship with the data to be traced in the preset knowledge graph as the target node; a third determination unit, used to determine the entity indicated by the target node as the object to be analyzed.

[0226] As an optional embodiment, the determination module includes: a first calculation unit, which is used to calculate the comprehensive score corresponding to each preset dimension based on the explanatory power and surprise of each object to be analyzed in the same preset dimension, wherein the explanatory power measures the explanatory ability of the preset dimension for the abnormal situation by comparing the difference between the observed value of the abnormal situation and the expected value of the normal situation, and the surprise is used to indicate the degree of deviation of the abnormal situation of the preset dimension from the normal situation. The comprehensive score is the weighted result of the explanatory power and surprise of the same preset dimension; a second calculation unit, which is used to perform a weighted sum operation on the comprehensive scores corresponding to the same object to be analyzed in multiple preset dimensions according to the dimension weights corresponding to the multiple preset dimensions, to obtain the weighted score corresponding to each object to be analyzed.

[0227] As an optional embodiment, the first calculation unit includes: a first calculation subunit, used to calculate the explanatory power corresponding to each preset dimension based on the observed value of the abnormal situation, the expected value of the normal situation, and the edge weight of each object to be analyzed in the same preset dimension, wherein the edge weight is pre-determined according to the degree of correlation between the preset dimension and the abnormal situation; a second calculation subunit, used to calculate the surprise corresponding to each preset dimension based on the observed value of the abnormal situation, and the mean and standard deviation of the normal situation of each object to be analyzed in the same preset dimension; a third calculation subunit, used to assign a first weight to the explanatory power, assign a second weight to the surprise, and perform weighted summation of the explanatory power and surprise of the same preset dimension to obtain a comprehensive score corresponding to each preset dimension, wherein, under the same preset dimension, the sum of the first weight and the second weight is 1.

[0228] As an optional embodiment, the device also includes: an identification subunit, which is used to identify the target abnormality type to which the object to be analyzed belongs among multiple preset abnormality types based on the observed value of the abnormal situation of each object to be analyzed in the same preset dimension, the expected value of the normal situation, and the edge weight, wherein each preset abnormality type is pre-configured with a corresponding edge weight calculation method; a determination subunit, which is used to determine the edge weight according to the edge weight calculation method corresponding to the target abnormality type.

[0229] An embodiment of the present invention may provide an electronic device, which may be a computer terminal, and the computer terminal may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0230] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0231] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the data tracing method: monitoring the data quality of the database to obtain data to be traced indicating that the data quality is abnormal; querying the data to be traced in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed, wherein the nodes in the preset knowledge graph are determined based on entities that affect the data quality of the database, and the node attributes of the preset knowledge graph include: entities correspond to dimension attributes respectively in multiple preset dimensions, and the edges in the preset knowledge graph are determined based on the association relationship between entities, and the entities at least include: data to be traced, and each object to be analyzed is an entity that has an association relationship with the data to be traced in the preset dimension; based on multiple preset dimensions, determining the weighted score corresponding to each object to be analyzed, wherein the weighted score is a comprehensive score obtained for the same object to be analyzed in multiple preset dimensions, and a weighted result of the dimension weight corresponding to each preset dimension; sorting the multiple objects to be analyzed in descending order according to the weighted score to obtain a root cause list of the data to be traced.

[0232] Figure 5 is a structural block diagram of a computer terminal according to an embodiment of the present invention, such as Figure 5 As shown, the computer terminal 50 may include: one or more (only one is shown in the figure) processors 52 and a memory 54.

[0233] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data tracing method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned data tracing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the terminal 50 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0234] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: monitor the data quality of the database to obtain data to be traced indicating that the data quality is abnormal; query the data to be traced in multiple preset dimensions through the preset knowledge graph to obtain multiple objects to be analyzed, wherein the nodes in the preset knowledge graph are determined based on the entities that affect the data quality of the database, and the node attributes of the preset knowledge graph include: the entities correspond to dimension attributes respectively in multiple preset dimensions, and the edges in the preset knowledge graph are determined based on the association relationship between entities, and the entities include at least: the data to be traced, and each object to be analyzed is an entity that has an association relationship with the data to be traced in the preset dimension; based on multiple preset dimensions, determine the weighted score corresponding to each object to be analyzed, wherein the weighted score is a comprehensive score obtained for the same object to be analyzed in multiple preset dimensions, and a weighted result of the dimension weight corresponding to each preset dimension; sort the multiple objects to be analyzed in descending order according to the weighted score to obtain a list of root causes of the data to be traced.

[0235] Optionally, the processor may also execute the program code of the following steps: collecting quality data for evaluating data quality; performing quality evaluation on the quality data to obtain an evaluation result, wherein the evaluation result is used to indicate whether the quality data has any abnormality; and determining the data to be traced based on the quality data having any abnormality.

[0236] Optionally, the processor may also execute the program code for the following steps: collecting quality data for evaluating data quality; determining entities and relationships between entities that affect the data quality of the database based on the quality data; and constructing a preset knowledge graph based on multiple entities and relationships.

[0237] Optionally, the processor may also execute the program code of the following steps: calling the dimension indexer corresponding to each preset dimension, wherein the dimension indexer is used to query based on the dimension attributes corresponding to the preset dimension; using the dimension indexer, in the preset knowledge graph, querying the node that has an association relationship with the data to be traced as the target node; and determining the entity indicated by the target node as the object to be analyzed.

[0238] Optionally, the processor may also execute the program code of the following steps: based on the explanatory power and surprise of each object to be analyzed in the same preset dimension, calculate the comprehensive score corresponding to each preset dimension, wherein the explanatory power measures the explanatory ability of the preset dimension for abnormal situations by comparing the difference between the observed value of the abnormal situation and the expected value of the normal situation, and the surprise is used to indicate the degree of deviation of the abnormal situation of the preset dimension from the normal situation, and the comprehensive score is the weighted result of the explanatory power and surprise of the same preset dimension; according to the dimension weights corresponding to the multiple preset dimensions, the comprehensive scores corresponding to the same object to be analyzed in the multiple preset dimensions are weighted and summed to obtain the weighted score corresponding to each object to be analyzed.

[0239] Optionally, the processor may also execute the program code of the following steps: calculating the explanatory power corresponding to each preset dimension based on the observed value of the abnormal situation, the expected value of the normal situation, and the edge weight of each object to be analyzed in the same preset dimension, wherein the edge weight is predetermined according to the degree of correlation between the preset dimension and the abnormal situation; calculating the surprise corresponding to each preset dimension based on the observed value of the abnormal situation, and the mean and standard deviation of the normal situation of each object to be analyzed in the same preset dimension; assigning a first weight to the explanatory power and a second weight to the surprise, and performing weighted summation on the explanatory power and the surprise of the same preset dimension to obtain a comprehensive score corresponding to each preset dimension, wherein, under the same preset dimension, the sum of the first weight and the second weight is 1.

[0240] Optionally, the processor may also execute the program code of the following steps: identifying the target abnormality type to which the object to be analyzed belongs among multiple preset abnormality types, wherein each preset abnormality type is pre-configured with a corresponding edge weight calculation method; and determining the edge weight according to the edge weight calculation method corresponding to the target abnormality type.

[0241] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 5 It does not limit the structure of the above electronic device. For example, the computer terminal 50 may also include Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 5 Different configurations shown.

[0242] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a computer program. The computer program can be stored in a non-volatile medium. The non-volatile storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0243] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the data tracing method provided in the embodiment.

[0244] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0245] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: monitoring the data quality of the database to obtain data to be traced indicating that the data quality is abnormal; querying the data to be traced in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed, wherein the nodes in the preset knowledge graph are determined based on entities that affect the data quality of the database, and the node attributes of the preset knowledge graph include: entities corresponding to dimension attributes in multiple preset dimensions, and edges in the preset knowledge graph are determined based on the association relationship between entities, and the entities include at least: data to be traced, and each object to be analyzed is an entity that has an association relationship with the data to be traced in the preset dimension; based on multiple preset dimensions, determining the weighted score corresponding to each object to be analyzed, wherein the weighted score is a comprehensive score obtained for the same object to be analyzed in multiple preset dimensions, and a weighted result of the dimension weight corresponding to each preset dimension; sorting the multiple objects to be analyzed in descending order according to the weighted score to obtain a root cause list of the data to be traced.

[0246] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: collecting quality data for evaluating data quality; performing quality evaluation on the quality data to obtain an evaluation result, wherein the evaluation result is used to indicate whether the quality data has an abnormality; and determining the data to be traced based on the quality data having an abnormality.

[0247] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: collecting quality data for evaluating data quality; determining entities and relationships between entities that affect the data quality of the database based on the quality data; and constructing a preset knowledge graph based on multiple entities and relationships.

[0248] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: calling the dimension indexer corresponding to each preset dimension, wherein the dimension indexer is used to query based on the dimension attributes corresponding to the preset dimension; using the dimension indexer, in the preset knowledge graph, querying the node that has an association relationship with the data to be traced as the target node; and determining the entity indicated by the target node as the object to be analyzed.

[0249] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: based on the explanatory power and surprise of each object to be analyzed in the same preset dimension, calculating the comprehensive score corresponding to each preset dimension, wherein the explanatory power measures the ability of the preset dimension to explain the abnormal situation by comparing the difference between the observed value of the abnormal situation and the expected value of the normal situation, and the surprise is used to indicate the degree of deviation of the abnormal situation of the preset dimension from the normal situation, and the comprehensive score is the weighted result of the explanatory power and surprise of the same preset dimension; according to the dimension weights corresponding to the multiple preset dimensions, the comprehensive scores corresponding to the same object to be analyzed in the multiple preset dimensions are weighted and summed to obtain the weighted score corresponding to each object to be analyzed.

[0250] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: calculating the explanatory power corresponding to each preset dimension based on the observed value of the abnormal situation, the expected value of the normal situation, and the edge weight of each object to be analyzed in the same preset dimension, wherein the edge weight is predetermined according to the degree of correlation between the preset dimension and the abnormal situation; calculating the surprise corresponding to each preset dimension based on the observed value of the abnormal situation, and the mean and standard deviation of the normal situation of each object to be analyzed in the same preset dimension; assigning a first weight to the explanatory power and a second weight to the surprise, and performing weighted summation on the explanatory power and the surprise of the same preset dimension to obtain a comprehensive score corresponding to each preset dimension, wherein, under the same preset dimension, the sum of the first weight and the second weight is 1.

[0251] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: identifying a target abnormality type to which the object to be analyzed belongs among multiple preset abnormality types, wherein each preset abnormality type is pre-configured with a corresponding edge weight calculation method; and determining the edge weight according to the edge weight calculation method corresponding to the target abnormality type.

[0252] An embodiment of the present invention further provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the steps of the data tracing method provided in the above embodiment are implemented.

[0253] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0254] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0255] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0256] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0257] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0258] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a non-volatile storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0259] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data tracing method, characterized in that: include: Monitor the data quality of the database and obtain data to be traced that indicates abnormal data quality; A plurality of objects to be analyzed are obtained by querying the data to be traced in a plurality of preset dimensions through a preset knowledge graph, wherein the nodes in the preset knowledge graph are determined based on entities that affect the data quality of the database, and the node attributes of the preset knowledge graph include: the entity corresponding to dimensional attributes in the plurality of preset dimensions, and the edges in the preset knowledge graph are determined based on the association relationship between the entities, and the entities include at least: the data to be traced, and each of the objects to be analyzed is the entity that has the association relationship with the data to be traced in the preset dimension; Determine a weighted score corresponding to each of the objects to be analyzed based on the multiple preset dimensions, wherein the weighted score is a weighted result of the comprehensive score obtained for the same object to be analyzed in the multiple preset dimensions and the dimensional weight corresponding to each preset dimension; The plurality of objects to be analyzed are sorted in descending order according to the weighted scores to obtain a root cause list of the data to be traced.

2. The method according to claim 1, characterized in that Monitor the data quality of the database and obtain data to be traced indicating abnormal data quality, including: Collecting quality data for evaluating the quality of the data; Performing a quality assessment on the quality data to obtain an assessment result, wherein the assessment result is used to indicate whether the quality data has any abnormality; The data to be traced is determined based on the quality data in which the abnormal situation occurs.

3. The method according to claim 1, characterized in that Before querying the data to be traced in multiple preset dimensions using a preset knowledge graph to obtain multiple objects to be analyzed, the method further includes: Collecting quality data for evaluating the quality of the data; determining, based on the quality data, the entities that affect the data quality of the database and the association relationships between the entities; The preset knowledge graph is constructed based on the multiple entities and the association relationships.

4. The method according to claim 1, wherein By querying the data to be traced in multiple preset dimensions through the preset knowledge graph, multiple objects to be analyzed are obtained, including: Calling a dimension indexer corresponding to each of the preset dimensions, wherein the dimension indexer is used to query according to the dimension attribute corresponding to the preset dimension; Using the dimension indexer, in the preset knowledge graph, searching for a node that has the association relationship with the data to be traced as a target node; The entity indicated by the target node is determined as the object to be analyzed.

5. The method according to claim 1, characterized in that Determining a weighted score corresponding to each of the objects to be analyzed based on the multiple preset dimensions includes: Based on the explanatory power and surprise of each object to be analyzed in the same preset dimension, a comprehensive score corresponding to each preset dimension is calculated, wherein the explanatory power measures the ability of the preset dimension to explain the abnormal situation by comparing the difference between the observed value of the abnormal situation and the expected value of the normal situation. The surprise is used to indicate the degree of deviation of the abnormal situation from the normal situation in the preset dimension. The comprehensive score is the weighted result of the explanatory power and surprise of the same preset dimension; According to the dimension weights corresponding to the multiple preset dimensions, the comprehensive scores corresponding to the multiple preset dimensions of the same object to be analyzed are weighted and summed to obtain the weighted score corresponding to each object to be analyzed.

6. The method according to claim 5, characterized in that Calculating the comprehensive score corresponding to each preset dimension based on the explanatory power and surprise of each object to be analyzed in the same preset dimension includes: Calculate the explanatory power corresponding to each preset dimension based on the observed value of the abnormal situation, the expected value of the normal situation, and the edge weight of each object to be analyzed in the same preset dimension, wherein the edge weight is pre-determined according to the degree of association between the preset dimension and the abnormal situation; Calculating the surprise corresponding to each preset dimension based on the observed value of the abnormal situation of each object to be analyzed in the same preset dimension, as well as the mean and standard deviation of the normal situation; A first weight is assigned to the explanatory power, a second weight is assigned to the surprise, and the explanatory power and the surprise of the same preset dimension are weighted and summed to obtain the comprehensive score corresponding to each preset dimension, wherein, under the same preset dimension, the sum of the first weight and the second weight is 1.

7. The method according to claim 6, characterized in that Before analyzing the observed value of the abnormal situation, the expected value of the normal situation, and the edge weight of each object to be analyzed in the same preset dimension, the method further includes: Identify a target abnormality type to which the object to be analyzed belongs among a plurality of preset abnormality types, wherein each of the preset abnormality types is pre-configured with a corresponding edge weight calculation method; The edge weight is determined according to the edge weight calculation method corresponding to the target anomaly type.

8. A data tracing device, characterized in that: include: The monitoring module is used to monitor the data quality of the database and obtain the data to be traced that indicates abnormal data quality; A query module is configured to query the data to be traced in multiple preset dimensions through a preset knowledge graph to obtain multiple objects to be analyzed, wherein the nodes in the preset knowledge graph are determined based on entities that affect the data quality of the database, and the node attributes of the preset knowledge graph include: the entity corresponding to dimensional attributes in multiple preset dimensions, and the edges in the preset knowledge graph are determined based on the association relationship between the entities, and the entities include at least: the data to be traced, and each of the objects to be analyzed is the entity that has the association relationship with the data to be traced in the preset dimension; a determination module, configured to determine a weighted score corresponding to each of the objects to be analyzed based on the plurality of preset dimensions, wherein the weighted score is a weighted result of the comprehensive score obtained for the same object to be analyzed in the plurality of preset dimensions and the dimensional weight corresponding to each of the preset dimensions; The sorting module is used to sort the multiple objects to be analyzed in descending order according to the weighted scores to obtain a root cause list of the data to be traced.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the data tracing method according to any one of claims 1 to 7 through the computer program.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the data tracing method described in any one of claims 1 to 7 are implemented.

Images