Data Quality Governance Methods and Systems Based on Artificial Intelligence

Abstract

This invention discloses a data quality governance method and system based on artificial intelligence, relating to the field of data governance technology. The governance method includes the following steps: pre-constructing a data graph and updating the data graph in each cycle; dividing the data graph into several data nodes; identifying abnormal data in the data graph and analyzing the diffusion of the impact of abnormal data; generating several influence links in the data graph to obtain the degree of influence between two adjacent data nodes; identifying anomalies in the diffusion of influence in each influence link; performing real-time quality assessment of data entities in each data node and identifying anomalies in the data nodes; performing diffusion analysis on the influence links of any abnormal data node and predicting anomalies in the remaining data nodes in any influence link; this helps to avoid business chain reactions caused by data problems and ensures the stability of system operation.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, specifically to a data quality governance method and system based on artificial intelligence. Background Technology

[0002] In today's data-driven decision-making era, data quality is directly related to the accuracy and reliability of downstream business systems. Traditional data quality governance methods often rely on setting static rules to audit single-point data. While these methods can detect explicit errors, they have significant limitations.

[0003] First, it cannot effectively depict the complex relationships between data entities. When a data source has quality anomalies, it is difficult to accurately and quickly assess the scope and extent of its cascading impact on downstream businesses through processing, transformation, and dependency links. Second, impact analysis relies heavily on manual experience to trace back, which is inefficient and prone to missing critical paths, resulting in inaccurate problem localization, incomplete impact assessment, and business departments being unable to obtain timely risk warnings. Summary of the Invention

[0004] The purpose of this invention is to provide a data quality governance method and system based on artificial intelligence to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data quality governance method based on artificial intelligence, the governance method comprising the following steps:

[0006] Step S100: Pre-construct a data graph, periodically collect and summarize data entities through several channels, and update the data graph in each period; based on the collection channels of existing data entities in the data graph, divide the data graph into several data nodes;

[0007] Step S200: Identify anomalous data in the data graph for any period, and analyze the impact diffusion of the anomalous data based on its distribution in the data graph; generate several impact links in the data graph based on the impact diffusion of the anomalous data.

[0008] Step S300: Perform correlation analysis on the abnormal data of two adjacent data nodes in any influence link to obtain the degree of influence between the two adjacent data nodes; based on the influence between any two data nodes, identify anomalies in the diffusion of influence in each influence link;

[0009] Step S400: Perform real-time quality assessment of data entities in each data node, identify anomalies in data nodes, conduct diffusion analysis on the impact chain of any abnormal data node, and make anomaly judgments and predictions for other data nodes in any impact chain.

[0010] Furthermore, step S100 includes the following steps:

[0011] Step S101: Preset a data structure for the data entities in the data graph. The data structure includes data attributes, data types and data sources. At the same time, preset several channels for the collection methods of data entities. Summarize the entity data collected from each channel every unit period, extract the data type and data attributes of any entity data, and generate the data structure for extracting entity data.

[0012] Step S102: Select an arbitrary unit period, extract the data source of each entity data from the selected unit period, divide all the entity data summarized in the selected unit period according to the data source to obtain several entity data sets, compare each entity data set with the existing entity data in the data graph, summarize the entity data sets with the same data source and the existing entity data, generate several entity data sets in the data graph of the selected unit period, and set each entity data set as a data node in the data graph;

[0013] Step S103: Randomly select a data node from the data graph, and randomly select two entity data from the selected data node. Extract the data structure of the two entity data. If the data structures of the two entity data are the same, divide the two entity data into a data subset of the selected data node. Divide all entity data according to the data structure to obtain several data subsets of the selected data node.

[0014] Furthermore, step S200 includes the following steps:

[0015] Step S201: Randomly select a data graph for a unit period, and randomly select a data subset of a data node from the selected unit period. Preset corresponding abnormal data identification rules for the selected data subset to obtain several abnormal data in the selected data subset. The abnormal data identification rules include checking whether key fields are empty or missing in the customer information table or order table; checking whether numerical fields are within a reasonable range in numerical data such as age, sales, and temperature; and checking whether the data is logically consistent with other related data in business data with dependencies such as order date and delivery date.

[0016] Step S202: Obtain the data source of each data subset in each data node; arbitrarily select two data nodes, and randomly select a data subset from each of the two data nodes to obtain the data source of each data subset and generate a data source group; pre-establish a business rule database, which stores several business rules, each containing business logic with two data sources; arbitrarily select a business rule; if the data source group is the same as the two data sources in the selected business rule, establish a connection relationship between the two data nodes and set the two data sources as a connection group between the two data nodes; business rules include customer information, transaction record association rules, order and inventory impact rules, product catalog and sales report mapping rules, etc.

[0017] Step S203: Obtain the connection relationship between each data node in the data graph. Randomly select two data nodes with abnormal data. If there is a connection relationship between the two data nodes, extract the data source of any two abnormal data. If the data source of the two abnormal data is the same as the connection group of the two data nodes, set the two data nodes as the influencing node group.

[0018] Step S204: Obtain each group of influencing nodes in the data graph, extract the connection relationship between two data nodes in each group, and set the two data nodes as the occurrence node and the influence node, respectively; arbitrarily select two groups of influencing nodes. If the influence node of one group of influencing nodes is the same as the occurrence node of the other group of influencing nodes, then an influence diffusion relationship is set between the two groups of influencing nodes; the occurrence node is the node that first generates an anomaly between the two nodes, and the influence node is the node that generates an anomaly due to the influence of the occurrence node. There is a cause-and-effect relationship between the two nodes.

[0019] Step S205: Summarize several groups of influencing nodes that have mutual influence and diffusion to obtain the connection relationship between each data node in the several groups of influencing nodes, and generate a corresponding influence link; summarize the influence diffusion between all groups of influencing nodes to obtain several influence links in the data map.

[0020] Furthermore, step S300 includes the following steps:

[0021] Step S301: Randomly select an influencing link, and randomly select an influencing node group from the selected influencing link. Obtain the number of connection groups s for each connection group, and summarize the data subsets contained in each connection group to obtain the number of data subsets r1. Set the number of data subsets contained in two data nodes in the selected influencing node group to R. totalThe calculated percentage of connections in the selected influential node group is f1 = r1 / R. total ;

[0022] Step S302: Randomly select the k-th connection group, and extract the data subsets of the occurrence node and the affected nodes in the k-th connection group to obtain the number of abnormal data in the data subset of the occurrence node as m. k The number of anomalous data in the subset of data affecting the node is n. k The percentage of abnormal influence f2 of the k-th connection group was calculated. k =n k / m k According to the formula:

[0023] ;

[0024] The influence degree Y of the selected influencing node group is calculated. The influence degree between two data nodes in the influencing node group mainly takes into account the number of associations between their subsets and the number of abnormal data. The more associations between subsets, the closer the relationship between the two data nodes and the higher the influence degree. The more abnormal data, the higher the influence degree.

[0025] Step S303: Obtain the influence degree of each group of influencing nodes in each influencing link. Randomly select a data node from the data graph to obtain the influencing links where the selected data node is located. Set the influence degree of the group of influencing nodes with the selected data node as the influencing node in the a-th influencing link as Y. a According to the formula:

[0026] ;

[0027] Where c represents the number of influencing links where the selected data node is located; the comprehensive influence value Y of the selected data node is calculated. com Preset an influence threshold Y th If Y com >Y th If so, the selected data node will be set as an abnormal data node;

[0028] Step S304: Randomly select an influencing link, extract the abnormal data nodes contained in the selected influencing link, count the number of abnormal data nodes as p, set the number of data nodes in the selected influencing link as q, and calculate the abnormal proportion of the selected influencing link as η = p / q; preset an abnormal proportion threshold η. th If η≥η thIf the selected influencing link is not an anomalous link, then the anomalous percentage of the selected influencing link in each unit period is extracted, and the anomalous percentage with the smallest value is selected as the anomalous judgment threshold η for the selected influencing link. th .

[0029] Furthermore, step S400 includes the following steps:

[0030] Step S401: Randomly select a data node from the current data map, extract the data entities collected in real time from the selected data node, and divide the selected data node into various data subsets; for any data subset, retrieve the corresponding abnormal data identification rules to obtain several abnormal data in each data subset, and count the number of abnormal data in the e-th data subset as n. e ;

[0031] Step S402: Obtain the influence links where the selected data node is located, and obtain the influence node groups with the data node as the originating node. Randomly select one influence node group, and obtain the number of connection groups s and the proportion of connection groups f1 of the selected influence node group. Set the influence threshold as Y. th Given the number of affected links as c, we obtain the expected percentage of abnormal impact for the selected affected node group as f. ex =Y th / (c×s×f1); According to the formula:

[0032] ;

[0033] The expected number of outliers, n, is calculated among the data subsets that are connected to the e-th data subset within the selected group of influencing nodes. ex Preset a threshold n for the number of abnormal data. th If n ex ≥n ex If so, the affected nodes in the selected affected node group will be set as the expected preset nodes, and an abnormal alert will be issued for the expected preset nodes; the expected abnormal impact ratio is mainly obtained by reverse calculation of the impact degree, so that the calculation process has uniformity.

[0034] Step S403: Randomly select an influencing link, and set the desired preset number of nodes in the selected influencing link to p. ex The expected proportion of anomalies affecting the selected links is η. ex Set the threshold for detecting anomalies affecting the link as η. th If η th ≥η ex If so, an anomaly alert will be sent to the selected affected link.

[0035] To better implement the above methods, a data quality governance system is also proposed, which includes a data graph analysis module, a data link analysis module, a link anomaly analysis module, and a real-time anomaly prediction module.

[0036] The data graph analysis module is used to pre-build a data graph, periodically collect and summarize data entities through several channels, and update the data graph in each period; based on the collection channels of the existing data entities in the data graph, the data graph is divided into several data nodes.

[0037] The data link analysis module is used to identify abnormal data in the data graph of any period and analyze the impact diffusion of abnormal data based on its distribution in the data graph; based on the impact diffusion of abnormal data, several impact links are generated in the data graph.

[0038] The link anomaly analysis module is used to perform correlation analysis on the abnormal data of two adjacent data nodes in any impact link to obtain the degree of impact between the two adjacent data nodes; based on the impact between any two data nodes, it identifies anomalies in the spread of impact in each impact link.

[0039] The real-time anomaly prediction module is used to perform quality assessment of data entities in each data node in real time, identify anomalies in data nodes, perform diffusion analysis on the impact chain of any abnormal data node, and predict anomalies in other data nodes in any impact chain.

[0040] Furthermore, the data graph analysis module includes a data graph construction unit and a graph node partitioning unit;

[0041] The data graph construction unit is used to pre-build a data graph, periodically collect and summarize data entities through several channels, and update the data graph in each period. The graph node division unit is used to divide the data graph into several data nodes based on the collection channels of the existing data entities in the data graph.

[0042] Furthermore, the data link analysis module includes an impact diffusion identification unit and an impact link generation unit;

[0043] The impact diffusion identification unit is used to identify abnormal data in the data map of any period and analyze the impact diffusion of abnormal data based on its distribution in the data map; the impact link generation unit is used to generate several impact links in the data map based on the impact diffusion of abnormal data.

[0044] Furthermore, the link anomaly analysis module includes an impact quantification unit and an anomaly link identification unit;

[0045] The influence degree quantification unit is used to perform correlation analysis on the abnormal data of two adjacent data nodes in any influence link to obtain the influence degree between the two adjacent data nodes; the abnormal link identification unit is used to identify the abnormal spread of influence in each influence link based on the influence between any two data nodes.

[0046] Furthermore, the real-time anomaly prediction module includes a real-time node analysis unit and an anomaly link prediction unit;

[0047] The real-time node analysis unit is used to perform quality assessment on data entities in each data node in real time and to identify anomalies in data nodes; the anomaly link prediction unit is used to perform diffusion analysis on the impact link of any anomaly data node and to make anomaly judgments and predictions on the remaining data nodes in any impact link.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] 1. This invention systematically integrates multi-source data entities by constructing a dynamically updated data graph, effectively identifying abnormal data and their relationships; compared with traditional methods, it can deeply mine the complex dependencies between data nodes, and achieve a comprehensive characterization and accurate positioning of the path of anomaly impact;

[0050] 2. This invention can track the diffusion path of abnormal data in the data graph in real time and quantify the degree of impact between nodes. It can identify potentially affected data nodes and business links in advance, thereby changing passive processing to proactive early warning, which greatly improves the response speed and decision-making basis of enterprises in data risk management.

[0051] 3. This invention organically combines data graphs, business rules, and real-time prediction modules to form a closed-loop governance system; it can dynamically adjust node relationships and judgment thresholds according to changes in business rules, which helps to avoid business chain reactions caused by data problems and ensures the stability of system operation. Attached Figure Description

[0052] Figure 1 This is a schematic diagram illustrating the steps of an AI-based data quality governance method.

[0053] Figure 2 This is a schematic diagram of the structure of an artificial intelligence-based data quality governance system. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example: Figures 1 to 2 As shown, this invention provides a data quality governance method based on artificial intelligence, the governance method including the following steps:

[0056] Step S100: Pre-construct a data graph, periodically collect and summarize data entities through several channels, and update the data graph in each period; based on the collection channels of existing data entities in the data graph, divide the data graph into several data nodes;

[0057] Step S100 includes the following steps:

[0058] Step S101: Preset a data structure for the data entities in the data graph. The data structure includes data attributes, data types and data sources. At the same time, preset several channels for the collection methods of data entities. Summarize the entity data collected from each channel every unit period, extract the data type and data attributes of any entity data, and generate the data structure for extracting entity data.

[0059] Step S102: Select an arbitrary unit period, extract the data source of each entity data from the selected unit period, divide all the entity data summarized in the selected unit period according to the data source to obtain several entity data sets, compare each entity data set with the existing entity data in the data graph, summarize the entity data sets with the same data source and the existing entity data, generate several entity data sets in the data graph of the selected unit period, and set each entity data set as a data node in the data graph;

[0060] Step S103: Randomly select a data node from the data graph, and randomly select two entity data from the selected data node. Extract the data structure of the two entity data. If the data structures of the two entity data are the same, divide the two entity data into a data subset of the selected data node. Divide all entity data according to the data structure to obtain several data subsets of the selected data node.

[0061] Step S200: Identify anomalous data in the data graph for any period, and analyze the impact diffusion of the anomalous data based on its distribution in the data graph; generate several impact links in the data graph based on the impact diffusion of the anomalous data.

[0062] Step S200 includes the following steps:

[0063] Step S201: Randomly select a data map of a unit period, randomly select a data subset of a data node from the selected unit period, preset corresponding abnormal data identification rules for the selected data subset, and obtain several abnormal data in the selected data subset;

[0064] Step S202: Obtain the data source of each data subset in each data node; arbitrarily select two data nodes, and randomly select a data subset from each of the two data nodes to obtain the data source of each data subset and generate a data source group; pre-establish a business rule database, which stores several business rules, each of which contains business logic with two data sources; arbitrarily select a business rule, and if the data source group is the same as the two data sources in the selected business rule, then establish a connection relationship between the two data nodes and set the two data sources as a connection group between the two data nodes;

[0065] Example 1: Rule A is defined in the business rule database as "Customer Information → Transaction Verification". Two data nodes are set as the customer information node and the transaction verification node. If they conform to rule A, a connection relationship is established between the two data nodes to form a connection group.

[0066] Step S203: Obtain the connection relationship between each data node in the data graph. Randomly select two data nodes with abnormal data. If there is a connection relationship between the two data nodes, extract the data source of any two abnormal data. If the data source of the two abnormal data is the same as the connection group of the two data nodes, set the two data nodes as the influencing node group.

[0067] Step S204: Obtain each group of influencing nodes in the data map, extract the connection relationship between two data nodes in each group of influencing nodes, and set the two data nodes as the occurrence node and the influence node respectively; arbitrarily select two groups of influencing nodes. If the influence node of one group of influencing nodes is the same as the occurrence node of the other group of influencing nodes, then the influence diffusion between the two groups of influencing nodes is set.

[0068] Step S205: Summarize several groups of influencing nodes that have mutual influence and diffusion to obtain the connection relationship between each data node in the several groups of influencing nodes, and generate a corresponding influence link; summarize the influence diffusion between all groups of influencing nodes to obtain several influence links in the data map.

[0069] Step S300: Perform correlation analysis on the abnormal data of two adjacent data nodes in any influence link to obtain the degree of influence between the two adjacent data nodes; based on the influence between any two data nodes, identify anomalies in the diffusion of influence in each influence link;

[0070] Step S300 includes the following steps:

[0071] Step S301: Randomly select an influencing link, and randomly select an influencing node group from the selected influencing link. Obtain the number of connection groups s for each connection group, and summarize the data subsets contained in each connection group to obtain the number of data subsets r1. Set the number of data subsets contained in two data nodes in the selected influencing node group to R. total The calculated percentage of connections in the selected influential node group is f1 = r1 / R. total ;

[0072] Example 2: Select the "Customer Information - Transaction Verification" affected node group in the affected link. Set that there are 10 subsets in the customer information node and 5 subsets in the transaction verification node, with a total of 15 subsets. Set that there are 3 connection groups between the two, i.e., r1=3, and get the connection ratio f1=3 / 15=0.2.

[0073] Step S302: Randomly select the k-th connection group, and extract the data subsets of the occurrence node and the affected nodes in the k-th connection group to obtain the number of abnormal data in the data subset of the occurrence node as m. k The number of anomalous data in the subset of data affecting the node is n. k The percentage of abnormal influence f2 of the k-th connection group was calculated. k =n k / m k According to the formula:

[0074] ;

[0075] The degree of influence Y of the selected influencing node group is calculated;

[0076] Example 3: In three connection groups, the number of abnormal data subsets of customer information nodes are set to 5, 8, and 3, respectively, and the number of abnormal data subsets of transaction verification nodes are set to 10, 16, and 6, respectively; the connection ratio is set to f1 = 0.2, and the influence degree of the node group is calculated to be Y = 0.2 × (5 / 10 + 8 / 16 + 3 / 6) = 0.2 × 1.5 = 0.3;

[0077] Step S303: Obtain the influence degree of each group of influencing nodes in each influencing link. Randomly select a data node from the data graph to obtain the influencing links where the selected data node is located. Set the influence degree of the group of influencing nodes with the selected data node as the influencing node in the a-th influencing link as Y. a According to the formula:

[0078] ;

[0079] Where c represents the number of influencing links where the selected data node is located; the comprehensive influence value Y of the selected data node is calculated. com Preset an influence threshold Y th If Y com >Y th If so, the selected data node will be set as an abnormal data node;

[0080] Step S304: Randomly select an influencing link, extract the abnormal data nodes contained in the selected influencing link, count the number of abnormal data nodes as p, set the number of data nodes in the selected influencing link as q, and calculate the abnormal proportion of the selected influencing link as η = p / q; preset an abnormal proportion threshold η. th If η≥η th If the selected influencing link is not an anomalous link, then the anomalous percentage of the selected influencing link in each unit period is extracted, and the anomalous percentage with the smallest value is selected as the anomalous judgment threshold η for the selected influencing link. th .

[0081] Step S400: Real-time quality assessment of data entities in each data node, anomaly identification of data nodes; diffusion analysis of the impact chain of any abnormal data node, and anomaly judgment and prediction of other data nodes in any impact chain.

[0082] Step S400 includes the following steps:

[0083] Step S401: Randomly select a data node from the current data map, extract the data entities collected in real time from the selected data node, and divide the selected data node into various data subsets; for any data subset, retrieve the corresponding abnormal data identification rules to obtain several abnormal data in each data subset, and count the number of abnormal data in the e-th data subset as n. e ;

[0084] Step S402: Obtain the influence links where the selected data node is located, and obtain the influence node groups with the data node as the originating node. Randomly select one influence node group, and obtain the number of connection groups s and the proportion of connection groups f1 of the selected influence node group. Set the influence threshold as Y. th Given the number of affected links as c, we obtain the expected percentage of abnormal impact for the selected affected node group as f. ex =Y th / (c×s×f1); According to the formula:

[0085] ;

[0086] The expected number of outliers, n, is calculated among the data subsets that are connected to the e-th data subset within the selected group of influencing nodes. ex Preset a threshold n for the number of abnormal data. th If n ex ≥n ex If so, the affected nodes in the selected affected node group will be set as the expected preset nodes, and an abnormal alert will be issued for the expected preset nodes;

[0087] Step S403: Randomly select an influencing link, and set the desired preset number of nodes in the selected influencing link to p. ex The expected proportion of anomalies affecting the selected links is η. ex Set the threshold for detecting anomalies affecting the link as η. th If η th ≥η ex If so, an anomaly alert will be sent to the selected affected link.

[0088] A data quality governance system, comprising a data graph analysis module, a data link analysis module, a link anomaly analysis module, and a real-time anomaly prediction module;

[0089] The data graph analysis module is used to pre-build a data graph, periodically collect and summarize data entities through several channels, and update the data graph in each period; based on the collection channels of the existing data entities in the data graph, the data graph is divided into several data nodes.

[0090] The data link analysis module is used to identify abnormal data in the data graph of any period and analyze the impact diffusion of abnormal data based on its distribution in the data graph; based on the impact diffusion of abnormal data, several impact links are generated in the data graph.

[0091] The link anomaly analysis module is used to perform correlation analysis on the abnormal data of two adjacent data nodes in any impact link to obtain the degree of impact between the two adjacent data nodes; based on the impact between any two data nodes, it identifies anomalies in the spread of impact in each impact link.

[0092] The real-time anomaly prediction module is used to perform quality assessment of data entities in each data node in real time, identify anomalies in data nodes, perform diffusion analysis on the impact chain of any abnormal data node, and predict anomalies in other data nodes in any impact chain.

[0093] The data graph analysis module includes a data graph construction unit and a graph node partitioning unit.

[0094] The data graph construction unit is used to pre-build a data graph, periodically collect and summarize data entities through several channels, and update the data graph in each period. The graph node division unit is used to divide the data graph into several data nodes based on the collection channels of the existing data entities in the data graph.

[0095] The data link analysis module includes an impact diffusion identification unit and an impact link generation unit.

[0096] The impact diffusion identification unit is used to identify abnormal data in the data map of any period and analyze the impact diffusion of abnormal data based on its distribution in the data map; the impact link generation unit is used to generate several impact links in the data map based on the impact diffusion of abnormal data.

[0097] The link anomaly analysis module includes an impact quantification unit and an anomaly link identification unit.

[0098] The influence degree quantification unit is used to perform correlation analysis on the abnormal data of two adjacent data nodes in any influence link to obtain the influence degree between the two adjacent data nodes; the abnormal link identification unit is used to identify the abnormal spread of influence in each influence link based on the influence between any two data nodes.

[0099] The real-time anomaly prediction module includes a real-time node analysis unit and an anomaly link prediction unit.

[0100] The real-time node analysis unit is used to perform quality assessment on data entities in each data node in real time and to identify anomalies in data nodes; the anomaly link prediction unit is used to perform diffusion analysis on the impact link of any anomaly data node and to make anomaly judgments and predictions on the remaining data nodes in any impact link.

[0101] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A data quality governance method based on artificial intelligence, characterized by: The governance method includes the following steps: Step S100: Pre-construct a data graph, periodically collect and summarize data entities through several channels, and update the data graph in each period; based on the collection channels of existing data entities in the data graph, divide the data graph into several data nodes; Step S200: Identify anomalous data in the data graph for any period, and analyze the impact diffusion of the anomalous data based on its distribution in the data graph; generate several impact links in the data graph based on the impact diffusion of the anomalous data. Step S300: Perform correlation analysis on the abnormal data of two adjacent data nodes in any influence link to obtain the degree of influence between the two adjacent data nodes; based on the influence between any two data nodes, identify anomalies in the diffusion of influence in each influence link; Step S400: Real-time quality assessment of data entities in each data node, anomaly identification of data nodes; diffusion analysis of the impact chain of any abnormal data node, and anomaly judgment and prediction of other data nodes in any impact chain. Step S200 includes the following steps: Step S201: Randomly select a data map of a unit period, randomly select a data subset of a data node from the selected unit period, preset corresponding abnormal data identification rules for the selected data subset, and obtain several abnormal data in the selected data subset; The rules for identifying abnormal data include checking whether key fields in customer information tables or order tables are empty or missing; checking whether numerical fields in numerical data are within a reasonable range, including age, sales revenue, and temperature; and checking whether the data in dependent business data is logically consistent with other related data, including order date and shipping date. Step S202: Obtain the data source of each data subset in each data node; arbitrarily select two data nodes, and randomly select a data subset from each of the two data nodes to obtain the data source of each data subset and generate a data source group; pre-establish a business rule database, which stores several business rules, each of which contains business logic with two data sources; arbitrarily select a business rule, and if the data source group is the same as the two data sources in the selected business rule, then establish a connection relationship between the two data nodes and set the two data sources as a connection group between the two data nodes; Step S203: Obtain the connection relationship between each data node in the data graph. Randomly select two data nodes with abnormal data. If there is a connection relationship between the two data nodes, extract the data source of any two abnormal data. If the data source of the two abnormal data is the same as the connection group of the two data nodes, set the two data nodes as the influencing node group. Step S204: Obtain each group of influencing nodes in the data map, extract the connection relationship between two data nodes in each group of influencing nodes, and set the two data nodes as the occurrence node and the influence node respectively; arbitrarily select two groups of influencing nodes. If the influence node of one group of influencing nodes is the same as the occurrence node of the other group of influencing nodes, then the influence diffusion between the two groups of influencing nodes is set. Step S205: Summarize several groups of influencing nodes that have mutual influence and diffusion to obtain the connection relationship between each data node in the several groups of influencing nodes, and generate a corresponding influence link; summarize the influence diffusion between all groups of influencing nodes to obtain several influence links in the data map.

2. The data quality governance method based on artificial intelligence according to claim 1, characterized in that: Step S100 includes the following steps: Step S101: Preset a data structure for the data entities in the data graph. The data structure includes data attributes, data types and data sources. At the same time, preset several channels for the collection methods of data entities. Summarize the entity data collected from each channel every unit period, extract the data type and data attributes of any entity data, and generate the data structure for extracting entity data. Step S102: Select an arbitrary unit period, extract the data source of each entity data from the selected unit period, divide all the entity data summarized in the selected unit period according to the data source to obtain several entity data sets, compare each entity data set with the existing entity data in the data graph, summarize the entity data sets with the same data source and the existing entity data, generate several entity data sets in the data graph of the selected unit period, and set each entity data set as a data node in the data graph; Step S103: Randomly select a data node from the data graph, and randomly select two entity data from the selected data node. Extract the data structure of the two entity data. If the data structures of the two entity data are the same, divide the two entity data into a data subset of the selected data node. Divide all entity data according to the data structure to obtain several data subsets of the selected data node.

3. The data quality governance method based on artificial intelligence according to claim 1, characterized in that: Step S300 includes the following steps: Step S301: Randomly select an influencing link, and randomly select an influencing node group from the selected influencing link. Obtain the number of connection groups s for each connection group, and summarize the data subsets contained in each connection group to obtain the number of data subsets r1. Set the number of data subsets contained in two data nodes in the selected influencing node group to R. total The calculated percentage of connections in the selected influential node group is f1 = r1 / R. total ; Step S302: Randomly select the k-th connection group, and extract the data subsets of the occurrence node and the affected nodes in the k-th connection group to obtain the number of abnormal data in the data subset of the occurrence node as m. k The number of anomalous data in the subset of data affecting the node is n. k The percentage of abnormal influence f2 of the k-th connection group was calculated. k =n k / m k According to the formula: ； The degree of influence Y of the selected influencing node group is calculated; Step S303: Obtain the influence degree of each group of influencing nodes in each influencing link. Randomly select a data node from the data graph to obtain the influencing links where the selected data node is located. Set the influence degree of the group of influencing nodes with the selected data node as the influencing node in the a-th influencing link as Y. a According to the formula: ； Where c represents the number of influencing links where the selected data node is located; the comprehensive influence value Y of the selected data node is calculated. com Preset an influence threshold Y th If Y com >Y th If so, the selected data node will be set as an abnormal data node; Step S304: Randomly select an influencing link, extract the abnormal data nodes contained in the selected influencing link, count the number of abnormal data nodes as p, set the number of data nodes in the selected influencing link as q, and calculate the abnormal proportion of the selected influencing link as η = p / q; preset an abnormal proportion threshold η. th If η≥η th If the selected influencing link is not an anomalous link, then the anomalous percentage of the selected influencing link in each unit period is extracted, and the anomalous percentage with the smallest value is selected as the anomalous judgment threshold η for the selected influencing link. th .

4. The data quality governance method based on artificial intelligence according to claim 3, characterized in that: Step S400 includes the following steps: Step S401: Randomly select a data node from the current data map, extract the data entities collected in real time from the selected data node, and divide the selected data node into various data subsets; for any data subset, retrieve the corresponding abnormal data identification rules to obtain several abnormal data in each data subset, and count the number of abnormal data in the e-th data subset as n. e ; Step S402: Obtain the influence links where the selected data node is located, and obtain the influence node groups with the data node as the originating node. Randomly select one influence node group, and obtain the number of connection groups s and the proportion of connection groups f1 of the selected influence node group. Set the influence threshold as Y. th Given the number of affected links as c, we obtain the expected percentage of abnormal impact for the selected affected node group as f. ex =Y th / (c×s×f1); According to the formula: ； The expected number of outliers, n, is calculated among the data subsets that are connected to the e-th data subset within the selected group of influencing nodes. ex Preset a threshold n for the number of abnormal data. th If n ex ≥n ex If so, the affected nodes in the selected affected node group will be set as the expected preset nodes, and an abnormal alert will be issued for the expected preset nodes; Step S403: Randomly select an influencing link, and set the desired preset number of nodes in the selected influencing link to p. ex The expected proportion of anomalies affecting the selected links is η. ex Set the threshold for detecting anomalies affecting the link as η. th If η th ≥η ex If so, an anomaly alert will be sent to the selected affected link.

5. A data quality governance system, used to execute the artificial intelligence-based data quality governance method according to any one of claims 1-4, characterized in that: The governance system includes a data graph analysis module, a data link analysis module, a link anomaly analysis module, and a real-time anomaly prediction module; The data graph analysis module is used to pre-build a data graph, periodically collect and summarize data entities through several channels, and update the data graph in each period; based on the collection channels of existing data entities in the data graph, the data graph is divided into several data nodes. The data link analysis module is used to identify abnormal data in the data graph of any period and analyze the impact diffusion of abnormal data based on its distribution in the data graph; and generate several impact links in the data graph based on the impact diffusion of abnormal data. The link anomaly analysis module is used to perform correlation analysis on the abnormal data of any two adjacent data nodes in an impact link to obtain the degree of impact between the two adjacent data nodes. Based on the influence between any two data nodes, anomalies in the spread of influence in each influence chain are identified. The real-time anomaly prediction module is used to perform real-time quality assessment of data entities in each data node, identify anomalies in data nodes, perform diffusion analysis on the impact chain of any abnormal data node, and predict anomalies in other data nodes in any impact chain.

6. A data quality governance system according to claim 5, characterized in that: The data graph analysis module includes a data graph construction unit and a graph node partitioning unit; The data graph construction unit is used to pre-build a data graph, periodically collect and summarize data entities through several channels, and update the data graph in each period. The graph node division unit is used to divide the data graph into several data nodes based on the collection channels of existing data entities in the data graph.

7. A data quality governance system according to claim 5, characterized in that: The data link analysis module includes an impact diffusion identification unit and an impact link generation unit; The influence diffusion identification unit is used to identify abnormal data in the data map of any period and analyze the influence diffusion of abnormal data based on its distribution in the data map; the influence link generation unit is used to generate several influence links in the data map based on the influence diffusion of abnormal data.

8. A data quality governance system according to claim 5, characterized in that: The link anomaly analysis module includes an impact quantification unit and an anomaly link identification unit; The influence degree quantification unit is used to perform correlation analysis on the abnormal data of two adjacent data nodes in any influence link to obtain the influence degree between the two adjacent data nodes. The abnormal link identification unit is used to identify abnormalities in the diffusion of influence in each influence link based on the influence between any two data nodes.

9. A data quality governance system according to claim 5, characterized in that: The real-time anomaly prediction module includes a real-time node analysis unit and an anomaly link prediction unit. The real-time node analysis unit is used to perform quality assessment on the data entities in each data node in real time and to identify anomalies in the data nodes; the anomaly link prediction unit is used to perform diffusion analysis on the impact link of any anomaly data node and to make anomaly judgment prediction on the remaining data nodes in any impact link.

Images