A data governance effect evaluation method

Abstract

The application discloses a data governance effect evaluation method, and relates to the technical field of data governance. Firstly, the application collects evaluation index time series of a complete business cycle before and after the implementation of a governance action, and then, after discretization processing, the application screens the correlation edges between indexes based on mutual information and marks the correlation polarity, so as to construct an undirected dependence network before and after the governance. Then, the application uses conditional mutual information to complete the discrimination of the causal direction, and generates a corresponding causal graph. Finally, the application compares the edge changes of the causal graph, identifies the effect offset phenomenon between indexes, calculates the net effect value, and realizes threshold early warning. The application can accurately capture the index causal dependence structure changes caused by the governance, and improves the accuracy and reliability of the data governance effect evaluation.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, specifically a method for evaluating the effectiveness of data governance. Background Technology

[0002] With the acceleration of the marketization of data elements, data governance has become a core element in ensuring data quality, and the evaluation of data governance effectiveness is a key step in verifying its effectiveness. However, existing evaluation methods still have the following shortcomings: First, most existing solutions focus on the year-on-year change analysis of single indicator values ​​before and after governance, or only construct the relationship between indicators based on simple correlation analysis. They cannot capture the structural changes in the dependence between indicators caused by governance actions, and it is difficult to identify the deep causal logic changes brought about by governance behavior. Secondly, existing technologies often use fixed threshold judgment methods in the indicator association screening process, which lacks adaptive consideration of the specificity of indicator association and is prone to introducing noisy associations, resulting in insufficient accuracy of the constructed indicator dependency network. Third, the existing evaluation system does not pay attention to the problem of the offsetting effects of multiple paths between indicators. It judges the effectiveness of governance based solely on changes in surface indicators, which is prone to misjudgment of governance effectiveness and fails to identify hidden risks that may exist after governance in advance. Therefore, there is an urgent need for a method to evaluate the effectiveness of data governance. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a data governance effectiveness evaluation method that solves the problems of insufficient causal identification and easy misjudgment of effectiveness in existing data governance effectiveness evaluation methods.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a data governance effectiveness evaluation method, comprising: Step 1: Collect continuous time series of all evaluation indicators within one complete business cycle before and after the implementation of the governance action. Discretize the continuous time series of each evaluation indicator by equal frequency binning. Connect any two indicators with undirected edges. Based on the mutual information value between the two indicators, select the related edges from all undirected edges and retain them. At the same time, mark the polarity of each related edge and output the undirected dependency network before and after governance respectively. Step 2: Based on the undirected dependent network before and after governance, for any two indicators X and Y connected by an undirected edge, traverse the third indicator Z in the network other than X and Y, and use it as a candidate condition variable. Calculate the conditional mutual information CMI(X,Y|Z) of each triple (X,Y,Z), and determine the causal direction based on the conditional mutual information. Assign a unique direction to each undirected edge to obtain the causal graph C_before before governance and the causal graph C_after after governance. Step 3: Iterate through each evaluation index A, extract all incoming and outgoing edges in C_before and C_after, and extract newly added edges, disappearing edges, and flipped edges. At the same time, identify all pairs (A,B) with mutual cancellation effects. For each identified pair, calculate the net effect value based on C_after. When any net effect value is lower than a preset threshold, trigger an alert for that pair and output the index pair (A,B) and the net effect value.

[0005] As a further aspect of the present invention, for each evaluation index D i Perform equal-frequency binning discretization, statistically analyze its probability distribution, and calculate its information entropy H. i , where i∈[1,N] and is an integer, and N is the number of evaluation indicators; For any two evaluation indicators D i and D j Statistically analyze their joint probability distribution and calculate the mutual information MI. ij Based on normalized mutual information values Obtain the candidate associated edge set C; For each evaluation indicator D i Calculate the variance Var of its normalized mutual information value with all other evaluation metrics. i ; Calculate the divergence penalty factor Where max(Var) represents the maximum variance among all evaluation indicators; For each edge (i,j) in the candidate associated edge set C, define an initial retention probability and obtain the final probability of the edge according to the iterative update rule; Retain all edges with a final probability greater than 0.5 to obtain the final set of associated edges E.

[0006] As a further aspect of the present invention, the specific operation for obtaining the candidate associated edge set C is as follows: For all The probability density function is obtained by Gaussian kernel density estimation, and the first derivative of its logarithmic density is calculated. The method that minimizes the first derivative is then found. Value, denoted as If there are multiple points with the minimum value, the largest one is taken. Value as ; Keep all The index pairs are used to obtain the candidate associated edge set C.

[0007] As a further aspect of the present invention, the specific steps for obtaining the final probability of the edge according to the iterative update rule are as follows: Set the iteration round r, with a value range of [0, ∞), and the update rules for each round are as follows: Calculate the competition pressure coefficient for each edge under the current probability. ,in, This represents the number of strongly related edges to node i in the r-th iteration, specifically expressed as: , The following represents the retention probability of edge (i,j) in the r-th iteration. This is an example function; it takes the value 1 when the condition within the parentheses is true, and 0 otherwise. Based on the competition pressure coefficient Update retention probability ,in, The learning rate; Among all candidate edges, if the maximum absolute value of the change in the retention probability between two adjacent iterations is less than a preset difference ε, then... When the iteration stops, the final probability is obtained, where max is the maximum value function.

[0008] As a further aspect of the present invention, the specific rules for marking the association polarity of each associated edge are as follows: For each of the selected associated edges, the two indicators D are connected. i and D j Calculate the joint probability matrix and the product matrix of independent probabilities, and obtain the difference matrix by subtracting the two. Then calculate the sum of the main diagonal and the sum of the secondary diagonal of the difference matrix respectively. If the sum of the main diagonals is greater than the sum of the secondary diagonals, then the polarity of the side is marked as positive; if the sum of the main diagonals is less than the sum of the secondary diagonals, then the polarity of the side is marked as negative; if the sum of the main diagonals is equal to the sum of the secondary diagonals, then the side is marked as non-polar.

[0009] As a further aspect of the present invention, the specific operation for determining causal direction based on conditional mutual information is as follows: For an edge connecting X and Y in an undirected dependency network, iterate through all third-party metrics Z and calculate the conditional dependency decay for each Z. Difference with asymmetric information flow ; Calculate the local support score of Z for the direction X→Y. Where CMI(X,Y|Z) is the conditional mutual information value, and sgn() is the sign function; The direction discrimination factor is calculated based on the average local support score of each Z: if the direction discrimination factor > η, the direction is determined to be X→Y; if the direction discrimination factor < -η, the direction is determined to be Y→X; if the absolute value of the direction discrimination factor ≤ η, it is marked as having no definite direction, where η is a preset threshold and is a positive number.

[0010] As a further aspect of the present invention, the number n that supports X→Y among all directions given by Z is counted. XY and the number n that supports Y→X YX If n XY >n YX Then assign the direction X→Y to the undirected edge; if n YX >n XY Then assign the direction Y→X to the undirected edge; if n XY =n YX If so, then the edge is temporarily kept as undirected.

[0011] As a further aspect of the present invention, the specific operation for identifying binary pairs (A, B) with mutually canceling effects is as follows: For the causal graph C_after after governance, a depth-first search is performed starting from each index node to enumerate all directed paths from that node to all other nodes, and the node sequence and path symbol of each path are recorded. The path symbol is defined as the product of the associative polarities of all directed edges on the path, where a positive associative polarity is 1 and a negative associative polarity is -1. If a path contains a directed edge marked as non-polar, skip that path and use only the edges marked as polar. At the same time, the maximum path length is limited to no more than the total number of nodes in the graph; After the enumeration is completed, for each pair (A,B), the set of path symbols is counted. If the set of path symbols contains both -1 and 1, then the pair (A,B) is determined to have a mutual cancellation effect.

[0012] As a further aspect of the present invention, the specific operation for calculating the net effect value based on C_after is as follows: For the binary pair (A, B) where mutual cancellation effects have been identified, all directed paths from A to B are extracted from the treated causal graph C_after. For each path, the path strength is calculated, which is the product of the mutual information values ​​of all edges on the path, multiplied by the path length attenuation factor. , where L is the number of edges in the path; Group all paths by sign and calculate the sum of positive path strength P for each group. zheng Sum of negative path strengths P fu Calculate the net effect value .

[0013] This invention provides a method for evaluating the effectiveness of data governance, which has the following advantages compared with existing technologies: (1) This invention uses equal-frequency bin discretization processing combined with an adaptive correlation edge screening mechanism of mutual information to accurately identify the real statistical correlation between indicators and simultaneously complete the correlation polarity labeling, constructing a high-confidence governance-before-after undirected dependency network, effectively avoiding noise interference and omission of real correlation caused by fixed threshold judgment; (2) Based on undirected dependency networks, this invention uses triplet conditional mutual information calculation and combines a global causal direction discrimination mechanism for multiple candidate conditional variables to assign a unique causal direction to undirected edges and construct a causal graph before and after governance, thereby capturing the structural changes in the causal dependency relationship of indicators caused by governance actions. (3) By comparing the edge changes of the causal graph before and after governance, this invention can accurately locate the dependency relationship brought about by governance, simultaneously identify the multi-path effect cancellation phenomenon between indicators, calculate the net effect value and set a threshold warning, comprehensively cover the explicit and implicit effects of governance, and effectively avoid misjudgment of governance effectiveness. Attached Figure Description

[0014] Figure 1 This is a flowchart of the steps of the present invention; Figure 2 This is a flowchart illustrating the steps for filtering associated edges in this invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] like Figure 1 This invention provides a method for evaluating the effectiveness of data governance; As an embodiment of this application, it includes: Step 1: Collect continuous time series of all evaluation indicators within one complete business cycle before and after the implementation of the governance action. Discretize the continuous time series of each evaluation indicator by equal frequency binning. Connect any two indicators with undirected edges. Based on the mutual information value between the two indicators, select the related edges from all undirected edges and retain them. At the same time, mark the polarity of each related edge and output the undirected dependency network before and after governance respectively. Step 2: Based on the undirected dependent network before and after governance, for any two indicators X and Y connected by an undirected edge, traverse the third indicator Z in the network other than X and Y, and use it as a candidate condition variable. Calculate the conditional mutual information CMI(X,Y|Z) of each triple (X,Y,Z), and determine the causal direction based on the conditional mutual information. Assign a unique direction to each undirected edge to obtain the causal graph C_before before governance and the causal graph C_after after governance. Step 3: Iterate through each evaluation index A, extract all incoming and outgoing edges in C_before and C_after, and extract newly added edges, disappearing edges, and flipped edges. At the same time, identify all pairs (A,B) with mutual cancellation effects. For each identified pair, calculate the net effect value based on C_after. When any net effect value is lower than a preset threshold, trigger an alert for that pair and output the index pair (A,B) and the net effect value.

[0017] As a second embodiment of this application, it is implemented based on the first embodiment, except that this embodiment includes: Step 1: Collect continuous time series data for all evaluation indicators for one complete business cycle before and after the implementation of the governance actions; Collecting time series data before governance is mainly to establish a baseline for comparison, thereby identifying the real changes brought about by governance actions. If only data after governance is collected, it is impossible to determine whether the improvement or deterioration of indicators is due to governance or to seasonal fluctuations in business or system version updates. At the same time, collecting data over the entire business cycle (such as a month or quarter) rather than a single point in time is to capture the complete fluctuation range and trend of the indicators, and to avoid misjudgment due to the randomness of the sampling time. The continuous time series of each evaluation indicator is discretized by equal frequency binning. The core purpose of equal frequency binning discretization is to convert the continuous time series into a discrete state so that the mutual information value of the two evaluation indicators can be calculated later. Since the definition of mutual information requires the variable to be discrete or already discretized, while the indicators in data governance (such as integrity percentage and processing time in seconds) are all continuous values, the joint probability distribution cannot be directly calculated. Equal-frequency binning can divide the value range of each evaluation index into several intervals, so that each interval contains approximately the same number of sample points, ensuring that the statistical confidence of each discrete state is similar. Connecting any two metrics with an undirected edge is to construct a complete graph, which serves as the initial framework for subsequent filtering of related edges. Before knowing which metrics have statistical dependencies, the most common approach is to assume that all metric pairs may be related, and then use data-driven methods (such as calculating mutual information) to prune weakly related or irrelevant edges. The resulting initial complete graph ensures that no potential dependencies will be missed. Based on the mutual information value between the two evaluation indicators, related edges are selected from all undirected edges and retained; Simultaneously, each associated edge in the associated edge set E is labeled with its associated polarity, and the undirected dependent network before and after governance is output respectively. The specific rules for marking the polarity of each associated edge are as follows: For each of the selected associated edges, the two indicators D are connected. i and D j Calculate the joint probability matrix U ab =p(D i =a,D j =b) and the product matrix of independent probabilities V ab =p(D i =a)×p(D j =b), thus obtaining the difference matrix CH=U ab -V ab Calculate the sum of the main diagonals S of the difference matrix CH. main The sum of the diagonals and the secondary diagonal S anti Where a, b∈[1,K], p(D) i =a,D j =b) represents the joint probability, that is, the probability that the discretized value of index Di is equal to that of the a-th bin (bin number a) and the discretized value of index Dj is equal to that of the b-th bin, p(D i =a), p(D j =b) represent the marginal probabilities, namely the probability that the individual index Di is in the a-th box and the probability that the individual index Dj is in the b-th box, respectively. If S main >S anti If the polarity of the edge is positive, then the index D is positive. i and D j Changes in the same direction; if S main <S anti If the polarity of the edge is negative, then the index D is marked as negative. i and D j Reverse change; if S main =S anti If the edge is nonpolar, then the edge is marked as nonpolar.

[0018] Step 2: Based on the undirected dependency network before and after governance, for any two indicators X and Y connected by an undirected edge, traverse the third indicator Z in the network other than X and Y, and use it as a candidate condition variable to calculate the conditional mutual information CMI(X,Y|Z) of each triple (X,Y,Z), where Z is all the remaining indicators in the entire evaluation indicator set after removing X and Y. In causal inference, the statistical dependence between two variables X and Y may stem from direct causation or be induced by other variables (such as mediating variables). However, a single Z can only provide local information, while global information can be obtained by traversing all Z. For example, in the true causal structure X→Y, the conditional mutual information CMI(X,Y|Z) will exhibit different patterns for different Z: if Z is the cause of X, the dependence between X and Y may be strengthened after Z is given; if Z is the result of Y, the dependence may be weakened after Z is given. By observing the changing trend of CMI under multiple Z, the direction can be judged more robustly. Here, CMI(X,Y|Z) represents the conditional mutual information between indicators X and Y given a third indicator Z. The specific operation for calculating the conditional mutual information CMI(X,Y|Z) for each triple (X,Y,Z) is as follows: For the indicators X and Y connected by the selected associated edges, and the third indicator Z traversed (all three have been discretized by equal-frequency binning in step 1, and the bin numbers are all {1,...,K}), count the joint frequency of the three indicators on the continuous time series. Where a, b, and c are all ∈ [1, K] and are integers, T is the length of the time series, and X is the length of the time series. t =a means that the value of Xt belongs to the interval corresponding to the a-th bin, Y t Z t The meaning is the same, where K is the number of boxes; Calculate joint probability and conditional probability , , ,in, Let be an estimate of the joint probability, representing the probability that index X takes the a-th bin, Y takes the b-th bin, and Z takes the c-th bin. This is an estimate of the marginal probability, representing the probability that index Z takes the c-th bin. Let Z be an estimate of the conditional probability, representing the probability that X takes the value a and Y takes the value b, given Z=c. Let Z be an estimate of the conditional probability, representing the probability that index X takes the a-th bin given Z=c. This represents the summation over all possible bin numbers b of index Y. Let be an estimate of the conditional probability, representing the probability that index Y takes the b-th bin given Z=c. This represents the summation over all possible bin numbers 'a' of index X; The marginal probability is Z=c; Calculate the conditional mutual information ,in, If there exists a c corresponding to If c is not included in the summation, then c will not be included in the summation. Causal direction determination is performed based on conditional mutual information. The specific operation is as follows: For an edge connecting X and Y in an undirected dependency network, iterate through all third-party metrics Z and calculate the conditional dependency decay for each Z. Difference with asymmetric information flow ; The specific expression for the conditionally dependent decay is as follows: Where MI(X,Y) is the mutual information between indicators X and Y, and CMI(X,Y|Z) is the conditional mutual information between X and Y given Z; Conditional decay This indicates the reduction in the dependency between X and Y after the introduction of Z. If this value is large, it means that Z can explain most of the dependency between X and Y. The specific expression for the asymmetric information flow difference is as follows: Where CMI(X,Z|Y) represents the conditional mutual information between X and Z given Y, and CMI(Y,Z|X) represents the conditional mutual information between Y and Z given X. Asymmetric information flow difference This reflects the difference in the strength of the relationship between X and Z relative to the relationship between Y and Z after excluding the influence of X and Y respectively. Calculate the local support score of Z for the direction X→Y. Its specific expression is , where sgn() is the sign function; The above formula means that if X→Y is true, then given Z, the dependency between X and Y should decrease significantly. (large), and the dependency of X on Z should be stronger than the dependency of Y on Z given Y (i.e., large). If the answer is positive, the score is positive; otherwise, the score is negative. Local support score for each Z obtained in the above steps Calculate its average value The direction discrimination factor is obtained. Its specific expression is :like If the direction is determined to be X→Y, then... If the direction is Y→X, then the direction is determined. If , it is marked as having no definite direction, where The preset threshold for small positive numbers; Count the number of directions given by Z that support X→Y, n XY and the number n that supports Y→X YX If n XY >n YX Then assign the direction X→Y to the undirected edge; if n YX >n XY Then assign the direction Y→X to the undirected edge; if n XY =n YX If so, then the edge is temporarily kept as undirected; After assigning a unique direction to each undirected edge, we obtain the causal graph C_before before governance and the causal graph C_after after governance.

[0019] Step 3: Iterate through each evaluation metric A, extract all incoming and outgoing edges in C_before and C_after, and extract newly added edges, disappearing edges, and flipped edges from them; The incoming edge refers to a directed edge that ends at A, meaning there exists an index B such that B→A. The outgoing edge refers to a directed edge originating from A, meaning there exists an index B such that A→B. The indices B connected by these edges are all nodes that are directly connected to A. The edges in the causal graph represent direct causal dependencies, not indirect paths. Therefore, when extracting the inbound and outbound edges, only the indices that are directly connected to A by directed edges are considered, and the indirect effects that are transferred through other nodes are not considered. The newly added edges refer to directed edges (regardless of direction) that did not exist in C_before but appeared in C_after, indicating that the governance action introduced new direct causal dependencies; The vanishing edge refers to a directed edge that exists in C_before but completely disappears in C_after (without both the direction and the reverse), indicating that the governance action has eliminated the original direct causal dependency; The flipped edge refers to an edge between the pair of indicators that exists in both C_before and C_after, but the causal direction has changed. For example, before the treatment it is A→B, and after the treatment it becomes B→A. If the direction is the same before and after the treatment, then the edge does not belong to any of the above categories and is regarded as a stable edge; The effectiveness of data governance is not only reflected in the changes in indicator values, but also in the reorganization of the causal dependencies between indicators. Among them, the addition of edges reflects that governance may have brought about new linkage effects (such as the original independent quality dimensions becoming mutually influential), while the disappearance of edges indicates that governance has successfully decoupled unnecessary dependencies (such as removing redundant data processing links), and the reversal of edges means the transfer of causal dominance. Simultaneously, identify all pairs (A, B) that exhibit mutual cancellation effects, specifically as follows: For the causal graph C_after after governance, a depth-first search is performed starting from each index node to enumerate all directed paths from that node to all other nodes, and the node sequence and path symbol of each path are recorded. The path symbol is defined as the product of the associative polarities of all directed edges on the path, where a positive associative polarity is 1 and a negative associative polarity is -1. If a path contains a directed edge marked as non-polar, skip that path and use only the edges marked as polar. At the same time, the maximum path length is limited to no more than the total number of nodes in the graph to avoid getting stuck in an infinite loop; After the enumeration is completed, for each pair (A, B), count the set of path symbols. The elements in this set are all -1 or 1. If the path symbol set contains both -1 and 1, then the pair (A,B) is determined to have a mutual cancellation effect. When both types of paths exist simultaneously, changes in A will affect B through both positive and negative channels. Ultimately, the actual change in B depends on the net difference between the positive and negative effects. For each identified pair, the net effect value is calculated based on C_after, according to the following rules: For the binary pair (A, B) that has been identified as having mutual cancellation effects, extract all directed paths from A to B from the treated causal graph C_after. For each path, define its path strength as the product of the mutual information values ​​of all edges on the path, multiplied by the path length attenuation factor. , where L is the number of edges (length) of the path; Group all paths by sign and calculate the sum of positive path strength P for each group. zheng Sum of negative path strengths P fu Calculate the net effect value ; When any net effect value is lower than a preset threshold, an alert is triggered for that pair, and the indicator pair (A,B) and the net effect value are output. If we only focus on the changes of a single indicator or simple correlations, we may easily overlook the hidden phenomenon of multiple paths canceling each other out. For example, after governance, data integrity is improved (i.e., positive path), but at the same time, data processing latency increases (i.e. negative path), resulting in a small net improvement in business efficiency. If no net effect threshold warning is set, managers may mistakenly believe that the governance is successful. In this case, by setting a threshold, once the net effect value does not meet the standard, the system will automatically issue a warning, indicating that there is a positive and negative path conflict in the tuple, so as to adjust the governance strategy in time.

[0020] As a third embodiment of this application, this embodiment further discloses a method for filtering associated edges from undirected edges, based on embodiments one and two. Figure 2 As shown, the specific content includes: For each evaluation indicator D i (i∈[1,N] and are integers) are discretized using equal-frequency binning, their probability distribution is statistically analyzed, and their information entropy H is calculated. i Where N is the number of evaluation indicators; For any two evaluation indicators D i and D j Statistically analyze their joint probability distribution and calculate the mutual information MI. ij ; The calculated mutual information values ​​are normalized, i.e. ,in, To evaluate the normalized mutual information between indicators Di and Dj, Hi is the information entropy of indicator Di, Hj is the information entropy of indicator Dj, and min() is the minimum value function used to take the smaller value between Hi and Hj. For all The probability density function is obtained by Gaussian kernel density estimation. Find the first derivative of its logarithmic density. , find Get the minimum value Value, denoted as If there are multiple points with the minimum value, the largest one is taken. Value as ,in, Normalized mutual information value (i.e.) (a general term) For probability density function Take the natural logarithm. For the independent variable The first derivative, It is the first derivative of the logarithmic probability density; This point corresponds to the curvature singularity of the probability density curve when it transitions from a high-density region (noise correlation) to a low-density region (true correlation), which is the position where the density decreases the most gently and is also the natural boundary between noise and signal. Keep all By using the index pairs, we obtain the candidate set of associated edges C; For each evaluation indicator D i Calculate the variance Vari of its normalized mutual information with all other evaluation metrics, specifically expressed as follows: ,in, Let Di be the average of the normalized mutual information between the indicator Di and all other indicators, and its specific expression is: , This represents the summation of all indices j except i; Calculate the divergence penalty factor Where max(Var) represents the maximum value among all Varis and the maximum variance among all evaluation metrics; The larger the penalty factor, the higher the index D. i The more significant the difference in the correlation strength among different indicators, the more specific and reliable the correlation is. For each edge (i,j) in the candidate associated edge set C, define an initial retention probability. ,in, Let be the initial retention probability of edge (i,j) before the start of the iteration. , Let Di and Dj be the divergence penalty factors. For all edges (p, q) in the candidate set C, compute... And take the maximum value among them as the denominator; Set the iteration round r, with a value range of [0, ∞), and the update rules for each round are as follows: Calculate the competition pressure coefficient for each edge under the current probability. Its specific expression is ,in, Let i be the number of strongly related edges to the index node i in the r-th iteration, and its specific expression is: The This is an example function; it takes the value 1 if the condition within the parentheses is true, and 0 otherwise. This represents summing over all nodes j connected to node i in the candidate set C. Let be the retention probability of edge (i,j) in the r-th round; Based on the competition pressure coefficient Update the retention probability of edge (i,j) in the (r+1)th round. Its specific expression is ,in, The learning rate is exp(), which is an exponential function. The specific meaning of this formula is: if the two endpoints of an edge already have many other strongly related edges, then the competition pressure coefficient... If the value is small, the exponent term is close to 1, and the probability remains unchanged; otherwise, the probability is suppressed. when When the time is reached, stop the iteration and denote the final probability as . ,in, This indicates that for all edges (i,j) in the candidate associated edge set C, take... The maximum value, This is the preset error value; Keep all By considering the edges, we obtain the final set of associated edges E.

[0021] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0022] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for evaluating the effectiveness of data governance, characterized in that, include: Step 1: Collect continuous time series of all evaluation indicators within one complete business cycle before and after the implementation of the governance action. Discretize the continuous time series of each evaluation indicator by equal frequency binning. Connect any two indicators with undirected edges. Based on the mutual information value between the two indicators, select the related edges from all undirected edges and retain them. At the same time, mark the polarity of each related edge and output the undirected dependency network before and after governance respectively. Step 2: Based on the undirected dependent network before and after governance, for any two indicators X and Y connected by an undirected edge, traverse the third indicator Z in the network other than X and Y, and use it as a candidate condition variable. Calculate the conditional mutual information CMI(X,Y|Z) of each triple (X,Y,Z), and determine the causal direction based on the conditional mutual information. Assign a unique direction to each undirected edge to obtain the causal graph C_before before governance and the causal graph C_after after governance. Step 3: Iterate through each evaluation index A, extract all incoming and outgoing edges in C_before and C_after, and extract newly added edges, disappearing edges, and flipped edges. At the same time, identify all pairs (A,B) with mutual cancellation effects. For each identified pair, calculate the net effect value based on C_after. When any net effect value is lower than a preset threshold, trigger an alert for that pair and output the index pair (A,B) and the net effect value.

2. The data governance effectiveness evaluation method according to claim 1, characterized in that, The specific operation for filtering out associated edges from all undirected edges is as follows: For each evaluation indicator D i Perform equal-frequency binning discretization, statistically analyze its probability distribution, and calculate its information entropy H. i , where i∈[1,N] and is an integer, and N is the number of evaluation indicators; For any two evaluation indicators D i and D j Statistically analyze their joint probability distribution and calculate the mutual information MI. ij Based on normalized mutual information values Obtain the candidate associated edge set C; For each evaluation indicator D i Calculate the variance Var of its normalized mutual information value with all other evaluation metrics. i ; Calculate the divergence penalty factor Where max(Var) represents the maximum variance among all evaluation indicators; For each edge (i,j) in the candidate associated edge set C, define an initial retention probability and obtain the final probability of the edge according to the iterative update rule; Retain all edges with a final probability greater than 0.5 to obtain the final set of associated edges E.

3. The data governance effectiveness evaluation method according to claim 2, characterized in that, The specific operation to obtain the candidate associated edge set C is as follows: For all The probability density function is obtained by Gaussian kernel density estimation, and the first derivative of its logarithmic density is calculated. The method that minimizes the first derivative is then found. Value, denoted as If there are multiple points with the minimum value, the largest one is taken. Value as ; All rights reserved The index pairs are used to obtain the candidate associated edge set C.

4. The data governance effectiveness evaluation method according to claim 2, characterized in that, The specific steps to obtain the final probability of the edge according to the iterative update rule are as follows: Set the iteration round r, with a value range of [0, ∞), and the update rules for each round are as follows: Calculate the competition pressure coefficient for each edge under the current probability. ,in, This represents the number of strongly related edges to node i in the r-th iteration, specifically expressed as: , The following represents the retention probability of edge (i,j) in the r-th iteration. This is an example function; it takes the value 1 when the condition within the parentheses is true, and 0 otherwise. Based on the competition pressure coefficient Update retention probability ,in, The learning rate; Among all candidate edges, if the maximum absolute value of the change in the retention probability between two adjacent iterations is less than a preset difference ε, then... When the iteration stops, the final probability is obtained, where max is the maximum value function.

5. The data governance effectiveness evaluation method according to claim 1, characterized in that, The specific rules for marking the polarity of each associated edge are as follows: For each of the selected associated edges, the two indicators D are connected. i and D j Calculate the joint probability matrix and the product matrix of independent probabilities, and obtain the difference matrix by subtracting the two. Then calculate the sum of the main diagonal and the sum of the secondary diagonal of the difference matrix respectively. If the sum of the main diagonals is greater than the sum of the secondary diagonals, then the polarity of the side is marked as positive; if the sum of the main diagonals is less than the sum of the secondary diagonals, then the polarity of the side is marked as negative; if the sum of the main diagonals is equal to the sum of the secondary diagonals, then the side is marked as non-polar.

6. The data governance effectiveness evaluation method according to claim 1, characterized in that, The specific operation for determining causal direction based on conditional mutual information is as follows: For an edge connecting X and Y in an undirected dependency network, iterate through all third-party metrics Z and calculate the conditional dependency decay for each Z. Difference with asymmetric information flow ; Calculate the local support score of Z for the direction X→Y. Where CMI(X,Y|Z) is the conditional mutual information value, and sgn() is the sign function; The direction discrimination factor is calculated based on the average local support score of each Z: if the direction discrimination factor > η, the direction is determined to be X→Y; if the direction discrimination factor < -η, the direction is determined to be Y→X; if the absolute value of the direction discrimination factor ≤ η, it is marked as having no definite direction, where η is a preset threshold and is a positive number.

7. The data governance effectiveness evaluation method according to claim 6, characterized in that, Count the number of directions given by Z that support X→Y, n XY and the number n that supports Y→X YX If n XY >n YX Then assign the direction X→Y to the undirected edge; if n YX >n XY Then assign the direction Y→X to the undirected edge; if n XY =n YX If so, then the edge is temporarily kept as undirected.

8. The data governance effectiveness evaluation method according to claim 1, characterized in that, The specific steps to identify pairs (A, B) with mutually canceling effects are as follows: For the causal graph C_after after governance, a depth-first search is performed starting from each index node to enumerate all directed paths from that node to all other nodes, and the node sequence and path symbol of each path are recorded. The path symbol is defined as the product of the associative polarities of all directed edges on the path, where a positive associative polarity is 1 and a negative associative polarity is -1. If a path contains a directed edge marked as non-polar, skip that path and use only the edges marked as polar. At the same time, the maximum path length is limited to no more than the total number of nodes in the graph; After the enumeration is completed, for each pair (A,B), the set of path symbols is counted. If the set of path symbols contains both -1 and 1, then the pair (A,B) is determined to have a mutual cancellation effect.

9. The data governance effectiveness evaluation method according to claim 1, characterized in that, The specific steps for calculating the net effect value based on C_after are as follows: For the binary pair (A, B) where mutual cancellation effects have been identified, all directed paths from A to B are extracted from the treated causal graph C_after. For each path, the path strength is calculated, which is the product of the mutual information values ​​of all edges on the path, multiplied by the path length attenuation factor. , where L is the number of edges in the path; Group all paths by sign and calculate the sum of positive path strength P for each group. zheng Sum of negative path strengths P fu Calculate the net effect value .

Images