A data quality inspection rule validity evaluation and feedback optimization method, storage medium and system

Abstract

The present invention provides a data quality check rule validity evaluation and feedback optimization method, storage medium and system. The method includes: selecting a first data quality check rule to be evaluated from a data quality check rule set, constructing a data quality check rule evaluation system according to multiple quality rule evaluation indicators, evaluating the first data quality check rule using each quality rule evaluation indicator to obtain a score, calculating the validity evaluation result based on the score and weight distribution of each quality rule evaluation indicator, if the validity evaluation result does not reach a first preset level, finding a second data quality check rule with the highest similarity from a data quality knowledge base, if the similarity reaches a second preset level, optimizing the first data quality check rule using optimization knowledge associated with the second data quality check rule, and then updating the data quality check rule set according to the optimized first data quality check rule.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data quality inspection rule validity evaluation and feedback optimization method, storage medium and system. Background Art

[0002] A data quality check rule set refers to a collection of data quality check rules defined by the user based on data standards. The data quality check rules can be read and executed by the data quality engine to perform data quality checks. The data quality engine then stores the results of the checks in the monitoring result table for user monitoring.

[0003] Different types of data may be associated with different data standards. If the data quality check rules read by the data quality engine during the data quality check process do not match the data standards associated with the data being checked, the data quality check results obtained using the read data quality check rules will deviate from the actual results. For example, if a field's character length is 10 and the associated data standard is that the character length must not exceed 9, and the data quality check rule defined according to this data standard checks whether the field's character length exceeds 9, then the data quality check result obtained using this data quality check rule should actually be a failure. However, the data quality check rule set may not contain a data quality check rule that checks whether the field's character length exceeds 9. In this case, during the data quality check, the data quality engine will read a different data quality check rule with the highest similarity, such as a rule that checks whether the field's character length exceeds 10. The data quality check result obtained using this read data quality check rule is a pass, which deviates from the actual results. Summary of the Invention

[0004] The technical problem to be solved by the present invention is how to reduce the deviation of the results of data quality inspection.

[0005] To solve the above technical problems, the present invention provides a data quality inspection rule effectiveness evaluation and feedback optimization method, comprising the following steps:

[0006] A. Selecting a first data quality check rule to be evaluated from the data quality check rule set;

[0007] B. Obtaining multiple quality rule evaluation indicators that can evaluate the first data quality inspection rule from different dimensions, and constructing a data quality inspection rule evaluation system based on the multiple quality rule evaluation indicators;

[0008] C. Evaluate the first data quality check rule using various quality rule evaluation indicators in the data quality check rule evaluation system to obtain a score for each quality rule evaluation indicator;

[0009] D. Obtaining the weight distribution of each quality rule evaluation indicator, and calculating the effectiveness evaluation result of the first data quality inspection rule based on the score of each quality rule evaluation indicator;

[0010] E. Obtaining multiple pieces of optimization knowledge generated by optimizing multiple preset data quality check rules, and constructing a data quality knowledge base based on the multiple pieces of optimization knowledge;

[0011] F. Determining whether the effectiveness evaluation result reaches a first preset level; if not, analyzing the similarity between other data quality check rules associated with each piece of optimization knowledge in the data quality knowledge base and the first data quality check rule, and finding a second data quality check rule with the highest similarity;

[0012] G. Determining whether the similarity between the first data quality check rule and the second data quality check rule reaches a second preset level; if so, optimizing the first data quality check rule using optimization knowledge associated with the second data quality check rule; if not, obtaining optimization knowledge for the first data quality check rule input by a user, optimizing the first data quality check rule, and adding the optimization knowledge for the first data quality check rule to the data quality knowledge base;

[0013] H. Update the data quality check rule set according to the optimized first data quality check rule.

[0014] Preferably, the quality rule evaluation indicators include normative indicators, operational indicators and audit indicators.

[0015] Preferably, the normative indicators include standard consistency indicators and integrity indicators, the operational indicators include operational efficiency indicators and operational result indicators, and the audit indicators include result accuracy indicators and business consistency indicators.

[0016] Preferably, in the step D, a hierarchical structure model is first constructed according to the data quality inspection rule evaluation system, and then the weight distribution of the indicators of each layer of the hierarchical structure model is calculated based on the hierarchical analysis method.

[0017] Preferably, in the step D, calculating the weight distribution of the indicators of each layer of the hierarchical structure model based on the hierarchical analysis method includes the following steps:

[0018] D1. Construct a judgment matrix based on the indicators of each layer in the hierarchical structure model;

[0019] D2. Calculate the eigenvector and maximum eigenvalue of the judgment matrix;

[0020] D3. Use the maximum eigenvalue to perform a consistency check on the judgment matrix. If the consistency check passes, use the elements in the eigenvector as the weight distribution of the indicators corresponding to the judgment matrix.

[0021] Preferably, in step D1, the judgment matrix is ​​constructed by using the importance comparison values ​​between each layer of indicators as elements of the judgment matrix.

[0022] Preferably, in step D2, the product square root of each row of the judgment matrix is ​​first calculated, and then the ratio of the product square root of each row of elements to the sum of all product square roots is calculated, and the ratio is used as the element of the eigenvector to form the eigenvector.

[0023] Preferably, in step F, if it is determined that the validity evaluation result reaches a first preset level, the second data quality check rule having the highest similarity with the first data quality check rule is not found, and steps G and H are not performed.

[0024] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned data quality check rule validity evaluation and feedback optimization method.

[0025] The present invention also provides a data quality inspection rule effectiveness evaluation and feedback optimization system, comprising a computer-readable storage medium and a processor connected to each other, wherein the computer-readable storage medium is as described above.

[0026] The present invention has the following beneficial effects: if the validity evaluation result of the first data quality check rule does not reach the first preset level, it means that the result obtained by using the first data quality check rule to perform data quality check will deviate from the actual situation, so the second data quality check rule with the highest similarity to the first data quality check rule is found. When the similarity reaches the second preset level, the first data quality check rule is optimized using the optimization knowledge associated with the second data quality check rule. When the similarity does not reach the second preset level, the optimization knowledge for the first data quality check rule is used for optimization, and the optimization knowledge for the first data quality check rule is added to the data quality knowledge base. Then, the data quality check rule set is updated according to the optimized first data quality check rule. In this way, the optimized first data quality check rule in the data quality check rule set can be used to perform data quality check in the future, and the deviation of the obtained result from the actual situation will be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of the effectiveness evaluation and feedback optimization method of data quality inspection rules;

[0028] Figure 2 This is a flowchart of the construction process of the data quality inspection rule evaluation system;

[0029] Figure 3 It is a schematic diagram of the hierarchical model;

[0030] Figure 4 It is a reference representation of the values ​​of the random consistency indicators of the same order in the hierarchical analysis method. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below in conjunction with specific embodiments.

[0032] A data quality check rule set refers to a collection of data quality check rules defined by the user according to data standards, wherein the data quality check rules can be read and executed by the data quality engine to perform data quality checks, and then the data quality engine stores the results of the checks in a monitoring result table for the user to monitor. In order to reduce the deviation of the results of the data quality check, it is necessary to evaluate the effectiveness and optimize the feedback of the data quality check rules in the data quality check rule set. To this end, this embodiment provides a data quality check rule effectiveness evaluation and feedback optimization system, which includes a computer-readable storage medium and a processor connected to each other, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the following is implemented: Figure 1 The data quality inspection rule effectiveness evaluation and feedback optimization method shown includes the following steps A, B, C, D, E, F, G, and H.

[0033] A. Select a first data quality check rule to be evaluated from the data quality check rule set.

[0034] There are multiple data quality check rules in the data quality check rule set. When the system evaluates the effectiveness of the data quality check rules, it selects the first data quality check rule to be evaluated from the data quality check rule set. The first data quality check rule, for example, checks whether the character length of the target data object exceeds 10.

[0035] B. Obtain multiple quality rule evaluation indicators that can evaluate the first data quality check rule from different dimensions, and construct a data quality check rule evaluation system based on the multiple quality rule evaluation indicators.

[0036] like Figure 2As shown, for the three dimensions of rule specification, rule operation, and audit results of the data quality check rules, the system obtains three quality rule evaluation indicators that can evaluate the first data quality check rule from these three dimensions respectively, and sorts out these three quality rule evaluation indicators to construct a data quality check rule evaluation system. In this way, the three quality rule evaluation indicators in the data quality check rule evaluation system are: normative indicators for the rule specification dimension, operational indicators for the rule operation dimension, and audit indicators for the audit result dimension.

[0037] The normative indicators are mainly analyzed from two aspects: the consistency of the relationship between the first data quality check rule and the data standard associated with the target data object it checks, which is also known as the standard consistency indicator; the completeness of the SQL script description specification of the first data quality check rule, which is also known as the integrity indicator. Specifically: the standard consistency indicator is to analyze whether the inspection condition parameters in the first data quality check rule are consistent with the inspection condition parameters defined by the data standard associated with the target data object checked by the first data quality check rule, such as analyzing whether the character length parameters in the data quality check rule of the character length detection type are consistent with the character length parameters defined by the data standard associated with the target data object checked by the rule; the integrity indicator is to analyze whether the description of the SQL script generated by the first data quality check rule complies with the SQL specification and whether it is complete, such as analyzing whether the SQL script of the first data quality check rule contains select, from, table name, table alias, field alias, conditional statements, etc.

[0038] The operation indicators are mainly analyzed from two aspects: the operation efficiency and the operation results of the first data quality check rule. They include operation efficiency indicators and operation result indicators. Specifically: the operation efficiency indicator refers to the analysis of the amount of data processed per unit time when the first data quality check rule is running, that is, the analysis of the data processing speed when the first data quality check rule is running, in units of items / minute; the operation result indicator refers to the analysis of the results of the success or failure of the first data quality check rule.

[0039] The audit indicators are mainly analyzed from the data quality problem results obtained from the operation of the first data quality check rule, including result accuracy indicators and business consistency indicators. Specifically: the result accuracy indicator refers to analyzing the proportion of operation results that are truly data quality problems in all operation result problem data by counting the operation result problem data of the first data quality check rule; the business consistency indicator refers to analyzing whether the inspection logic described in the first data quality check rule and the business inspection rule of the target data object it checks (a business logic that checks data quality rules described in text) is consistent, that is, analyzing whether the inspection condition parameters of the first data quality check rule meet the condition limitation requirements described in the business inspection rule.

[0040] C. Evaluate the first data quality check rule using various quality rule evaluation indicators in the data quality check rule evaluation system to obtain scores for various quality rule evaluation indicators.

[0041] After building the data quality check rule evaluation system, the system sorts out different quality rule evaluation indicators into quantifiable standards, and evaluates the first data quality check rule based on the quantified quality rule evaluation indicators to obtain scores for each quality rule evaluation indicator, for example:

[0042] Taking the standard consistency indicator as an example, the system analyzes whether the check condition parameters of the first data quality check rule are consistent with the check condition parameters defined in the data standard associated with the target data object checked by the first data quality check rule. If they are consistent, the standard consistency indicator score is 1; if they are inconsistent, the standard consistency indicator score is 0. In this embodiment, the character length parameter of the first data quality check rule is 10, while the character length parameter defined in the data standard associated with the target data object checked by the first data quality check rule is 9. If the two are inconsistent, the standard consistency indicator score c1 of the first data quality check rule is 0.

[0043] Taking the integrity indicator as an example, the system analyzes the proportion of descriptions in the SQL script generated by the first data quality check rule that comply with the SQL specification. This proportion is used as the integrity indicator score. The integrity indicator score is between [0, 1]. If the SQL script description fully complies with the SQL specification, the integrity indicator score is 1. If the SQL script description does not comply with the SQL specification at all, the integrity indicator score is 0. In this embodiment, if the description of the SQL script generated by the first data quality check rule fully complies with the SQL specification, the integrity indicator score of the first data quality check rule is c2 = 1.

[0044] Taking the operational efficiency index as an example, the system analyzes the amount of data processed per unit time when the first data quality check rule is running, that is, analyzes the data processing speed when the first data quality check rule is running: if the data processing speed is greater than 100 items / minute, the operational efficiency index score is 1; if the data processing speed is greater than 40 items / minute but less than 100 items / minute, the operational efficiency index score is 0.8; if the data processing speed is greater than 10 items / minute but less than 40 items / minute, the operational efficiency index score is 0.6; if the data processing speed is greater than 5 items / minute but less than 10 items / minute, the operational efficiency index score is 0.4; if the data processing speed is greater than 1 item / minute but less than 5 items / minute, the operational efficiency index score is 0.2; if the data processing speed is less than 1 item / minute, the operational efficiency index score is 0.1. In this embodiment, if the data processing speed of the first data quality check rule is greater than 40 items / minute but less than 100 items / minute, the operational efficiency index score c3 of the first data quality check rule is 0.8.

[0045] Taking the execution result indicator as an example, the system analyzes whether the first data quality check rule is executed successfully or failed. If the execution is successful, the execution result indicator score is 1, and if the execution fails, the execution result indicator score is 0. In this embodiment, if the first data quality check rule is executed successfully, the execution result indicator score of the first data quality check rule is c4 = 1.

[0046] Taking the result accuracy index as an example, the system collects statistics on the problem result data of the first data quality check rule and analyzes the proportion of the problem result data that are truly data quality issues. This proportion is used as the score of the result accuracy index. The score of the result accuracy index is between [0, 1]. If all the problem result data are data quality issues, the result accuracy index score is 1. If all the problem result data are not data quality issues, the result accuracy index score is 0. In this embodiment, among the problem result data of the first data quality check rule, the proportion of the problem result data that are truly data quality issues is 50%. Therefore, the result accuracy index score of the first data quality check rule is c5 = 0.5.

[0047] Taking the business consistency indicator as an example, the system analyzes whether the check logic described in the first data quality check rule and the business check rule of the target data object it checks is consistent. If they are consistent, the business consistency indicator score is 1; if they are inconsistent, the business consistency indicator score is 0. In this embodiment, the check logic described in the business check rule of the first data quality check rule and the business check rule of the target data object it checks is completely consistent, and the business consistency indicator score c6 of the first data quality check rule is 1.

[0048] D. Obtain the weight distribution of each quality rule evaluation indicator, and calculate the effectiveness evaluation result of the first data quality inspection rule based on the score of each quality rule evaluation indicator.

[0049] After obtaining the scores of each quality rule evaluation indicator, the system calculates the relative weights of each quality rule evaluation indicator in the data quality inspection rule evaluation system based on the Analytic Hierarchy Process (AHP). The Analytic Hierarchy Process (AHP) is a multi-objective decision analysis method that combines qualitative and quantitative analysis methods. The main idea of ​​this method is to decompose complex problems into several levels and several factors, make comparative judgments on the importance of each pair of indicators, establish a judgment matrix based on the indicators, and calculate the approximate solution and maximum eigenvalue of the eigenvector of the judgment matrix to obtain the weights of the importance of different solutions, providing a basis for selecting the best solution.

[0050] In the process of calculating the relative weights of various quality rule evaluation indicators, the system constructs the following according to the data quality inspection rule evaluation system: Figure 3 The hierarchical model shown in the figure includes a target layer, a criterion layer, and a solution layer. The target layer refers to the purpose of the decision and the problem to be solved, specifically the effectiveness of the data quality inspection rules evaluated by the data quality inspection rule evaluation system. The criterion layer refers to the factors considered and the criteria for decision-making, specifically the rule specifications, rule operation, and audit results of the data quality inspection rule evaluation system. The solution layer refers to the alternative solutions for decision-making, specifically the standard consistency, completeness, operation efficiency, operation results, result accuracy, and business consistency of the data quality inspection rule evaluation system. The system specifically performs the following steps D1, D2, and D3 to calculate the influence weights of the indicators at each layer of the hierarchical model:

[0051] D1. The system constructs a judgment matrix based on the criteria layer and solution layer in the hierarchical model. Specifically:

[0052] Since the three indicators in the criterion layer correspond to one indicator in the target layer, we only need to analyze the influence weight of the criterion layer on one indicator, and construct a total of one criterion layer matrix. Since the six indicators in the solution layer correspond to three indicators in the criterion layer, we need to analyze the influence weight of the solution layer on the three indicators, and construct a total of three solution layer matrices. Taking the criterion layer matrix as an example, the rule specification is recorded as x1, the rule operation is recorded as x2, and the audit result is recorded as x3. The criterion layer matrix is ​​constructed as follows:

[0053] In the judgment matrix, element x ij is x i with xj The importance comparison value of x i with x j Which one is more important? If x ij The value of is greater than 1, that is, x i ∶x j The value of is greater than 1, which means that x i x j Important, if x ij The value of is less than 1, that is, x i ∶x j The value of is less than 1, which means that x j x i important; and, x ij with x ji The values ​​of are reciprocal to each other, that is, x ij *x ji = 1. And, the element x ij The i in also reflects the element x ij The row number in the criterion layer matrix X, element x ij The j in also reflects the element x ij The column number in the criteria layer matrix X.

[0054] Then, in the above-mentioned criterion layer matrix X: the element x 11 Is the importance comparison value of x1 and x1, element x 12 Is the importance comparison value of x1 and x2, element x 13 Is the importance comparison value of x1 and x3, element x 21 Is the importance comparison value of x2 and x1, element x 22 Is the importance comparison value of x2 and x2, element x 23 is the importance comparison value of x2 and x3, element x 31 Is the importance comparison value of x3 and x1, element x 32 is the importance comparison value of x3 and x2, element x 33 is the importance comparison value of x3 and x3; and x 12 *x 21 =1,x 13 *x 31 =1,x 23 *x 32 =1,x 11 *x 11 =1,x 22 *x 22 =1,x 33 *x 33 =1, that is, 2x 11 =2x 22 =2x 33 =1, so x 11=x 22 =x 33 =1.

[0055] In this embodiment, the criterion layer matrix is, for example, Similarly, three program-level matrices can be constructed based on program-level indicators, for example: Scenario layer matrix corresponding to operating indicators Program-level matrix corresponding to audit indicators

[0056] D2. The system uses the analytic hierarchy process to calculate the eigenvector and maximum eigenvalue of the criterion layer matrix X. Specifically:

[0057] First calculate the product of each row of elements in the criterion layer matrix X, and then find the nth root of the product of each row of elements to get the square root of the product of each row of elements but Where n is the order of the criterion layer matrix X, specifically 3.

[0058] The above criterion layer matrix The square root of the product of the first row of elements Square root of the product of the elements in the second row The square root of the product of the elements in the third row

[0059] Then calculate the product square root of each row element The ratio w to the sum of the square roots of all products b i ,but With this ratio w i As the elements of the eigenvector W, the eigenvector W=(w1, w2, ..., w n ) T .

[0060] The above criterion layer matrix The square root of the product of the first row of elements The ratio to the sum of the square roots of all products b Square root of the product of the elements in the second row The ratio to the sum of the square roots of all products b The square root of the product of the elements in the third row The ratio to the sum of the square roots of all products b This can form the eigenvector W of the criterion layer matrix X = (0.7153, 0.0977, 0.1870) T .

[0061] Then calculate the vector XW obtained by multiplying the criterion layer matrix X and the eigenvector W, and then calculate the ratio of the vector XW to the eigenvector W. This ratio is the maximum eigenvalue λ of the criterion layer matrix X max The specific calculation formula is

[0062] The above criterion layer matrix and the eigenvector W = (0.7153, 0.0977, 0.1870) T Multiplying them together gives vector XW = (2.1472, 0.2934, 0.5612) T , calculate the ratio of vector XW to eigenvector W to get the maximum eigenvalue of the criterion layer matrix X

[0063] D3. Using the maximum eigenvalue λ of the criterion layer matrix X max Perform a consistency check on the criterion layer matrix X. If the consistency check passes, the weight distribution of each quality rule evaluation indicator in the criterion layer is obtained based on the elements in the feature vector W. Specifically:

[0064] In order to check the consistency of the criterion layer matrix X, we first define the consistency index CI, which is: Then, the same-order random consistency index RI is introduced to calculate the consistency ratio CR. If the consistency ratio CR is less than 0.1, the criterion layer matrix X passes the consistency check. If the consistency ratio CR is not less than 0.1, the criterion layer matrix X fails the consistency check.

[0065] The calculation formula of consistency ratio CR is as follows: The reference table of the values ​​of the same-order random consistency index RI of the hierarchical analysis method is as follows: Figure 4 , since the order of the criterion layer matrix X is 3, the value of the random consistency index RI of the same order is 0.58. Therefore, the consistency ratio is

[0066] Since the consistency ratio CR=0.0017<0.1, the criterion layer matrix X passes the consistency check, so the element w in the feature vector W i They are the weighted approximate solutions of the quality rule evaluation indicators in the criterion layer, that is, the weight of the influence of rule specifications on the effectiveness of data quality inspection rules is w1=0.7153, the weight of the influence of rule operation on the effectiveness of data quality inspection rules is w2=0.0977, and the weight of the influence of audit results on the effectiveness of data quality inspection rules is w3=0.1870.

[0067] In other embodiments, if the consistency ratio CR of the criterion layer matrix X is ≥ 0.1, the criterion layer matrix X fails the consistency check, and the element w in the feature vector W isi Since it is not an approximate solution to the weights of the quality rule evaluation indicators in the criterion layer, the criterion layer matrix X needs to be adjusted until it passes the consistency check.

[0068] Then, the system executes steps D2 and D3 for each quality rule evaluation indicator in the solution layer, that is, the weight distribution of each quality rule evaluation indicator in the solution layer is calculated using the same method. Specifically:

[0069] Program-level matrix corresponding to normative indicators The square root of the product of the first row of elements Square root of the product of the elements in the second row Scenario layer matrix corresponding to operating indicators The square root of the product of the first row of elements Square root of the product of the elements in the second row Program-level matrix corresponding to audit indicators The square root of the product of the first row of elements Square root of the product of the elements in the second row

[0070] Therefore, the solution layer matrix The square root of the product of the first row of elements The ratio to the sum of the square roots of all products b Square root of the product of the elements in the second row The ratio to the sum of the square roots of all products b This can form the eigenvector K1 = (0.75, 0.25) of the solution layer matrix Y1 T ; Solution layer matrix The square root of the product of the first row of elements The ratio to the sum of the square roots of all products b 0.2, the square root of the product of the elements in the second row The ratio to the sum of the square roots of all products b This can form the eigenvector K2 = (0.2, 0.8) of the solution layer matrix Y2 T . Solution level matrix The square root of the product of the first row of elements The ratio to the sum of the square roots of all products b Square root of the product of the elements in the second row The ratio to the sum of the square roots of all products b This can form the eigenvector K3 of the solution layer matrix Y3 = (0.1667, 0.8333) T ;

[0071] Since the solution layer matrix Y1, solution layer matrix Y2 and solution layer matrix Y3 are all second-order matrices, and second-order matrices do not have the problem of inconsistent element relationships, there is no need to perform consistency check. Therefore: the element k in the eigenvector K1 of the solution layer matrix Y1 is i They are the approximate solutions of the influence weights of the standard consistency index and the integrity index in the solution layer on the rules and specifications in the target layer, that is, the influence weight of the standard consistency index on the rules and specifications is k1 = 0.75, and the influence weight of the integrity index on the rules and specifications is k2 = 0.25; the element k in the eigenvector K2 of the solution layer matrix Y2 is i They are the approximate solutions of the influence weights of the operation efficiency index and the operation result index in the solution layer on the rule operation in the target layer, that is, the influence weight of the operation efficiency index on the rule operation is k3 = 0.2, and the influence weight of the operation result index on the rule operation is k4 = 0.8; the element k in the eigenvector K3 of the solution layer matrix Y3 is i They are the approximate solutions of the influence weights of the result accuracy index and the business consistency index in the solution layer on the audit results in the target layer, that is, the influence weight of the result accuracy index on the audit results is k5=0.1667, and the influence weight of the business consistency index on the audit results is k6=0.8333.

[0072] After calculating the weight distribution of each quality rule evaluation indicator at the criterion level and the weight distribution of each quality rule evaluation indicator at the solution level, the system combines these weight distributions with the scores of each quality rule evaluation indicator at the solution level to calculate the effectiveness evaluation result of the first data quality check rule. Specifically:

[0073] First, combine the weight distribution and scores of each quality rule evaluation indicator at the solution level to calculate the scores of each quality rule evaluation indicator at the criterion level. Specifically, the scores of the normative indicators at the criterion level are: The scores of the operating indicators in the criterion layer are as follows: The scores of the audit indicators in the criteria layer are as follows:

[0074] Then, according to the weight distribution and scores of each quality rule evaluation indicator at the solution layer, the total score of the effectiveness of the data quality check rules at the target layer is calculated, thereby obtaining the effectiveness evaluation result of the first data quality check rule. Specifically, the total score of the effectiveness of the data quality check rules at the target layer is

[0075] It should be noted that if the total score S of the target layer is greater than 0.8, the data quality check rule validity evaluation result is excellent; if the total score S of the target layer is greater than 0.6 but not greater than 0.8, the data quality check rule validity evaluation result is good; if the total score S of the target layer is greater than 0.4 but not greater than 0.6, the data quality check rule validity evaluation result is fair; if the total score S of the target layer is not greater than 0.4, the data quality check rule validity evaluation result is poor. In this embodiment, the total score S of the target layer = 0.44403055, then 0.4 < S ≤ 0.6, so the validity evaluation result of the first data quality check rule is fair.

[0076] E. Acquire multiple pieces of optimization knowledge generated by optimizing multiple preset data quality check rules, and construct a data quality knowledge base based on the multiple pieces of optimization knowledge.

[0077] In this embodiment, the system uses optimization knowledge in the data quality knowledge base to optimize data quality check rules whose effectiveness is assessed as fair or poor. It should be noted that the data quality knowledge base contains multiple pieces of optimization knowledge, each corresponding to a data quality check rule. For example, the optimization knowledge generation process involves the following: For example, the correct data quality check rule should be "Check whether the character length of field data exceeds 9," but the data quality engine reads the incorrect data quality check rule "Check whether the character length of field data exceeds 11." Therefore, the incorrect data quality check rule must be optimized. Initially, the user manually searches the database and finds that the correct data quality check rule is "Check whether the character length of field data exceeds 9." Therefore, the character length parameter "11" in the incorrect data quality check rule is manually changed to "9." This generates an optimization knowledge piece, "Reduce the character length parameter to 9." This optimization knowledge piece is associated with the data quality check rule "Check whether the character length of field data exceeds 11." This optimization knowledge piece is then added to the knowledge base. In this way, after obtaining multiple pieces of optimization knowledge generated by optimizing multiple data quality check rules, a data quality knowledge base containing multiple pieces of optimization knowledge can be constructed.

[0078] F. Determine whether the effectiveness evaluation result reaches the first preset level. If not, analyze the similarity between the first data quality check rule and other data quality check rules associated with each optimization knowledge in the data quality knowledge base, and find the second data quality check rule with the highest similarity.

[0079] In this embodiment, if the effectiveness evaluation result is excellent or good (i.e., S > 0.6), it means that the first preset level is reached; if the effectiveness evaluation result is average or poor (i.e., S ≤ 0.6), it means that the first preset level is not reached. The effectiveness evaluation result of the first data quality inspection rule is average (S = 0.44403055, 0.4 < S ≤ 0.6), and the first preset level is not reached, which means that the result obtained by performing data quality inspection using the first data quality inspection rule will deviate from the actual situation. Therefore, it is necessary to optimize the first data quality inspection rule. For this purpose, the system calculates the similarity between the data quality inspection rules associated with each optimization knowledge in the data quality knowledge base and the first data quality inspection rule based on the Euclidean distance algorithm. The specific calculation formula is So as to find the second data quality inspection rule with the highest similarity to the first data quality inspection rule. For example, the optimization knowledge associated with the found second data quality inspection rule is the above-mentioned "reduce the character length parameter to 9". It should be noted that the Euclidean distance algorithm is a conventional similarity algorithm in this field, and its calculation process will not be elaborated here.

[0080] In other embodiments, instead of the system finding the second data quality inspection rule with the highest similarity to the first data quality inspection rule, the user retrieves the second data quality inspection rule with the highest similarity to the first data quality inspection rule in the data quality knowledge base through manual database query.

[0081] G. Determine whether the similarity between the first data quality inspection rule and the second data quality inspection rule reaches the second preset level. If it reaches, optimize the first data quality inspection rule using the optimization knowledge associated with the second data quality inspection rule; if it does not reach, obtain the optimization knowledge input by the user for the first data quality inspection rule for optimization, and add the optimization knowledge for the first data quality inspection rule to the data quality knowledge base;

[0082] After finding the second data quality check rule with the highest similarity to the first data quality check rule, it is determined whether the similarity reaches a second preset degree, which is 80% for example. In this way, if the similarity reaches 80%, it means that the optimization knowledge associated with the second data quality check rule is applicable to the first data quality check rule, so the optimization knowledge associated with the second data quality check rule is used to reduce the character length parameter in the first data quality check rule to 9; if the similarity does not reach 80%, it means that the optimization knowledge associated with the second data quality check rule is not applicable to the first data quality check rule, and since the data quality check rules associated with all the optimization knowledge in the data quality knowledge base are Among them, the second data quality check rule has the highest similarity with the first data quality check rule, which means that the similarity between the data quality check rules associated with all the optimization knowledge in the data quality knowledge base and the first data quality check rule does not reach 80%, that is, all the optimization knowledge in the data quality knowledge base is not applicable to the first data quality check rule. Therefore, the user can input the optimization knowledge for the first data quality check rule according to experience, and the system will then use the optimization knowledge input by the user to optimize the first data quality check rule. Then, the system will add the optimization knowledge for the first data quality check rule input by the user to the data quality knowledge base, thereby expanding and improving the data quality knowledge base.

[0083] H. Update the data quality check rule set according to the optimized first data quality check rule.

[0084] After optimizing the first data quality check rule, the optimized first data quality check rule is collected into the data quality check rule set to update the data quality check rule set. In this way, the optimized first data quality check rule in the data quality check rule set can be used to perform data quality checks in the future, and the deviation between the obtained results and the actual situation will be reduced.

[0085] It should be noted that in the above step F, if the effectiveness evaluation result of the first data quality check rule is judged to be excellent or good (S>0.6), reaching the first preset level, it means that the result obtained by using the first data quality check rule to perform data quality check is no deviation from the actual situation, then there is no need to optimize the first data quality check rule, and there is no need to find the second data quality check rule with the highest similarity from the data quality knowledge base, and there is no need to execute the above steps G and H.

[0086] The above is only an embodiment of the invention and does not limit the scope of patent protection. Those skilled in the art can make non-substantial changes or substitutions based on the invention and still fall within the scope of patent protection.

Claims

1. A data quality inspection rule effectiveness evaluation and feedback optimization method, characterized by: The steps include: A. Selecting a first data quality check rule to be evaluated from the data quality check rule set; B. Obtaining multiple quality rule evaluation indicators that can evaluate the first data quality inspection rule from different dimensions, and constructing a data quality inspection rule evaluation system based on the multiple quality rule evaluation indicators; C. Evaluate the first data quality check rule using various quality rule evaluation indicators in the data quality check rule evaluation system to obtain a score for each quality rule evaluation indicator; D. Obtaining the weight distribution of each quality rule evaluation indicator, and calculating the effectiveness evaluation result of the first data quality inspection rule based on the score of each quality rule evaluation indicator; E. Obtaining multiple pieces of optimization knowledge generated by optimizing multiple preset data quality check rules, and constructing a data quality knowledge base based on the multiple pieces of optimization knowledge; F. Determining whether the effectiveness evaluation result reaches a first preset level; if not, analyzing the similarity between other data quality check rules associated with each piece of optimization knowledge in the data quality knowledge base and the first data quality check rule, and finding a second data quality check rule with the highest similarity; G. Determining whether the similarity between the first data quality check rule and the second data quality check rule reaches a second preset level; if so, optimizing the first data quality check rule using optimization knowledge associated with the second data quality check rule; if not, obtaining optimization knowledge for the first data quality check rule input by a user, optimizing the first data quality check rule, and adding the optimization knowledge for the first data quality check rule to the data quality knowledge base; H. updating the data quality check rule set according to the optimized first data quality check rule; In the step D, a hierarchical model is first constructed according to the data quality inspection rule evaluation system, and then the weight distribution of the indicators of each layer of the hierarchical model is calculated based on the hierarchical analysis method. The weight distribution of the indicators of each layer of the hierarchical model is calculated based on the hierarchical analysis method, which includes the following steps: D1. Construct a judgment matrix based on the indicators of each layer in the hierarchical structure model; D2. Calculate the eigenvector and maximum eigenvalue of the judgment matrix; D3. Use the maximum eigenvalue to perform a consistency check on the judgment matrix. If the consistency check passes, the elements in the eigenvector are used as the weight distribution of the indicators corresponding to the judgment matrix; In step D1, the importance comparison values ​​between each layer of indicators are used as elements of the judgment matrix to construct the judgment matrix; in step D2, the product square root of each row of elements in the judgment matrix is ​​first calculated, and then the ratio of the product square root of each row of elements to the sum of all product square roots is calculated, and the ratio is used as the element of the eigenvector to construct the eigenvector; In step F, if it is determined that the validity evaluation result reaches the first preset level, the second data quality check rule having the highest similarity with the first data quality check rule is not found, and steps G and H are not performed.

2. The data quality inspection rule effectiveness evaluation and feedback optimization method according to claim 1 is characterized in that: The quality rule evaluation indicators include normative indicators, operational indicators and audit indicators.

3. The data quality inspection rule effectiveness evaluation and feedback optimization method according to claim 2 is characterized in that: The normative indicators include standard consistency indicators and integrity indicators, the operational indicators include operational efficiency indicators and operational result indicators, and the audit indicators include result accuracy indicators and business consistency indicators.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data quality check rule effectiveness evaluation and feedback optimization method according to any one of claims 1 to 3 are implemented.

5. A data quality check rule effectiveness evaluation and feedback optimization system, comprising a computer-readable storage medium and a processor connected to each other, characterized in that: The computer-readable storage medium as claimed in claim 4.

Images