Data quality assessment method, device, storage medium and electronic device

Abstract

The data quality assessment method, device, storage medium and electronic device provided by the embodiment of the present application belong to the field of data governance technology. The method can obtain data to be evaluated, and perform data exploration on the data to be evaluated, obtain data information corresponding to the data to be evaluated, respond to the selection instruction of the target evaluation index, determine the target evaluation index corresponding to the data to be evaluated, according to the correlation between the target evaluation index and the evaluation rule, determine the target evaluation rule corresponding to the data to be evaluated, and perform quality assessment on the data to be evaluated according to the target evaluation rule and data information, obtain the rule evaluation score corresponding to the data to be evaluated, according to the rule weight corresponding to each target evaluation rule, respectively determine each indicator evaluation score corresponding to the data to be evaluated, and according to the indicator weight corresponding to each target evaluation index, determine the quality assessment score corresponding to the data to be evaluated. Thus, the efficiency and accuracy of quality assessment of data can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data governance technology, and in particular, to a data quality assessment method, device, storage medium and electronic device. Background Art

[0002] With the advent of the big data era, there is an explosive growth of various data. At the same time, the value of data has been widely recognized in enterprises. However, not all data is valuable. At present, the common data quality problems in enterprises include: non-standard data, inaccurate data, incomplete data, inconsistent data, etc. The quality of data directly affects the value of data. High-quality data plays an indispensable role in business, management, and decision-making.

[0003] Most of the related technologies use manual methods to perform statistical analysis on data to obtain quality scores for data quality assessment. However, the manual method is inefficient and prone to erroneous assessment results. Summary of the invention

[0004] To solve the existing technical problems, the embodiments of the present application provide a data quality assessment method, device, storage medium and electronic device, which can improve the efficiency and accuracy of data quality assessment.

[0005] To achieve the above purpose, the technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a data quality assessment method, the method comprising:

[0007] Acquire the data to be evaluated, perform data exploration on the data to be evaluated, and obtain data information corresponding to the data to be evaluated;

[0008] In response to an instruction for selecting a target evaluation indicator, determining at least one target evaluation indicator corresponding to the data to be evaluated;

[0009] Determine the target evaluation rule corresponding to the data to be evaluated according to the association relationship between the target evaluation indicator and the evaluation rule;

[0010] Performing a quality assessment on the data to be assessed according to the target assessment rule and the data information, and obtaining a rule assessment score and quality assessment information corresponding to the data to be assessed;

[0011] According to the rule weights corresponding to the various target evaluation rules included in the target evaluation indicators, the indicator evaluation scores of the various target evaluation indicators corresponding to the data to be evaluated are determined, and according to the indicator weights corresponding to the various target evaluation indicators, the quality evaluation scores corresponding to the data to be evaluated are determined.

[0012] The data quality assessment method provided by the embodiment of the present application can obtain the data to be assessed, and perform data exploration on the data to be assessed to obtain the data information corresponding to the data to be assessed, and in response to the selection instruction of the target assessment index, at least one target assessment index corresponding to the data to be assessed can be determined, and the target assessment rule corresponding to the data to be assessed can be determined according to the association relationship between the target assessment index and the assessment rule, and the quality assessment of the data to be assessed can be performed according to the target assessment rule and the data information, and the rule assessment score and quality assessment information corresponding to the data to be assessed can be obtained, and then the index assessment score of each target assessment index corresponding to the data to be assessed can be determined according to the rule weight corresponding to each target assessment rule included in the target assessment index, and the quality assessment score corresponding to the data to be assessed can be determined according to the index weight corresponding to each target assessment index. Since the data can be automatically evaluated for quality according to the target assessment rules corresponding to multiple target assessment indicators, the efficiency of quality assessment of the data can be improved, and a more accurate data quality assessment result can be obtained.

[0013] In an optional embodiment, the performing quality assessment on the data to be assessed according to the target assessment rule and the data information includes:

[0014] According to the description information in the target evaluation rule, the rule content corresponding to the description information is determined, and the quality of the data to be evaluated is evaluated according to the rule content and the data information.

[0015] In this embodiment, the rule content corresponding to the description information can be determined according to the description information in the target evaluation rule, and the quality of the data to be evaluated can be evaluated according to the rule content and data information. Since data information can be obtained by performing data exploration on the data, and the quality of the data can be evaluated more accurately according to the rule content and data information corresponding to multiple evaluation rules, the accuracy of the quality evaluation of the data can be improved.

[0016] In an optional embodiment, determining the rule content corresponding to the description information according to the description information in the target evaluation rule includes:

[0017] If the description information indicates that the rule content is located in the data standard knowledge base, the rule content is searched from the data standard knowledge base.

[0018] In this embodiment, if the description information indicates that the rule content is located in the data standard knowledge base, the rule content is found from the data standard knowledge base, thereby expanding the evaluation rules for data quality evaluation, thereby discovering more quality problems in the data and improving the accuracy of data quality evaluation.

[0019] In an optional embodiment, searching the rule content from the data standard knowledge base includes:

[0020] Matching the data to be evaluated with the standard data format and standard data type in the data standard knowledge base, determining the target standard data format and target standard data type corresponding to the data to be evaluated, and determining the rule content corresponding to the target evaluation rule according to the target standard data format and target standard data type; or

[0021] The data to be evaluated is matched with the standard data format and standard data type in the data standard knowledge base, the target standard data format and target standard data type corresponding to the data to be evaluated are determined, and the rule content matching the target standard data format and the target standard data type is searched in the data element rule knowledge base.

[0022] In this embodiment, the data to be evaluated can be matched with the standard data format and standard data type in the data standard knowledge base to determine the target standard data format and target standard data type corresponding to the data to be evaluated, and the rule content corresponding to the target evaluation rule can be determined according to the target standard data format and target standard data type, or the data to be evaluated can be matched with the standard data format and standard data type in the data standard knowledge base to determine the target standard data format and target standard data type corresponding to the data to be evaluated, and the rule content matching the target standard data format and target standard data type can be searched in the data element rule knowledge base. Since the data standard knowledge base and the data element rule knowledge base are added, the detection scope of data quality evaluation can be expanded to discover more quality problems in the data.

[0023] In an optional embodiment, determining the indicator evaluation score of each target evaluation indicator corresponding to the data to be evaluated according to the rule weight corresponding to each target evaluation rule included in the target evaluation indicator, and determining the quality evaluation score corresponding to the data to be evaluated according to the indicator weight corresponding to each target evaluation indicator, includes:

[0024] According to the rule weights corresponding to the various target evaluation rules included in the target evaluation index, weighted average is performed on the evaluation scores of the various rules to obtain the indicator evaluation scores of the various target evaluation indicators corresponding to the data to be evaluated;

[0025] According to the indicator weight corresponding to each target evaluation indicator, each indicator evaluation score is weighted averaged to obtain the quality evaluation score corresponding to the data to be evaluated.

[0026] In this embodiment, the weighted average of each rule evaluation score can be performed according to the rule weight corresponding to each target evaluation rule included in the target evaluation index to obtain the indicator evaluation score of each target evaluation index corresponding to the data to be evaluated, and the weighted average of each indicator evaluation score can be performed according to the indicator weight corresponding to each target evaluation index to obtain the quality evaluation score corresponding to the data to be evaluated. Since the data quality evaluation score under each target evaluation index can be obtained and the comprehensive quality evaluation score of the data quality evaluation can be obtained according to each target evaluation index, a more comprehensive and accurate data quality evaluation result can be obtained.

[0027] In an optional embodiment, the rule weights corresponding to the various target evaluation rules included in the target evaluation index and the indicator weights corresponding to the various target evaluation indicators are obtained by constructing a judgment matrix through a hierarchical analysis method.

[0028] In this embodiment, since the rule weights corresponding to each target evaluation rule and the indicator weights corresponding to each target evaluation indicator included in the target evaluation index are obtained by constructing a judgment matrix through the hierarchical analysis method, more reasonable rule weights and indicator weights can be obtained.

[0029] In an optional embodiment, after determining the quality assessment score corresponding to the data to be assessed, the method further includes:

[0030] Output the quality assessment result corresponding to the data to be evaluated; the quality assessment result includes the indicator assessment scores of each target assessment indicator corresponding to the data to be evaluated, the quality assessment score corresponding to the data to be evaluated, and part or all of the quality assessment information corresponding to the data to be evaluated.

[0031] In this embodiment, after determining the quality assessment score corresponding to the data to be assessed, the quality assessment result corresponding to the data to be assessed can also be output. Among them, the quality assessment result can include the indicator assessment score of each target assessment indicator corresponding to the data to be assessed, the quality assessment score corresponding to the data to be assessed, and part or all of the quality assessment information corresponding to the data to be assessed. Since the output quality assessment result includes the indicator assessment score, the comprehensive quality assessment score and the quality assessment information, the accuracy of the quality assessment of the data can be improved.

[0032] In a second aspect, an embodiment of the present application further provides a data quality assessment device, comprising:

[0033] A data information acquisition unit, used to acquire the data to be evaluated, perform data exploration on the data to be evaluated, and obtain data information corresponding to the data to be evaluated;

[0034] An evaluation indicator determination unit, configured to respond to an instruction for selecting a target evaluation indicator and determine at least one target evaluation indicator corresponding to the data to be evaluated;

[0035] An evaluation rule determination unit, used to determine the target evaluation rule corresponding to the data to be evaluated according to the association relationship between the target evaluation indicator and the evaluation rule;

[0036] A first result determination unit is used to perform quality assessment on the data to be assessed according to the target assessment rule and the data information, and obtain a rule assessment score and quality assessment information corresponding to the data to be assessed;

[0037] The second result determination unit is used to determine the indicator evaluation score of each target evaluation indicator corresponding to the data to be evaluated according to the rule weights corresponding to each target evaluation rule included in the target evaluation indicator, and determine the quality evaluation score corresponding to the data to be evaluated according to the indicator weights corresponding to each target evaluation indicator.

[0038] In an optional embodiment, the first result determination unit is specifically configured to:

[0039] According to the description information in the target evaluation rule, the rule content corresponding to the description information is determined, and the quality of the data to be evaluated is evaluated according to the rule content and the data information.

[0040] In an optional embodiment, the first result determination unit is further configured to:

[0041] If the description information indicates that the rule content is located in the data standard knowledge base, the rule content is searched from the data standard knowledge base.

[0042] In an optional embodiment, the first result determination unit is further configured to:

[0043] Matching the data to be evaluated with the standard data format and standard data type in the data standard knowledge base, determining the target standard data format and target standard data type corresponding to the data to be evaluated, and determining the rule content corresponding to the target evaluation rule according to the target standard data format and target standard data type; or

[0044] The data to be evaluated is matched with the standard data format and standard data type in the data standard knowledge base, the target standard data format and target standard data type corresponding to the data to be evaluated are determined, and the rule content matching the target standard data format and the target standard data type is searched in the data element rule knowledge base.

[0045] In an optional embodiment, the second result determination unit is specifically configured to:

[0046] According to the rule weights corresponding to the various target evaluation rules included in the target evaluation index, weighted average is performed on the evaluation scores of the various rules to obtain the indicator evaluation scores of the various target evaluation indicators corresponding to the data to be evaluated;

[0047] According to the indicator weight corresponding to each target evaluation indicator, each indicator evaluation score is weighted averaged to obtain the quality evaluation score corresponding to the data to be evaluated.

[0048] In an optional embodiment, the rule weights corresponding to the various target evaluation rules included in the target evaluation index and the indicator weights corresponding to the various target evaluation indicators are obtained by constructing a judgment matrix through a hierarchical analysis method.

[0049] In an optional embodiment, the device further includes an evaluation result output unit, which is used to:

[0050] Output the quality assessment result corresponding to the data to be evaluated; the quality assessment result includes the indicator assessment scores of each target assessment indicator corresponding to the data to be evaluated, the quality assessment score corresponding to the data to be evaluated, and part or all of the quality assessment information corresponding to the data to be evaluated.

[0051] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data quality assessment method of the first aspect is implemented.

[0052] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the processor implements the data quality assessment method of the first aspect.

[0053] The technical effects brought about by any one of the implementation methods in the second to fourth aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1 A flowchart of a data quality assessment method provided in an embodiment of the present application;

[0056] Figure 2 A flowchart of another data quality assessment method provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of the structure of a data quality assessment device provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of the structure of another data quality assessment device provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0061] It should be noted that the terms "including" and "having" and their variations involved in the documents of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0062] The technical solution provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0063] The present application embodiment provides a data quality assessment method, such as Figure 1 As shown, the following steps are included:

[0064] Step S101, obtaining the data to be evaluated, performing data exploration on the data to be evaluated, and obtaining data information corresponding to the data to be evaluated.

[0065] The data to be evaluated can be obtained, and data exploration can be performed on the data to be evaluated to obtain the data exploration and analysis results corresponding to the data to be evaluated, that is, the data information corresponding to the data to be evaluated. Specifically, data exploration is a way to provide data definition through automatic exploration, algorithm assistance, manual confirmation and modifiable methods. The data exploration and analysis results mainly include: basic data information (data volume, storage size, storage location), data table information (data table classification, data table structure information, data table partition information), field information (primary key information, foreign key information, key business fields), data standard exploration (standard data element, standard data item), etc.

[0066] Step S102, in response to an instruction to select a target evaluation indicator, determining at least one target evaluation indicator corresponding to the data to be evaluated.

[0067] The evaluation indicators may include a normative indicator, an accuracy indicator, a consistency indicator, and an integrity indicator. After the data to be evaluated is acquired, in response to a user selecting an instruction to select at least one evaluation indicator from among the normative indicator, the accuracy indicator, the consistency indicator, and the integrity indicator, at least one target evaluation indicator corresponding to the data to be evaluated may be determined.

[0068] Step S103, determining the target evaluation rule corresponding to the data to be evaluated according to the association relationship between the target evaluation indicator and the evaluation rule.

[0069] Each evaluation index may include multiple evaluation rules, and there is an association relationship between the evaluation rules and the evaluation index. Therefore, the target evaluation rules corresponding to the target evaluation index can be determined according to the association relationship between the evaluation rules and the evaluation index. For example, the definition of the normative index can be to detect whether the data complies with the data specification, and the evaluation rules included in the normative index can be to detect the standardization of the table name, table annotation, field name, field annotation, and field type of the data respectively; the definition of the accuracy index can be based on the specification requirements of the data element value domain, to detect whether the data content is accurate, and the evaluation rules included in the accuracy index can be to detect the length, format, value domain, type, and logic of the data respectively; the definition of the consistency index can be to detect whether the fields with an association relationship are consistent, and the evaluation rules included in the consistency index can be to detect the consistency of the code value association; the definition of the integrity index can be to detect whether the data is complete, and the evaluation rules included in the integrity index can be to detect the loss and omission of data. Assuming that the target evaluation indicators corresponding to the data to be evaluated are determined to be standardization indicators and accuracy indicators, the target evaluation rules corresponding to the data to be evaluated can be determined to be to respectively detect the standardization of the table name, table comments, and field name of the data to be evaluated, and to respectively detect the accuracy of the length, format, and type of the data to be evaluated, based on the association between the standardization indicators and accuracy indicators and the evaluation rules.

[0070] Step S104: perform quality assessment on the data to be assessed according to the target assessment rules and data information, and obtain rule assessment scores and quality assessment information corresponding to the data to be assessed.

[0071] According to the description information in the target evaluation rule, the rule content corresponding to the description information can be determined, and the quality of the data to be evaluated can be evaluated according to the rule content and the data information corresponding to the data to be evaluated. If the description information indicates that the rule content is located in the data standard knowledge base, the rule content can be found in the data standard knowledge base.

[0072] Specifically, when the target evaluation rule is an evaluation rule under a normative index, a consistency index, and an integrity index, the data to be evaluated can be matched with the standard data format and the standard data type in the data standard knowledge base to determine the target standard data format and the target standard data type corresponding to the data to be evaluated, and the rule content corresponding to the target evaluation rule can be determined based on the target standard data format and the target standard data type. For example, the description information of the target evaluation rule is to detect the normativeness of the naming of the data to be evaluated, and the description information indicates that the standard table name corresponding to the data needs to be obtained from the data standard knowledge base, then the data to be evaluated can be matched with the standard data format and the standard data type in the data standard knowledge base to determine the target standard data format and the target standard data type corresponding to the data to be evaluated, that is, to determine what kind of data the data to be evaluated is and the detailed data information of the data to be evaluated, and according to the target standard data format and the target standard data type, the corresponding rule content can be found in the data standard knowledge base, and then the data to be evaluated can be detected based on the rule content.

[0073] When the target evaluation rule is an evaluation rule under the accuracy index, the data to be evaluated can be matched with the standard data format and standard data type in the data standard knowledge base to determine the target standard data format and target standard data type corresponding to the data to be evaluated, and the rule content matching the target standard data format and target standard data type can be found in the data element rule knowledge base. For example, the description information of the target evaluation rule is to detect the accuracy of the length of the data to be evaluated, and the description information indicates that the accurate data length needs to be obtained from the data standard knowledge base, then the data to be evaluated can be matched with the standard data format and standard data type in the data standard knowledge base to determine the target standard data format and target standard data type corresponding to the data to be evaluated, and according to the target standard data format and target standard data type, the corresponding rule content can be found in the data standard knowledge base. If the rule content is located in the data element rule knowledge base, it is necessary to find the corresponding rule content in the data element rule knowledge base, and then detect the data to be evaluated according to the rule content.

[0074] In one embodiment, if the description information indicates that the rule content is not located in the data standard knowledge base, that is, the corresponding rule content can be directly determined based on the description information, then the quality assessment of the data to be assessed can be performed based on the rule content and the data information corresponding to the data to be assessed. For example, the target assessment rule can be used to detect the consistency of the code value association, and the data information corresponding to the data to be assessed can be obtained by performing data exploration on the data to be assessed, and the code and value included in the data to be assessed can be determined based on the data information, and then the target assessment rule can be used to perform a quality assessment on the consistency of the code value association of the code and value in the data to be assessed.

[0075] After quality assessment of the data to be assessed according to the target assessment rule and the data information corresponding to the data to be assessed, the rule assessment score and quality assessment information corresponding to the data to be assessed can be obtained. The rule assessment score corresponding to the data to be assessed is the score calculated according to the score calculation logic corresponding to the target assessment rule. For example, when the target assessment rule is to detect the accuracy of the length of the data, the logic for calculating the length validity score of the data can be: the total amount of data that meets the length detection standard / (the number of fields involved in the detection*the number of detection rows). The quality assessment information corresponding to the data to be evaluated is the problem information existing in the data to be evaluated determined according to the rule content corresponding to the target assessment rule. The problem information may include the problem level of the data to be evaluated determined according to the rule evaluation score corresponding to the data to be evaluated. For example, according to the rule evaluation score corresponding to the data to be evaluated, the problem level of the data to be evaluated can be determined to be high, medium, low, or information. Moreover, if the rule evaluation score corresponding to the data to be evaluated is (0 to 20) points, it can be determined that the problem level corresponding to the data to be evaluated is high. If the rule evaluation score corresponding to the data to be evaluated is [20 to 60) points, it can be determined that the problem level corresponding to the data to be evaluated is medium. If the rule evaluation score corresponding to the data to be evaluated is [60 to 100) points, it can be determined that the problem level corresponding to the data to be evaluated is low. If the rule evaluation score corresponding to the data to be evaluated is 100 points, it can be determined that the problem level corresponding to the data to be evaluated is information.

[0076] Step S105, determining the indicator evaluation scores of the target evaluation indicators corresponding to the data to be evaluated according to the rule weights corresponding to the target evaluation rules included in the target evaluation indicators, and determining the quality evaluation scores corresponding to the data to be evaluated according to the indicator weights corresponding to the target evaluation indicators.

[0077] According to the rule weights corresponding to the target evaluation rules included in the target evaluation index, the evaluation scores of each rule can be weighted averaged to obtain the indicator evaluation scores of each target evaluation index corresponding to the data to be evaluated. Then, according to the indicator weights corresponding to each target evaluation index, the evaluation scores of each indicator can be weighted averaged to obtain the quality evaluation score corresponding to the data to be evaluated.

[0078] For example, the target evaluation indicators corresponding to the data to be evaluated are indicator A and indicator B, and the evaluation rules included in indicator A are A1 and A2, and the evaluation rules included in indicator B are B1, B2 and B3. The rule evaluation score of evaluation rule A1 corresponding to the data to be evaluated is 90 points, the rule evaluation score of evaluation rule A2 corresponding to the data to be evaluated is 60 points, the rule evaluation score of evaluation rule B1 corresponding to the data to be evaluated is 40 points, the rule evaluation score of evaluation rule B2 corresponding to the data to be evaluated is 100 points, and the rule evaluation score of evaluation rule B3 corresponding to the data to be evaluated is 80 points. The indicator weight corresponding to indicator A is 0.7, the indicator weight corresponding to indicator B is 0.3, the rule weight corresponding to evaluation rule A1 is 0.6, the rule weight corresponding to evaluation rule A2 is 0.4, the rule weight corresponding to evaluation rule B1 is 0.4, the rule weight corresponding to evaluation rule B2 is 0.3, and the rule weight corresponding to evaluation rule B3 is 0.3. Then we can get the indicator evaluation score of indicator A corresponding to the data to be evaluated is 90×0.6+60×0.4=78 points, the indicator evaluation score of indicator B corresponding to the data to be evaluated is 40×0.4+100×0.3+80×0.3=70 points, and the quality evaluation score corresponding to the data to be evaluated is 78×0.7+70×0.3=75.6 points.

[0079] The target evaluation index includes rule weights corresponding to each target evaluation rule and indicator weights corresponding to each target evaluation indicator, which are obtained by constructing a judgment matrix through the hierarchical analysis method.

[0080] The Analytic Hierarchy Process (AHP) is a simple, flexible and practical multi-criteria decision-making method for quantitative analysis of qualitative problems. Its main content is the construction of the judgment matrix and its consistency test.

[0081] Experts scored the actual perceived quality of measurement data and the theoretically considered importance. The scoring used a 1-9 geometric scaling method to form a judgment matrix. The scale and meaning of the judgment matrix are shown in the following table. After forming the judgment matrix A according to the table, the consistency test process of the hierarchical analysis method weight solution is as follows:

[0082] The scale and meaning of the judgment matrix can be shown in Table 1:

[0083]

[0084] The judgment matrix A is

[0085] The eigenvector and the maximum eigenvalue are calculated, and the indicator weight corresponding to each evaluation indicator and the rule weight corresponding to each evaluation rule are calculated through data normalization.

[0086] The square root method is used to solve the approximate solution and maximum eigenvalue of the eigenvector of the judgment matrix A. The solution process first calculates the nth root of the product of each row of the judgment matrix:

[0087] Perform normalization to obtain the indicator weight corresponding to each evaluation indicator and the rule weight corresponding to each evaluation rule. Calculate the quotient of the nth root of the product of each row and the sum of the square roots to form the feature vector W i =(w1, w2, ..., w n ) T ,

[0088] Calculate the maximum eigenvalue λ of the judgment matrix A max , Where A is the judgment matrix, (AW) i Represents the i-th element of the vector AW.

[0089] Check the consistency of the judgment matrix. The calculation formula of the consistency test index CR is: Among them, RI is the random consistency index, which can be obtained by looking up the table. If CR<0.1, A passes the consistency test; if it fails, the original judgment matrix must be modified until the consistency test passes. At this time, the approximate solution of the eigenvector is the weight coefficient of each element.

[0090] In one embodiment, after determining the quality assessment score corresponding to the data to be assessed, the quality assessment result corresponding to the data to be assessed may also be output. The quality assessment result may include part or all of the indicator assessment scores of each target assessment indicator corresponding to the data to be assessed, the quality assessment score corresponding to the data to be assessed, and the quality assessment information corresponding to the data to be assessed.

[0091] In another embodiment, the data to be evaluated can be obtained before quality evaluation is performed on the data to be evaluated. Specifically, when the data to be evaluated is a table, the data to be evaluated can be obtained according to the following process: First, enter the basic information of the data source: the basic information of the data source is mainly the database configuration information used by the business system to store data, including but not limited to: IP address, database name, user name, password, etc.; secondly, collect the table structure: through the data source configuration information, collect the table structure information in the database for subsequent data exploration; finally, add the table to be evaluated: the table to be evaluated can be added individually or in batches, and added to the list to be explored and evaluated.

[0092] The data quality assessment method provided by the embodiment of the present application can obtain the data to be assessed, and perform data exploration on the data to be assessed, obtain the data information corresponding to the data to be assessed, and in response to the selection instruction of the target assessment index, determine at least one target assessment index corresponding to the data to be assessed, and determine the target assessment rule corresponding to the data to be assessed according to the association relationship between the target assessment index and the assessment rule, and perform quality assessment on the data to be assessed according to the target assessment rule and the data information, and obtain the rule assessment score and quality assessment information corresponding to the data to be assessed, and then determine the index assessment score of each target assessment index corresponding to the data to be assessed according to the rule weight corresponding to each target assessment rule included in the target assessment index, and determine the quality assessment score corresponding to the data to be assessed according to the index weight corresponding to each target assessment index. Since the data can be automatically evaluated for quality according to the assessment rules corresponding to multiple assessment indicators, the efficiency of quality assessment of the data can be improved, and a more accurate data quality assessment result can be obtained.

[0093] In some embodiments, the data quality assessment method proposed in this application can also be based on Figure 2 The process shown in is implemented. Figure 2 As shown, the following steps may be included:

[0094] Step S201, constructing a data standard knowledge base and a data element rule knowledge base.

[0095] The data standard knowledge base may include: data element, qualifier, data dictionary, data item, etc. The data element rule knowledge base may include: general data quality rules, such as the 18-digit length verification rule of the ID card, the mailbox format verification rule, the date type data format verification rule, etc., and custom data quality rules. When the general data quality rules do not meet the data quality test, the data quality can be tested through user-defined quality rules.

[0096] Furthermore, the knowledge included in the data standard knowledge base and the data element rule knowledge base can be continuously updated.

[0097] Step S202: construct a data quality assessment indicator system.

[0098] The data quality assessment indicator system can include multiple assessment indicators, and each assessment indicator includes multiple assessment rules. The data quality can be assessed according to the rule content corresponding to each assessment rule. When assessing the data quality, users can select the indicator system for data quality assessment according to business needs.

[0099] In the embodiment of the present application, the data quality assessment system may include: standardization, accuracy, consistency, and integrity. Among them, standardization mainly verifies whether the data table name, table annotation, field name, field annotation, and field type are standardized based on data governance specifications, data element standards, data item standards, etc. Accuracy mainly verifies whether the data content is accurate in terms of length, format, value range, type, and logic based on data element value domain specification requirements; consistency mainly detects whether the fields with associated relationships are consistent, such as: consistency of code value association; integrity mainly detects whether the data information is complete and whether the data is lost or omitted.

[0100] Step S203: Calculate index weights based on AHP.

[0101] Through the hierarchical relationship between the data quality assessment indicator systems and by constructing a judgment matrix, the indicator weight corresponding to each assessment indicator and the rule weight corresponding to each assessment rule are calculated.

[0102] Among them, the data quality evaluation index system can be composed of a three-level index, the first-level index is the evaluation index, the second-level index is the intermediate evaluation index, and the third-level index is the evaluation rule. For example, if the evaluation index is index A, index B and index C, then index A can include 2 intermediate evaluation indicators A1 and A2, index B can include 1 intermediate evaluation indicator B1, and index C can include 3 intermediate evaluation indicators C1, C2 and C3. The intermediate evaluation index A1 can include 2 evaluation rules A11 and A12, the intermediate evaluation index A2 can include 3 evaluation rules A21, A22 and A23, the intermediate evaluation index B1 can include 2 evaluation rules B11 and B12, the intermediate evaluation index C1 can include 3 evaluation rules C11, C12 and C13, the intermediate evaluation index C2 can include 2 evaluation rules C21 and C22, and the intermediate evaluation index C3 can include 4 evaluation rules C31, C32, C33 and C34. Then, based on AHP, the indicator weights corresponding to the evaluation indicators A, B and C, the indicator weights corresponding to the intermediate evaluation indicators A1, A2, B1, C1, C2 and C3, and the rule weights corresponding to the evaluation rules A11, A12, A21, A22, A23, B11, B12, C11, C12 and C13 can be calculated respectively. In addition, the sum of the rule weights corresponding to the evaluation rules included in each intermediate evaluation indicator is equal to 1, the sum of the indicator weights corresponding to the intermediate evaluation indicators included in each evaluation indicator is equal to 1, and the sum of the indicator weights corresponding to each evaluation indicator is equal to 1.

[0103] Step S204: construct a data quality assessment model.

[0104] Based on the quality assessment index system selected by the user, the index weights and rule weights are normalized, and the optimal model is established by combining the characteristics of different data.

[0105] Step S205: perform quality assessment on the data based on the data quality assessment model.

[0106] The constructed data quality assessment model can be: G=f(W, E, V).

[0107] The specific formula can be: G = (W 1 ⊙F 1 ⊙E 1 )(W 2 ⊙F 2 ⊙E 2 )(W 3 ⊙F 3 ⊙E 3 )V.

[0108] in,

[0109]

[0110]

[0111]

[0112] G is the quality assessment score, W is the weight, which can include three levels of weights, namely, the indicator weight corresponding to the evaluation indicator, the indicator weight corresponding to the intermediate evaluation indicator, and the rule weight corresponding to the evaluation rule, and the weight size represents the importance of the indicator. E is the indicator set marker, through which the effect of dynamic selection of evaluation indicators can be achieved, that is, any level of the three-level indicators can be selected through E. V is the quantification result of the three-level indicators. F is a normalized matrix, which is used to adjust the weight changes caused by the change of the selection matrix.

[0113] After the data quality assessment model is constructed, the data quality can be assessed based on the data quality assessment model.

[0114] Step S206: output the quality assessment result.

[0115] By using the data quality assessment model to assess the data quality, the quality assessment results corresponding to the data can be obtained. The quality assessment results may include the comprehensive data quality score, the scores of each quality assessment indicator, the problem level and the problem data details. The problem data details may include the amount of data to be checked, the number of problematic data items, the problem code, the problem data example, etc. In addition, the obtained quality assessment results can be displayed in a visual way.

[0116] Compared with the prior art, the data quality assessment method provided in the embodiment of the present application may have the following advantages:

[0117] First, by constructing a knowledge base, we can continuously accumulate knowledge about data standards and quality inspection rules, and the knowledge base can be reused with projects, greatly improving the efficiency of the entire data governance system.

[0118] Second, incorporate the standardization of data into data quality testing to constrain user behavior to a certain extent and promote data standardization.

[0119] Third, by taking data exploration as a prerequisite for data quality assessment, we can deepen our understanding of data information and data. We can also perform data benchmarking through data exploration, and match the corresponding data element detection rules with the benchmarked data elements without presetting the detection fields, thus expanding the scope of data quality detection and discovering more data quality problems.

[0120] and Figure 1The data quality assessment method shown is based on the same inventive concept, and a data quality assessment device is also provided in the embodiment of the present application. Since the device is a device corresponding to the data quality assessment method of the present application, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated.

[0121] Figure 3 A schematic diagram of the structure of a data quality assessment device provided in an embodiment of the present application is shown. Figure 3 As shown, the data quality assessment device includes a data information acquisition unit 301 , an assessment index determination unit 302 , an assessment rule determination unit 303 , a first result determination unit 304 and a second result determination unit 305 .

[0122] The data information acquisition unit 301 is used to acquire the data to be evaluated, perform data exploration on the data to be evaluated, and obtain data information corresponding to the data to be evaluated;

[0123] An evaluation indicator determination unit 302, in response to an instruction for selecting a target evaluation indicator, determines at least one target evaluation indicator corresponding to the data to be evaluated;

[0124] An evaluation rule determination unit 303 is used to determine the target evaluation rule corresponding to the data to be evaluated according to the association relationship between the target evaluation index and the evaluation rule;

[0125] The first result determination unit 304 is used to perform quality assessment on the data to be assessed according to the target assessment rule and the data information, and obtain the rule assessment score and quality assessment information corresponding to the data to be assessed;

[0126] The second result determination unit 305 is used to determine the indicator evaluation score of each evaluation indicator corresponding to the data to be evaluated according to the rule weights corresponding to each evaluation rule included in the evaluation indicator, and determine the quality evaluation score corresponding to the data to be evaluated according to the indicator weights corresponding to each evaluation indicator.

[0127] In an optional embodiment, the first result determination unit 304 is specifically configured to:

[0128] According to the description information in the target evaluation rule, the rule content corresponding to the description information is determined, and the quality of the data to be evaluated is evaluated based on the rule content and data information.

[0129] In an optional embodiment, the first result determination unit 304 is further configured to:

[0130] If the description information indicates that the rule content is located in the data standard knowledge base, the rule content is found from the data standard knowledge base.

[0131] In an optional embodiment, the first result determination unit 304 is further configured to:

[0132] Matching the data to be evaluated with the standard data format and standard data type in the data standard knowledge base, determining the target standard data format and target standard data type corresponding to the data to be evaluated, and determining the rule content corresponding to the second target evaluation rule according to the target standard data format and target standard data type; or

[0133] The data to be evaluated is matched with the standard data formats and standard data types in the data standard knowledge base, the target standard data format and target standard data type corresponding to the data to be evaluated are determined, and the rule content matching the target standard data format and target standard data type is searched in the data element rule knowledge base.

[0134] In an optional embodiment, the second result determination unit 305 is specifically configured to:

[0135] According to the rule weights corresponding to the various evaluation rules included in the evaluation index, the evaluation scores of the various rules are weighted averaged to obtain the index evaluation scores of the various evaluation indicators corresponding to the data to be evaluated;

[0136] According to the indicator weights corresponding to each evaluation indicator, the evaluation scores of each indicator are weighted averaged to obtain the quality evaluation score corresponding to the data to be evaluated.

[0137] In an optional embodiment, the rule weights corresponding to the various evaluation rules included in the evaluation index and the indicator weights corresponding to the various evaluation indicators are obtained by constructing a judgment matrix through a hierarchical analysis method.

[0138] In an optional embodiment, if Figure 4 As shown, the above device may further include an evaluation result output unit 401, which is used to:

[0139] Output the quality assessment result corresponding to the data to be assessed; the quality assessment result includes the indicator assessment score of each assessment indicator corresponding to the data to be assessed, the quality assessment score corresponding to the data to be assessed, and part or all of the quality assessment information corresponding to the data to be assessed.

[0140] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. The electronic device can be used to evaluate the quality of data. In one embodiment, the electronic device can be a server, or a terminal device or other electronic device. In this embodiment, the structure of the electronic device can be as follows: Figure 5 As shown, it includes a memory 501 , a communication module 503 and one or more processors 502 .

[0141] The memory 501 is used to store computer programs executed by the processor 502. The memory 501 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0142] The memory 501 may be a volatile memory, such as a random-access memory (RAM); the memory 501 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 501 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 501 may be a combination of the above memories.

[0143] The processor 502 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 502 is used to implement the above data quality assessment method when calling the computer program stored in the memory 501 .

[0144] The communication module 503 is used to communicate with terminal devices and other servers.

[0145] The specific connection medium between the memory 501, the communication module 503 and the processor 502 is not limited in the embodiment of the present application. Figure 5 In the embodiment, the memory 501 and the processor 502 are connected via a bus 504. The bus 504 is connected to the processor 502 via a bus 504. Figure 5 The connections between other components are shown in bold lines, which are only for illustration and are not intended to be limiting. Bus 504 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0146] According to one aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device performs the data quality assessment method in the above-mentioned embodiment. The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0147] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A data quality assessment method, characterized in that: The method comprises: Acquire the data to be evaluated, perform data exploration on the data to be evaluated, and obtain data information corresponding to the data to be evaluated; wherein data exploration is a method of providing data definition through automatic exploration, algorithm assistance, manual confirmation and modifiable methods; In response to an instruction for selecting a target evaluation indicator, determining at least one target evaluation indicator corresponding to the data to be evaluated; Determine the target evaluation rule corresponding to the data to be evaluated according to the association relationship between the target evaluation indicator and the evaluation rule; If the description information in the target evaluation rule indicates that the rule content is located in the data standard knowledge base, the data to be evaluated is matched with the standard data format and standard data type in the data standard knowledge base to determine the target standard data format and target standard data type corresponding to the data to be evaluated, and the rule content corresponding to the target evaluation rule is determined according to the target standard data format and target standard data type, and the quality of the data to be evaluated is evaluated according to the rule content and the data information to obtain the rule evaluation score and quality evaluation information corresponding to the data to be evaluated; or, the data to be evaluated is matched with the standard data format and standard data type in the data standard knowledge base to determine the target standard data format and target standard data type corresponding to the data to be evaluated, and the rule content matching the target standard data format and the target standard data type is searched in the data element rule knowledge base, and the quality of the data to be evaluated is evaluated according to the rule content and the data information to obtain the rule evaluation score and quality evaluation information corresponding to the data to be evaluated; wherein the data standard knowledge base includes but is not limited to: data element, qualifier, data dictionary, data item; the data element rule knowledge base includes but is not limited to: general data quality rules; According to the rule weights corresponding to the various target evaluation rules included in the target evaluation indicators, the indicator evaluation scores of the various target evaluation indicators corresponding to the data to be evaluated are determined, and according to the indicator weights corresponding to the various target evaluation indicators, the quality evaluation scores corresponding to the data to be evaluated are determined, and then the quality evaluation results corresponding to the data to be evaluated are output; the quality evaluation results include part or all of the indicator evaluation scores of the various target evaluation indicators corresponding to the data to be evaluated, the quality evaluation scores corresponding to the data to be evaluated, and the quality evaluation information corresponding to the data to be evaluated.

2. The method according to claim 1, characterized in that The step of determining the indicator evaluation score of each target evaluation indicator corresponding to the data to be evaluated according to the rule weights corresponding to each target evaluation rule included in the target evaluation indicator, and determining the quality evaluation score corresponding to the data to be evaluated according to the indicator weights corresponding to each target evaluation indicator, comprises: According to the rule weights corresponding to the various target evaluation rules included in the target evaluation index, weighted average is performed on the evaluation scores of the various rules to obtain the indicator evaluation scores of the various target evaluation indicators corresponding to the data to be evaluated; According to the indicator weight corresponding to each target evaluation indicator, each indicator evaluation score is weighted averaged to obtain the quality evaluation score corresponding to the data to be evaluated.

3. The method according to claim 2, characterized in that The rule weights corresponding to the various target evaluation rules included in the target evaluation index and the indicator weights corresponding to the various target evaluation indicators are obtained by constructing a judgment matrix through a hierarchical analysis method.

4. A data quality assessment device, characterized in that: include: A data information acquisition unit, used to acquire the data to be evaluated, perform data exploration on the data to be evaluated, and obtain data information corresponding to the data to be evaluated; wherein data exploration is a method of providing data definition through automatic exploration, algorithm assistance, manual confirmation and modifiable methods; An evaluation indicator determination unit, configured to determine at least one target evaluation indicator corresponding to the data to be evaluated in response to an instruction for selecting a target evaluation indicator; An evaluation rule determination unit, used to determine the target evaluation rule corresponding to the data to be evaluated according to the association relationship between the target evaluation indicator and the evaluation rule; A first result determination unit is used for matching the data to be evaluated with the standard data format and standard data type in the data standard knowledge base, if the description information in the target evaluation rule indicates that the rule content is located in the data standard knowledge base, determining the target standard data format and target standard data type corresponding to the data to be evaluated, and determining the rule content corresponding to the target evaluation rule according to the target standard data format and target standard data type, and performing quality assessment on the data to be evaluated according to the rule content and the data information, to obtain the rule assessment score and quality assessment information corresponding to the data to be evaluated; or, matching the data to be evaluated with the standard data format and standard data type in the data standard knowledge base, determining the target standard data format and target standard data type corresponding to the data to be evaluated, and searching for the rule content matching the target standard data format and the target standard data type in the data element rule knowledge base, and performing quality assessment on the data to be evaluated according to the rule content and the data information, to obtain the rule assessment score and quality assessment information corresponding to the data to be evaluated; wherein the data standard knowledge base includes but is not limited to: data element, qualifier, data dictionary, data item; the data element rule knowledge base includes but is not limited to: general data quality rules; The second result determination unit is used to determine the indicator evaluation scores of each target evaluation indicator corresponding to the data to be evaluated according to the rule weights corresponding to each target evaluation rule included in the target evaluation indicator, and after determining the quality evaluation score corresponding to the data to be evaluated according to the indicator weights corresponding to each target evaluation indicator, output the quality evaluation result corresponding to the data to be evaluated; the quality evaluation result includes part or all of the indicator evaluation scores of each target evaluation indicator corresponding to the data to be evaluated, the quality evaluation score corresponding to the data to be evaluated, and the quality evaluation information corresponding to the data to be evaluated.

5. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

6. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the method according to any one of claims 1 to 3 is implemented.

Images