Data quality evaluation method, system and related device

Abstract

The invention discloses a data quality evaluation method and system and a related device. The method comprises the following steps: constructing a sub-service attention matrix and a data set attention matrix based on a pre-obtained evaluation demand, and determining a demand index according to the evaluation demand; obtaining an evaluation index library, and extracting corresponding evaluation indexes from the evaluation index library; determining and constructing a data element attention matrix based on the evaluation indexes; obtaining data quality elements, setting constraint conditions, and constructing a constraint condition attention matrix; obtaining a first weight sequence according to the demand index; obtaining a second weight matrix according to the constraint condition attention matrix; collecting a target data set; obtaining a problem number, and calculating a constraint rule evaluation score based on the problem number, the attention matrix and a rule algorithm; calculating a quality element evaluation score according to the second weight matrix and the constraint rule evaluation score; and calculating a data set evaluation score according to the first weight sequence and the quality element evaluation score.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data quality evaluation method, system and related devices. Background Art

[0002] The rapid development of information technology has led to the widespread application of information systems in the public service field of government departments, ranging from government office automation systems to various livelihood service platforms, such as social security inquiry systems, provident fund management systems, etc. As information construction continues to deepen, the quality of government services depends on the quality of data services. High-quality data can ensure the scientific nature of government decision-making and the efficiency of public services, thereby improving public satisfaction.

[0003] In existing technologies, data quality evaluation usually requires manual participation in some processes. For example, staff need to collect data samples from numerous data sources and then check and verify the data one by one according to the evaluation standards within their respective departments.

[0004] However, the huge amount of data samples and the lack of unified evaluation standards lead to inaccurate data quality evaluation, which in turn affects the quality of government services. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a data quality evaluation method, system and related devices.

[0006] The technical solution provided in this application is described below: The first aspect of the present application provides a data quality evaluation method, the method comprising: Constructing a sub-service attention matrix and a data set attention matrix based on the pre-acquired evaluation requirements, and determining demand indicators according to the evaluation requirements; Acquire an evaluation index library, and extract an evaluation index corresponding to the demand index from the evaluation index library; Determine and construct a data element attention matrix based on the evaluation index; Acquire data quality elements, set constraints for each of the data quality elements, and construct a constraint attention matrix; Allocating a first weight to each of the data quality elements according to the demand indicator to obtain a first weight sequence; According to the constraint condition attention matrix, a second weight is set for each of the data quality elements to obtain a second weight matrix; Collecting a target data set, wherein the target data set includes multiple data tuples, and each data tuple includes multiple data elements; Obtaining the number of problems of the data tuple that do not meet the constraint condition, and calculating the constraint rule evaluation score based on the number of problems, the data element attention matrix and the constraint rule algorithm; Calculate the quality element evaluation score according to the second weight matrix and the constraint rule evaluation score; A data set evaluation score is calculated based on the first weight sequence and the quality element evaluation score.

[0007] Optionally, obtaining the number of problems of the data tuple that does not meet the constraint condition, and calculating the constraint rule evaluation score based on the number of problems, the data element attention matrix and the constraint rule algorithm includes: Obtaining the number of problems with the data tuples that violate the constraint condition; Calculate the problem tuple coverage rate according to the number of problems and the total number of tuples in the target data set; Acquire a verification result matrix of the target data set, and convert the verification result matrix into a constraint condition vector, wherein one or more constraint condition vectors correspond to a constraint rule; A constraint rule evaluation score is calculated based on the constraint rule, the constraint rule algorithm and the data element attention matrix.

[0008] Optionally, the problem tuple coverage is calculated by the following formula: ; in, represents the problem tuple coverage, represents the number of questions for the data tuple, Represents the total number of tuples of the target data set; The constraint rule matrix is ​​calculated by the following formula: ; in, represents the Chinese name constraint vector, represents the English name constraint vector, represents the abbreviated constraint vector, represents the definition describing the constraint vector, represents the data type constraint vector, represents the range constraint vector, represents the control constraint vector, represents the maximum occurrence constraint vector, represents other constraint vectors, represents the constraint rule matrix; The constraint rule evaluation score is calculated by the following formula: ; in, represents the evaluation score of the constraint rule, represents the constraint rule algorithm, represents the constraint rule matrix, Represents the data element attention matrix.

[0009] Optionally, the constraint rule algorithm includes the requirement indicators, and the requirement indicators include a standardization indicator, a completeness indicator, an accuracy indicator, a consistency indicator, a timeliness indicator, and an accessibility indicator.

[0010] Optionally, the normative indicators include: Data standard constraints, data model constraints, metadata constraints, business rule constraints, authoritative reference data constraints, and security specification constraints; The integrity indicators include: data element integrity constraints and data record integrity constraints; The accuracy indicators include: data content correctness constraints, data format compliance constraints, data duplication rate constraints, data uniqueness constraints, and dirty data occurrence rate constraints; The consistency indicators include: same data consistency constraints and associated data consistency constraints; The timeliness index includes: correctness constraint based on time period, timeliness constraint based on time point and timing constraint; The accessibility index includes: accessibility constraints and usability constraints.

[0011] Optionally, the score of the data standard constraint is calculated by the following formula: ; in, represents the score of the data standard constraint, represents the constraint rule matrix, The rows of the constraint rule matrix, represents the columns of the constraint rule matrix, represents the column number of the constraint rule matrix, represents the row number of the constraint rule matrix, Represents the constraint rule matrix OK The elements of the column, represents the number of the data tuples of the target data set; The score of the data element integrity constraint is calculated by the following formula: ; in, represents the score of the integrity constraint of the data element, represents the range vector of the constraint rule matrix, The constraint rule matrix is ​​represented by elements, represents the number of data elements concerned by the constraint rule matrix, Represents the value range vector values, express A data set containing the data elements.

[0012] Optionally, the quality element evaluation score is calculated by the following formula: ; in, represents the evaluation score of the quality element, express The data quality elements The constraint rule evaluation score, Indicates the second weight matrix The data quality elements The constraints mentioned; The dataset evaluation score is calculated by the following formula: ; in, represents the evaluation score of the dataset, express The target dataset The quality element evaluation score, Indicates the first weight matrix The target dataset The weight of each of the data quality elements.

[0013] Optionally, the method further includes: Calculate the sub-service evaluation score according to the data set attention matrix and the data set evaluation score; The information service evaluation score is calculated based on the sub-service evaluation score and the sub-service attention matrix.

[0014] A second aspect of the present application provides a data quality evaluation system, the system comprising: A first construction unit is used to construct a sub-service attention matrix and a data set attention matrix based on pre-acquired evaluation requirements, and determine a demand index according to the evaluation requirements; An extraction unit, used to obtain an evaluation index library, and extract an evaluation index corresponding to the demand index from the evaluation index library; A second construction unit is used to determine and construct a data element attention matrix based on the evaluation index; The third construction unit is used to obtain data quality elements, set constraints for each of the data quality elements, and construct a constraint attention matrix.

[0015] an allocating unit, configured to allocate a first weight to each of the data quality elements according to the demand indicator to obtain a first weight sequence; A setting unit, configured to set a second weight for each of the data quality elements according to the constraint condition attention matrix to obtain a second weight matrix; A collection unit, used for collecting a target data set, wherein the target data set includes a plurality of data tuples, and each data tuple includes a plurality of data elements; A first calculation unit is used to obtain the number of problems of the data tuple that do not meet the constraint condition, and calculate the constraint rule evaluation score based on the number of problems, the data element attention matrix and the constraint rule algorithm; A second calculation unit, configured to calculate a quality element evaluation score according to the second weight matrix and the constraint rule evaluation score; A third calculation unit, configured to calculate a data set evaluation score according to the first weight sequence and the quality element evaluation score; A third aspect of the present application provides a data quality evaluation device, the device comprising: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0016] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0017] It can be seen from the above technical solutions that this application has the following advantages: 1. Constructing the sub-service attention matrix and the data set attention matrix based on the pre-acquired evaluation requirements, and determining the demand indicators according to the evaluation requirements, is conducive to the reasonable allocation of attention, ensuring that the evaluation results accurately reflect the evaluation requirements, avoiding the blindness and inaccuracy of data evaluation, and establishing the standardization and unification of data quality evaluation.

[0018] 2. By obtaining an evaluation index library and extracting evaluation indexes corresponding to the demand indexes from the evaluation index library, the use of irrelevant evaluation indexes is avoided, and the evaluation results are more focused on the aspects concerned by the demand indexes, thereby further improving the accuracy of the evaluation.

[0019] 3. Constructing a data element attention matrix based on the evaluation indicators can finely manage data quality, provide a more targeted framework for subsequent data quality evaluation, and more accurately measure the importance of each data element to the overall data quality evaluation.

[0020] 4. By obtaining data quality elements, setting constraints for each of the data quality elements, and constructing a constraint attention matrix, it is possible to accurately reflect the degree of compliance of the data under different constraints and provide a more reliable data basis.

[0021] 5. By assigning a first weight to each of the data quality elements according to the demand indicators to obtain a first weight sequence, the important performance of the data quality elements in meeting key requirements can be more accurately measured, and the relative importance of each data quality element in the overall evaluation can be accurately reflected.

[0022] 6. By setting a second weight for each of the data quality elements according to the constraint condition attention matrix, a second weight matrix is ​​obtained, which provides another dimension of quantitative basis for the evaluation of data quality elements, complements the first weight sequence, avoids the one-sidedness of single-dimensional evaluation, and makes the evaluation more comprehensive and detailed.

[0023] 7. By obtaining the number of problems of the data tuples that do not meet the constraints, and calculating the constraint rule evaluation score based on the number of problems, the data element attention matrix and the constraint rule algorithm, the performance of the target data set in satisfying the constraints can be quantitatively evaluated, intuitively reflecting the degree of compliance of the data.

[0024] 8. By calculating the quality element evaluation score based on the second weight matrix and the constraint rule evaluation score, the evaluation of data quality elements can be made more comprehensive and accurate, further avoiding single-dimensional evaluation, and helping to more accurately locate the importance of each data quality element to the overall data quality, providing support for subsequent evaluation.

[0025] 9. By calculating the data set evaluation score according to the first weight sequence and the quality element evaluation score, the quality status and score of the target data set can be fully and clearly understood, further improving the accuracy of data evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A schematic diagram of an embodiment of the data quality evaluation method provided in this application; Figure 2 A flow chart of another embodiment of the data quality evaluation method provided in this application; Figure 3 A schematic diagram of the structure of an embodiment of the data quality evaluation system provided by this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the data quality evaluation device provided in this application. DETAILED DESCRIPTION

[0028] The present application provides a data quality evaluation method, which can unify the evaluation standard and improve the accuracy of data quality evaluation. It should be noted that the data quality evaluation method of the present application is applied to a terminal.

[0029] See also Figure 1 The present application first provides an embodiment of a data quality evaluation method, which includes: S101, constructing a sub-service attention matrix and a data set attention matrix based on the pre-acquired evaluation requirements, and determining demand indicators according to the evaluation requirements; In this embodiment, after clarifying the specific content of the pre-acquired evaluation requirements, a sub-service attention matrix is ​​constructed based on the evaluation requirements. When there are multiple sub-services, such as data storage sub-service, data query sub-service, data processing sub-service, etc. For each sub-service, it is necessary to determine whether to pay attention to the attributes, services or functions of the sub-service according to the evaluation requirements. If it is concerned, it is recorded as 1, and if it is not concerned, it is recorded as 0. The sub-service attention matrix with multiple sub-services can be expressed by the following formula: ; in, represents the sub-service attention matrix, Indicates whether to follow this sub-service. Indicates Individual service.

[0030] At the same time, in this embodiment, a data set attention matrix is ​​constructed based on its evaluation requirements when there are multiple data sets of different types, such as user information data sets, transaction record data sets, etc.

[0031] For each data set, it is necessary to determine whether to pay attention to the characteristics of the data set according to the evaluation requirements. If it is concerned, it is recorded as 1, and if it is not concerned, it is recorded as 0. The data set attention matrix with multiple different types of data sets can be expressed by the following formula: ; in, represents the dataset attention matrix, Indicates whether to pay attention to this dataset. Indicates data sets.

[0032] Furthermore, in this embodiment, demand indicators are determined according to evaluation requirements, and the demand indicators are key measurement indicators extracted from the evaluation requirements, including normative indicators, completeness indicators, accuracy indicators, consistency indicators, timeliness indicators, and accessibility indicators.

[0033] This embodiment constructs a sub-service attention matrix and a data set attention matrix based on pre-acquired evaluation requirements, which can focus on the key parts of the evaluation requirements and reasonably allocate attention, clearly showing the attention of each sub-service and data set in the entire evaluation requirements, and providing support for subsequent calculations. Determining demand indicators based on evaluation requirements provides a clear direction for the entire evaluation process, avoids the blindness of data evaluation, and ensures that the evaluation results can accurately reflect the degree to which the evaluation requirements are met.

[0034] S102, obtaining an evaluation index library, and extracting evaluation indexes corresponding to the demand indexes from the evaluation index library; In this embodiment, the evaluation index library contains a variety of indicators for evaluation. After obtaining the evaluation index library, the demand indicators determined in step S101 are used as screening basis. For example, if the demand indicators are the integrity and timeliness of the evaluation data, then indicators corresponding to or highly correlated with the data integrity and timeliness are searched from the evaluation index library, such as the integrity rate of data elements, the integrity rate of data records, the update frequency of data, the time interval from the generation of data to its availability, etc. At the same time, the measurement scale and evaluation method can be determined based on the extracted evaluation indicators.

[0035] This embodiment can ensure the pertinence of the evaluation process by extracting indicators corresponding to the demand indicators from the evaluation indicator library, avoid using irrelevant evaluation indicators, and make the evaluation results more focused on the aspects concerned by the demand indicators, which is conducive to improving the accuracy of the evaluation.

[0036] S103, constructing a data element attention matrix based on the evaluation index; A data element is the basic unit of data, which is a data unit whose definition, identification, expression and allowed values ​​are described by a set of attributes. For example, in a user information database, user name, age, contact information, etc. are all data elements.

[0037] In this embodiment, after the evaluation index is determined, the data element attention is constructed based on the evaluation index. First, the importance of each data element is analyzed according to the determined evaluation index. The important data that plays a key role is given a high attention, and the auxiliary or unimportant data is given a low attention or no attention. For example, the determined evaluation index focuses on the accuracy and completeness of the data. For data elements such as user names that play a key role in user identification, their accuracy and completeness are crucial to the entire system, so they are given attention; for user interests and hobbies, which are not the focus of the evaluation index, no attention is given. Finally, these different data elements and their corresponding attention are arranged according to a certain structure to construct a data element attention matrix.

[0038] For each data element in the data element attention matrix, it is necessary to determine whether to pay attention to the data element according to the evaluation index. If it is paid attention to, it is recorded as 1, and if it is not paid attention to, it is recorded as 0. The data element attention matrix can be expressed by the following formula: ; in, represents the data element attention matrix, Indicates whether to pay attention to this data element. Indicates data element.

[0039] This embodiment determines that constructing a data element attention matrix based on evaluation indicators helps to manage and evaluate data more finely. Furthermore, the matrix provides a more targeted framework for subsequent data quality evaluation, more accurately measures the contribution of each data element to the overall data quality evaluation, helps to discover the root causes of data quality problems, and can correct them in time.

[0040] S104, obtaining data quality elements, setting constraints for each data quality element, and constructing a constraint attention matrix; In this embodiment, data quality elements are determined based on evaluation indicators. After obtaining the data quality elements, constraints are set for each data quality element. For example, in a sales data system, data quality elements include the accuracy of sales data, the completeness of sales records, and the consistency of data from different sales channels. After obtaining these data quality elements, for the data quality element of accuracy of sales data, the constraint is that the error range cannot exceed 5%; for the completeness of sales records, the constraint is that all sales transactions must have corresponding records and no omissions are allowed; for data consistency, the constraint is that the sales calculation method for the same product in different sales channels must be the same.

[0041] After setting the constraints, assign attention to each constraint based on factors such as demand and data importance, and arrange these constraints and their attention according to a certain regular structure to construct a constraint attention matrix.

[0042] For each constraint in the constraint attention matrix, it is necessary to determine whether to pay attention to the constraint based on the data quality element. If it is paid attention to, it is recorded as 1, and if it is not paid attention to, it is recorded as 0. The constraint attention matrix can be expressed by the following formula: ; in, represents the constraint attention matrix, Indicates whether to pay attention to this constraint. Indicates constraints.

[0043] This embodiment can clarify the data quality standards by setting the constraints of the data quality elements, so that the management and evaluation of data can be governed by rules. Constructing a constraint attention matrix further optimizes resource allocation and key focus areas. When evaluating data quality, the matrix can make the evaluation more comprehensive and targeted, and can accurately reflect the degree of compliance of the data under different constraints, providing a more reliable data basis.

[0044] S105, assigning a first weight to each data quality element according to the demand indicator to obtain a first weight sequence; In this embodiment, based on the demand indicators extracted from the evaluation requirements, a first weight is assigned to each acquired data quality element. For example, the data quality elements include accuracy, timeliness, and accessibility. Among the demand indicators, accuracy accounts for the highest proportion, followed by timeliness, and accessibility is the lowest. For the importance of accuracy in the demand indicators, a higher weight, such as 0.5, will be assigned; for timeliness, a weight of 0.3 will be assigned; for accessibility, a weight of 0.2 will be assigned. Finally, these weights assigned according to the data quality elements are arranged in sequence to obtain the first weight sequence.

[0045] The first weight sequence can be expressed by the following formula: ; in, represents the first weight sequence, Indicates the weight of an indicator.

[0046] This embodiment assigns a first weight to each data quality element according to the demand index, which can more accurately measure the important performance of the data quality element in meeting key requirements, and the first weight sequence makes the evaluation of data quality elements more standardized, accurately reflecting the relative importance of each data quality element in the overall evaluation.

[0047] S106, setting a second weight for each data quality element according to the constraint condition attention matrix to obtain a second weight matrix; In this embodiment, the constraint condition attention matrix includes the constraint conditions set for each data quality element and the corresponding attention. The second weight is set for each data quality element according to the constraint condition attention matrix. For example, the data quality elements include accuracy, completeness, and consistency. The constraint condition attention of accuracy is high, the completeness attention is medium, and the consistency attention is the lowest. A higher weight, such as 0.5, is set for accuracy; for completeness, a weight of 0.3 is assigned; for consistency, a weight of 0.2 is assigned. Finally, the second weights of these set data quality elements are arranged in a certain structure to obtain a second weight matrix.

[0048] The second weight matrix can be expressed by the following formula: ; in, represents the second weight matrix, Indicates the weight of a data quality element.

[0049] This embodiment sets a second weight for each data quality element according to the constraint condition attention matrix to obtain a second weight matrix, which provides another dimension of quantitative basis for the evaluation of data quality elements, complements the first weight sequence, avoids the one-sidedness of single-dimensional evaluation, and makes the evaluation more comprehensive and detailed.

[0050] S107, collecting a target data set, wherein the target data set includes multiple data tuples, and each data tuple includes multiple data elements; In this embodiment, data sets are collected from multiple data sources through corresponding interfaces or data extraction tools. After the target data is collected, it is preliminarily cleaned to remove duplicate data, erroneous data, etc., and finally a target data set is obtained. The target data set contains multiple data tuples, each of which contains multiple data elements. For example, in an order management system, a data tuple is information related to each order, and the data elements contained therein include order number, order time, product name, product quantity, customer name, customer contact information, etc.

[0051] This embodiment can provide a reliable data source for subsequent analysis and decision-making by accurately collecting the target data set, and the preliminary data cleaning helps to improve data quality and reduce evaluation and analysis deviations caused by erroneous data or duplicate data.

[0052] S108, obtaining the number of questions of the data tuples that do not meet the constraint conditions, and calculating the constraint rule evaluation score based on the number of questions, the data element attention matrix and the constraint rule algorithm; In this embodiment, to obtain the number of problems of data tuples that do not meet the constraint conditions, we first need to traverse each data tuple in the collected target data set, and for each data tuple, check it according to the previously set constraint conditions to determine the data tuples that do not meet the constraint conditions. For example, the constraint condition stipulates that the age of a data element in the data tuple must be a positive integer. During the inspection process, it is found that there are data tuples that do not meet this condition, then these non-compliant data tuples are counted to obtain the number of problems of the data tuple. Then, according to the data element attention matrix, the importance weight of each data element in the whole is determined, and then combined with the constraint rule algorithm to calculate, and finally the constraint rule evaluation score is obtained.

[0053] Furthermore, the constraint rule algorithm includes demand indicators, and the demand indicators include standardization indicators, completeness indicators, accuracy indicators, consistency indicators, timeliness indicators, and accessibility indicators.

[0054] Further, normative indicators include: Data standard constraints, data model constraints, metadata constraints, business rule constraints, authoritative reference data constraints, and security specification constraints; The score of the data standard constraint is calculated by the following formula: ; in, represents the score of the data standard constraint, represents the constraint rule matrix, The rows of the constraint rule matrix are expressed as, represents the columns of the constraint rule matrix, represents the column number of the constraint rule matrix, represents the row number of the constraint rule matrix, Represents the constraint rule matrix OK The elements of the column, Represents the number of data tuples of the target dataset; Furthermore, integrity indicators include: data element integrity constraints and data record integrity constraints; The score of the data element integrity constraint is calculated as follows: ; in, represents the score of the data element integrity constraint, represents the range vector of the constraint rule matrix, The constraint rule matrix elements, Indicates the number of data elements that the constraint rule matrix focuses on, The first values, express A data set of data elements.

[0055] Furthermore, accuracy indicators include: data content correctness constraints, data format compliance constraints, data duplication rate constraints, data uniqueness constraints, and dirty data occurrence rate constraints; The score of the data content correctness constraint is calculated by the following formula: ; in, represents the score of the data element integrity constraint, represents the value of the range vector, The constraint rule matrix elements, Indicates the number of data elements that the constraint rule matrix focuses on, The first values, express A data set of data elements.

[0056] The score of the dirty data occurrence rate constraint is calculated using the following formula: ; in, represents the score of the data element integrity constraint, Indicates that A dataset of tuples, Indicates the number of dirty data tuples.

[0057] Furthermore, consistency indicators include: same data consistency constraints and associated data consistency constraints; The score of the same data consistency constraint is calculated by the following formula: ; in, represents the score of the same data consistency constraint, express OK The constraint rule matrix of columns, represents the column number of the constraint rule matrix, represents the row number of the constraint rule matrix, Represents the constraint rule matrix OK The elements of the column, Represents the number of data tuples in the target dataset, Indicates the number of data elements.

[0058] Furthermore, timeliness indicators include: correctness constraints based on time periods, timeliness constraints based on time points, and timing constraints; The score of the time period-based correctness constraint is calculated as follows: ; in, represents the score of the correctness constraint based on the time period, The constraint rule matrix values, Represents the number of all data tuples.

[0059] Furthermore, accessibility indicators include: accessibility constraints and usability constraints.

[0060] The score of the accessibility constraint is calculated as follows: ; in, represents the score of the accessibility constraint, Indicates that A dataset of tuples, Indicates the number of all data tuples that can be obtained.

[0061] This embodiment obtains the number of problems of data tuples that do not meet the constraint conditions, and calculates the constraint rule evaluation score based on the number of problems, the data element attention matrix and the constraint rule algorithm. It can quantitatively evaluate the performance of the target data set in satisfying the constraint conditions, intuitively reflect the compliance level of the data, and help make the data quality evaluation of the target data set more scientific and reasonable.

[0062] S109, calculating the quality element evaluation score according to the second weight matrix and the constraint rule evaluation score; In this embodiment, the quality element evaluation score is calculated based on the second weight matrix and the constraint rule evaluation score. First, the second weight matrix and the constraint rule evaluation score that have been obtained need to be obtained. The second weight matrix contains the weights set for each data quality element according to the constraint condition attention matrix. The constraint rule evaluation score is calculated by counting the number of problems of data tuples that do not meet the constraints, and combining the data element attention matrix and the constraint rule algorithm. Then, the weight of each quality element in the second weight matrix and the constraint rule evaluation score are calculated to obtain the quality element evaluation score. For example, for a certain quality element, the weight in the second weight matrix is ​​0.3, and the constraint rule evaluation score is 80 points. Then the calculated quality element evaluation score is 0.3×80=24 points, that is, the quality element evaluation score of a certain quality element is 24 points.

[0063] The quality element evaluation score can be expressed by the following formula: ; in, represents the quality element evaluation score, express The first element of data quality Constraint rule evaluation scores, represents the second weight matrix The first element of data quality A constraint rule.

[0064] The calculation of the quality element evaluation score in this embodiment combines the second weight matrix and the constraint rule evaluation score, making the evaluation of data quality elements more comprehensive and accurate, avoiding single-dimensional evaluation, and helping to more accurately locate the importance of each data quality element to the overall data quality, providing support for subsequent evaluation.

[0065] S110. Calculate a data set evaluation score according to the first weight sequence and the quality element evaluation score.

[0066] In this embodiment, the data set evaluation score is calculated based on the first weight sequence and the quality element evaluation score. First, the first weight sequence and the quality element evaluation score are obtained. The first weight sequence is the weight assigned to each data quality element according to the demand index. The quality element evaluation score is calculated by the second weight matrix and the constraint rule evaluation score. Then, according to the correspondence between the first weight sequence and the quality element evaluation score, each weight in the first weight sequence is calculated with the corresponding quality element evaluation score to obtain the data set evaluation score. For example, the first weight of the quality element A in the target data set is 0.4, and the corresponding quality element evaluation score is 60 points. Then, the data set evaluation score of the quality element A of the target data set is 0.4×60=24 points.

[0067] The quality element evaluation score can be expressed by the following formula: ; in, represents the evaluation score of the dataset, express The target dataset Quality element evaluation scores, Represents the first weight matrix The target dataset The weight of each data quality element.

[0068] This embodiment calculates a data set evaluation score based on the first weight sequence and the quality element evaluation score. The score can fully and clearly understand the quality status of the data set, further improving the accuracy of data evaluation.

[0069] The specific steps for calculating the constraint rule evaluation score, sub-service evaluation score, and information service evaluation score based on the number of questions, data element attention matrix, and constraint rule algorithm are described as follows: See also Figure 2 , Figure 2 Another embodiment of a data quality evaluation method provided by the present application includes: S201, constructing a sub-service attention matrix and a data set attention matrix based on the pre-acquired evaluation requirements, and determining demand indicators according to the evaluation requirements; S202, obtaining an evaluation index library, and extracting evaluation indexes corresponding to the demand indexes from the evaluation index library; S203, constructing a data element attention matrix based on the evaluation index; S204, obtaining data quality elements, setting constraints for each data quality element, and constructing a constraint attention matrix; S205, assigning a first weight to each data quality element according to the demand indicator to obtain a first weight sequence; S206, setting a second weight for each data quality element according to the constraint condition attention matrix to obtain a second weight matrix; S207, collecting a target data set, wherein the target data set includes multiple data tuples, and each data tuple includes multiple data elements; In this embodiment, steps S201 to S207 are similar to steps S101 to S107 in the aforementioned embodiment, and are not described again here.

[0070] S208, obtaining the number of problems with data tuples that violate constraint conditions; In this embodiment, in order to obtain the number of data tuples that violate the constraints, it is first necessary to obtain the set constraints, and then check each data tuple in the target data set one by one. During the checking process, the data elements in each data tuple are compared with the corresponding constraints. By traversing and comparing all data tuples, the number of data tuples that violate the constraints can be counted. For example, the constraints stipulate that the age data in a data tuple must be 18 years old. When it is checked that the age data in a data tuple is 15 years old, it is determined that the data tuple has violated the constraints.

[0071] S209, calculating the problem tuple coverage rate according to the number of problems and the total number of tuples in the target data set; In this embodiment, the number of problems with data tuples that violate the constraint conditions and the total number of tuples in the target data set are first obtained, and then the coverage of the percentage of problem tuples is calculated based on the number of problems and the total number of tuples.

[0072] The tuple coverage of this problem is calculated by the following formula: ; in, represents the problem tuple coverage, represents the number of questions for the data tuple, Indicates the total number of tuples in the target data set; S210, obtaining a verification result matrix of the target data set, and converting the verification result matrix into a constraint condition vector, wherein one or more constraint condition vectors correspond to one constraint rule; In this embodiment, the verification result matrix of the target data set is first obtained from the problem tuple coverage of the target data set, and then the verification result matrix is ​​converted into a constraint condition vector according to a specific mapping rule. For example, each row or column in the verification result matrix may represent a verification result of a data attribute, and the constraint condition vector is formed by extracting and combining the data of these rows or columns in a certain order, and one or more such constraint condition vectors correspond to a constraint rule, that is, a constraint rule matrix is ​​constructed.

[0073] The constraint vector is expressed as follows: ; in, represents the constraint vector, represents the problem tuple coverage, express A dataset of data elements.

[0074] The constraint rule matrix is ​​calculated as follows: ; in, represents the Chinese name constraint vector, represents the English name constraint vector, represents the abbreviated constraint vector, represents the definition describing the constraint vector, represents the data type constraint vector, represents the range constraint vector, represents the control constraint vector, represents the maximum occurrence constraint vector, represents other constraint vectors, Represents the constraint rule matrix.

[0075] S211, calculating the quality element evaluation score according to the second weight matrix and the constraint rule evaluation score; S212: Calculate a data set evaluation score based on the first weight sequence and the quality element evaluation score.

[0076] In this embodiment, steps S211 to S212 are similar to steps S109 to S110 in the aforementioned embodiment, and are not described in detail here.

[0077] S213, calculating the sub-service evaluation score according to the data set attention matrix and the data set evaluation score; In this embodiment, the sub-service evaluation score is calculated based on the data set attention matrix and the data set evaluation score. First, it is necessary to obtain the constructed data set attention matrix and the calculated data set evaluation score, and then each element in the data set attention matrix is ​​associated with the corresponding data set evaluation score to obtain the sub-service evaluation score. For example, if the attention weight of a data set in the data set attention matrix is ​​0.2, and the corresponding data set evaluation score is 80 points, then the calculation of the sub-service evaluation score is: 0.2×80=16 points.

[0078] The sub-service evaluation score is calculated using the following formula: ; in, Indicates the evaluation score of the dataset of this sub-service. represents the attention weight of the dataset, Indicates the sub-service evaluation score, Indicates Datasets, express A subservice for a dataset.

[0079] S214. Calculate the information service evaluation score based on the sub-service evaluation score and the sub-service attention matrix.

[0080] In this embodiment, the information service evaluation score is calculated based on the sub-service evaluation score and the sub-service attention matrix. First, the sub-service evaluation score and the sub-service attention matrix need to be obtained, and then each element in the sub-service attention matrix is ​​calculated with the corresponding sub-service evaluation score. For example, a sub-service in the sub-service attention matrix has an attention weight of 0.3 and a corresponding sub-service evaluation score of 50 points. Then the information service evaluation score is calculated as: 0.3×50=15 points.

[0081] The information service evaluation score is calculated using the following formula: ; in, Indicates the sub-service evaluation score of the information service. represents the attention weight of the dataset, Indicates the information service evaluation score, Indicates Sub-service attention weight, express Information service for individual sub-services.

[0082] This embodiment can intuitively reflect the preliminary status of the target data set in satisfying the constraints by obtaining the number of problems with data tuples that violate the constraints, and provide a data basis for subsequent calculations of coverage. The problem tuple coverage is calculated based on the number of problems and the total number of tuples in the target data set, which can clearly reflect the proportion of data tuples with problems in the data set, and provide a data basis for subsequent calculations. By obtaining the verification result matrix of the target data set and converting the verification result matrix into a constraint vector, wherein one or more constraint vectors correspond to a constraint rule, the relationship between the target data set and the constraint rule can be more intuitively reflected, the accuracy and efficiency of the entire evaluation process can be improved, and the constraint vector can provide a more accurate data basis, so that the evaluation result can better reflect the true quality status of the data set. By calculating the sub-service evaluation score based on the data set attention matrix and the data set evaluation score, the comprehensive quality of the sub-service can be more comprehensively and systematically reflected, avoiding the one-sidedness of considering the quality of the data set alone, and providing a data basis for subsequent calculations. By calculating the information service evaluation score based on the sub-service evaluation score and the sub-service attention matrix, the data quality of the entire information service can be evaluated, which is convenient for improving the overall quality of the information service according to the evaluation and meeting user needs.

[0083] The following is a detailed description of the data quality evaluation system provided by this application. Figure 3 , Figure 3 An embodiment of a data quality evaluation system provided by the present application, the system comprises: The first construction unit 301 is used to construct a sub-service attention matrix and a data set attention matrix based on the pre-acquired evaluation requirements, and determine the demand index according to the evaluation requirements; An extraction unit 302 is used to obtain an evaluation index library and extract an evaluation index corresponding to the demand index from the evaluation index library; The second construction unit 303 is used to determine and construct a data element attention matrix based on the evaluation index; The third construction unit 304 is used to obtain data quality elements, set constraints for each data quality element, and construct a constraint attention matrix; An allocating unit 305, configured to allocate a first weight to each data quality element according to the demand indicator to obtain a first weight sequence; A setting unit 306, configured to set a second weight for each data quality element according to the constraint condition attention matrix to obtain a second weight matrix; The acquisition unit 307 is used to acquire a target data set, wherein the target data set includes a plurality of data tuples, and each data tuple includes a plurality of data elements; A first calculation unit 308 is used to obtain the number of problems of the data tuples that do not meet the constraint conditions, and calculate the constraint rule evaluation score based on the number of problems, the data element attention matrix and the constraint rule algorithm; A second calculation unit 309 is used to calculate the quality element evaluation score according to the second weight matrix and the constraint rule evaluation score; The third calculation unit 310 is used to calculate the data set evaluation score according to the first weight sequence and the quality element evaluation score.

[0084] Optionally, the first calculation unit 308 is further configured to: Get the number of problems with data tuples that violate constraints; The question tuple coverage is calculated based on the number of questions and the total number of tuples in the target data set; Obtaining a verification result matrix of the target data set, and converting the verification result matrix into a constraint condition vector, wherein one or more constraint condition vectors correspond to one constraint rule; The constraint rule evaluation score is calculated based on the constraint rules, constraint rule algorithm and data element attention matrix.

[0085] Optionally, the problem tuple coverage is calculated as follows: ; in, represents the problem tuple coverage, represents the number of questions for the data tuple, Indicates the total number of tuples in the target data set; The constraint rule matrix is ​​calculated as follows: ; in, represents the Chinese name constraint vector, represents the English name constraint vector, represents the abbreviated constraint vector, represents the definition describing the constraint vector, represents the data type constraint vector, represents the range constraint vector, represents the control constraint vector, represents the maximum occurrence constraint vector, represents other constraint vectors, represents the constraint rule matrix; The constraint rule evaluation score is calculated using the following formula: ; in, represents the constraint rule evaluation score, represents the constraint rule algorithm, represents the constraint rule matrix, Represents the data element attention matrix.

[0086] Optionally, the constraint rule algorithm includes requirement indicators, and the requirement indicators include a standardization indicator, a completeness indicator, an accuracy indicator, a consistency indicator, a timeliness indicator, and an accessibility indicator.

[0087] Optional, normative indicators include: Data standard constraints, data model constraints, metadata constraints, business rule constraints, authoritative reference data constraints, and security specification constraints; Integrity indicators include: data element integrity constraints and data record integrity constraints; Accuracy indicators include: data content correctness constraints, data format compliance constraints, data duplication rate constraints, data uniqueness constraints, and dirty data occurrence rate constraints; The consistency indicators include: same data consistency constraints and associated data consistency constraints; Timeliness indicators include: correctness constraints based on time periods, timeliness constraints based on time points, and timing constraints; Accessibility indicators include: accessibility constraints and usability constraints.

[0088] Optionally, the score of the data standard constraint is calculated as follows: ; in, represents the score of the data standard constraint, represents the constraint rule matrix, represents the rows of the constraint rule matrix, represents the columns of the constraint rule matrix, represents the column number of the constraint rule matrix, represents the row number of the constraint rule matrix, Represents the constraint rule matrix OK The elements of the column, Represents the number of data tuples of the target dataset; The score of the data element integrity constraint is calculated as follows: ; in, represents the score of the data element integrity constraint, represents the range vector of the constraint rule matrix, The constraint rule matrix elements, Indicates the number of data elements that the constraint rule matrix focuses on, The first values, express A data set of data elements.

[0089] Optionally, the quality element evaluation score is calculated by the following formula: ; in, represents the quality element evaluation score, express The first element of data quality Constraint rule evaluation scores, represents the second weight matrix The first element of data quality Constraint rules; The evaluation score of the dataset is calculated by the following formula: ; in, represents the evaluation score of the dataset, express The target dataset Quality element evaluation scores, Represents the first weight matrix The target dataset The weight of each data quality element.

[0090] Optionally, a fourth calculating unit 311 is further included, which is specifically configured to: The sub-service evaluation score is calculated based on the dataset attention matrix and the dataset evaluation score; The information service evaluation score is calculated based on the sub-service evaluation score and the sub-service attention matrix.

[0091] This application also provides a data quality evaluation device, see Figure 4 , Figure 4 An embodiment of a data quality evaluation device provided in the present application includes: Processor 401, memory 402, input and output unit 403, bus 404; The processor 401 is connected to the memory 402, the input and output unit 403 and the bus 404; The memory 402 stores a program, and the processor 401 calls the program to execute any of the above methods.

[0092] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above methods.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0094] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0096] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A data quality evaluation method, characterized in that: The method comprises: Constructing a sub-service attention matrix and a data set attention matrix based on the pre-acquired evaluation requirements, and determining demand indicators according to the evaluation requirements; Acquire an evaluation index library, and extract an evaluation index corresponding to the demand index from the evaluation index library; Determine and construct a data element attention matrix based on the evaluation index; Acquire data quality elements, set constraints for each of the data quality elements, and construct a constraint attention matrix; Allocating a first weight to each of the data quality elements according to the demand indicator to obtain a first weight sequence; According to the constraint condition attention matrix, a second weight is set for each of the data quality elements to obtain a second weight matrix; Collecting a target data set, wherein the target data set includes multiple data tuples, and each data tuple includes multiple data elements; Obtaining the number of problems of the data tuple that do not meet the constraint condition, and calculating the constraint rule evaluation score based on the number of problems, the data element attention matrix and the constraint rule algorithm; Calculate the quality element evaluation score according to the second weight matrix and the constraint rule evaluation score; A data set evaluation score is calculated based on the first weight sequence and the quality element evaluation score.

2. The method according to claim 1, characterized in that: The obtaining of the number of problems of the data tuple that does not meet the constraint condition, and calculating the constraint rule evaluation score based on the number of problems, the data element attention matrix and the constraint rule algorithm comprises: Obtaining the number of problems with the data tuples that violate the constraint condition; Calculate the problem tuple coverage rate according to the number of problems and the total number of tuples in the target data set; Acquire a verification result matrix of the target data set, and convert the verification result matrix into a constraint condition vector, wherein one or more constraint condition vectors correspond to a constraint rule; A constraint rule evaluation score is calculated based on the constraint rule, the constraint rule algorithm and the data element attention matrix.

3. The method according to claim 2, characterized in that The problem tuple coverage is calculated by the following formula: ； in, represents the problem tuple coverage, represents the number of questions for the data tuple, Represents the total number of tuples of the target data set; The constraint rule matrix is ​​calculated by the following formula: ； in, represents the Chinese name constraint vector, represents the English name constraint vector, represents the abbreviated constraint vector, It represents the definition and description of the constraint vector. represents the data type constraint vector, represents the range constraint vector, represents the control constraint vector, represents the maximum occurrence constraint vector, represents other constraint vectors, represents the constraint rule matrix; The constraint rule evaluation score is calculated by the following formula: ； in, represents the evaluation score of the constraint rule, represents the constraint rule algorithm, represents the constraint rule matrix, Represents the data element attention matrix.

4. The method according to any one of claims 1 to 3, characterized in that The constraint rule algorithm includes the requirement indicators, which include a standardization indicator, a completeness indicator, an accuracy indicator, a consistency indicator, a timeliness indicator, and an accessibility indicator.

5. The method according to claim 4, characterized in that The normative indicators include: Data standard constraints, data model constraints, metadata constraints, business rule constraints, authoritative reference data constraints, and security specification constraints; The integrity indicators include: data element integrity constraints and data record integrity constraints; The accuracy indicators include: data content correctness constraints, data format compliance constraints, data duplication rate constraints, data uniqueness constraints, and dirty data occurrence rate constraints; The consistency indicators include: same data consistency constraints and associated data consistency constraints; The timeliness index includes: correctness constraint based on time period, timeliness constraint based on time point and timing constraint; The accessibility index includes: accessibility constraints and usability constraints.

6. The method according to claim 5, characterized in that The score of the data standard constraint is calculated by the following formula: ； in, represents the score of the data standard constraint, represents the constraint rule matrix, The rows of the constraint rule matrix, represents the columns of the constraint rule matrix, represents the column number of the constraint rule matrix, represents the row number of the constraint rule matrix, Represents the constraint rule matrix OK The elements of the column, represents the number of the data tuples of the target data set; The score of the data element integrity constraint is calculated by the following formula: ； in, represents the score of the integrity constraint of the data element, represents the range vector of the constraint rule matrix, The constraint rule matrix is ​​represented by elements, represents the number of data elements concerned by the constraint rule matrix, Represents the value range vector values, express A data set containing the data elements.

7. The method according to claim 1, characterized in that The quality element evaluation score is calculated by the following formula: ； in, represents the evaluation score of the quality element, express The data quality elements The constraint rule evaluation score, Indicates the second weight matrix The data quality elements The constraints mentioned; The dataset evaluation score is calculated by the following formula: ； in, represents the evaluation score of the dataset, express The target dataset The quality element evaluation score, Indicates the first weight matrix The target dataset The weight of each of the data quality elements.

8. The method according to claim 1, characterized in that: The method further comprises: Calculate the sub-service evaluation score according to the data set attention matrix and the data set evaluation score; The information service evaluation score is calculated based on the sub-service evaluation score and the sub-service attention matrix.

9. A data quality evaluation system, characterized in that: include: A first construction unit is used to construct a sub-service attention matrix and a data set attention matrix based on pre-acquired evaluation requirements, and determine a demand index according to the evaluation requirements; An extraction unit, used to obtain an evaluation index library, and extract an evaluation index corresponding to the demand index from the evaluation index library; A second construction unit is used to determine and construct a data element attention matrix based on the evaluation index; A third construction unit is used to obtain data quality elements, set constraints for each of the data quality elements, and construct a constraint condition attention matrix; an allocating unit, configured to allocate a first weight to each of the data quality elements according to the demand indicator to obtain a first weight sequence; A setting unit, configured to set a second weight for each of the data quality elements according to the constraint condition attention matrix to obtain a second weight matrix; A collection unit, used for collecting a target data set, wherein the target data set includes a plurality of data tuples, and each data tuple includes a plurality of data elements; A first calculation unit is used to obtain the number of problems of the data tuple that do not meet the constraint condition, and calculate the constraint rule evaluation score based on the number of problems, the data element attention matrix and the constraint rule algorithm; A second calculation unit, configured to calculate a quality element evaluation score according to the second weight matrix and the constraint rule evaluation score; The third calculation unit is used to calculate the data set evaluation score according to the first weight sequence and the quality element evaluation score.

10. A data quality evaluation device, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 8.

Images