A data quality evaluation method, device and medium

Abstract

This application discloses a data quality assessment method, device, and medium. The method includes: establishing a data quality assessment index system based on multiple preset assessment dimensions; obtaining the data table and metadata table corresponding to the data to be assessed, and determining the measured values ​​of each assessment index for the data to be assessed under the data quality assessment index system; determining the assessment index weight matrix and assessment index selection matrix corresponding to the data to be assessed; and constructing a data quality assessment model based on the assessment index weight matrix, assessment index selection matrix, and measured values, so as to conduct data quality assessment on the data to be assessed according to the data quality assessment model. By establishing a comprehensive multi-dimensional, multi-indicator quality assessment index system and constructing a reasonable and effective quality assessment analysis model, quantitative assessment and analysis of data quality can be achieved, providing targeted guidance for data development and use, and laying the foundation for improving data quality.

Description

Technical Field

[0001] This application relates to the field of computers, specifically to a data quality assessment method, device, and medium. Background Technology

[0002] As a crucial information carrier, data is a factor of production and a vital productive force. It is the cornerstone of socio-economic development and a fundamental and strategic resource of modern society. Widely applied in numerous industries such as retail, telecommunications, manufacturing, agriculture, and finance, it generates immense social value and industrial potential. Its practical application value is primarily reflected in two important aspects: firstly, data can help enterprises analyze markets and development trends to enhance their innovation capabilities and core competitiveness; secondly, it can assist regulatory authorities in their oversight and decision-making, improving the quality and efficiency of their services.

[0003] However, in many real-world scenarios, data quality issues arise, including incomplete data, inconsistencies, missing metadata, inaccurate data types, non-standard data formats, unreasonable data values, duplicate or redundant data, and invalid data. These problems significantly impact the reliability of the information contained within the data, thereby affecting its actual value. Therefore, it is necessary to assess and analyze data quality issues. Summary of the Invention

[0004] To address the aforementioned issues, this application proposes a data quality assessment method, comprising:

[0005] A data quality assessment index system is established based on multiple preset assessment dimensions, with each assessment dimension containing multiple assessment indicators.

[0006] Obtain the data table and metadata table corresponding to the data to be evaluated, and based on the attributes of the data table and the metadata table, determine the measured values ​​of each evaluation indicator corresponding to the data to be evaluated under the data quality evaluation indicator system.

[0007] Determine the weight matrix and selection matrix of the evaluation indicators corresponding to the data to be evaluated;

[0008] A data quality assessment model is constructed based on the assessment indicator weight matrix, the assessment indicator selection matrix, and the measured values, so as to assess the data quality of the data to be assessed based on the data quality assessment model.

[0009] On the other hand, this application also proposes a data quality assessment device, comprising:

[0010] At least one processor; and,

[0011] A memory communicatively connected to the at least one processor; wherein,

[0012] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform a data quality assessment method as described in the above example.

[0013] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as the data quality assessment method described in the above example.

[0014] The data quality assessment method proposed in this application can bring the following beneficial effects:

[0015] By establishing a comprehensive, multi-dimensional, and multi-indicator quality assessment system and constructing a reasonable and effective quality assessment and analysis model, we can achieve quantitative assessment and analysis of data quality, provide targeted guidance for data development and use, and lay the foundation for improving data quality. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a flowchart illustrating the data quality assessment method in the embodiments of this application;

[0018] Figure 2 This is a schematic diagram of a data quality assessment method in one scenario of this application embodiment;

[0019] Figure 3 This is a schematic diagram of the data quality assessment device in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0022] like Figure 1 As shown in the embodiments of this application, a data quality assessment method is provided, including:

[0023] S101: Based on multiple preset evaluation dimensions, establish a data quality evaluation index system, wherein each evaluation dimension contains multiple evaluation indicators.

[0024] Figure 2 This is a schematic diagram of a data quality assessment method in one scenario of this application embodiment. The following is in conjunction with... Figure 1 and Figure 2 Provide an explanation.

[0025] Specifically, multiple pre-defined evaluation dimensions are determined, and each evaluation dimension contains multiple evaluation indicators.

[0026] Furthermore, the evaluation dimensions include: completeness evaluation, validity evaluation, uniqueness evaluation, and consistency evaluation. For the completeness evaluation dimension, the evaluation indicators include: non-null value rate and attribute data non-null value rate; for the validity evaluation dimension, the evaluation indicators include: data type validity, data format validity, and data value retrieval validity; for the uniqueness evaluation dimension, the evaluation indicators include: primary key data uniqueness rate and non-primary key data uniqueness rate; for the consistency evaluation dimension, the evaluation indicators include: data equivalence consistency rate, data logical consistency rate, and data existence consistency rate. The data quality evaluation indicator system can be shown in the table below.

[0027]

[0028]

[0029] The evaluation indicators in the data quality assessment indicator system can be divided into two groups based on whether the scope of investigation requires additional settings: one group consists of basic evaluation indicators, including non-null rate of data, non-null rate of attribute data, and data type effectiveness rate, which are evaluation indicators for the overall data by default; the other group consists of pending evaluation indicators, including other evaluation indicators besides the basic evaluation indicators, which need to be selected and set according to the characteristics of the data itself. Basic evaluation indicators are default and fixed, used to evaluate each data table to be evaluated, while pending evaluation indicators need to be flexibly selected and used according to the characteristics of the data table to be evaluated. All evaluation indicators are positive evaluation indicators defined by the ratio method, taking values ​​between 0 and 1. The corresponding measured values ​​obtained are standardized and can be used directly without standardization.

[0030] In the constructed data quality assessment index system, each assessment index reasonably describes the quality problems of the data to be assessed in certain specific aspects. For example, data with format rule constraints such as mobile phone numbers, email addresses, postal codes, IP addresses, and ID card numbers can be quantitatively described in terms of format standardization using data format effectiveness rate; data with enumerated values ​​such as gender, education level, grade classification, and marital status, as well as data with reasonable value patterns such as height, weight, scores, and salaries, can be quantitatively described in terms of value standardization using data value effectiveness rate; data with computational dependency relationships such as total scores and individual subject scores can be quantitatively described in terms of computational correlation consistency using data equivalence consistency rate; data with logical dependency relationships such as annual sales and monthly sales, PV and UV can be quantitatively described in terms of logical correlation consistency using data logical consistency rate; and data with matching dependency relationships such as ID card numbers and birth dates can be quantitatively described in terms of matching correlation consistency using data existence consistency rate.

[0031] S102: Obtain the data table and metadata table corresponding to the data to be evaluated, and based on the attributes of the data table and the metadata table, determine the measured values ​​of each evaluation indicator corresponding to the data to be evaluated under the data quality evaluation indicator system.

[0032] Specifically, the data to be evaluated is acquired, and an evaluation data table is generated based on this data. The data to be evaluated can be obtained from the relevant system. To enable quantitative evaluation and analysis of the data based on the data quality evaluation index system, it is also necessary to determine the metadata table recording relevant metadata information of the data table to be evaluated by automatically collecting relevant data tables from the data source system library containing the created database, tables, and field information, and by manually filling in the metadata. The metadata table, which consists of the metadata used to record the metadata of the data table to be evaluated, includes table name, field name, field type, field length, format constraints, value constraints, field description, business definition, business rules, creation time, and update time. This is based on the number of rows N in the data table to be evaluated. r ,Number of fields c and the number of fields N in the metadata table m The actual measured values ​​of each evaluation indicator under the data quality evaluation indicator system are determined for the data to be evaluated.

[0033] Based on this, calculation rules can be designed for each evaluation indicator in the data table to be evaluated. The actual measured values ​​of each indicator can be calculated through these calculation rules.

[0034] The non-null rate calculation rule is as follows: Use the `count(probe field)` function to obtain the number of records in the table to be evaluated whose probe fields are not null or not empty characters. Add the corresponding record counts for each probe field to obtain the total number of records. Calculate the measured value corresponding to the non-null rate of the data.

[0035] The rule for calculating the non-null value rate of attribute data is as follows: Use the `count(probe field)` function to obtain the number of records in the metadata table where the probe field is not null or not an empty character. Then, sum the records corresponding to each probe field to obtain the total number of records. Calculate the measured value corresponding to the non-null rate of attribute data.

[0036] Data type efficiency calculation rules: Based on the unique characteristics or specific operations of the corresponding data type, set corresponding filtering conditions. Use these conditions to filter the corresponding probe fields and use the count(probe field) function to obtain the number of records that meet the conditions. Add up the record counts for each probe field to obtain the total number of records. Calculate the measured value corresponding to the data type effectiveness

[0037] Data format efficiency calculation rules: Based on the characteristics of the data table to be evaluated and the metadata table, set S of fields in the data table to be evaluated that participate in the exploration of this indicator is defined. f Based on the format constraint rules corresponding to the probe fields, set corresponding regular expressions as filtering conditions. Use these conditions to filter the corresponding probe fields and use the count(probe field) function to obtain the number of records that meet the conditions. Add up the record counts for each probe field to get the total number of records. Calculate the measured value corresponding to the data format effectiveness.

[0038] Data value retrieval efficiency calculation rules: Based on the characteristics of the data table to be evaluated and the metadata table, set S of fields in the data table to be evaluated that participate in the exploration of this indicator is defined. v Based on the value constraints of the probe field or the value range of other reference data tables, a combination of comparison inequalities, IN, BETWEEN, and other operations is set as the filtering condition. This condition is used to filter the corresponding probe field, and the COUNT(probe field) function is used to obtain the number of records that meet the condition. The total number of records is obtained by summing the counts for each probe field. Calculate the measured value corresponding to the effectiveness of the data.

[0039] Primary key single key data uniqueness calculation rule: Use the count(distinct primary key field) function to obtain the number of records in the table to be evaluated whose primary key field is not null or not an empty character. Calculate the measured value corresponding to the single primary key data rate.

[0040] Non-primary key data singleness rate calculation rule: The set of fields S participating in the index exploration is set according to the characteristics of the data table to be evaluated. np Use the `count(distinct probe field)` function to get the number of records where the probe field is not null or not an empty character. Add the records corresponding to each probe field to get the total number. Calculate the measured value corresponding to the singleness rate of non-primary key data.

[0041] Data consistency rate calculation rules: The set of fields S participating in the index investigation is set according to the characteristics of the data table to be evaluated. e The set of associated fields corresponding to each probe field; using the count(probe field - f(associated field1, ..., associated field s)) function (where f is an expression based on the calculation rules between the probe field and each associated field) to obtain the number of records in the data table to be evaluated that satisfy "probe field - f(associated field1, ..., associated field s) = 0", and summing the number of records corresponding to each probe field to obtain the total number of records. Calculate the measured value corresponding to the data equivalence rate.

[0042] Data logical consistency rate calculation rules: The set of fields S participating in the exploration of this indicator is set according to the characteristics of the data table to be evaluated. l For each probe field, identify its associated field; use the `count(probe field - associated field)` function to obtain the number of records in the data table to be evaluated that satisfy "probe field - associated field" being greater than or less than 0; sum the records corresponding to each probe field to obtain the total number of records. Calculate the measured value corresponding to the logical consistency rate of the data.

[0043] Data consistency calculation rules: The set of fields S participating in the exploration of this indicator is set according to the characteristics of the data table to be evaluated. x For each probe field, define the associated fields; set a regular expression based on the matching relationship as the filtering condition, and use this condition to filter the corresponding probe fields. Use `count(probe field)` to get the number of records that meet the condition for that probe field. Add up the records corresponding to each probe field to get the total number of records. The calculated data has a consistency rate corresponding to the measured values.

[0044] S103: Determine the evaluation index weight matrix and evaluation index selection matrix corresponding to the data to be evaluated.

[0045] To perform data quality assessment and analysis, it is necessary to calculate the weight data of the assessment indicators, the indicator selection data of the data table to be assessed, and the measured values ​​of the assessment indicators (the measured values ​​have been obtained through the calculation rules mentioned above).

[0046] The weights of the evaluation indicators are used to characterize the relative importance of each evaluation indicator in the data quality assessment analysis. Consider using the evaluation indicator weight matrix W = (w... i,j ) 4×3 As the main form for storing and subsequently calculating relevant weight data values, its element w i,j Indicators In dimension D i The corresponding weight data value, w 1, and w 3, The first element is always 0, and the remaining elements take values ​​in the interval (0,1) with the sum of the elements in each row being 1. Through professional surveys or scoring, the weight matrix of the evaluation indicators can be reasonably determined using the analytic hierarchy process (AHP).

[0047] The data selected for the evaluation table is used to mark the actual selection of each evaluation indicator in the data quality assessment analysis. Consider using an indicator selection matrix M = (m... i,j ) 4×3 As the main form for storing and subsequently calculating the selected marker values ​​of relevant indicators, its element m i,j This indicates the selection of a marker value for the indicator. This marker value is a Boolean value and can only take the values ​​1 or 0, corresponding to whether the indicator is considered or not when conducting a quality assessment of the data table to be evaluated. These are the two scenarios.

[0048] Based on the grouping characteristics of the evaluation indicators, the selection matrix for the evaluation indicators varies depending on the specific evaluation indicators included in the data quality evaluation indicator system. Therefore, the evaluation mode can be set into two categories: the basic evaluation mode uses only basic evaluation indicators to evaluate data quality; the setting evaluation mode uses both basic evaluation indicators and undetermined evaluation indicators to evaluate data quality. Correspondingly, the indicator selection matrix M corresponding to the data table to be evaluated can take the following values:

[0049] The indicator selection marker value b is a Boolean value, and the corresponding value needs to be set according to whether the corresponding evaluation indicator is considered.

[0050] For each given data table to be evaluated, the basic evaluation mode is used by default to evaluate its quality. In this case, the indicator selection matrix M corresponding to the data table to be evaluated is assumed to be M. basicBased on this, a more detailed quality assessment can be conducted by combining the metadata table with the specific characteristics of the data table to be evaluated and selecting the appropriate assessment mode. In this case, the indicator matrix M corresponding to the data table to be evaluated is M. defined The indicator selection marker value b can be set to 1 according to the undetermined evaluation indicators used in this evaluation mode.

[0051] The measured data for the evaluation indicators are the actual values ​​of each evaluation indicator after exploring the data table and metadata table to be evaluated according to the corresponding rules. Based on the calculation rules of the evaluation indicators, an SQL-based indicator data calculation algorithm can be determined. This algorithm can explore the data table and metadata table to be evaluated according to the indicator exploration range and corresponding indicator constraint rules, and obtain the corresponding measured indicator data.

[0052] For each given data table to be evaluated, if the evaluation mode defaults to the basic evaluation mode, the corresponding indicator data calculation algorithm can be used directly to obtain the corresponding measured indicator data and synchronize it to the indicator data table. If the evaluation mode is set to the evaluation mode, the exploration field of the selected indicator to be evaluated and the corresponding constraint rules need to be specified first. Based on this, the corresponding indicator data calculation algorithm can be used to obtain the corresponding measured indicator data and synchronize it to the indicator data table.

[0053] S104: Construct a data quality assessment model based on the assessment index weight matrix, the assessment index selection matrix, and the measured values, so as to assess the data quality of the data to be assessed based on the data quality assessment model.

[0054] For data quality assessment processes involving multiple dimensions and indicators, based on the weight matrix of assessment indicators, the indicator selection matrix of the data table to be assessed, and the indicator data table, the following data quality assessment and analysis model is given using the weighted average method.

[0055] The formula for calculating the data quality score of the data table to be evaluated is:

[0056]

[0057] And dimension D i The formula for calculating the impact of (=1,2,3,4) on data quality is:

[0058]

[0059] in This represents the Hadamard product of matrices. Representation matrix The element in the i-th row and j-th column, Representing dimension D i The actual measured data of the corresponding indicators.

[0060] The data quality assessment and analysis model can score the data quality of the data table to be evaluated based on the relevant data of the assessment indicators. It can also further measure the degree of influence of the assessment dimensions, helping to understand the overall data quality status and pinpoint data quality problems. Based on the data quality assessment and analysis model, for each given data table to be evaluated, by substituting its corresponding indicator selection matrix and measured indicator data into the model, the quality score and dimensional influence data of the data table to be evaluated can be calculated, and then synchronized to the evaluation data table.

[0061] As a crucial method for measuring data quality and identifying data problems, and an important form of reflecting the value of data assets, data quality assessment and analysis involves using scientific and reasonable assessment methods based on multiple assessment dimensions and constraint rules to quantitatively evaluate and analyze data in databases or data warehouses, obtaining the quality level or score of the data to be assessed and its quality analysis status. The data quality assessment dimensions and constraint rules need to be selected according to the characteristics of the data being assessed and the focus of the assessment objectives. Commonly used assessment dimensions include completeness, accuracy, uniqueness, validity, consistency, and timeliness, while commonly used constraint rules include primary key constraints, uniqueness constraints, value constraints, format constraints, type constraints, consistency constraints, and non-null constraints.

[0062] Data quality assessment and analysis is a systematic and comprehensive task. Its essence lies in determining the data quality score and status by quantifying the degree to which the data being assessed conforms to all constraints associated with the assessment indicators under each assessment dimension. This includes an assessment indicator system, an assessment analysis model, and the implementation techniques of the assessment methods. Through data quality assessment and analysis, we can obtain information about the data quality status, understand the data quality level, and identify data quality problems. This provides targeted guidance for various data usage and development efforts, lays the foundation for improving data quality, and enhances the effectiveness of data-driven initiatives. Furthermore, as people's requirements for data quality and their focus on data value evolve, data quality assessment must meet higher-level requirements, placing greater emphasis on the applicability and accuracy of data quality assessment and analysis methods, and establishing a systematic, comprehensive, and flexible data quality assessment and analysis methodology system.

[0063] like Figure 3 As shown in the illustration, this application also provides a data quality assessment device, including:

[0064] At least one processor; and,

[0065] A memory communicatively connected to the at least one processor; wherein,

[0066] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform a data quality assessment method as described in any of the above embodiments.

[0067] This application also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as the data quality assessment method described in any of the above embodiments.

[0068] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0069] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0070] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0072] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0075] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0076] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0077] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A data quality assessment method, characterized in that, include: A data quality assessment index system is established based on multiple preset assessment dimensions, with each assessment dimension containing multiple assessment indicators. Obtain the data table and metadata table corresponding to the data to be evaluated, and based on the attributes of the data table and the metadata table, determine the measured values ​​of each evaluation indicator corresponding to the data to be evaluated under the data quality evaluation indicator system. Determine the weight matrix and selection matrix of the evaluation indicators corresponding to the data to be evaluated; Based on the evaluation index weight matrix, the evaluation index selection matrix, and the measured values, a data quality evaluation model is constructed so as to evaluate the data to be evaluated based on the data quality evaluation model. Based on the evaluation index weight matrix, the evaluation index selection matrix, and the measured values, a data quality evaluation model is constructed, specifically including: pass Obtain the data quality score corresponding to the data table to be evaluated; pass The evaluation dimensions are obtained from the data table to be evaluated. The degree of impact on data quality; in, The data quality score corresponding to the data table to be evaluated. Evaluation dimensions in the data table to be evaluated The degree of impact on data quality Represents the Hadamard product of matrices. Representation matrix No. Line number Column elements, Indicates evaluation dimensions The measured values ​​of the corresponding evaluation indicators. To evaluate the indicator weight matrix, Select a matrix for the evaluation indicators.

2. The method according to claim 1, characterized in that, A data quality assessment indicator system is established based on multiple pre-defined assessment dimensions, specifically including: Multiple preset evaluation dimensions are determined, and each evaluation dimension contains multiple evaluation indicators; The types of the evaluation indicators include: basic evaluation indicators and undetermined evaluation indicators, wherein the undetermined evaluation indicators are obtained based on the requirements; Establish a data quality assessment indicator system based on all identified assessment indicators.

3. The method according to claim 2, characterized in that, The evaluation dimensions include: completeness evaluation dimension, effectiveness evaluation dimension, uniqueness evaluation dimension, and consistency evaluation dimension; For the aforementioned integrity assessment dimensions, the assessment metrics include: data non-null rate and attribute data non-null rate. The evaluation indicators for the aforementioned validity assessment dimensions include: data type validity, data format validity, and data value validity. For the uniqueness assessment dimensions, the assessment indicators include: primary key data uniqueness rate and non-primary key data uniqueness rate. The evaluation metrics for the aforementioned consistency assessment dimensions include: data equivalence consistency rate, data logical consistency rate, and data existence consistency rate.

4. The method according to claim 3, characterized in that, Obtain the data table and metadata table corresponding to the data to be evaluated, and based on the attributes of the data table and the metadata table, determine the measured values ​​of each evaluation indicator for the data to be evaluated under the data quality evaluation indicator system, specifically including: Obtain the data to be evaluated, and generate an evaluation data table based on the data to be evaluated; A metadata table is determined, which is used to record the metadata of the data table to be evaluated. The metadata table includes table name, field name, field type, field length, format constraints, value constraints, field description, business definition, business rules, creation time, and update time. Based on the number of rows and fields in the data table to be evaluated, and the number of fields in the metadata table, determine the measured values ​​of each evaluation indicator corresponding to the data to be evaluated under the data quality evaluation indicator system.

5. The method according to claim 4, characterized in that, Based on the number of rows and fields in the data table to be evaluated, and the number of fields in the metadata table, determine the measured values ​​of each evaluation indicator corresponding to the data to be evaluated under the data quality evaluation indicator system, specifically including: Determine the number of rows in the data table to be evaluated. , number of fields and the number of fields in the metadata table. ; Regarding the non-null rate of the data, through The corresponding measured values ​​were obtained, among which, The non-null rate of the data. This refers to the sum of all non-empty record values ​​in all probe fields of the data table to be evaluated; Regarding the non-null rate of the aforementioned attribute data, through The corresponding measured values ​​were obtained, among which, The percentage of non-null values ​​in attribute data. This refers to the sum of all non-empty record values ​​in all probe fields of the metadata table. For the efficiency of the aforementioned data type, through The corresponding measured values ​​were obtained, among which, For data type efficiency, The sum of the values ​​of the probe fields in the data table to be evaluated that meet the preset filtering conditions after filtering. Regarding the efficiency of the aforementioned data format, through The corresponding measured values ​​were obtained, among which, For efficient data formatting, This refers to the set of fields in the data table to be evaluated that participate in data format efficiency detection, based on the metadata table. The sum of record values ​​in the probe field of the data table to be evaluated that meet the filtering conditions after filtering by setting the corresponding regular expression according to the format constraint rules; Regarding the effectiveness of the data retrieval, through The corresponding measured values ​​were obtained, among which, To ensure efficient data retrieval, This is a set of fields in the data table to be evaluated that participate in the data value retrieval efficiency detection, based on the metadata table. The sum of record values ​​that meet the filtering conditions in the probe field of the data table to be evaluated after filtering by a combination of preset operations based on value constraint rules or value range settings, wherein the preset operations include at least one of comparison inequalities, in, and between. Regarding the single-value rate of the primary key data, through The corresponding measured values ​​were obtained, among which, Single primary key data rate The number of records in the data table to be evaluated where the primary key field is not empty; Regarding the single-value rate of the non-primary key data, through The corresponding measured values ​​were obtained, among which, The singleness rate of non-primary key data. The set of fields in the data table to be evaluated that participate in the detection of singleness rate of non-primary key data. The sum of all non-empty records in the probe fields of the data table to be evaluated; Regarding the data equivalence rate, through The corresponding measured values ​​were obtained, among which, For data consistency rate, The set of fields in the data table to be evaluated that participate in the data equivalence consistency rate detection. The sum of the record values ​​that meet the conditions in the probe field of the data table to be evaluated after filtering according to the filtering conditions set by the calculation rules of the probe field and each associated field; Regarding the data logical consistency rate, through The corresponding measured values ​​were obtained, among which, For data logical consistency rate, The set of fields in the data table to be evaluated that participate in the data logic consistency rate detection. The sum of record values ​​in the probe field of the data table to be evaluated that meet the filtering conditions after filtering based on the logical rules set by the probe field and the logical association field; Regarding the consistency rate of the data, through The corresponding measured values ​​were obtained, among which, For the data to have a consistency rate, The set of fields in the data table to be evaluated that participate in the data consistency rate detection. This refers to the sum of record values ​​in the probe field of the data table to be evaluated that meet the filtering criteria after filtering using regular expressions based on matching relationships.

6. The method according to claim 2, characterized in that, Determining the evaluation index weight matrix and evaluation index selection matrix corresponding to the data to be evaluated specifically includes: Based on the importance of each evaluation indicator to the data to be evaluated, determine the weight matrix of the evaluation indicators corresponding to the data to be evaluated. , among which, element Indicators In evaluation dimensions The corresponding weight data values, and The constant value is 0, and the remaining elements take values ​​within the range. And the sum of the elements in each row is 1; Based on the selection of each evaluation indicator for the data to be evaluated, determine the evaluation indicator selection matrix corresponding to the data to be evaluated. , among which, element This indicates the selection of a label value for the evaluation indicator.

7. The method according to claim 6, characterized in that, The evaluation index selection matrix is ​​determined based on the actual selection of evaluation indicators during the data quality assessment process. Wherein, when the evaluation indicators selected in the data quality evaluation index system only include the basic evaluation indicators and do not include the pending evaluation indicators, it is the basic evaluation mode; when the evaluation indicators selected in the data quality evaluation index system include the basic evaluation indicators and at least one of the pending evaluation indicators, it is the setting evaluation module. The evaluation index selection matrix ; Among them, the evaluation indicators select labeled values. This is a Boolean value, set according to your needs.

8. A data quality assessment device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform a data quality assessment method as described in any one of claims 1 to 7.

9. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured as follows: the data quality assessment method as described in any one of claims 1 to 7.

Images