A data quality assessment system
By constructing a data quality assessment system, the problems of insufficient scalability and universality in existing technologies are solved, enabling efficient and flexible assessment of different datasets and improving the applicability and practicality of data quality assessment.
Patent Information
- Application Number
- CN202310394294.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-13
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2043-04-13
AI Technical Summary
Existing data quality assessment methods lack scalability and universality, with overlapping and repetitive assessment indicators, making it difficult to promote and apply them on a larger scale.
A data quality assessment system is constructed, including a data dictionary metadata model, a metadata model, a data quality assessment model, an assessment task management module, an assessment task scheduling module, a quality monitoring and alarm module, and an assessment result storage module. The system extracts metadata for analysis and management, calculates the quantitative quality status under each assessment indicator dimension, and provides alarm prompts.
It achieves the universality and scalability of data quality assessment, enabling effective assessment of data from different sources and formats, thus improving the practicality and applicability of data quality assessment.
Smart Images
Figure CN116701890B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to data quality, in particular to a data quality evaluation system. BACKGROUND
[0002] In the era of big data, data has penetrated into various industries and fields and become an important production factor. With the explosive growth of data volume, more and more invalid, erroneous, redundant, non-standard and inconsistent data problems occur. Good data quality is the basis for ensuring data application, and low-quality data will lead us to make wrong decisions.
[0003] With the use of massive multi-source and heterogeneous data, the importance of data quality is further highlighted. Data quality evaluation is a process of determining the current data quality status of a data set, and is the implementation of data quality evaluation process and evaluation result analysis process under different evaluation index dimensions.
[0004] The existing data quality evaluation methods are basically constructed from business to build data quality evaluation models, which are highly targeted, but lack scalability and universality, have too many evaluation index dimensions, and some evaluation index dimensions have problems such as overlapping and repetition. The understanding and quantification methods of some evaluation index dimensions of the model in different data quality evaluation tasks are prone to differences, which reduces the scalability and practicability of the model and makes it difficult to be widely applied. SUMMARY
[0005] (I) Technical problems to be solved
[0006] In view of the above-mentioned shortcomings of the prior art, the present application provides a data quality evaluation system, which can effectively overcome the defects of poor scalability and universality of the prior art.
[0007] (II) Technical solutions
[0008] To achieve the above-mentioned purposes, the present application is realized by the following technical solutions:
[0009] A data quality evaluation system comprises a data dictionary metadata model, a metadata model, a data quality evaluation model, an evaluation task management module, an evaluation task scheduling module, a quality monitoring and alarm module, and an evaluation result storage module.
[0010] The data dictionary metadata model extracts metadata from a database metadata table and manually improves it, and is used for database connection and testing to provide support for driving data quality evaluation.
[0011] A metadata model, including a constraint rule metadata model containing constraint rules corresponding to evaluation indexes, a rule analysis metadata model containing constraint rule analysis functions, and an index evaluation metadata model containing evaluation index calculation functions;
[0012] A data quality evaluation model, which determines a data set to be evaluated according to the metadata, selects evaluation indexes, analyzes constraint rules corresponding to the evaluation indexes, updates the analysis results to the database through an evaluation result storage module, and finally calculates quantitative quality conditions under each evaluation index dimension;
[0013] An evaluation task management module, which creates a data quality evaluation task and manages the data quality evaluation task;
[0014] An evaluation task scheduling module, which sets parameters for the execution of the data quality evaluation task and updates the execution results of the data quality evaluation task to the database through the evaluation result storage module;
[0015] A quality monitoring and alarming module, which judges whether the data quality of the data set to be evaluated meets the actual business requirements, alarms and prompts when an abnormal situation occurs in the execution process of the data quality evaluation task, and updates to the database through the evaluation result storage module.
[0016] Preferably, the data quality evaluation model includes a constraint rule analysis module and an evaluation index processing module;
[0017] The constraint rule analysis module queries constraint rules corresponding to the evaluation indexes through the constraint rule metadata model, analyzes the constraint rules through the rule analysis metadata model, calls corresponding constraint rule analysis functions, updates analysis progress information, running exception information and data violating the constraint rules generated in the analysis process to the database through the evaluation result storage module, and stores summary information and evaluation index information analyzed and calculated to the index evaluation metadata model;
[0018] The evaluation index processing module calculates quantitative quality conditions under each evaluation index dimension by calling corresponding evaluation index calculation functions through the index evaluation metadata model.
[0019] Preferably, the data quality evaluation model is composed of a data set to be evaluated, a set of evaluation indexes, a set of constraint rules, a set of evaluation instances and a set of evaluation algorithms, and the data quality evaluation model is represented by a six-tuple as follows:
[0020] QA<S,D,R,W,I,A>
[0021] Wherein, S is the data set to be evaluated, D is the evaluation index set corresponding to the data set S to be evaluated, R is the constraint rule set corresponding to the evaluation index set D, W is the weight set of the weight of each constraint rule in the constraint rule set R, which describes the proportion of each constraint rule in the constraint rule set R, I is the evaluation instance set, and A is the evaluation algorithm set.
[0022] Preferably, the data quality evaluation model calculates the quantitative quality status under each evaluation index dimension, including the following steps:
[0023] S1, selecting the metadata MD extracted from the database metadata table, determining the data set S to be evaluated and selecting the evaluation index set D;
[0024] S2, traversing each constraint rule in the constraint rule set R corresponding to the evaluation index set D and the weight of each constraint rule in the weight set W, analyzing the constraint rule on the evaluation instance set I through the constraint rule analysis module;
[0025] S3, calculating the quantitative quality status under each evaluation index dimension through the evaluation index processing module.
[0026] Preferably, the data dictionary metadata model represents the metadata through the following four-tuple:
[0027] MD<db,dt,cl,ci>
[0028] Wherein, db is the description information of the database, including the database physical location information, the database type, the database version number, the database name, the URL of the database, and the username and password for accessing the database;
[0029] dt is the description information of the metadata table, including the table name, the table description, and the database information to which the table belongs;
[0030] cl is the description information of the field, including the field name, the field code, and the metadata table information to which the field belongs;
[0031] ci is the constraint information of the field, including the data type of the field value, the value range, the integrity constraint, and the field primary foreign key.
[0032] Preferably, the data quality evaluation model calculates the quantitative quality status under each evaluation index dimension, including:
[0033] Taking the accuracy in the evaluation index as an example, the quantitative quality status under the accuracy dimension is calculated, specifically including:
[0034] The number of problem classification S q : The number of constraint rule violation categories when data quality evaluation is performed on the data set to be evaluated;
[0035] Number of problem records R q : When performing data quality assessment on the data set to be assessed, the number of records that violate constraint rules, and the same record that violates multiple constraint rules is not repeated calculation;
[0036] Number of problem data D q : When performing data quality assessment on the data set to be assessed, the number of records that violate constraint rules, and the same record that violates multiple constraint rules is repeated calculation;
[0037] Number of problem data items C q : When performing data quality assessment on the data set to be assessed, the number of data items that violate constraint rules, and one attribute that violates multiple constraint rules is not repeated calculation;
[0038] Number of missing data items C s : The difference between the number of metadata defined data items and the actual number of data items in the data set to be assessed;
[0039] Assuming that the number of records of the data set to be assessed is C r , the number of metadata defined data items is C qd , then the data accuracy is calculated as follows:
[0040]
[0041] Preferably, the evaluation indicators include kernel mode layer evaluation indicators and extended instance layer evaluation indicators, the kernel mode layer evaluation indicators include integrity, consistency, uniqueness and validity, and the extended instance layer evaluation indicators include timeliness, security and accuracy.
[0042] Preferably, the evaluation task management module acquires the data set to be assessed, the data dictionary metadata model corresponding to the data set to be assessed, and the metadata model, creates a data quality assessment task, and adds data quality assessment task information for management;
[0043] The evaluation task scheduling module sets parameters including execution period, execution mode and exception handling of the data quality assessment task, and updates the execution progress information, data that violates constraint rules and the quantified quality status under each evaluation indicator dimension generated during the execution of the data quality assessment task to the database through the evaluation result warehousing module.
[0044] Preferably, the quality monitoring and alarming module quantitatively and qualitatively analyzes the quantified quality status under each evaluation indicator dimension to obtain the data quality assessment level of the data set to be assessed, and judges whether the data quality of the data set to be assessed meets the actual business requirements according to the data quality assessment level;
[0045] The quality monitoring alarm module alarms when an abnormal situation occurs during the execution of the data quality evaluation task, and the evaluation result storage module updates the data quality evaluation level and the data quality evaluation task execution abnormal situation to the database.
[0046] Preferably, a visualization module is further included, which analyzes and statistically processes the data information fed back by the evaluation task scheduling module and the quality monitoring alarm module, and visually displays the data quality condition, the data quality analysis report, the constraint rule violation data statistics, the problem rectification suggestion, and the progress information and abnormal information of the data quality evaluation task execution.
[0047] (III) Beneficial Effects
[0048] Compared with the prior art, the data quality evaluation system provided by the application constructs a complete data quality evaluation system for the data quality problems of the data center, including the establishment and classification of evaluation indexes and constraint rules, the system structure and process of data quality evaluation, and the like.
[0049] Based on the data dictionary metadata model and the metadata model, the data quality evaluation process is abstracted by the data quality evaluation model, the specific objects and the related attributes of the objects involved in the data quality evaluation are separated from the evaluation process, the data quality evaluation is separated from the specific database, the evaluation indexes and the constraint rules can be added and customized according to the requirements, the evaluation indexes and the evaluation algorithms can be expanded, the generality of the same data quality evaluation index system for the data quality evaluation of different to-be-evaluated data sets is improved, and the goal of evaluating the quality of data of different sources and formats is achieved. BRIEF DESCRIPTION OF DRAWINGS
[0050] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.
[0051] Figure 1 The system and flowchart of the application;
[0052] Figure 2 The schematic diagram of the data quality evaluation index system in the application;
[0053] Figure 3 The classification schematic diagram of the evaluation indexes in the application. Figure 2 DETAILED DESCRIPTION
[0054] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.
[0055] A data quality evaluation system, as shown in Figure 1 includes a data dictionary metadata model, a metadata model, a data quality evaluation model, an evaluation task management module, an evaluation task scheduling module, a quality monitoring alarm module and an evaluation result storage module.
[0056] The data dictionary metadata model extracts metadata from a database metadata table and is manually improved, used for database connection and testing, and provides support for driving data quality evaluation.
[0057] The metadata model includes a constraint rule metadata model containing constraint rules corresponding to evaluation indexes, a rule analysis metadata model containing constraint rule analysis functions, and an index evaluation metadata model containing evaluation index calculation functions.
[0058] The data quality evaluation model determines a data set to be evaluated according to metadata, selects evaluation indexes, analyzes constraint rules corresponding to the evaluation indexes, updates the analysis results to the database through the evaluation result storage module, and finally calculates the quantitative quality status under each evaluation index dimension.
[0059] The evaluation task management module creates a data quality evaluation task and manages the data quality evaluation task.
[0060] The evaluation task scheduling module sets parameters for the execution of the data quality evaluation task, and updates the data quality evaluation task execution results to the database through the evaluation result storage module.
[0061] The quality monitoring alarm module judges whether the data quality of the data set to be evaluated meets the actual business requirements, and gives an alarm prompt when an abnormal situation occurs in the data quality evaluation task execution process, and updates to the database through the evaluation result storage module.
[0062] The data dictionary metadata model represents metadata through the following four tuples:
[0063] MD<db,dt,cl,ci>
[0064] Wherein, db is the description information of the database, including the database physical location information, database type, database version number, database name, database URL, username and password for accessing the database;
[0065] dt is the description information of the metadata table, including the table name, table description, and database information to which the table belongs;
[0066] cl is the description information of the field, including the field name, field code, and metadata table information to which the field belongs;
[0067] ci is the constraint information of the field, including the data type of the field value, value range, integrity constraint, and field primary foreign key.
[0068] 2) The data quality evaluation model includes a constraint rule analysis module and an evaluation index processing module;
[0069] The constraint rule analysis module queries the constraint rules corresponding to the evaluation indexes through the constraint rule metadata model, analyzes the constraint rules through the rule analysis metadata model and the corresponding constraint rule analysis function, updates the analysis progress information, running exception information, and data violating the constraint rules to the database through the evaluation result storage module, and stores the summary information and evaluation index information analyzed and calculated to the index evaluation metadata model;
[0070] The evaluation index processing module calculates the quantitative quality status under each evaluation index dimension through the corresponding evaluation index calculation function of the index evaluation metadata model.
[0071] 1) The data quality evaluation model is composed of a data set to be evaluated, an evaluation index set, a constraint rule set, an evaluation instance set, and an evaluation algorithm set. The data quality evaluation model is represented by six tuples as follows:
[0072] QA<S,D,R,W,I,A>
[0073] Wherein, S is the data set to be evaluated, D is the evaluation index set corresponding to the data set to be evaluated S, R is the constraint rule set corresponding to the evaluation index set D, W is the weight set of the weight of each constraint rule in the constraint rule set R, which describes the proportion of each constraint rule in the constraint rule set R, I is the evaluation instance set, and A is the evaluation algorithm set.
[0074] 2) The data quality evaluation model calculates the quantitative quality status under each evaluation index dimension, including the following steps:
[0075] S1, select the metadata MD extracted from the database metadata table, determine the data set S to be evaluated and select the evaluation index set D;
[0076] S2, traverse each constraint rule in the constraint rule set R corresponding to the evaluation index set D and the weight of each constraint rule in the weight set W, analyze the constraint rule on the evaluation instance set I through the constraint rule analysis module;
[0077] S3, calculate the quantified quality status under each evaluation index dimension through the evaluation index processing module.
[0078] Specifically, the data quality evaluation model calculates the quantified quality status under each evaluation index dimension, including:
[0079] Taking accuracy in the evaluation index as an example, the quantified quality status under the accuracy dimension is calculated, specifically including:
[0080] The number of problem categories S q : the number of constraint rule categories violated when the data quality of the to-be-evaluated data set is evaluated;
[0081] The number of problem records R q : the number of constraint rule records violated when the data quality of the to-be-evaluated data set is evaluated, and the same record violating multiple constraint rules is not repeated;
[0082] The number of problem data D q : the number of constraint rule records violated when the data quality of the to-be-evaluated data set is evaluated, and the same record violating multiple constraint rules is repeated;
[0083] The number of problem data items C q : the number of data items violating constraint rules when the data quality of the to-be-evaluated data set is evaluated, and one attribute violating multiple constraint rules is not repeated;
[0084] The number of missing data items C s : the difference between the number of metadata-defined data items and the number of actual data items in the to-be-evaluated data set;
[0085] Suppose the number of records of the to-be-evaluated data set is C r , the number of metadata-defined data items is C qd , then the data accuracy is calculated as follows:
[0086]
[0087] ③The evaluation task management module obtains the to-be-evaluated data set, the data dictionary metadata model corresponding to the to-be-evaluated data set, and the metadata model, creates a data quality evaluation task, and adds data quality evaluation task information for management.
[0088] The evaluation task management module creates a data quality evaluation task, and the data quality evaluation process is in units of data quality evaluation tasks, one evaluation task can include several sub-tasks, and the sub-tasks are combinations of data sets to be evaluated and data quality evaluation indexes.
[0089] (4) The evaluation task scheduling module sets parameters of the data quality evaluation task, including execution period, execution mode and exception handling, and updates execution progress information, data violating constraint rules and quantitative quality status of each evaluation index dimension generated in the data quality evaluation task execution process to the database through the evaluation result warehousing module.
[0090] The evaluation task scheduling module can be implemented based on an open source scheduling tool, such as Apache DolphinScheduler.
[0091] (5) The quality monitoring and alarm module performs quantitative and qualitative analysis on the quantitative quality status of each evaluation index dimension to obtain a data quality evaluation level of the data set to be evaluated, and judges whether the data quality of the data set to be evaluated meets the actual business requirements according to the data quality evaluation level;
[0092] The quality monitoring and alarm module alarms when an abnormal situation occurs in the data quality evaluation task execution process, and updates the data quality evaluation level and the data quality evaluation task execution abnormal situation to the database through the evaluation result warehousing module.
[0093] (6) In the technical solution of the present application, the main functions of the evaluation result warehousing module include:
[0094] 1) Update the analysis progress information, running exception information and data violating the constraint rules generated in the constraint rule analysis process to the database;
[0095] 2) Update the execution progress information, data violating the constraint rules and quantitative quality status of each evaluation index dimension generated in the data quality evaluation task execution process to the database;
[0096] 3) Update the data quality evaluation level and the data quality evaluation task execution abnormal situation to the database.
[0097] The evaluation result warehousing module supports output data sources such as mysql, elasticsearch and hdfs.
[0098] (7) In the technical solution of the present application, a visualization module is also included, which analyzes and statistics the data information fed back by the evaluation task scheduling module and the quality monitoring and alarm module, and visually displays the data quality status, data quality analysis report, constraint rule violation data statistics, problem rectification suggestions, and progress information and exception information of the data quality evaluation task execution.
[0099] In the technical solution of the present application, the evaluation indexes include core mode layer evaluation indexes and extended instance layer evaluation indexes, the core mode layer evaluation indexes include integrity, consistency, uniqueness and validity, and the extended instance layer evaluation indexes include timeliness, security and accuracy, as shown in the following table:
[0100] Table 1: Classification of evaluation indexes
[0101]
[0102]
[0103] Data quality evaluation is a process of measuring data quality based on various evaluation indexes and constraint rules. First, the constraint rules that must be met by data itself and other data under the index dimension are constructed. Data quality is multi-faceted and multi-angle, and needs to be reflected by the joint action of multi-dimensional evaluation indexes.
[0104] The data quality evaluation index system is a framework for evaluating data quality under actual business needs. In order to ensure the effectiveness of the evaluation indexes, the definition of the evaluation indexes needs to start in the quality planning design phase. Data quality constraint rules are a further refinement of data quality evaluation indexes. Based on the constraint rules, specific quantitative analysis is performed on each evaluation index. The constraint rules included in the evaluation indexes are defined to evaluate the quality of data from different sources and determine the applicability of data quality to meet actual business needs. In practice, evaluation indexes can be selected according to different business needs and corresponding constraint rules can be developed.
[0105] Data quality evaluation indexes include integrity, uniqueness, validity, consistency, accuracy, timeliness and security. Data quality constraint rules include existence constraint, non-empty constraint, primary key uniqueness constraint, entity uniqueness constraint, length constraint, content specification constraint, value range constraint, flag value constraint, code value domain constraint, existence consistency constraint, equality consistency constraint, logical consistency constraint, value accuracy constraint, timeliness constraint, data encryption constraint and data desensitization constraint, etc.
[0106] Data quality evaluation indexes include core mode layer evaluation indexes and extended instance layer evaluation indexes. The core mode layer evaluation indexes are basic evaluation indexes, mainly describing data quality problems caused by design defects, including integrity, consistency, uniqueness and validity. The core mode layer evaluation indexes are mandatory indexes for data quality evaluation. Their meanings are not affected by the external environment of metadata, so the quantification method is easier to reach an agreement in a larger range.
[0107] The extended instance layer evaluation index is an optional evaluation index, mainly describing data quality problems in data recording, including timeliness, security and accuracy, and other optional evaluation indexes, which are closely related to external attributes of metadata. Since it is difficult to completely separate the optional evaluation indexes such as timeliness, security and accuracy from the metadata specification outline, metadata development and application environment, and these optional evaluation indexes have different quantification methods when data quality evaluation is performed for different business requirements, the optional evaluation indexes can be extended based on the purpose, object and method of data quality evaluation and the kernel mode layer evaluation index, which can guarantee the universality of the kernel mode layer evaluation index and meet the individualized evaluation requirements of different data sets.
[0108] The above examples are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A data quality assessment system, characterized by: The data dictionary metadata model, the metadata model, the data quality evaluation model, the evaluation task management module, the evaluation task scheduling module, the quality monitoring alarm module and the evaluation result storage module are included. The data dictionary metadata model extracts metadata from a database metadata table and is manually improved, is used for database connection and testing, and provides support for driving data quality evaluation. The metadata model includes a constraint rule metadata model including constraint rules corresponding to evaluation indexes, a rule analysis metadata model including constraint rule analysis functions, and an index evaluation metadata model including evaluation index calculation functions. The data quality evaluation model determines a data set to be evaluated according to metadata, selects evaluation indexes, analyzes constraint rules corresponding to the evaluation indexes, updates the analysis results to a database through an evaluation result storage module, and finally calculates the quantitative quality status under each evaluation index dimension through the following steps: S1, selecting metadata MD extracted from a database metadata table, determining a data set S to be evaluated and selecting an evaluation index set D; S2, traversing each constraint rule in the constraint rule set R corresponding to the evaluation index set D and the weight of each constraint rule in the weight set W, analyzing the constraint rules on the evaluation instance set I through a constraint rule analysis module; S3, calculating the quantitative quality status under each evaluation index dimension through an evaluation index processing module; The evaluation task management module creates a data quality evaluation task and manages the data quality evaluation task. The evaluation task scheduling module sets parameters for the execution of the data quality evaluation task and updates the data quality evaluation task execution results to the database through the evaluation result storage module. The quality monitoring alarm module judges whether the data quality of the data set to be evaluated meets the actual business requirements, alarms when an abnormal situation occurs in the data quality evaluation task execution process, and updates to the database through the evaluation result storage module. The evaluation task management module obtains the data set to be evaluated, the data dictionary metadata model corresponding to the data set to be evaluated, and the metadata model, creates a data quality evaluation task, and adds data quality evaluation task information for management. The evaluation task scheduling module sets parameters for the data quality evaluation task including execution period, execution mode and abnormal processing, and updates the execution progress information generated in the data quality evaluation task execution process, the data violating the constraint rules and the quantitative quality status under each evaluation index dimension to the database through the evaluation result storage module. The quality monitoring alarm module quantitatively and qualitatively analyzes the quantitative quality status under each evaluation index dimension to obtain the data quality evaluation level of the data set to be evaluated, and judges whether the data quality of the data set to be evaluated meets the actual business requirements according to the data quality evaluation level. The quality monitoring alarm module alarms when an abnormal situation occurs in the data quality evaluation task execution process, and updates the data quality evaluation level and the data quality evaluation task execution abnormal situation to the database through the evaluation result storage module.
2. The data quality assessment system of claim 1, wherein: The data quality evaluation model comprises a constraint rule analysis module and an evaluation index processing module. The constraint rule analysis module queries the constraint rules corresponding to the evaluation indexes through the constraint rule metadata model, analyzes the constraint rules through a corresponding constraint rule analysis function of the rule analysis metadata model, updates the analysis progress information, running abnormal information and data violating the constraint rules generated in the analysis process to the database through the evaluation result warehousing module, and stores the summary information and evaluation index information analyzed and calculated to the index evaluation metadata model. The evaluation index processing module calculates the quantitative quality status under each evaluation index dimension through a corresponding evaluation index calculation function of the index evaluation metadata model.
3. The data quality assessment system of claim 2, wherein: The data quality evaluation model is composed of a data set to be evaluated, an evaluation index set, a constraint rule set, an evaluation instance set and an evaluation algorithm set, and is expressed by a six-tuple as follows: QA<S,D,R,W,I,A> S is the data set to be evaluated, D is the evaluation index set corresponding to the data set to be evaluated S, R is the constraint rule set corresponding to the evaluation index set D, W is the weight set of the weight of each constraint rule in the constraint rule set R, which describes the proportion of each constraint rule in the constraint rule set R, I is the evaluation instance set, and A is the evaluation algorithm set.
4. The data quality assessment system of claim 1, wherein: The data dictionary metadata model is expressed by a four-tuple as follows: MD<db,dt,cl,ci> db is the description information of the database, including the physical location information of the database, the type of the database, the version number of the database, the name of the database, the URL of the database, the username and password for accessing the database; dt is the description information of the metadata table, including the table name, the description of the table, and the database information to which the table belongs; cl is the description information of the field, including the field name, the field code, and the metadata table information to which the field belongs; ci is the constraint information of the field, including the data type of the field value, the value range, the integrity constraint, and the field primary and foreign key.
5. The data quality assessment system of claim 2, wherein: The data quality evaluation model calculates the quantitative quality status under each evaluation index dimension, which comprises: Taking the accuracy in the evaluation index as an example, the quantitative quality status under the accuracy dimension is calculated, which specifically comprises: Number of problem categories S q : Number of constraint rule violations that occur when performing data quality assessment on the data set under evaluation; Number of problem records R q : When data quality assessment is performed on the data set to be evaluated, the number of records that violate constraint rules is recorded, and the same record does not repeat the calculation of violating multiple constraint rules. Number of problem data D q : When data quality assessment is performed on the data set to be assessed, the number of records violating constraint rules is recorded, and the same record appears repeatedly in the calculation of violating multiple constraint rules; Number of problem data items C q : Number of data items that violate constraint rules when data quality assessment is performed on the data set to be assessed, and one attribute does not repeat the calculation of violating multiple constraint rules; Number of missing data items C s Difference between number of metadata defined data items and number of actual data items in the data set to be evaluated; Assume the number of records of the data set to be evaluated is C r , the number of data items defined by the metadata is C qd , then the data accuracy is calculated by the following formula:
6. The data quality assessment system of any of claims 1-5, wherein: The evaluation index comprises a kernel mode layer evaluation index and an extended instance layer evaluation index, the kernel mode layer evaluation index comprises integrity, consistency, uniqueness and validity, and the extended instance layer evaluation index comprises timeliness, security and accuracy.
7. The data quality assessment system of claim 1, wherein: The visualization module analyzes and calculates the data information fed back by the evaluation task scheduling module and the quality monitoring and alarm module, and visually displays the data quality status, the data quality analysis report, the statistical data violating the constraint rules, the problem rectification suggestions, and the progress information and abnormal information of the data quality evaluation task execution.