Data quality management method and system

Abstract

The invention discloses a data quality management method and system, belongs to the technical field of data management, and aims to solve the technical problems of how to improve the data quality management effect and improve the data service quality. Comprising the following steps: during data acquisition, performing quality detection through a corresponding quality detection rule, and performing data partitioning and data labeling according to an initial data quality detection evaluation result; performing corresponding preprocessing on the data of the to-be-used data partition according to the initial data quality detection and evaluation result; evaluating data quality according to the data preprocessing result to obtain an evaluation result, and adjusting and optimizing a quality detection rule according to the evaluation result; a user is supported to evaluate data quality through a data quality evaluation service, a data problem report rate of the user is formed, the data quality is evaluated according to the data problem report rate of the user, and data acquisition quality detection and data integration preprocessing operation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data quality management method and system. Background Art

[0002] With the advent of the big data era, Internet applications have extended from consumption to industry. Various types of smart devices are becoming more and more popular, and the application of the Internet of Things is becoming more extensive and in-depth, which has led to an explosive growth in the amount of data in various industries. Data services have emerged as the times require. By collecting and aggregating business data, analyzing and mining various data, users can be provided with a more accurate and personalized service experience. Enterprises can obtain key information such as market trends and user behavior through data services, so as to make more informed decisions. However, the huge increase in data volume also brings challenges to data quality management. In the process of implementing data services, data quality will directly affect the value of data and the accuracy of data analysis and prediction, which is crucial for supporting decision-making, driving innovation, optimizing processes and improving competitiveness.

[0003] Data quality management is a complex and ongoing process that requires a variety of measures to ensure data quality. However, traditional data quality management methods are difficult to cope with the growing data sources and data volumes, and current data needs have become more real-time and complex and diverse. It is necessary to start from multiple aspects such as business, technology, monitoring, and tools to continuously improve the level of data quality management in order to provide reliable data support for business decisions and operations.

[0004] How to improve the effectiveness of data quality management and the quality of data services is a technical problem that needs to be solved. Summary of the invention

[0005] The technical task of the present invention is to provide a data quality management method and system to solve the technical problem of how to improve the data quality management effect and improve the quality of data services in view of the above shortcomings.

[0006] In a first aspect, the present invention provides a data quality management method, comprising the following steps:

[0007] Data collection quality detection: According to the enterprise's business system and actual business, corresponding multiple enterprise management nodes are built, and adaptive quality detection rules are built for the management nodes. During data collection, quality detection is performed using the corresponding quality detection rules to obtain initial data quality detection and evaluation results, and data partitioning and data labeling are performed based on the initial data quality detection and evaluation results;

[0008] Data integration preprocessing: When integrating data from different systems, departments or external data, the data of the data partition to be used is preprocessed accordingly according to the initial data quality detection and evaluation results to obtain the data preprocessing results;

[0009] Data governance quality control: Evaluate data quality based on data preprocessing results, obtain evaluation results, and adjust and optimize quality detection rules based on the evaluation results;

[0010] Data usage quality feedback mechanism: Build a data quality problem feedback mechanism. During the data usage process, support users to evaluate data quality through data quality evaluation services, form users' data problem reporting rate, evaluate data quality based on users' data problem reporting rate, and improve data collection quality detection and data integration preprocessing operations.

[0011] As a priority, quality inspection rules are constructed based on business requirements, data characteristics, data quality dimensions, technical factors, organizational and cultural factors, and other factors;

[0012] Quality inspection rules include:

[0013] Uniqueness detection, including primary key detection, composite field detection and duplicate data detection, uses the uniqueness detection function in the ETL tool to perform uniqueness detection while collecting data;

[0014] Integrity testing, including data missing check, record integrity check and range integrity check;

[0015] Accuracy testing, including data comparison checking, outlier checking, and logical relationship checking;

[0016] Standardization testing, including format standardization testing, coding standardization checking, and naming standardization checking;

[0017] Consistency testing, including data comparison checking, data association checking, and redundant data checking;

[0018] Timeliness detection, including data update frequency check and timestamp check;

[0019] Validity check, which includes data validity period check and data source check.

[0020] Preferably, the preprocessing includes taking the latest data for duplicate values, marking data failure, marking data differences, marking abnormal values, deleting data redundancy, and marking timestamp abnormalities.

[0021] Preferably, during data governance quality control, adjusting and optimizing quality detection rules includes modifying rule logic, adjusting thresholds, and adding new rules.

[0022] As a preferred option, the data quality problem feedback mechanism provides alarm notification services and problem reporting services. The problem reporting service supports users in evaluating data quality and forming users' data problem reporting rates through data quality evaluation services. The alarm notification service provides the following operations: pushing users' data problem reporting rates to relevant personnel, supporting relevant personnel in evaluating data quality based on users' data problem reporting rates, and improving data collection quality detection and data integration preprocessing operations.

[0023] In a second aspect, the present invention provides a data quality management system for providing data quality management through a data quality management method as described in any one of the first aspects, wherein the system includes a data collection quality detection module, a data integration preprocessing module, a data governance quality control module, and a data usage quality feedback mechanism module;

[0024] The data collection quality detection module is used to perform the following: construct multiple corresponding enterprise management nodes according to the enterprise's business system and actual business, and construct adaptive quality detection rules for the management nodes. During data collection, quality detection is performed using the corresponding quality detection rules to obtain initial data quality detection and evaluation results, and data partitioning and data labeling are performed based on the initial data quality detection and evaluation results;

[0025] The data integration preprocessing module is used to perform the following: when integrating data from different systems, departments or external data, the data of the data partition to be used is preprocessed accordingly according to the initial data quality detection and evaluation results to obtain the data preprocessing results;

[0026] The data governance quality control module is used to perform the following: evaluate data quality based on data preprocessing results, obtain evaluation results, and adjust and optimize quality detection rules based on the evaluation results;

[0027] The data usage quality feedback mechanism module is used to perform the following: build a data quality problem feedback mechanism. During the data usage process, it supports users in evaluating data quality through data quality evaluation services, forms users' data problem reporting rates, evaluates data quality based on users' data problem reporting rates, and improves data collection quality detection and data integration preprocessing operations.

[0028] Preferably, the data collection quality detection module is used to construct quality detection rules based on business requirements, data characteristics, data quality dimensions, technical factors, organizational and cultural factors and other factors;

[0029] Quality inspection rules include:

[0030] Uniqueness detection, including primary key detection, composite field detection and duplicate data detection, uses the uniqueness detection function in the ETL tool to perform uniqueness detection while collecting data;

[0031] Integrity testing, including data missing check, record integrity check and range integrity check;

[0032] Accuracy testing, including data comparison checking, outlier checking, and logical relationship checking;

[0033] Standardization testing, including format standardization testing, coding standardization checking, and naming standardization checking;

[0034] Consistency testing, including data comparison checking, data association checking, and redundant data checking;

[0035] Timeliness detection, including data update frequency check and timestamp check;

[0036] Validity check, which includes data validity period check and data source check.

[0037] Preferably, the preprocessing includes taking the latest data for duplicate values, marking data failure, marking data differences, marking abnormal values, deleting data redundancy, and marking timestamp abnormalities.

[0038] Preferably, during data governance quality control, adjusting and optimizing quality detection rules includes modifying rule logic, adjusting thresholds, and adding new rules.

[0039] As a preferred option, the data quality problem feedback mechanism provides alarm notification services and problem reporting services. The problem reporting service supports users in evaluating data quality and forming users' data problem reporting rates through data quality evaluation services. The alarm notification service provides the following operations: pushing users' data problem reporting rates to relevant personnel, supporting relevant personnel in evaluating data quality based on users' data problem reporting rates, and improving data collection quality detection and data integration preprocessing operations.

[0040] The data quality management method and system of the present invention have the following advantages: the data life cycle includes three stages: source, processing and consumption. It provides quality detection in data collection, data preprocessing during data integration, quality control in the data governance process, and quality feedback during data use. Based on the three stages of the data life cycle, the quality feedback link is integrated to achieve continuous improvement of the overall data service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] The present invention is further described below in conjunction with the accompanying drawings.

[0043] Figure 1 This is a flowchart of a data quality management method in Example 1. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0045] The embodiments of the present invention provide a data quality management method and system for solving the technical problem of how to improve the data quality management effect and improve the quality of data services.

[0046] Embodiment 1:

[0047] The present invention provides a data quality management method, which includes four steps: data collection quality detection, data integration preprocessing, data governance quality control, and data usage quality feedback mechanism.

[0048] Step S100: Data collection quality inspection: Based on the enterprise's business system and actual business, corresponding multiple enterprise management nodes are constructed, and adaptive quality inspection rules are constructed for the management nodes. During data collection, quality inspection is performed using the corresponding quality inspection rules to obtain initial data quality inspection evaluation results, and data partitioning and data labeling are performed based on the initial data quality inspection evaluation results.

[0049] In this embodiment, quality detection rules are constructed based on business requirements, data characteristics, data quality dimensions, technical factors, organizational and cultural factors, and other factors. The quality detection rules include the following:

[0050] Uniqueness detection, including primary key detection, composite field detection and duplicate data detection, uses the uniqueness detection function in the ETL tool to perform uniqueness detection while collecting data;

[0051] Integrity testing, including data missing check, record integrity check and range integrity check;

[0052] Accuracy testing, including data comparison, outlier and logical relationship checks, such as whether the order amount and order quantity are correct, and whether the age in the personal information is negative or exceeds the limit;

[0053] Standardization testing, including format standardization testing, coding standardization checking, and naming standardization checking;

[0054] Consistency testing, which includes data comparison checking, data association checking, and redundant data checking, such as whether the relationship between the order and the order details is correct;

[0055] Timeliness detection, including data update frequency check and timestamp check;

[0056] Validity check, which includes data validity period check and data source check.

[0057] Step S200: Data integration preprocessing: When data from different systems, departments or external data are integrated, the data of the data partition to be used is preprocessed accordingly according to the initial data quality detection and evaluation results to obtain data preprocessing results.

[0058] In this embodiment, the preprocessing includes taking the latest data for duplicate values, marking data failure, marking data differences, marking abnormal values, deleting data redundancy, and marking timestamp abnormalities.

[0059] Data integration mainly collects data from different systems, departments or external data to provide rich data resources for business innovation. According to the quality inspection results of the previous step, the data of different data partitions should be preprocessed accordingly during data integration. This embodiment takes customer, product and order data as an example to introduce the specific preprocessing scheme as shown in Table 1.

[0060] Table 1:

[0061]

[0062]

[0063] Step S300: Data governance quality control: Evaluate data quality based on data preprocessing results, obtain evaluation results, and adjust and optimize quality detection rules based on the evaluation results.

[0064] In this embodiment, when performing data governance quality control, adjusting and optimizing quality detection rules includes modifying rule logic, adjusting thresholds, and adding new rules.

[0065] Data governance is the foundation for enterprises to achieve data strategy. It is a management system that includes organizations, systems, processes and tools to ensure the quality, security, compliance and effectiveness of data. It is very important to monitor the effectiveness of quality control during the data governance process to ensure that the quality of data services is continuously improved and meets business needs.

[0066] According to the data preprocessing results of the previous step and the data quality assessment, if the rules fail to effectively prevent data quality issues or there are too many false positives or false negatives, the rules need to be adjusted and optimized in a timely manner. Use monitoring tools to collect monitoring data related to data quality, including data anomalies, rule violations, etc.; according to the monitoring results and feedback, timely adjust and optimize data quality rules. Adjustments can include modifying rule logic, adjusting thresholds, adding new rules, etc. Ensure that the monitoring system can cover all key data sources and data processing links to avoid missing important data quality issues.

[0067] Step S400: Data usage quality feedback mechanism: Build a data quality problem feedback mechanism. During the data usage process, support users to evaluate data quality through data quality evaluation services, form users' data problem reporting rate, evaluate data quality based on users' data problem reporting rate, and improve data collection quality detection and data integration preprocessing operations.

[0068] In this embodiment, the data quality problem feedback mechanism provides alarm notification services and problem reporting services. The problem reporting service supports users to evaluate data quality and form users' data problem reporting rates through data quality evaluation services. The alarm notification service provides the following operations: push users' data problem reporting rates to relevant personnel, support relevant personnel to evaluate data quality based on users' data problem reporting rates, and improve data collection quality detection and data integration preprocessing operations.

[0069] Establish a feedback mechanism for data quality issues to ensure that relevant personnel can promptly understand data quality issues and their impacts. The feedback mechanism includes alarm notification services, problem reporting services, etc., and incorporates monitoring results and rule adjustments into the continuous improvement process of data quality management to continuously improve the level and effectiveness of data quality management.

[0070] Data quality evaluation: Provide data quality evaluation function during data use, allowing users to score and evaluate the accuracy, completeness, and validity of the data, and collect data quality feedback.

[0071] Problem reporting rate: Evaluate data quality based on the user's data problem reporting rate, deeply explore the causes of the problems, and continuously improve the detection and preprocessing links to make detection more accurate and data processing more effective.

[0072] Embodiment 2:

[0073] The present invention provides a data quality management system, comprising a data acquisition quality detection module, a data integration preprocessing module, a data governance quality control module and a data usage quality feedback mechanism module.

[0074] The data collection quality detection module is used to perform the following: construct multiple corresponding enterprise management nodes based on the enterprise's business system and combined with actual business, and construct adaptive quality detection rules for the management nodes. During data collection, quality detection is performed using the corresponding quality detection rules to obtain initial data quality detection evaluation results, and data partitioning and data labeling are performed based on the initial data quality detection evaluation results.

[0075] In this embodiment, the data collection quality detection module is used to construct quality detection rules based on business requirements, data characteristics, data quality dimensions, technical factors, organizational and cultural factors, and other factors. The quality detection rules include the following:

[0076] Uniqueness detection, including primary key detection, composite field detection and duplicate data detection, uses the uniqueness detection function in the ETL tool to perform uniqueness detection while collecting data;

[0077] Integrity testing, including data missing check, record integrity check and range integrity check;

[0078] Accuracy testing, including data comparison, outlier and logical relationship checks, such as whether the order amount and order quantity are correct, and whether the age in the personal information is negative or exceeds the limit;

[0079] Standardization testing, including format standardization testing, coding standardization checking, and naming standardization checking;

[0080] Consistency testing, which includes data comparison checking, data association checking, and redundant data checking, such as whether the relationship between the order and the order details is correct;

[0081] Timeliness detection, including data update frequency check and timestamp check;

[0082] Validity check, which includes data validity period check and data source check.

[0083] The data integration preprocessing module is used to perform the following: when integrating data from different systems, departments or external data, the data of the data partition to be used is preprocessed accordingly according to the initial data quality detection and evaluation results to obtain the data preprocessing results.

[0084] In this embodiment, the preprocessing includes taking the latest data for duplicate values, marking data failure, marking data differences, marking abnormal values, deleting data redundancy, and marking timestamp abnormalities.

[0085] The data governance quality control module is used to perform the following: evaluate data quality based on data preprocessing results, obtain evaluation results, and adjust and optimize quality detection rules based on the evaluation results.

[0086] In this embodiment, when performing data governance quality control, adjusting and optimizing quality detection rules includes modifying rule logic, adjusting thresholds, and adding new rules.

[0087] Data governance is the foundation for enterprises to achieve data strategy. It is a management system that includes organizations, systems, processes and tools to ensure the quality, security, compliance and effectiveness of data. It is very important to monitor the effectiveness of quality control during the data governance process to ensure that the quality of data services is continuously improved and meets business needs.

[0088] According to the data preprocessing results and the data quality assessment, if the rules fail to effectively prevent data quality problems or there are too many false positives or false negatives, the rules need to be adjusted and optimized in a timely manner. Use monitoring tools to collect monitoring data related to data quality, including data anomalies, rule violations, etc.; according to the monitoring results and feedback, timely adjust and optimize data quality rules. Adjustments can include modifying rule logic, adjusting thresholds, adding new rules, etc. Ensure that the monitoring system can cover all key data sources and data processing links to avoid missing important data quality issues.

[0089] The data usage quality feedback mechanism module is used to perform the following: build a data quality problem feedback mechanism. During the data usage process, it supports users in evaluating data quality through data quality evaluation services, forms users' data problem reporting rates, evaluates data quality based on users' data problem reporting rates, and improves data collection quality detection and data integration preprocessing operations.

[0090] In this embodiment, the data quality problem feedback mechanism provides alarm notification services and problem reporting services. The problem reporting service supports users to evaluate data quality and form users' data problem reporting rates through data quality evaluation services. The alarm notification service provides the following operations: push users' data problem reporting rates to relevant personnel, support relevant personnel to evaluate data quality based on users' data problem reporting rates, and improve data collection quality detection and data integration preprocessing operations.

[0091] Establish a feedback mechanism for data quality issues to ensure that relevant personnel can promptly understand data quality issues and their impacts. The feedback mechanism includes alarm notification services, problem reporting services, etc., and incorporates monitoring results and rule adjustments into the continuous improvement process of data quality management to continuously improve the level and effectiveness of data quality management.

[0092] Data quality evaluation: Provide data quality evaluation function during data use, allowing users to score and evaluate the accuracy, completeness, and validity of the data, and collect data quality feedback.

[0093] Problem reporting rate: Evaluate data quality based on the user's data problem reporting rate, deeply explore the causes of the problems, and continuously improve the detection and preprocessing links to make detection more accurate and data processing more effective.

[0094] The system of this embodiment can execute the method disclosed in Example 1 to achieve data quality management.

[0095] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A data quality management method, characterized in that: The steps include: Data collection quality detection: According to the enterprise's business system and actual business, corresponding multiple enterprise management nodes are built, and adaptive quality detection rules are built for the management nodes. During data collection, quality detection is performed using the corresponding quality detection rules to obtain initial data quality detection and evaluation results, and data partitioning and data labeling are performed based on the initial data quality detection and evaluation results; Data integration preprocessing: When integrating data from different systems, departments or external data, the data of the data partition to be used is preprocessed accordingly according to the initial data quality detection and evaluation results to obtain the data preprocessing results; Data governance quality control: Evaluate data quality based on data preprocessing results, obtain evaluation results, and adjust and optimize quality detection rules based on the evaluation results; Data usage quality feedback mechanism: Build a data quality problem feedback mechanism. During the data usage process, support users to evaluate data quality through data quality evaluation services, form users' data problem reporting rate, evaluate data quality based on users' data problem reporting rate, and improve data collection quality detection and data integration preprocessing operations.

2. The data quality management method according to claim 1, characterized in that: Build quality inspection rules based on business requirements, data characteristics, data quality dimensions, technical factors, organizational and cultural factors, and other factors; Quality inspection rules include: Uniqueness detection, including primary key detection, composite field detection and duplicate data detection, uses the uniqueness detection function in the ETL tool to perform uniqueness detection while collecting data; Integrity testing, including data missing check, record integrity check and range integrity check; Accuracy testing, including data comparison checking, outlier checking, and logical relationship checking; Standardization testing, including format standardization testing, coding standardization checking, and naming standardization checking; Consistency testing, including data comparison checking, data association checking, and redundant data checking; Timeliness detection, including data update frequency check and timestamp check; Validity check, which includes data validity period check and data source check.

3. The data quality management method according to claim 1, characterized in that: Preprocessing includes taking the latest data for duplicate values, marking data invalidation, marking data differences, marking outliers, deleting data redundancy, and marking timestamp anomalies.

4. The data quality management method according to claim 1, characterized in that: When performing data governance quality control, adjusting and optimizing quality detection rules includes modifying rule logic, adjusting thresholds, and adding new rules.

5. The data quality management method according to claim 1, characterized in that: The data quality problem feedback mechanism provides alarm notification services and problem reporting services. The problem reporting service supports users in evaluating data quality and forming users' data problem reporting rates through data quality evaluation services. The alarm notification service provides the following operations: pushing users' data problem reporting rates to relevant personnel, supporting relevant personnel in evaluating data quality based on users' data problem reporting rates, and improving data collection quality detection and data integration preprocessing operations.

6. A data quality management system, characterized in that: Used to provide data quality management through a data quality management method according to any one of claims 1 to 5, the system comprising a data acquisition quality detection module, a data integration preprocessing module, a data governance quality control module and a data usage quality feedback mechanism module; The data collection quality detection module is used to perform the following: construct multiple corresponding enterprise management nodes according to the enterprise's business system and actual business, and construct adaptive quality detection rules for the management nodes. During data collection, quality detection is performed using the corresponding quality detection rules to obtain initial data quality detection and evaluation results, and data partitioning and data labeling are performed based on the initial data quality detection and evaluation results; The data integration preprocessing module is used to perform the following: when integrating data from different systems, departments or external data, the data of the data partition to be used is preprocessed accordingly according to the initial data quality detection and evaluation results to obtain the data preprocessing results; The data governance quality control module is used to perform the following: evaluate data quality based on data preprocessing results, obtain evaluation results, and adjust and optimize quality detection rules based on the evaluation results; The data usage quality feedback mechanism module is used to perform the following: build a data quality problem feedback mechanism. During the data usage process, it supports users in evaluating data quality through data quality evaluation services, forms users' data problem reporting rates, evaluates data quality based on users' data problem reporting rates, and improves data collection quality detection and data integration preprocessing operations.

7. The data quality management system according to claim 6, characterized in that: The data collection quality detection module is used to build quality detection rules based on business requirements, data characteristics, data quality dimensions, technical factors, organizational and cultural factors, and other factors; Quality inspection rules include: Uniqueness detection, including primary key detection, composite field detection and duplicate data detection, uses the uniqueness detection function in the ETL tool to perform uniqueness detection while collecting data; Integrity testing, including data missing check, record integrity check and range integrity check; Accuracy testing, including data comparison checking, outlier checking, and logical relationship checking; Standardization testing, including format standardization testing, coding standardization checking, and naming standardization checking; Consistency testing, including data comparison checking, data association checking, and redundant data checking; Timeliness detection, including data update frequency check and timestamp check; Validity check, which includes data validity period check and data source check.

8. The data quality management system according to claim 6, characterized in that: Preprocessing includes taking the latest data for duplicate values, marking data invalidation, marking data differences, marking outliers, deleting data redundancy, and marking timestamp anomalies.

9. The data quality management system according to claim 6, characterized in that: When performing data governance quality control, adjusting and optimizing quality detection rules includes modifying rule logic, adjusting thresholds, and adding new rules.

10. The data quality management system according to claim 6, characterized in that: The data quality problem feedback mechanism provides alarm notification services and problem reporting services. The problem reporting service supports users in evaluating data quality and forming users' data problem reporting rates through data quality evaluation services. The alarm notification service provides the following operations: pushing users' data problem reporting rates to relevant personnel, supporting relevant personnel in evaluating data quality based on users' data problem reporting rates, and improving data collection quality detection and data integration preprocessing operations.

Images