Data quality verification and restoration method based on big data

Abstract

The invention relates to the technical field of data processing, and discloses a data quality verification and restoration method based on big data, comprising the following steps: S1, setting a data quality assessment rule; s2, defining related rules, indexes, quality templates and quality tasks, and evaluating the accuracy, integrity, consistency and credibility of the big data set; s3, recovering data integrity and accuracy by adopting a proper recovery technology; and S4, generating a data quality report and visual content, presenting a data quality condition for the user, assisting the user in discovering problems and formulating improvement measures, and supporting subsequent data analysis and decision. According to the method, the data quality can be improved, the data credibility can be enhanced, data analysis and decision can be optimized, automatic and intelligent processing can be realized, flexible rule customization and management can be provided, the effects are beneficial to improving the data quality and availability, and more efficient, accurate and reliable data processing and decision support can be brought to users.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data quality verification and repair method based on big data. Background Art

[0002] Big data has become an important resource and driving force in today's information society. The widespread collection and application of big data has provided huge opportunities for enterprises and organizations, but it has also brought challenges to data quality. Data quality issues include data accuracy, completeness, consistency, and credibility, and abnormal data is a common data quality problem. Abnormal data may be data errors, noise, or outliers caused by various reasons. These abnormal data may have a negative impact on data analysis, decision-making, and business processes. For example, in the financial field, abnormal data may lead to incorrect risk assessment and trading decisions. In the manufacturing industry, abnormal data may lead to inaccurate quality control and product defects.

[0003] Currently, some methods and technologies have been proposed for data quality verification and abnormal data repair for big data. Data quality verification methods can evaluate the quality of data by defining and applying a series of rules and indicators. These methods can help discover problems such as inconsistencies, missing values, duplicates, and outliers in the data. However, existing methods face challenges in dealing with large-scale and high-dimensional data, as the size and complexity of the data may lead to difficulties in calculation and storage.

[0004] Therefore, an innovative data quality verification and abnormal data repair method is needed to effectively cope with the challenges in the big data environment. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a data quality verification and repair method based on big data. The present invention aims to solve the problems existing in the prior art and provide a comprehensive solution that can accurately evaluate data quality and automatically repair damaged or incomplete data, thereby improving the credibility and availability of big data.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a data quality verification and repair method based on big data, comprising the following steps:

[0007] S1. Users select the quality rule templates and / or custom rules provided by the system according to their own needs to define quality tasks and set rules for data quality assessment.

[0008] S2. Define relevant rules, indicators, quality templates and quality tasks, evaluate the accuracy, completeness, consistency and credibility of large data sets, and users can customize rules and indicators according to actual conditions and apply them to the verification process, and automatically detect abnormal data through technical means.

[0009] S3. The user selects a custom repair template and / or repairs abnormal data based on algorithmic learning of normal patterns, using appropriate repair techniques to restore data integrity and accuracy.

[0010] S4. Generate data quality reports and visualization content to present data quality status to users, assist them in identifying problems and formulating improvement measures, and support subsequent data analysis and decision-making.

[0011] Preferably, when the user customizes the rules in S1, the value range of the data field, the data format requirements, and the conditional restrictions on the repeated occurrence of data can be set in detail.

[0012] Preferably, the self-defined rules and indicators in S2 include but are not limited to an upper limit on the proportion of missing values ​​in the data, a reasonable interval of numerical distribution, and requirements on the degree of correlation between data.

[0013] Preferably, the repair method in S3 includes technologies such as data interpolation, data inference and data error correction to restore the integrity and accuracy of the data, and predict the reasonable value range of abnormal data by establishing a complex data model and combining historical data characteristics and trends.

[0014] Preferably, the data quality report generated in S4 includes a comprehensive score of data quality, detailed scores of various verification indicators, and specific distribution locations and quantities of abnormal data.

[0015] Preferably, it also includes real-time monitoring of dynamic changes in data during the entire data quality verification and repair process, and automatically adjusting verification rules and repair strategies according to operations such as data addition, modification and deletion.

[0016] Preferably, after completing the data quality verification and repair operations, strict retrospective verification of the repaired results is also included, and by comparing with the original data and the expected quality standards, it is ensured that the data quality is significantly and stably improved.

[0017] Preferably, a data quality verification and repair system based on big data comprises:

[0018] Rule template module, used to support users to define and manage data quality verification rules;

[0019] Data quality verification module, used to evaluate the data quality in large data sets;

[0020] Anomaly data repair module, used to automatically repair abnormal data in large data sets;

[0021] The data quality report and visualization module is used to generate detailed data quality reports based on the results of data quality verification and abnormal data detection, and provide visual charts and statistical information.

[0022] The present invention provides a data quality verification and repair method based on big data. It has the following beneficial effects:

[0023] 1. The present invention can effectively improve the data quality of large data sets through data quality verification and abnormal data repair. By accurately evaluating the accuracy, completeness, consistency and credibility of data and repairing abnormal data, the impact of data errors and noise can be reduced, and the accuracy and reliability of data can be improved.

[0024] 2. The data quality verification and abnormal data repair process of the present invention can help users find and solve anomalies and errors in the data, ensure the reliability and consistency of the data, provide a reliable data foundation, help users make accurate data analysis and decisions, and enhance the credibility of large data sets.

[0025] 3. The present invention can provide reliable support for data analysis and decision-making. By improving data quality and removing the interference of abnormal data, users can obtain more accurate, consistent and reliable data results. This helps users make more informed decisions and improve work efficiency and business competitiveness.

[0026] 4. The present invention uses advanced technologies such as machine learning, data mining and pattern recognition to achieve automation and intelligence in abnormal data detection and repair. Users do not need to manually process abnormal data in large data sets, but rely on the algorithms and models in the invention to achieve automatic repair, saving time and labor costs.

[0027] 5. The rule template module in the present invention provides flexible rule customization and management functions, and users can customize and adjust data quality verification rules according to specific needs and application scenarios. This enables the invention to adapt to different industries and fields and meet the ever-changing data quality requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 Schematic diagram of the rule template of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Example:

[0032] Please see attached Figure 1-2 The embodiment of the present invention provides a data quality verification and repair method based on big data, comprising the following steps:

[0033] S1. Users select the quality rule templates and / or custom rules provided by the system to define quality tasks and set rules for data quality assessment based on their own needs. When users customize rules, they can set the value range of data fields, data format requirements, and conditions for repeated data in detail. Flexible rule setting methods are provided, allowing users to accurately customize data quality assessment rules that suit their own needs, including detailed restrictions on data fields, thereby improving the pertinence and effectiveness of data assessment and laying the foundation for subsequent processing.

[0034] S2. Define relevant rules, indicators, quality templates and quality tasks, and evaluate the accuracy, completeness, consistency and credibility of large data sets. Users can customize rules and indicators based on actual conditions and apply them to the verification process. Abnormal data can be automatically detected through technical means. Customized rules and indicators include but are not limited to the upper limit of the missing value ratio of data, the reasonable range of numerical distribution, and the degree of correlation between data. Comprehensively evaluate the multi-faceted quality of large data sets. User-defined rules and indicators make the evaluation more realistic, accurately identify problems such as missing data, numerical deviation and consistency, and automatically detect abnormal data to improve efficiency and accuracy.

[0035] S3. Users choose custom repair templates and / or learn normal patterns based on algorithms to repair abnormal data, and use appropriate repair techniques to restore data integrity and accuracy. Repair methods include data interpolation, data inference, and data error correction to restore data integrity and accuracy. By establishing complex data models, combining historical data characteristics and trends, predicting the reasonable value range of abnormal data, providing multiple repair options, using appropriate technologies to restore data integrity and accuracy, and combining data models to predict the value range of abnormal data, the rationality and reliability of repair are improved to provide high-quality data for subsequent applications.

[0036] S4. Generate data quality reports and visualizations to present data quality status to users, assist them in identifying problems and developing improvement measures, and support subsequent data analysis and decision-making. Data quality reports include comprehensive scores for data quality, detailed scores for various verification indicators, and specific distribution locations and quantities of abnormal data. Intuitive reports and visualizations are generated to help users quickly understand data quality status, identify problems, develop measures, and support decision-making. Real-time monitoring and automatic adjustment strategies adapt to data changes, and backtracking verification ensures significant repair effects and stable improvement of data quality.

[0037] During the entire data quality verification and repair process, the dynamic changes of data are monitored in real time, and the verification rules and repair strategies are automatically adjusted according to operations such as data addition, modification and deletion.

[0038] Specifically, real-time monitoring of dynamic changes in data ensures that the latest status of data can be captured in a timely manner, avoiding deviations in quality verification and repair based on outdated data. Automatic adjustment of verification rules and repair strategies based on the addition, modification and deletion of data enables the system to flexibly adapt to data changes and maintain the accuracy and effectiveness of data processing. This dynamic adjustment mechanism reduces the need for manual intervention, improves work efficiency, and can better cope with the ever-changing data environment, ensuring that data quality is always under good control.

[0039] After completing the data quality verification and repair operations, the repaired results are strictly retrospectively verified, and by comparing them with the original data and expected quality standards, it is ensured that the data quality is significantly and stably improved.

[0040] Specifically, strict retrospective verification provides reliable guarantee for data quality. By comparing with the original data and expected quality standards, the repair effect can be objectively evaluated to ensure that the repair work has achieved the expected goal. This verification mechanism helps to discover possible residual problems or new problems, and conduct further processing and optimization in a timely manner. This ensures that data quality is continuously and steadily improved, enhances the reliability and credibility of data, and provides a solid guarantee for the effective use and value of data.

[0041] A data quality verification and repair system based on big data, including

[0042] Rule template module:

[0043] Provides a user-friendly interface for defining and managing data quality verification rules. Templates can be provided from dimensions such as timeliness and completeness. Figure 1,Users can select the quality rule templates provided by the system, or customize the rules to evaluate the quality of the data.,It supports a flexible and scalable approach, allowing users to define and adjust quality rules according to specific needs and application scenarios.

[0044] Data quality verification module:

[0045] Provides quality task management. Users can use rule templates to form different quality tasks to perform quality verification on different data sets. Quality tasks support manual scheduling and periodic scheduling based on cron expressions, and support user-defined rules and indicators to adapt to different data characteristics and quality requirements. When executing quality tasks, users can use SQL-based execution methods to execute quality tasks, or they can use artificial intelligence methods to execute them, using supervised learning algorithms such as support vector machines (SVM) or random forests, using labeled training data, and training classification models to identify the differences between normal data and abnormal data, classify them, and automatically discover outliers and errors.

[0046] Abnormal data repair module:

[0047] Provides the function of customizing repair templates to repair abnormal data. It also provides SQL and artificial intelligence methods for data repair; the SQL method has higher accuracy but greater limitations and can only be based on the repair method provided by the system. The artificial intelligence method is based on machine learning and pattern recognition algorithms and is more widely used. It learns normal patterns from data and applies repair strategies to repair data. It supports technologies such as data interpolation, data inference, and data error correction to restore the integrity and accuracy of data.

[0048] Data Quality Reporting and Visualization Module:

[0049] Provides a page to display detailed data quality reports, including data quality scores, distribution of abnormal data, and repair status. Provides visual charts and statistical information to intuitively display the quality status of data and the characteristics of abnormal data, as well as abnormal data export, repair data details and other functions to help users find problems and improvement measures, and support data analysis and decision-making processes.

[0050] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data quality verification and repair method based on big data, It is characterized in that The following steps are involved: S1. Users select the quality rule templates and / or custom rules provided by the system according to their own needs to define quality tasks and set rules for data quality assessment. S2. Define relevant rules, indicators, quality templates and quality tasks, evaluate the accuracy, completeness, consistency and credibility of large data sets, and users can customize rules and indicators according to actual conditions and apply them to the verification process, and automatically detect abnormal data through technical means. S3. The user selects a custom repair template and / or repairs abnormal data based on algorithmic learning of normal patterns, using appropriate repair techniques to restore data integrity and accuracy. S4. Generate data quality reports and visualization content to present data quality status to users, assist them in identifying problems and formulating improvement measures, and support subsequent data analysis and decision-making.

2. According to the data quality verification and repair method based on big data as described in claim 1, It is characterized in that When the user customizes the rules in S1, the value range of the data field, the data format requirements, and the conditional restrictions on the repeated appearance of data can be set in detail.

3. According to the data quality verification and repair method based on big data as described in claim 1, It is characterized in that The self-defined rules and indicators in S2 include, but are not limited to, an upper limit on the proportion of missing values ​​in the data, a reasonable range of numerical distribution, and requirements on the degree of correlation between data.

4. According to the data quality verification and repair method based on big data as described in claim 1, It is characterized in that The repair method in S3 includes data interpolation, data inference, and data error correction techniques to restore the integrity and accuracy of the data. By establishing a complex data model and combining historical data characteristics and trends, the reasonable value range of abnormal data is predicted.

5. According to the data quality verification and repair method based on big data as described in claim 1, It is characterized in that The data quality report generated in S4 includes a comprehensive score of data quality, detailed scores of various verification indicators, and the specific distribution location and quantity of abnormal data.

6. According to the data quality verification and repair method based on big data as described in claim 1, It is characterized in that It also includes real-time monitoring of dynamic changes in data during the entire data quality verification and repair process, and automatic adjustment of verification rules and repair strategies based on operations such as data addition, modification, and deletion.

7. According to the data quality verification and repair method based on big data as described in claim 1, It is characterized in that It also includes strict retrospective verification of the repaired results after completing the data quality verification and repair operations, and ensuring that the data quality is significantly and stably improved by comparing it with the original data and expected quality standards.

8. A data quality verification and repair system based on big data, using a data quality verification and repair method based on big data as described in any one of claims 1 to 7, It is characterized in that Includes a rule template module to support users in defining and managing data quality verification rules; Data quality verification module, used to evaluate the data quality in large data sets; Anomaly data repair module, used to automatically repair abnormal data in large data sets; The data quality report and visualization module is used to generate detailed data quality reports based on the results of data quality verification and abnormal data detection, and provide visual charts and statistical information.

Images