A data quality intelligent detection and optimization method, system, device and medium
Patent Information
- Application Number
- CN202611147409.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-30
- Publication Date
- 2026-10-30
AI Technical Summary
[0005]有鉴于此，有必要提供一种数据质量智能检测与优化方法、系统、设备及介质，用以解决数据质量检测与优化的效率和精确度不高的技术问题
[0016]本发明的有益效果是：基于初始数据质量检查规则对待检测数据集进行数据质量检测，获得数据质量评估结果；构建与数据质量评估结果对应的智能优化算法库，基于数据质量评估结果以及智能优化算法库对待检测数据集进行优化处理，获得优化后的数据集以及优化报告，当优化后的数据集不满足预设的优化目标时，基于优化报告对初始数据质量检测规则进行修正，获得修正后的数据质量检测规则，基于修正后的数据质量检测规则对初始数据质量检测规则进行更新，当优化后的数据集满足优化目标时，将优化后的数据集作为最优质量的数据集，通过规则对数据进行检测，并通过算法对数据进行优化，并通过优化报告优化规则，提高了数据处理的效率与准确性。
Smart Images
Figure CN123011874A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data quality management technology, and in particular to a method, system, device and medium for intelligent detection and optimization of data quality. Background Technology
[0002] With the widespread application of information technology, a large amount of data, such as professional business data, is collected, transmitted and stored. However, due to the complexity of data sources, the diversity of collection devices and the uncertainty of data processing, data quality problems (such as missing data, redundancy, inconsistent formats, noisy data, etc.) are becoming increasingly prominent in practical applications and pose great risks to decision-making and resource coordination.
[0003] Data quality improvement is mainly based on data quality detection with set rules and data quality optimization with relevant methods. Currently, the set rules are mainly set manually, such as data type, range check, dependency rules, etc.; data optimization methods mainly include traditional methods such as feature selection, data standardization, and handling of single values, missing values, and outliers.
[0004] Traditional manual rules struggle to handle complex and dynamically changing data and lack intelligent detection mechanisms, making it difficult to automate the detection and optimization of data quality. Data quality optimization methods primarily rely on machine learning and artificial intelligence technologies to clean data based on data quality detection results, but they lack adaptability and intelligence, and cannot provide feedback to established manual rules, making it difficult to improve efficiency in complex business scenarios. Summary of the Invention
[0005] In view of this, it is necessary to provide a data quality intelligent detection and optimization method, system, device and medium to solve the technical problems of low efficiency and accuracy in data quality detection and optimization.
[0006] To address the aforementioned problems, in a first aspect, the present invention provides a data quality intelligent detection and optimization method, comprising: Dynamically configure initial data quality detection rules based on different data sources, data types, and application scenarios; Based on the initial data quality inspection rules, data quality inspection is performed on the dataset to be tested to obtain data quality assessment results; A smart optimization algorithm library corresponding to the data quality assessment results is constructed. Based on the data quality assessment results and the smart optimization algorithm library, the dataset to be tested is optimized to obtain the optimized dataset and optimization report. The smart optimization algorithm library is used to optimize the integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks of the dataset to be tested. When the optimized dataset does not meet the preset optimization objective, the initial data quality detection rules are corrected based on the optimization report to obtain the corrected data quality detection rules, and the initial data quality detection rules are updated based on the corrected data quality detection rules. When the optimized dataset meets the optimization objective, the optimized dataset is taken as the dataset with the best quality.
[0007] In one possible implementation, the initial data quality detection rules include at least one or more of the following: uniqueness rules, consistency rules, value range rules, null value rules, regular expression rules, integrity rules, precision rules, data format rules, accuracy rules, referential rules, volatility rules, relational rules, and logical rules.
[0008] In one possible implementation, the step of performing data quality checks on the dataset to be tested based on the initial data quality check rules to obtain data quality assessment results includes: Based on the initial data quality inspection rules, each item in the dataset to be inspected is checked to identify quality problems in the dataset and generate data quality assessment results corresponding to the quality problems.
[0009] In one possible implementation, the data quality assessment results include one or more of the following: integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks.
[0010] In one possible implementation, the intelligent optimization algorithm library includes one or more of the following: linear interpolation algorithm, KNN algorithm, timestamp-based priority determination mechanism, clustering algorithm, wavelet transform smoothing algorithm, quantile truncation method, data fusion algorithm, offline batch processing, replacement algorithm, and masking algorithm.
[0011] In one possible implementation, the step of optimizing the dataset to be detected based on the data quality assessment results and an intelligent optimization algorithm library to obtain an optimized dataset and an optimization report includes: When the data quality assessment result is an integrity defect, the dataset to be detected is optimized using a linear interpolation algorithm and / or a KNN algorithm to obtain an optimized dataset and an optimization report. When the data quality assessment result is a unique conflict, a timestamp-based priority determination mechanism is used to optimize the dataset to be tested, resulting in an optimized dataset and an optimization report. When the data quality assessment result is an accuracy anomaly, the wavelet transform smoothing algorithm is used to optimize the dataset to be detected, and an optimized dataset and optimization report are obtained. When the data quality assessment result is a consistency deviation, a data fusion algorithm is used to optimize the dataset to be tested, resulting in an optimized dataset and an optimization report. When the data quality assessment result is outdated, offline batch processing is used to optimize the dataset to be tested, resulting in an optimized dataset and an optimization report. When the data quality assessment result indicates a compliance risk, a replacement algorithm is used to optimize the dataset to be tested, resulting in an optimized dataset and an optimization report.
[0012] In one possible implementation, the step of revising the initial data quality detection rules based on the optimization report to obtain revised data quality detection rules includes: Based on the optimization objectives of the dataset (completeness, accuracy, consistency, uniqueness, validity, and relevance), the initial data quality detection rules are modified according to the optimization report and the optimization objectives to obtain the modified data quality detection rules.
[0013] Secondly, the present invention also provides a data quality intelligent detection and optimization system, comprising: The detection rule configuration module is used to dynamically configure initial data quality detection rules based on different data sources, data types, and application scenarios. The detection module is used to perform data quality detection on the dataset to be detected based on the initial data quality check rules, and obtain data quality assessment results. The optimization module is used to construct an intelligent optimization algorithm library corresponding to the data quality assessment results. Based on the data quality assessment results and the intelligent optimization algorithm library, the dataset to be tested is optimized to obtain the optimized dataset and optimization report. The intelligent optimization algorithm library is used to optimize the integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks of the dataset to be tested. The rule correction module is used to correct the initial data quality detection rules based on the optimization report when the optimized dataset does not meet the preset optimization target, obtain the corrected data quality detection rules, and update the initial data quality detection rules based on the corrected data quality detection rules. When the optimized dataset meets the optimization target, the optimized dataset is taken as the dataset with the best quality.
[0014] Thirdly, the present invention also provides an electronic device, comprising: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the data quality intelligent detection and optimization method described above.
[0015] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, wherein when the program or instruction is executed by a processor, it is able to implement the steps of the data quality intelligent detection and optimization method described in any one of the above method items.
[0016] The beneficial effects of this invention are as follows: Data quality is checked on the dataset to be tested based on initial data quality inspection rules to obtain data quality assessment results; an intelligent optimization algorithm library corresponding to the data quality assessment results is constructed; the dataset to be tested is optimized based on the data quality assessment results and the intelligent optimization algorithm library to obtain an optimized dataset and an optimization report; when the optimized dataset does not meet the preset optimization target, the initial data quality inspection rules are corrected based on the optimization report to obtain corrected data quality inspection rules; the initial data quality inspection rules are updated based on the corrected data quality inspection rules; when the optimized dataset meets the optimization target, the optimized dataset is taken as the dataset of optimal quality; data is detected through rules, optimized through algorithms, and the rules are optimized through the optimization report, thereby improving the efficiency and accuracy of data processing. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 A flowchart illustrating an embodiment of the intelligent data quality detection and optimization method provided by the present invention; Figure 2 A schematic diagram of an embodiment of the intelligent data quality detection and optimization system provided by the present invention; Figure 3 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation
[0019] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0021] This invention discloses a method, system, device, and medium for intelligent data quality detection and optimization, which can be used in computers. The method, device, or computer-readable storage medium involved in this invention can be integrated with the aforementioned device or operate relatively independently.
[0022] One specific embodiment of the present invention discloses a data quality intelligent detection and optimization method, which can be executed by a computer, specifically by one or more processors of the computer. For example... Figure 1 As shown, the intelligent data quality detection and optimization method includes: S101. Dynamically configure initial data quality detection rules based on different data sources, data types, and application scenarios.
[0023] It should be noted that the initial data quality inspection rules include multiple data quality inspection rules, which are configured according to the data source and usage requirements.
[0024] S102. Based on the initial data quality inspection rules, perform data quality inspection on the dataset to be tested to obtain data quality assessment results.
[0025] S103. Construct an intelligent optimization algorithm library corresponding to the data quality assessment results. Based on the data quality assessment results and the intelligent optimization algorithm library, optimize the dataset to be tested to obtain the optimized dataset and optimization report. The intelligent optimization algorithm library is used to optimize the integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks of the dataset to be tested.
[0026] S104. When the optimized dataset does not meet the preset optimization objective, the initial data quality detection rules are modified based on the optimization report to obtain the modified data quality detection rules. The initial data quality detection rules are then updated based on the modified data quality detection rules. When the optimized dataset meets the optimization objective, the optimized dataset is taken as the dataset with the best quality.
[0027] It should be noted that by revising the data quality inspection rules, data quality can be improved, manual intervention can be reduced, and the efficiency and accuracy of data quality detection and optimization can be increased.
[0028] In some embodiments, data quality detection rules and an intelligent optimization algorithm library are constructed in the host computer. The dataset to be detected is input into the host computer, the dataset to be detected is detected by the data quality detection rules, and an optimization report is obtained by the intelligent optimization algorithm library. The data quality detection rules are corrected by the optimization report, thereby obtaining the dataset with the best quality.
[0029] In some embodiments, in step S101, initial data quality detection rules are dynamically configured based on different data sources, data types, and application scenarios. Multiple data quality detection rules are defined and configured, serving as initial data quality detection rules. The initial data quality detection rules are dynamically configured based on different data sources, data types, and application scenarios to construct a rule engine. The rule engine includes basic configuration and quality rule configuration. Basic configuration management presets time variables and regular expression function variables, which can be directly referenced when configuring quality rules, simplifying user operations. Quality rule configuration supports configuring quality audit tasks from both object and rule perspectives. It supports adding individual rules and batch importing via templates. After rule configuration, online testing is possible, and quality audit tasks are automatically scheduled and executed after the rules are deployed. Rules are configured according to data sources and usage requirements, with initial data... The quality inspection rules should include at least one or more of the following: unique rules, consistency rules, value range rules, null value rules, regular expression rules, integrity rules, precision rules, data format rules, accuracy rules, referential rules, volatility rules, relational rules, and logical rules. For example, integrity rules can be constructed by checking whether data is indeed as specified in the fields and records; unique rules can be constructed by checking whether data records are duplicated; consistency rules can be constructed by checking whether field data is within the reasonable range of business requirements; and accuracy rules can be constructed by checking field format. A rule database can be built using multiple rules, while supporting rule version management, access control, and release management to ensure the accuracy and effectiveness of the rules. The rules can be dynamically adjusted according to actual conditions. For example, data from professional communications requires specific format verification rules; data from time-series systems requires time consistency and chain integrity verification.
[0030] In some embodiments, in step S102, data quality inspection is performed on the dataset to be inspected based on the initial data quality inspection rules to obtain data quality assessment results. Each item in the dataset to be inspected is then checked based on the initial data quality inspection rules to identify quality problems and generate corresponding data quality assessment results. For the dataset to be inspected, a data quality assessment is first performed. A rule engine checks each item in the data to identify quality problems such as missing data, duplication, inconsistency, and anomalies. The system supports access from multiple data sources (such as relational, non-relational, and file formats) and supports multiple inspection methods including full, incremental, and sampling inspections. The data quality assessment includes quality monitoring and management processes and quality report management processes. Quality monitoring and management executes rules in batches at regular intervals, generating detailed quality inspection results for each batch. The quality inspection results support viewing details from both rule and object perspectives. The data quality report management system allows users to view statistical analysis of rule execution across different timeframes from various dimensions, including rules, clusters, and teams. It supports custom configuration of analysis dimensions and exporting quality reports. Data quality assessment results include one or more of the following: integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks. The system uses initial data quality check rules in the rule engine to examine each data entry and identify quality issues. Integrity defects are marked as "Null" or "Empty"; uniqueness conflicts are marked as "Duplicate"; and consistency deviations are marked as "Formatting Inconsistencies," ... Error or LogicConflict; Accuracy anomaly: refers to data values exceeding the reasonable business range or being noisy data, the assessment result is marked as "Outlier" or "Noise"; Timeliness lag: refers to data update timestamps being delayed and not arriving within the predetermined time window, the assessment result is marked as "Stale Data"; Compliance risk: refers to sensitive data not being desensitized or violating security policies, the assessment result is marked as "Security Violation".
[0031] In some embodiments, in step S103, an intelligent optimization algorithm library corresponding to the data quality assessment results is constructed. Based on the data quality assessment results and the intelligent optimization algorithm library, the dataset to be tested is optimized to obtain the optimized dataset and optimization report. The intelligent optimization algorithm library is used to optimize the dataset to be tested for integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks. For different data quality assessment results, corresponding intelligent optimization algorithm libraries are provided. These libraries include linear interpolation algorithms, KNN algorithms, timestamp-based priority determination mechanisms, clustering algorithms, wavelet transform smoothing algorithms, quantile truncation methods, data fusion algorithms, offline batch processing, replacement algorithms, and masking algorithms. One or more of the masking algorithms are used. Data quality optimization includes various methods such as data cleaning, data transformation, data standardization, and data completion. Basic data optimization includes data duplication, data type conversion, date standardization, and unit standardization. Under the premise of basic data optimization, data quality problems such as missing data can be solved through data mining, machine learning, and other technologies. For optimization of "integrity defects", when the data quality assessment result is integrity defects, linear interpolation algorithm and / or KNN algorithm are used to optimize the dataset to be tested, resulting in an optimized dataset and optimization report. The judgment logic is that the identification field is empty and the field is marked as "required". The implementation process is to first determine whether the field has time series characteristics. If the data is a time series, a linear interpolation algorithm is used to fill in the missing data based on the values at the preceding and following time points. If the data does not have time series characteristics but has a correlated field, the KNN (K-Nearest Neighbor) algorithm is used to calculate the mean of similar records for filling in the missing data. If the data is a categorical field with a high missing rate, it is marked as "Unknown" and an alert is generated, rather than being directly deleted. When the data quality assessment result is a uniqueness conflict, a timestamp-based priority determination mechanism is used to optimize the dataset to be tested, resulting in an optimized dataset and optimization report. The optimization implementation for "uniqueness conflicts" is as follows: the judgment logic is based on hash comparison using the primary key or unique index. If duplicates are found in the data, the implementation process is to activate the timestamp-based priority determination mechanism and retain the data when... The latest record is interpolated, and other duplicate records are marked as "SoftDelete" (logical deletion) and written to the historical archive instead of being physically deleted to prevent accidental deletion of important historical states. When the data quality assessment result is an accuracy anomaly, the wavelet transform smoothing algorithm is used to optimize the dataset to be tested, resulting in an optimized dataset and optimization report. Optimization implementation for "accuracy anomaly" and "noise": The judgment logic is to calculate the degree of deviation of the data point from the mean. If the absolute value of the Z-score is greater than 3, or the DBSCAN clustering algorithm determines it as an isolated point, the implementation process is as follows: For high-frequency time-series data, the wavelet transform smoothing algorithm is used to eliminate high-frequency noise while preserving the data trend characteristics.For outliers, a quantile truncation method is used to forcibly correct values exceeding the 1%-99th quantile to boundary values, preventing extreme values from interfering with subsequent analysis. When the data quality assessment result is a consistency deviation, a data fusion algorithm is used to optimize the dataset to be tested, resulting in an optimized dataset and optimization report. The optimization implementation for "consistency deviation" is as follows: the judgment logic is based on regular expression matching failure or dictionary mapping mismatch. The implementation process involves calling the data fusion engine; for example, for inconsistent date formats (such as "2023 / 1 / 1" and "01-01-2023"), numerical features are automatically extracted and reconstructed. It uses the ISO standard format; for inconsistent units (such as "km" and "m"), it automatically identifies the unit field and performs conversion, uniformly storing standard units; when the data quality assessment result is time-delayed, offline batch processing is used to optimize the dataset to be tested, resulting in an optimized dataset and optimization report; for the optimization implementation of "time-delayed": the judgment logic is that the difference between the current system time and the data business time exceeds a preset threshold (such as 15 minutes). The implementation process is to judge the business value of the delayed data. If the delay time is within an acceptable range (such as <1 hour), it is directly written to the real-time database and marked "Late". The "Arriving" tag is used; if the delay is too long, the data is routed to an offline batch processing queue for historical data recalculation to avoid polluting real-time metrics; when the data quality assessment result indicates compliance risk, a replacement algorithm is used to optimize the dataset to be tested, resulting in an optimized dataset and optimization report. Optimization implementation for "compliance risk": The judgment logic is to identify unmasked plaintext sensitive data through sensitive data identification rules (such as regular expression matching of ID card and mobile phone number formats) or data fingerprinting technology, or to detect data access permission configurations that do not comply with security policies. The implementation process involves triggering a dynamic desensitization engine. For data fields identified as sensitive (such as personal privacy and trade secrets), a replacement algorithm (e.g., using a random mapping table to replace the real value) or a masking algorithm (e.g., replacing the middle 8 digits of the ID card number with "") is automatically applied. Simultaneously, the data's metadata tags are automatically corrected, marking it as "desensitized," and unauthorized access requests are blocked to ensure data security and compliance during the flow process.
[0032] Data quality is monitored and detected through rules. Based on the results of the quality detection report, corresponding intelligent optimization algorithms are provided to process the data. After processing, the data quality optimization results and optimization report are obtained.
[0033] In some embodiments, in step S104, when the optimized dataset does not meet the preset optimization objective, the initial data quality detection rules are modified based on the optimization report to obtain the modified data quality detection rules, and the initial data quality detection rules are updated based on the modified data quality detection rules. When the optimized dataset meets the optimization objective, the optimized dataset is taken as the dataset of optimal quality. Adaptive feedback and rule modification are performed according to the optimization report. The optimization objective is preset based on the completeness, accuracy, consistency, uniqueness, validity, and relevance of the optimized dataset. The initial data quality detection rules are modified based on the optimization report and the optimization objective to obtain the modified data quality detection rules. After obtaining the modified data quality detection rules, the initial data quality check rules in the rule engine are updated, and data quality detection and data optimization are performed again until the optimized dataset meets the optimization objective and the dataset of optimal quality is obtained. The adaptive feedback and rule modification focus on setting optimization objectives from multiple dimensions and angles such as the completeness, accuracy, consistency, uniqueness, validity, and relevance of the data quality optimization results. By continuously iterating and adaptively adjusting the predetermined rules, the data quality optimization achieves the best effect.
[0034] During the data cleaning and optimization process, optimization reports are automatically generated and fed back to the rule engine through a self-learning mechanism. The rules are automatically adjusted based on the cleaning effect to improve the efficiency and accuracy of subsequent data processing. A rule performance evaluation model is established to automatically eliminate inefficient rules. Subsequently, the data processing rules are rearranged, and machine learning algorithms such as decision trees can be used to generate rule priority rankings. When rules conflict, the optimal approach is selected based on historical processing results.
[0035] During data inspection and optimization, data security and integrity assurance mechanisms are constructed, such as encrypted storage, audit logs, and access control, to ensure that data is not subject to tampering or leakage during processing. For encrypted storage, transmission encryption, static data encryption, and key management can be implemented. For transmission encryption, TLS / SSL protocols are used to encrypt data transmission, ensuring that data is not tampered with during transmission. For static data encryption, sensitive stored data is encrypted using AES-256, and encryption keys are managed independently. Key management utilizes a key management system to rotate and manage keys. For audit logs, log recording, log protection, and regular auditing and analysis can be implemented. Log recording records... For critical user behaviors, log protection involves signing and encrypting logs for storage, while periodic auditing and analysis utilizes automated tools to analyze abnormal behavior. Regarding access control, the principles of least privilege, dynamic access control, and periodic access review are implemented. The principle of least privilege assigns permissions based on roles or attributes; dynamic access control implements real-time permission verification, requiring secondary authentication for critical operations; and periodic access review periodically cleans up expired permissions. Encryption technology is used to encrypt and store data, preventing tampering during processing. Access management and audit logs are set up to ensure the traceability and security of data processing, and a distributed storage solution is implemented to ensure high availability of data in case of failure.
[0036] In summary, the intelligent data quality detection and optimization method provided by this invention dynamically configures initial data quality detection rules based on different data sources, data types, and application scenarios; performs data quality detection on the dataset to be tested based on the initial data quality inspection rules to obtain data quality assessment results; constructs an intelligent optimization algorithm library corresponding to the data quality assessment results; optimizes the dataset to be tested based on the data quality assessment results and the intelligent optimization algorithm library to obtain an optimized dataset and an optimization report; when the optimized dataset does not meet the preset optimization target, the initial data quality detection rules are modified based on the optimization report to obtain modified data quality detection rules, and the initial data quality detection rules are updated based on the modified data quality detection rules; when the optimized dataset meets the optimization target, the optimized dataset is taken as the dataset of optimal quality, thereby improving the efficiency and accuracy of data processing.
[0037] To better implement the intelligent data quality detection and optimization method in the embodiments of the present invention, based on the intelligent data quality detection and optimization method, correspondingly, as follows: Figure 2 As shown, this embodiment of the invention also provides a data quality intelligent detection and optimization system 200, which includes: The detection rule configuration module 201 is used to dynamically configure the initial data quality detection rules based on different data sources, data types and application scenarios. The detection module 202 is used to perform data quality detection on the dataset to be detected based on the initial data quality inspection rules, and obtain data quality assessment results. The optimization module 203 is used to construct an intelligent optimization algorithm library corresponding to the data quality assessment results. Based on the data quality assessment results and the intelligent optimization algorithm library, the dataset to be tested is optimized to obtain the optimized dataset and optimization report. The intelligent optimization algorithm library is used to optimize the integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks of the dataset to be tested. The rule correction module 204 is used to correct the initial data quality detection rules based on the optimization report when the optimized dataset does not meet the preset optimization goal, obtain the corrected data quality detection rules, and update the initial data quality detection rules based on the corrected data quality detection rules. When the optimized dataset meets the optimization goal, the optimized dataset is taken as the dataset with the best quality.
[0038] like Figure 3 As shown, the present invention also provides an electronic device 300, which can be a mobile terminal, desktop computer, laptop, handheld computer, server, or other computing device. The electronic device 300 includes a processor 301, a memory 302, and a display 303. Figure 3 Only some components of the electronic device 300 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0039] In some embodiments, memory 302 may be an internal storage unit of the electronic device 300, such as a hard disk or memory of the electronic device 300. In other embodiments, memory 302 may be an external storage device of the electronic device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 300. Furthermore, memory 302 may include both internal and external storage units of the electronic device 300. Memory 302 is used to store application software and various types of data installed on the electronic device 300, such as program code installed on the electronic device 300. Memory 302 may also be used to temporarily store data that has been output or will be output. In one embodiment, memory 302 stores a data quality intelligent detection and optimization program, which can be executed by processor 301 to implement the data quality intelligent detection and optimization method of various embodiments of the present invention.
[0040] In some embodiments, processor 301 may be a central processing unit (CPU), microprocessor or other data processing chip, used to run program code stored in memory 302 or process data, such as intelligent data quality detection and optimization methods.
[0041] In some embodiments, display 303 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 303 is used to display identification information of the intelligent data quality detection and optimization program and to display a visual user interface. Components 301-303 of electronic device 300 communicate with each other via a system bus.
[0042] In some embodiments, when the processor 301 executes the data quality intelligent detection and optimization program in the memory 302, it implements each step of the data quality intelligent detection and optimization method as described in the above embodiments. Since the data quality intelligent detection and optimization method has been described in detail above, it will not be repeated here.
[0043] Accordingly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps or functions of the data quality intelligent detection and optimization method provided in the above-described method embodiments.
[0044] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0045] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A data quality intelligent detection and optimization method, characterized in that, include: Dynamically configure initial data quality detection rules based on different data sources, data types, and application scenarios; Based on the initial data quality inspection rules, data quality inspection is performed on the dataset to be tested to obtain data quality assessment results; A smart optimization algorithm library corresponding to the data quality assessment results is constructed. Based on the data quality assessment results and the smart optimization algorithm library, the dataset to be tested is optimized to obtain the optimized dataset and optimization report. The smart optimization algorithm library is used to optimize the integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks of the dataset to be tested. When the optimized dataset does not meet the preset optimization objective, the initial data quality detection rules are corrected based on the optimization report to obtain the corrected data quality detection rules, and the initial data quality detection rules are updated based on the corrected data quality detection rules. When the optimized dataset meets the optimization objective, the optimized dataset is taken as the dataset with the best quality.
2. The intelligent data quality detection and optimization method according to claim 1, characterized in that, The initial data quality detection rules include one or more of the following: uniqueness rules, consistency rules, value range rules, null value rules, regular expression rules, integrity rules, precision rules, data format rules, accuracy rules, referential rules, volatility rules, relational rules, and logical rules.
3. The intelligent data quality detection and optimization method according to claim 2, characterized in that, The process of performing data quality checks on the dataset to be tested based on the initial data quality check rules to obtain data quality assessment results includes: Based on the initial data quality inspection rules, each item in the dataset to be inspected is checked to identify quality problems in the dataset and generate data quality assessment results corresponding to the quality problems.
4. The intelligent data quality detection and optimization method according to claim 3, characterized in that, The data quality assessment results include one or more of the following: integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks.
5. The intelligent data quality detection and optimization method according to claim 4, characterized in that, The intelligent optimization algorithm library includes one or more of the following: linear interpolation algorithm, KNN algorithm, timestamp-based priority determination mechanism, clustering algorithm, wavelet transform smoothing algorithm, quantile truncation method, data fusion algorithm, offline batch processing, replacement algorithm, and masking algorithm.
6. The intelligent data quality detection and optimization method according to claim 5, characterized in that, The optimization process, based on the data quality assessment results and the intelligent optimization algorithm library, optimizes the dataset to be detected to obtain an optimized dataset and an optimization report, including: When the data quality assessment result is an integrity defect, the dataset to be detected is optimized using a linear interpolation algorithm and / or a KNN algorithm to obtain an optimized dataset and an optimization report. When the data quality assessment result is a unique conflict, a timestamp-based priority determination mechanism is used to optimize the dataset to be tested, resulting in an optimized dataset and an optimization report. When the data quality assessment result is an accuracy anomaly, the wavelet transform smoothing algorithm is used to optimize the dataset to be detected, and an optimized dataset and optimization report are obtained. When the data quality assessment result is a consistency deviation, a data fusion algorithm is used to optimize the dataset to be tested, resulting in an optimized dataset and an optimization report. When the data quality assessment result is outdated, offline batch processing is used to optimize the dataset to be tested, resulting in an optimized dataset and an optimization report. When the data quality assessment result indicates a compliance risk, a replacement algorithm is used to optimize the dataset to be tested, resulting in an optimized dataset and an optimization report.
7. The intelligent data quality detection and optimization method according to claim 1, characterized in that, The step of revising the initial data quality detection rules based on the optimization report to obtain revised data quality detection rules includes: Based on the optimization objectives of the dataset (completeness, accuracy, consistency, uniqueness, validity, and relevance), the initial data quality detection rules are modified according to the optimization report and the optimization objectives to obtain the modified data quality detection rules.
8. A data quality intelligent detection and optimization system, characterized in that, include: The detection rule configuration module is used to dynamically configure initial data quality detection rules based on different data sources, data types, and application scenarios. The detection module is used to perform data quality detection on the dataset to be detected based on the initial data quality check rules, and obtain data quality assessment results. The optimization module is used to construct an intelligent optimization algorithm library corresponding to the data quality assessment results. Based on the data quality assessment results and the intelligent optimization algorithm library, the dataset to be tested is optimized to obtain the optimized dataset and optimization report. The intelligent optimization algorithm library is used to optimize the integrity defects, uniqueness conflicts, consistency deviations, accuracy anomalies, timeliness delays, and compliance risks of the dataset to be tested. The rule correction module is used to correct the initial data quality detection rules based on the optimization report when the optimized dataset does not meet the preset optimization target, obtain the corrected data quality detection rules, and update the initial data quality detection rules based on the corrected data quality detection rules. When the optimized dataset meets the optimization target, the optimized dataset is taken as the dataset with the best quality.
9. An electronic device, characterized in that, Including memory and processor; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps of the data quality intelligent detection and optimization method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the intelligent data quality detection and optimization method according to any one of claims 1-7.