Data quality detection method and device

By dynamically selecting data quality detection models and filtering algorithms and switching models according to data flow rate, the problem of balancing detection accuracy and throughput is solved, efficient data quality detection is achieved, and computing costs and resource waste are reduced.

CN120658649APending Publication Date: 2025-09-16GRG BANKING IT
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511010885.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-07-22
Publication Date
2025-09-16

AI Technical Summary

Technical Problem

Existing data quality detection methods find it difficult to balance detection accuracy and data throughput, and are unable to achieve a dynamic balance between the two, resulting in high computational costs and waste of resources.

Method used

By dynamically selecting the data quality detection model based on the flow rate of the target data stream, different filtering algorithms are adopted to balance detection accuracy and throughput, including moving window average filtering algorithm and sliding window average filtering algorithm, and the model is switched according to the flow rate range to optimize resource utilization.

Benefits of technology

It achieves a dynamic balance between detection accuracy and throughput, reduces computing costs, avoids systematic errors, improves storage efficiency, and reduces resource waste.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120658649A_ABST
    Figure CN120658649A_ABST
Patent Text Reader

Abstract

The invention discloses a data quality detection method and device, and belongs to the technical field of computers. The data quality detection method comprises the following steps: acquiring a target data stream; based on the flow velocity of the target data stream, determining a data quality detection model corresponding to the flow velocity of the target data stream in the plurality of data quality detection models as a target model; and detecting the data quality of the target data stream based on the target model. According to the data quality detection method and device provided by the invention, the data quality detection model corresponding to the flow rate of the target data stream is determined as the target model for detecting the data quality of the target data stream, and the data quality detection model is dynamically selected based on the flow rate of the target data stream; according to the method, the precision of data quality detection and the data throughput can be considered, dynamic balance is achieved between the precision of data quality detection and the data throughput, the workflow model can be improved, the storage efficiency can be improved, the calculation cost can be reduced, and systematic errors can be effectively avoided.
Need to check novelty before this filing date? Find Prior Art

Claims

1. A data quality detection method, characterized in that: include: Get the target data stream; Based on the flow rate of the target data flow, determining the data quality detection model corresponding to the flow rate of the target data flow among the multiple data quality detection models as a target model; Based on the target model, the data quality of the target data stream is detected.

2. The data quality detection method according to claim 1, characterized in that: Each of the data quality detection models includes multiple detection indicators and the weight corresponding to each detection indicator; for any two of the data quality detection models, the number of detection indicators included is different, or the same number of detection indicators are included and at least one of the detection indicators is different, or the same detection indicators are included and the weight corresponding to at least one of the detection indicators is different.

3. The data quality detection method according to claim 2, characterized in that: The detection indicators include: redundancy indicators and / or deviation indicators; the redundancy indicators are used to indicate the redundancy level of data fields in the target data stream; the deviation indicators are used to indicate the degree of deviation between the distribution of data in the target data stream and the target distribution corresponding to the target data stream.

4. The data quality detection method according to any one of claims 1 to 3, characterized in that: The step of determining, based on the flow rate of the target data flow, the data quality detection model corresponding to the flow rate of the target data flow among the multiple data quality detection models as the target model comprises: Obtaining the flow rate of the target data flow; Determining a target flow rate interval corresponding to the flow rate of the target data flow; The data quality detection model corresponding to the target flow rate interval among the multiple data quality detection models is determined as the target model.

5. The data quality detection method according to claim 4, characterized in that: The determining of the target flow rate interval corresponding to the flow rate of the target data flow includes: determining a target filtering algorithm based on a flow rate of the target data stream, a first flow rate threshold, and a second flow rate threshold; wherein the first flow rate threshold is less than the second flow rate threshold; Performing filtering processing on the flow rate of the target data stream based on a target filtering algorithm; The target flow rate interval corresponding to the flow rate of the target data flow after filtering is determined.

6. The data quality detection method according to claim 5, characterized in that: The determining of a target filtering algorithm based on the flow rate of the target data flow, a first flow rate threshold, and a second flow rate threshold includes: When the flow rate of the target data stream is greater than or equal to the first flow rate threshold, determining the first filtering algorithm as the target filtering algorithm; When the flow rate of the target data flow is less than the second flow rate threshold, determining the second filtering algorithm as the target filtering algorithm; The computing resources consumed by the first filtering algorithm are less than those consumed by the second filtering algorithm, and the accuracy of the second filtering algorithm is higher than that of the first filtering algorithm.

7. The data quality detection method according to claim 6, characterized in that: The first filtering algorithm includes a moving window average filtering algorithm; the second filtering algorithm includes a sliding window average filtering algorithm.

8. A data quality detection device, characterized in that: include: Acquisition module, used to obtain target data stream; a determination module, configured to determine, based on the flow rate of the target data flow, the data quality detection model corresponding to the flow rate of the target data flow among the multiple data quality detection models as a target model; A detection module is used to detect the data quality of the target data stream based on the target model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data quality detection method according to any one of claims 1 to 7 is implemented.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data quality detection method according to any one of claims 1 to 7 is implemented.