Data cleaning method, device and equipment, and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to data cleaning method, device and equipment, and a storage medium. The data cleaning method comprises the steps: obtaining to-be-cleaned data, converting the to-be-cleaned data into decimal data, and generating a to-be-cleaned data table corresponding to a data source terminal from the decimal to-be-cleaned data; calling a preset data cleaning rule file, extracting a data cleaning rule corresponding to the table name of the to-be-cleaned data table, and generating a corresponding data cleaning execution code for each data cleaning rule; labeling each piece of to-be-cleaned data in the to-be-cleaned data table, and matching a corresponding data cleaning rule for the to-be-cleaned data; analyzing the label of each piece of to-be-cleaned data, executing a data cleaning execution code, and cleaning the to-be-cleaned data to obtain to-be-stored data; and storing the to-be-stored data in the form of binary data.According to the technical scheme, the data cleaning accuracy and efficiency are improved, and the storage space is saved.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a data cleaning method, device, equipment and storage medium. Background technique [0002] With the rapid development of computer technology and communication technology, people can obtain more and more digital information, but at the same time, they need to invest more time in organizing and organizing the digital information. For example, in business systems, some irregular data is often generated due to factors such as code defects, business definition changes, and network delays. For example, the payment time of an order is earlier than the creation time of the order. This is a piece of data that does not conform to business logic. . Before performing statistical analysis on the data, it is necessary to filter out these irregular data to ensure the accuracy of the statistics. Data cleaning is a process of reducing data errors and inconsistencies. The ...

AI Technical Summary

Problems solved by technology

However, it is obvious that the output efficiency of manual input and writing code is low; the efficiency of code operation depends on the personal ability of engineers, and optimization experience is difficult to promote on a large scale; irregular data is directly discarded, which is an irreversible operation. Both are disadvantageous; at the same time, it is not easy to ensure the standardization and consistency of the code

Images