Dynamic configurable rule-based data cleaning framework under big data background

Abstract

The invention belongs to the field of big data processing and analysis, and discloses a dynamic configurable rule-based data cleaning framework under a big data background. According to the dynamic configurable rule-based data cleaning framework under the big data background, a new method which is interdisciplinary, reusable and configurable and which integrates the data conversion, data check and data recovery is adopted, so that the description ability and execution efficiency of the cleaning process are improved. Experimental results of a plurality of real data sets indicate that the data cleaning framework is capable of integrating dynamic configurable rules to a plurality of data sources and a plurality of different application fields in a seamless manner, and can be implemented in a plurality of projects, so that the effective functions of the data cleaning framework in real scenes are further verified.

Description

technical field [0001] The invention belongs to the field of big data processing and analysis, and is a data cleaning framework based on dynamically configurable rules used in a big data environment. Background technique [0002] Research on the data of several well-known companies, 25% of the important data is flawed. A survey found that "dirty data" caused American companies to pay about 600 billion US dollars (600 billion dollars) in losses every year. A recent survey by Experian QAS Inc found that British companies were in 2011 because of quality problems with customer data. Lost £8 billion. In fact, the market for data cleaning tools is growing at a rate of 17% per year, which is higher than the average growth rate of 7% for other sectors of the IT industry. Although data cleaning research is constantly advancing, there is still no ready-made solution that can be directly used and deployed in different application fields without complex customization to automatically ...

AI Technical Summary

Problems solved by technology

[0005] Most of the existing research on data cleaning focuses on character data, and the processing of numerical (such as numerical fields falling within a certain interval), enumerated, Boolean and other fields is not mature and practical; most Data cleaning tools or frameworks are aimed at certain specific fields. If users need to introduce new rules or reuse some rules from other fields (for example, ID number rules are common in many fields), it becomes very difficult to expand existing solutions. Or it becomes very difficult to deploy these solutions to your own system; at present, there are still some cleaning tools whose cleaning detection and cleaning modification are implemented by hard coding, which will lead to poor scalability and flexibility of the system. When the cleaning rules change The code of the cleaning part needs to be re-implemented, and the descriptiveness of the hard-coded method for data cleaning is weak, especially in the implementation of complex logic data cleaning, which is relatively lacking in execution and understanding; there are also some cleaning tools In the process of detection, cleaning and modification, manual judgment is used. This method has the advantage of high accuracy when t

Images