Mass data system with data cleaning function

Abstract

The invention discloses a mass data system with the data cleaning function. The mass data system comprises a data collecting module, a data processing module, a data cleaning module, a data memory management module, a data service module and a data monitoring module, wherein the data collecting module collects data from all kinds of data sources to a data processing center; data collected by the data collecting module and primarily processed is decoded and subjected to format conversion through the data processing module; the data cleaning module firstly completes data analysis and error type definition, secondly completes error logging searching and indentifying and finally completes error correcting; the data memory management module carries out memory management on data processed by the data processing module; the data service module is used for achieving the accessing requirements of clients for data; the data monitoring module monitors, records and processes data in the data collecting module, the data processing module and the data service module. By means of the mass data system with the data cleaning function, structural level errors and recording level errors can be found and corrected, and value losses, value errors, duplicate records and disagreement errors between the insides of data sources and data sources are avoided.

Description

technical field [0001] The invention relates to a data system, in particular to a massive data system with data cleaning. Background technique [0002] In recent years, with the rapid development of information technology, the amount of data collected, stored, processed and analyzed is increasing. The processing of massive data is becoming more and more popular. Different from the traditional data structure characteristics, big data has three characteristics, including massiveness, distribution, and heterogeneity. Its massiveness mainly refers to the huge scale of data and its growth rate continues to increase; its distribution is mainly reflected in the fact that the huge amount of data cannot be stored, calculated and analyzed on a single machine; its heterogeneity is mainly reflected in the differences in data types and data sources diversification. Using the traditional centralized processing method for structured data, it is difficult to solve the problems caused by b...

AI Technical Summary

Problems solved by technology

Using the traditional centralized processing method for structured data, it is difficult to solve the problems caused by big data. In view of these three characteristics, the integration and cleaning of big data becomes particularly important

Big data also includes uncertain data. At this stage, the causes of uncertain data are relatively diverse, mainly reflected in inaccurate original data, use of coarse-grained data sets, missing data fields, and data integration.

Images