Data cleaning system and method for aiming at big data

Abstract

The invention discloses a data cleaning system and method for aiming at big data. A system application layer comprises a data analysis and extraction module, a similar joins module, a similar subgraph gathering module, an entity sampling module and a probability and entity query module, a storage layer stores a structural data record, a similar data record pair and a similar communication subgraph generated in a data cleaning process by utilizing a distributed storage tool HDFS (Hadoop Distributed File System) provided by Hadoop, and the cleaned structural data record is stored by utilizing a distributed storage tool HBase provided by the Hadoop. The method comprises the following steps: obtaining data to be cleaned, carrying out similar joins, enabling the similar subgrpahs to be gathered, sampling an entity, and carrying out probability calculation and entry query. The invention is a data cleaning system for aiming at the big data and an uncertain data certainty method, solves a problem that traditional centralized similar joins can not adapt to large-scale data operation, fully utilizes graphs and relevant knowledge to creatively finish big data cleaning, and provides a data preparation for the analysis of mass data.

Description

technical field [0001] The invention belongs to the technical field of data mining, and in particular relates to a big data-oriented data cleaning system and method. Background technique [0002] In recent years, with the rapid development of information technology, the amount of data collected, stored, processed and analyzed is increasing. The processing of big data is becoming more and more popular. Different from the traditional data structure characteristics, big data has three characteristics, including massiveness, distribution, and heterogeneity. Its massiveness mainly refers to the huge scale of data and its growth rate continues to increase; its distribution is mainly reflected in the fact that the huge amount of data cannot be stored, calculated and analyzed on a single machine; its heterogeneity is mainly reflected in the differences in data types and data sources diversification. Using the traditional centralized processing method for structured data, it is dif...

AI Technical Summary

Problems solved by technology

Using the traditional centralized processing method for structured data, it is difficult to solve the problems caused by big data. In view of these three characteristics, the integration and cleaning of big data becomes particularly important

Big data also includes uncertain data. At this stage, the causes of uncertain data are relatively diverse, mainly reflected in inaccurate original data, use of coarse-grained data sets, missing data fields, and data integration.

Images