A data cleaning system and method for big data

Abstract

A data cleaning system and method for big data. The application layer of the system includes a data analysis and extraction module, a similar connection module, a similar subgraph aggregation module, an entity sampling module, a probability calculation and an entity query module. The storage layer uses the distribution provided by Hadoop. The storage tool HDFS stores structured data records, similar data record pairs, and similar connected subgraphs generated during the data cleaning process. The distributed storage tool HBase provided by Hadoop is used to store the cleaned structured data records. The method includes obtaining data to be cleaned; similar connection; similar subgraph aggregation; entity sampling; probability calculation and entity query. The invention is a data cleaning system and uncertainty data determination method for big data, which solves the problem that the previous centralized similarity connection cannot adapt to large-scale data operations, and makes full use of graphs and related knowledge to creatively complete big data. Cleans and provides data preparation for massive data analysis.

Description

technical field [0001] The invention belongs to the technical field of data mining, and in particular relates to a big data-oriented data cleaning system and method. Background technique [0002] In recent years, with the rapid development of information technology, the amount of data collected, stored, processed and analyzed is increasing. The processing of big data is becoming more and more popular. Different from the traditional data structure characteristics, big data has three characteristics, including massiveness, distribution, and heterogeneity. Its massiveness mainly refers to the huge scale of data and its growth rate continues to increase; its distribution is mainly reflected in the fact that the huge amount of data cannot be stored, calculated and analyzed on a single machine; its heterogeneity is mainly reflected in the differences in data types and data sources diversification. Using the traditional centralized processing method for structured data, it is dif...

AI Technical Summary

Problems solved by technology

Using the traditional centralized processing method for structured data, it is difficult to solve the problems caused by big data. In view of these three characteristics, the integration and cleaning of big data becomes particularly important

Big data also includes uncertain data. At this stage, the causes of uncertain data are relatively diverse, mainly reflected in inaccurate original data, use of coarse-grained data sets, missing data fields, and data integration.

Images