A large data set clustering method based on MapReduce

Abstract

The invention belongs to the technical field of big data processing, and discloses a large data set clustering method based on MapReduce and an application thereof. Raw data input and format conversion; Canopy partition and screening to get the initial clustering partition; K-Means iteration, using the result of Canopy clustering as the initial clustering partition; data point allocation, after K-Means iteration, the complete information of k clusters is obtained. For the problem of initial cluster center selection and large iterative computation amount existing in a traditional K-Means algorithm. a K-Means improved algorithm based on Canopy partitioning and filtering is proposed, the algorithm is realized within the technical framework of MapReduce, and in-depth research is carried out. The results show that the improved algorithm has obvious performance improvement in clustering accuracy and computational overhead.

Description

technical field [0001] The invention belongs to the technical field of big data processing, and in particular relates to a clustering method for large data sets based on MapReduce. Background technique [0002] With the advent of the era of big data, in more and more application scenarios, the scale of data that people need to process expands to TB or even PB level, and they hope to quickly and effectively mine reliable and useful hidden information (AlexeyB et al.2018) . Therefore, how to quickly and accurately mine valuable information from big data is of great significance at present. As one of the core technologies in the field of data mining, cluster analysis can often be used as the pre-processing of other data mining algorithms (Treu T et al.2018). However, in the face of such a huge data scale, traditional clustering methods cannot meet the practical needs in terms of data storage and computing overhead (Efstathiou G et al. 2018). [0003] The MapReduce computing ...

AI Technical Summary

Problems solved by technology

[0005] To sum up, the problems existing in the existing technology are: in the face of huge data scale, the traditional clustering method cannot meet the actual needs in terms of data storage and computing overhead, and the accuracy rate is low, so it cannot effectively dig out reliable and useful clustering methods. hidden information of

[0010] (4) The decrease speed of the sum of squared errors in the iterative process of the improved algorithm is not faster than that of the traditional algorithm

Images