A large-scale data distributed clustering processing method based on mapreduce

Abstract

The invention provides a large-scale data distributed clustering processing method based on MapReduce, which includes sampling large-scale data on the principle of non-repetition of equal scales; inputting sampled data into the MapReduce distributed parallel framework and calculating the local density and sum of the sampled data. Average density; find all sampled data whose local density is greater than the average density as a set of candidate points for the initial cluster center point of each cluster and feed it back to the main node, and select the distance between every two adjacent candidate points to be greater than 2 times the setting All candidate points in the range are used as the initial clustering center points; the MapReduce distributed parallel framework is used to perform parallel clustering tasks, and the average distance between data is calculated for each cluster to update the clustering center point; the error sum of squares criterion function is applied to the child nodes Determine whether to continue iteration; each sub-node clusters large-scale data based on the cluster center point. The invention realizes parallel clustering, reduces the number of clustering iterations, and improves clustering accuracy and parallel clustering efficiency.

Description

technical field [0001] The invention belongs to the technical field of parallel clustering, in particular to a large-scale data distributed clustering processing method based on MapReduce. Background technique [0002] With the rapid development of information technology, the scale of data continues to increase, and the use of parallel mechanisms to effectively mine and analyze large-scale data sets can promote the development and progress of Internet technology. Cluster analysis is an important data processing technology, and one of the important topics in the field of machine learning and artificial intelligence. It is widely used in data mining, information retrieval and other research. The main task is to divide the data set into multiple subsets, so that the similarity between the data objects in the subset is high, and the difference between the data objects in different subsets is relatively large. Due to the increase of data scale, the traditional stand-alone cluste...

Images