MapReduce-based distributed cluster processing method for large-scale data

Abstract

The invention provides a MapReduce-based distributed cluster processing method for large-scale data. The method comprises the steps that the large-scale data is sampled according to equal-scale non-repetition principles; sampled data is input into a MapReduce distributed parallel framework, and local density and average density of the sampled data are calculated; all the sampled data with the local density greater than the average density is found out to serve as a candidate point set of initial cluster center points of all clusters, the candidate point set is fed back to a master node, and every two adjacent candidate points in a distance two times that of a set range among all candidate points are selected to service as the initial cluster center points; the MapReduce distributed parallel framework is utilized to perform a parallel clustering task, and an average value of the distance between the data is calculated for each cluster to update the cluster center points; whether child nodes continue iteration is judged by use of a sum-of-squared-error criterion function; and all the child nodes perform clustering on the large-scale data according to the cluster center points. Through the method, parallel clustering is realized, clustering iteration times are reduced, and clustering accuracy and parallel clustering efficiency are improved.

Description

technical field [0001] The invention belongs to the technical field of parallel clustering, in particular to a large-scale data distributed clustering processing method based on MapReduce. Background technique [0002] With the rapid development of information technology, the scale of data continues to increase, and the use of parallel mechanisms to effectively mine and analyze large-scale data sets can promote the development and progress of Internet technology. Cluster analysis is an important data processing technology, and one of the important topics in the field of machine learning and artificial intelligence. It is widely used in data mining, information retrieval and other research. The main task is to divide the data set into multiple subsets, so that the similarity between the data objects in the subset is high, and the difference between the data objects in different subsets is relatively large. Due to the increase of data scale, the traditional stand-alone cluste...

AI Technical Summary

Problems solved by technology

The K-Means method combines the Canopy method Canopy-Kmeans, uses the characteristics of Canopy to calculate the similarity of objects, and preprocesses the data. The advantage is that the initial clustering center point can be given to avoid falling into local optimum, but the disadvantage is that the distance between objects can be calculated. The time consumption of the similarity is relatively large

The method based on data density calculation is to calculate the density of all data, and then select the data with the highest density as the cluster center point to avoid the problem of random selection, and it is more accurate, but the traditional calculation cost is also large, and it is easy to cause node clustering. The load is heavy, reducing the overall efficiency of parallel clustering

Images