GMM clustering method for high-dimensional massive data under hadoop framework

Abstract

The invention discloses a high-dimensional mass data GMM (Gaussian Mixture Model) clustering method under a Hadoop framework. The high-dimensional mass data GMM clustering method is used for clustering high-dimensional mass data by structuring the clustering problem of mass data on a distributed platform, mainly aiming at the defects of the existing clustering algorithms. The high-dimensional mass data GMM clustering method comprises the following specific steps of: 1. constructing a local area network; 2. establishing a Hadoop platform; 3. uploading data to a cluster; 4. initially clustering, 5. calculating the parameter and the discrimination function of each cluster; 6. discriminating whether clustering is completed or not; 7. clustering again; 8. calculating the mean value and the weight of each class in a new cluster; 9. calculating the variance of each class in the new cluster; and 10. outputting clustering results. By using the characteristics of a MapReduce operation model in the Hadoop framework, using a Map parallelizing method to process parallelizable parts in the cluster and adopting two Map / Reduce tasks to respectively calculate the mean value and the variance, high-efficiency and high-accuracy clustering can be realized, and the scalability and the fault tolerance are better.

Description

technical field [0001] The invention belongs to the field of computer technology, and further relates to a Gaussian Mixture Model (GMM) clustering method for high-dimensional and massive data under the Hadoop framework in the field of data mining. The present invention can conveniently and efficiently complete the clustering of high-dimensional and massive data, and overcome the problems of inefficiency and dimension disaster in the processing of massive data in the stand-alone mode. technical background [0002] A computing framework widely used in the process of massive data processing MR "Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters [J]. Communications of the ACM, 2005, 51 (1): 107-113" the The computing framework was invented by Google. It is a parallel programming model that has emerged in recent years. It puts parallelization, fault tolerance, data distribution, load balancing, etc. in a library, and boils down all operations of the syste...

AI Technical Summary

Problems solved by technology

The shortcomings of this method are: in the clustering process, it cannot effectively deal with massive data, and the algorithm efficiency is limited by time and space complexity.

However, the disadvantage of this clustering method is that this clustering method does not solve the problem of dimensionality disaster in large-scale data processing, and its scalability and scalability are not strong.

Images