High-dimensional mass data GMM (Gaussian Mixture Model) clustering method under Hadoop framework

Abstract

The invention discloses a high-dimensional mass data GMM (Gaussian Mixture Model) clustering method under a Hadoop framework. The high-dimensional mass data GMM clustering method is used for clustering high-dimensional mass data by structuring the clustering problem of mass data on a distributed platform, mainly aiming at the defects of the existing clustering algorithms. The high-dimensional mass data GMM clustering method comprises the following specific steps of: 1. constructing a local area network; 2. establishing a Hadoop platform; 3. uploading data to a cluster; 4. initially clustering, 5. calculating the parameter and the discrimination function of each cluster; 6. discriminating whether clustering is completed or not; 7. clustering again; 8. calculating the mean value and the weight of each class in a new cluster; 9. calculating the variance of each class in the new cluster; and 10. outputting clustering results. By using the characteristics of a MapReduce operation model in the Hadoop framework, using a Map parallelizing method to process parallelizable parts in the cluster and adopting two Map/Reduce tasks to respectively calculate the mean value and the variance, high-efficiency and high-accuracy clustering can be realized, and the scalability and the fault tolerance are better.

Description

Technical field [0001] The invention belongs to the field of computer technology, and further relates to a Gaussian Mixtures Model (GMM) clustering method for high-dimensional and massive data under the Hadoop framework in the data mining field. The invention can conveniently and efficiently complete the clustering of high-dimensional and massive data areas, and overcome the low-efficiency and dimensional disaster problems of massive data processing in the stand-alone mode. technical background [0002] A computing framework MR widely used in the process of massive data processing "Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters[J]. Communications of the ACM, 2005, 51(1): 107-113" The computing framework was invented by Google. In recent years, the emerging parallel programming model puts parallelization, fault tolerance, data distribution, load balancing, etc. in a library, and summarizes all operations on data by the system into two steps: The Map (...

AI Technical Summary

Problems solved by technology

The shortcomings of this method are: in the clustering process, it cannot effectively deal with massive data, and the algorithm efficiency is limited by time and space complexity.

However, the disa...

Images