High-dimensional mass data GMM (Gaussian Mixture Model) clustering method under Hadoop framework

Abstract

The invention discloses a high-dimensional mass data GMM (Gaussian Mixture Model) clustering method under a Hadoop framework. The high-dimensional mass data GMM clustering method is used for clustering high-dimensional mass data by structuring the clustering problem of mass data on a distributed platform, mainly aiming at the defects of the existing clustering algorithms. The high-dimensional mass data GMM clustering method comprises the following specific steps of: 1. constructing a local area network; 2. establishing a Hadoop platform; 3. uploading data to a cluster; 4. initially clustering, 5. calculating the parameter and the discrimination function of each cluster; 6. discriminating whether clustering is completed or not; 7. clustering again; 8. calculating the mean value and the weight of each class in a new cluster; 9. calculating the variance of each class in the new cluster; and 10. outputting clustering results. By using the characteristics of a MapReduce operation model in the Hadoop framework, using a Map parallelizing method to process parallelizable parts in the cluster and adopting two Map / Reduce tasks to respectively calculate the mean value and the variance, high-efficiency and high-accuracy clustering can be realized, and the scalability and the fault tolerance are better.

Description

Technical field [0001] The invention belongs to the field of computer technology, and further relates to a Gaussian Mixtures Model (GMM) clustering method for high-dimensional and massive data under the Hadoop framework in the data mining field. The invention can conveniently and efficiently complete the clustering of high-dimensional and massive data areas, and overcome the low-efficiency and dimensional disaster problems of massive data processing in the stand-alone mode. technical background [0002] A computing framework MR widely used in the process of massive data processing "Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters[J]. Communications of the ACM, 2005, 51(1): 107-113" The computing framework was invented by Google. In recent years, the emerging parallel programming model puts parallelization, fault tolerance, data distribution, load balancing, etc. in a library, and summarizes all operations on data by the system into two steps: The Map (...

AI Technical Summary

Problems solved by technology

The shortcomings of this method are: in the clustering process, it cannot effectively deal with massive data, and the algorithm efficiency is limited by time and space complexity.

However, the disadvantage of this clustering method is that this clustering method does not solve the problem of dimensionality disaster in large-scale data processing, and its scalability and scalability are not strong.

Images