Big-data clustering algorithm based on cloud computing platform

Abstract

The invention discloses a large-data clustering algorithm based on a cloud computing platform. Primitive data is pre-processed; data are divided into M sub-data and distributed to M Map functions and local clustering is carried out on the sub-data; clusters with the same key are combined; if the number R of practical clusters is smaller than the number k of clusters, the number c of representative points and a constriction factor a are regulated and clustering is carried out again until termination conditions are achieved. if a new data set is generated, local clustering is carried out according to judgment conditions that the number K of new data source centers is larger than the obtained number K of clusters before updating or the number of new data source points is larger than the number of data source points before updating. According to the large-data clustering algorithm based on the cloud computing platform, the parallel computing capacity of a high-performance clustering system of cloud computing is used for solving the problem that mass data need to be processed in clustering, and therefore a relation of the data can be rapidly and efficiently dug up.

Description

technical field [0001] The invention belongs to the technical field of data mining and relates to a big data clustering algorithm based on a cloud computing platform. Background technique [0002] As an interdisciplinary subject in the fields of statistics, machine learning and data mining, cluster analysis has attracted many researchers to join in it, making it a very active research topic in the field of data mining research. So far, researchers at home and abroad have proposed many clustering algorithms. The main clustering methods can be divided into: partition-based methods, hierarchical-based methods, density-based methods, grid-based methods, and model-based methods. . [0003] At the "Sixth International Symposium on Mobile Internet" held on August 21, 2012, Deng Kan, a Ph.D. in computer robotics from Carnegie Mellon, said that discovering the value in big data depends on data mining algorithms, and There must be data mining algorithms plus cloud computing parallel...

AI Technical Summary

Problems solved by technology

In the face of big data processing, the method based on sample sampling probability is generally adopted, but the sampling method does not consider the overall relative distance between data points or intervals and the uneven distribution of data, resulting in the problem of hard division of intervals

Although later, clustering, fuzzy concepts, and cloud models were introduced to improve the problem of interval division and achieved good results, but these methods did not consider the different effects of big data data points on knowledge discovery tasks

Images