Big data clustering method based on cloud computing platform

Abstract

The invention discloses a big data clustering method based on a cloud computing platform. The method comprises the following steps: 1: deploying a data set X = (x1, x2,..., xn) to an HDFS; 2, reading each record i in sequence, and calculating the similarity between the sample i and each sample to obtain a similarity matrix S of the data set, wherein the similarity calculation of the ith sample and the other samples and the similarity calculation of the jth sample and the other samples can be performed in parallel; 3, obtaining a weighted connection matrix W and a degree matrix D of the data set according to the matrix S; and 4, calculating a Laplacian matrix L. Compared with a traditional spectral clustering method, the parallel spectral clustering method has the advantages that the clustering precision of parallel spectral clustering is improved, meanwhile, the data processing speed can be improved through a MapReduce calculation framework used when the parallel spectral clustering is used for calculation, the method mainly depends on the number of calculation nodes in a Hadoop cluster, namely the number of task-trackers, and the clustering speed of the whole spectral clustering is greatly improved.

Description

technical field [0001] The invention relates to the field of data mining, in particular to a big data clustering method based on a cloud computing platform. Background technique [0002] Cloud computing (Cloud Computing) is a delivery model of computing resources, which usually virtualize resources. Simply put, cloud computing is the provision of computing services (including servers, storage, databases, networking, software, analytics, and intelligence)—providing rapid innovation, elastic resources, and economies of scale over the Internet. [0003] Big data, a term in the IT industry, refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain period of time. It requires a new processing model to have stronger decision-making power, insight, and Massive, high growth rate and diverse information assets of process optimization capabilities. [0004] The spectral clustering algorithm is based on the spectral...

AI Technical Summary

Problems solved by technology

[0005] However, the traditional spectral clustering algorithm is to solve the first k eigenvalues ​​and their corresponding eigenvectors in the constructed Laplacian matrix according to the number k of clusters and construct the eigenvector space, and then use the K-means algorithm to The eigenvectors in the vector space are clustered. In practical applications, as the size of the data set increases, the convergence speed of spectral clustering becomes very slow. However, using the traditional spectral clustering algorithm will reduce the running speed and configuration. Higher requirements, not conducive to universal use

Images