Improved k-means clustering method based on distributed computing platform

Abstract

The invention discloses an improved k-means clustering method based on a distributed computing platform, and introduces a distributed computing platform Spark for the problem of slow processing of mass data, the Kruskal's algorithm for the problem of too many iterations and the Tanimoto distance for the problem of giving no consideration to the correlation among features of a vector. First, a minimum spanning tree is constructed for the randomly selected k points by using the Kruskal's algorithm, the corresponding weight sum is obtained, and the process is repeated for n times. Then, according to weight sums obtained within the n times, the maximum weight is selected thereform and that distance values between edges composed of the k points are not much different is ensured. In this way, the relatively uniform distribution of cluster centers can be guaranteed. Finally, a clustering algorithm is performed using k-means algorithm improved by using the Tanimoto distance.

Description

technical field [0001] The invention relates to an improved k-means clustering method suitable for distributed computing platform Spark in machine learning, and belongs to the technical field of data mining. Background technique [0002] The rapid development of Internet technology and information technology has led to a sharp increase in information resources, which has caused a serious problem of information overload. How to dig out hidden and useful information from massive data has aroused more and more people's attention, and machine learning technology has emerged from this. Cluster analysis is a very important part of it. It forms a collection of abstract or physical objects into multiple classes, so that objects of the same class have a high degree of similarity, and objects of different classes have as low a degree of similarity as possible. In the field of machine learning, partition-type clustering, density-type clustering, and network-type clustering algorithms...

AI Technical Summary

Problems solved by technology

Aiming at the problem of selecting the maximum value of the minimum spanning tree sum in the prior art without ensuring the relatively uniform distribution of the cluster centers, the present invention proposes that if the distances between the points in the selected cluster center point set cannot be guaranteed to be similar, here it is necessary to set A threshold, then remove the cluster center point set, select the point set corresponding to the minimum spanning tree weight and the maximum value from the remaining point sets as the cluster center point set, and judge the difference between the points in the cluster center point set at this time Whether the distance is guaranteed to be not much different, repeat the above process, which can effectively reduce the number of iterations

Aiming at the problem that the prior art two-time selection does not solve the problem of randomness well, the present invention selects the initial center through Closcal, and reduces the problems caused by randomness as much as possible by repeating n times, which can better reduce iterations

Images