Data clustering method based on dimensionality reduction and sampling

Abstract

The invention discloses a data clustering method based on dimensionality reduction and sampling, comprising steps of performing dimensionality reduction processing on a data set through a piecewise mean value algorithm, constructing a random function, performing random sampling from a large-scale clustering data set to obtain a working set with a relatively small scale, performing k-means clustering on the working set to obtain a random sampling clustering result, and performing classification on residual samples through measuring a relation between the residual samples and the obtained sampling clustering result. The data clustering method based on dimensionality reduction and sampling adopts the dimensionality reduction and sampling to reduce the number and dimensionalities of data samples which participate iteration, greatly reduces the complexity of the k-means algorithm under the condition that a good clustering effect is maintained, and realizes high efficiency clustering for big-scale data.

Description

technical field [0001] The invention relates to a method capable of efficiently clustering large-scale data, belonging to the technical field of data processing. Background technique [0002] At present, the commonly used data clustering methods include classic k-means, FCM, hierarchical clustering and self-organizing neural mapping, among which k-means is the most classic and widely used partitioning clustering method. The K-means clustering method dynamically iteratively adjusts the cluster center, and iterates continuously according to the similarity between the sample and each sub-cluster center to obtain the clustering result. However, since k-means needs to repeatedly calculate the similarity between each sample and the center, the complexity of the algorithm will increase exponentially with the increase in the dimension of the data set (time complexity: O(tkmn), space complexity : O((m+k)n). Among them, t is the number of iterations, k is the number of clusters, m is...

AI Technical Summary

Problems solved by technology

However, since k-means needs to repeatedly calculate the similarity between each sample and the center, the complexity of the algorithm will increase exponentially with the increase in the dimension of the data set (time complexity: O(tkmn), space complexity : O((m+k)n)

Among them, t is the number of iterations, k is the number of clusters, m is the number of records, n is the dimension), when the sample size is large, it cannot be processed effectively

Images