Improved Kmeans clustering method based on SimHash

Abstract

The invention discloses an improved Kmeans clustering method based on SimHash. The method includes the following steps: 1. using the SimHash algorithm to compute the fingerprint of each to-be-clustered file, generating one SimHash table; 2. selecting one to-be-clustered file, based on the fingerprint of the file, checking the SimHash table to obtain one similar file set S0; 3. putting the files from the S0 that have an Overlap value with the to-be-clustered file greater than a threshold [alpha] to a similar file S1; 4. back to a cluster C0 belonged to the S1, computing the distances between the to-be-clustered file and the centers of mass of all clusters in the C0, classifying the to-be-clustered file to a cluster I which is closest to the to-be-clustered file and smaller than the threshold and updating the center of mass of the cluster I, or creating a new cluster k, and taking the to-be-clustered file as a first element of the cluster k; if C0 is empty, creating a cluster j, taking the to-be-filed cluster as a first element of the cluster j. According to the invention, the method increases the effects of clustering.

Description

technical field [0001] The invention relates to an improved Kmeans clustering method based on SimHash, belonging to the technical field of computer networks. Background technique [0002] In today's era of explosive growth of Internet information, deduplication and clustering of information, as an important technical means, has been researched and favored by more and more scholars. The deduplication and clustering of text information is an important part of it. Therefore, the present invention only involves deduplication and clustering of text information. The clustering of text information mainly involves two aspects: the first is the text representation mode, that is, what to use to represent the text; the second is to choose the appropriate clustering algorithm. At present, there are many kinds of text representation modes, and the popular ones are Vector Space Model (VSM), probability model, language model, etc.; similarly, in the selection of clustering algorithm, the...

AI Technical Summary

Problems solved by technology

The clustering problem is a typical partition-based problem. The simplest and most commonly used algorithm in the partition-based clustering algorithm is the Kmeans clustering algorithm, but the pure Kmeans algorithm has two obvious disadvantages: one is that it needs to Specify the number of clusters beforehand; if the number of clusters is not known in advance, the clustering quality is difficult to guarantee; second, it is necessary to iteratively calculate the centroid. When the sample data is large, not only the calculation cost is large but also time-consuming , cannot be applied to large amounts of data

Images