An internet data clustering method and system

Abstract

The invention relates to an internet text data clustering method and system. The internet text data clustering method comprises the steps of: a, training text data by using a topic model to obtain a probability distribution matrix of all key words in each topic, and grouping the key words in a text set; b, rearranging feature sets of the text data according to the grouping of the key works to obtain new document data containing key word grouping feature information; c, operating a double-layer soft subspace clustering algorithm on the new document data containing the key word grouping information to generate a clustering center matrix and a sample ownership matrix; d, repeating the steps a-c for n times to obtain a plurality of clustering results; e, operating a clustering integration algorithm on a model set to integrate the multiple clustering results to obtain a final clustering result. The method and the system can reduce the instability of an FG-k-means algorithm effectively.

Description

technical field [0001] The invention belongs to the technical field of data mining, in particular to an internet text data clustering method and system. Background technique [0002] With the advent of the era of big data, the data faced in the field of data mining has become more and more complex. Especially Internet text data, in addition to the huge amount, the text data constructed by the Vector Space Model (Vector SpaceModel) also has ultra-high dimensionality and sparsity. Existing data mining clustering algorithms, such as k-means, hierarchical clustering There are generally deficiencies and limitations when applied to text clustering such as general clustering and general clustering. [0003] Aiming at the problem of subspace clustering of high-dimensional sparse data, academia has proposed many related subspace clustering algorithms (Subspace Clustering), and soft subspace clustering algorithm is one of them. According to different weighted layers, soft subspace c...

AI Technical Summary

Problems solved by technology

[0007] The existing FG-k-means algorithm is superior to other algorithms in terms of clustering performance, but there are still the following problems: the FG-k-means algorithm needs to use the information of the feature group to complete the purpose of two-level clustering optimization, but generally Such information is not provided in the text data, which limits the scope of its application; FG-k-means has the problem of instability of clustering results caused by different selection of initial center points

Images