Text clustering method based on random neighbor embedding

Abstract

The invention discloses a text clustering method based on random neighbor embedding. The text clustering method includes the following step that the text set is preprocessed and the text set is expressed as a normalized word-text co-occurrence matrix; high-dimensional text data is embedded into low-dimensional space by t-Distribution random neighbor embedding (t-SNE) to make the low-dimensional embedding points corresponding to the text with low similarity in high-dimensional space be far in distance and the low-dimensional embedding points corresponding to the text with high similarity be close in distance; and the plurality of low-dimensional embedding points are taken as the initial mass center of the K-means algorithm, and the K-means algorithm is adopted to perform clustering based on the low-dimensional space mapping point coordinates. According to the invention, the dimension disaster problem caused by text high-dimension sparse characteristics can be solved, the dimension of the text data is reduced, the operation time of the clustering algorithm is shortened, and the precision of the clustering algorithm is improved.

Description

technical field [0001] The invention relates to a text clustering integration method, in particular to a text clustering method based on random neighbor embedding. Background technique [0002] With the rapid growth of network information and the maturity of technologies such as search engines, the main problem facing human society is no longer information scarcity, but how to improve the efficiency of information acquisition and information access. At present, most of the information on the Internet is presented in the form of text, so how to effectively organize large-scale text collections has become a very challenging problem. [0003] Text / document clustering (text / document clustering) is based on the well-known clustering assumption: the similarity of texts of the same type is greater, and the similarity of texts of different types is smaller. As one of the most important unsupervised machine learning methods, clustering does not require training, nor does it require ...

AI Technical Summary

Problems solved by technology

The existing clustering algorithms are difficult to meet the following two requirements when processing text data: (1) high clustering accuracy; (2) fast running speed

Overall, fast clustering algorithms sacrifice accuracy, while high-precision clustering algorithms run slowly

Images