Chinese short text clustering method

Abstract

The invention relates to a Chinese short text clustering method, and in particular relates to a Chinese short text clustering method based on word vectors and similarity calculation thereof. The Chinese short text clustering method comprises the following specific steps of: obtaining needed word vectors by utilizing a Word2Vec word vector training model; obtaining weights of all words in a short text set by utilizing a word weight calculation algorithm; according to the word vectors and the weights of all the words, calculating the similarity value between every two texts in the short text set through a short text similarity algorithm; and, according to the similarity value between every two texts in the short text set, clustering short texts. The invention provides a carrying optimization type short text similarity calculation method; the problems of sparse short text grammar characteristics, semantic loss and the like can be solved; on the basis of a graph model, the weights of the words are continuously calculated iteratively, so that the sentence similarity calculation accuracy is increased; and, a density peak clustering method is applied in short text clustering, so that the efficiency of the clustering method is effectively increased.

Description

technical field [0001] The invention relates to the field of natural language processing, in particular to a Chinese short text clustering method based on word vectors and similarity calculations. Background technique [0002] In recent years, due to the popularization of mobile devices and the update of social media platforms, the manifestations of social media have undergone tremendous changes. The changes are specifically reflected in: the transition from long text to short text has been realized, and the focus has also shifted from traditional long text social media platforms such as blogs and forums to short text social media platforms such as Sina Weibo and Twitter. [0003] The immediacy and convenience of short-text social media platforms have greatly contributed to the growth of information volume. Compared with traditional texts, short texts in social networks have the characteristics of short texts, diverse topics, more garbage, and emotional tendencies, which po...

AI Technical Summary

Problems solved by technology

[0005] Therefore, the present invention aims at the overload of short text information in current social media, and the calculation method of existing short text similarity mainly includes calculating the same number of words, Jaccard similarity coefficient, cosine similarity, etc. These algorithms do not take into account The problem of semantic knowledge of short texts provides a Chinese short text clustering method, specifically a Chinese short text clustering method based on word vectors and their similarity calculations

Images