Short text topic model mining method based on word network to extend characteristics

Abstract

A short text topic model mining method based on a word network to extend characteristics comprises a weighted word network construction step, a short text characteristics extending step, and a topic mining step. The weighted word network construction step comprises preprocessing a text, performing Chinese words segmentation on the text in a short text corpus, and deleting stop words; establishing a weighted word network from a document after the Chinese words segmentation is performed, wherein nodes in the weighted word network are words, each edge between the nodes is cooccurrence relation of two words in the same document, and the weight of the edge is the cooccurrence time of the two words in the whole corpus; and ending. The short text characteristics extending step comprises using the word nodes included by each short text after the Chinese words segmentation is performed as a community of the established weighted word network. According to the short text characteristics sparsity solution method based on word network community module degree, the problem that the effect of applying an LDA topic model to the short text is poor is solved. Accuracy of a short text topic model is increased.

Description

technical field [0001] The invention relates to the field of short text text topic models and complex network analysis, and is a method for expanding short text features by using weighted word networks, so as to solve the feature sparsity problem existing in the application of LDA topic models to short texts. Background technique [0002] The features of short texts are very sparse, resulting in the poor performance of generative models such as LDA for topic modeling based on word co-occurrence relations on short texts. In the current environment, short texts have incomparable advantages over long texts. Compared with long texts, short texts express concise semantics and transmit information quickly. People are more and more inclined to use short texts to transmit information. , short texts are becoming one of the most important information carriers in today's society, such as online advertisements, short messages, and popular social media such as Weibo, Twitter, etc. These ...

AI Technical Summary

Problems solved by technology

Another way to expand short texts is to use search engines or wiki databases to find words or sentences related to the current text in search engines or wiki databases to expand the original text. This method can to a certain extent Solve the data sparsity problem of short text, but on the one hand, external databases will have a great impact on the expansion results, on the other hand, the final model quality depends on the quality of the expansion strategy

Images