Topic model-based text representation method

Abstract

The invention belongs to the technical field of text mining, and discloses a topic model-based text representation method, which comprises the following steps of S1, preprocessing a text; S2, performing LDA topic modeling of Gibbs sampling on the preprocessed text; S3, selecting a theme keyword of each theme; carrying out normalization processing on the theme keywords; S4, mapping each topic keyword into a word2vec space; S5, setting a distance threshold value, and deleting the theme keywords of which the cosine distances with other theme keywords are greater than the distance threshold value;S6, calculating the coordinates of each theme in the word2vec space; calculating the coordinates of each text in the word2vec space; and calculating the Euclidean distance between each text and eachtopic, and classifying each text to the topic with the minimum Euclidean distance to the text. According to the method, modeling is carried out through the topic model, and then the topic model is mapped into the Word2vec space to carry out vectorized representation on the text, so that the defects of topic ambiguity distinguishing in the LDA topic model are overcome, and the problem of word sequentiality is solved.

Description

technical field [0001] The invention belongs to the technical field of text mining, and in particular relates to a text representation method based on a topic model. The topic model is modeled and then mapped to Word2vec space to perform vectorized representation of the text. Background technique [0002] In recent years, the number of network texts (such as news, blogs, etc.) has exploded, and they have extremely high research value and commercial value. How to better dig out the hidden value in massive network texts is a key issue that researchers, enterprises and countries are concerned about. [0003] Generally speaking, the traditional unsupervised clustering algorithm based on machine learning does not work well when dealing with text clustering. This is because text clustering uses word bag method and TF-IDF for feature extraction. These methods represent an article as a text-to-word vector based on a corpus. The word vector generated by this text representation is ...

AI Technical Summary

Problems solved by technology

[0004] In order for the present invention to overcome the deficiencies in the prior art, a text representation method based on a topic model is provided, through which the topic model is modeled, and then mapped to the Word2vec space to represent the text in a vectorized representation, which solves the topic ambiguity in the LDA topic model Distinguish between flaws and word sequence problems that existing bag-of-words models ignore

Images