Short text topic recognition method based on Dirichlet variational auto-encoder

Abstract

The invention provides a short text topic recognition method based on Dirichlet variational auto-encoder, which comprises the following steps of S1, performing preprocessing, word segmentation, stop word removal, punctuation mark and number removal on a short text data set to obtain a text feature vector of the data set; s2, performing training, performing clustering, determining the category of each short text in the short text set, and taking the category as supplementary feature information of the short text; s3, constructing a conditional variational neural topic model to obtain document-topic distribution and topic-word distribution in the corpus set; and S4, carrying out short text topic recognition to obtain supplementary feature information of the short text as feature representation of the short text, wherein the supplementary feature information is used for text classification and clustering. According to the short text topic recognition method based on the Dirichlet variational auto-encoder, model training is accelerated, the problem of sparse features of a short text topic model is solved, and the classification and clustering effects of short texts are enhanced whileshort text topic recognition is performed.

Description

technical field [0001] The present invention relates to the technical field of short texts, and more specifically, relates to a short text topic recognition method based on a Dirichlet variational autoencoder. Background technique [0002] With the vigorous development of the Internet, the Internet has become an important source of information for people. As the main information carrier, text plays an important role in the dissemination of network information. Many data analysis applications such as Weibo, SMS, and comments involve extracting topic information from short texts, and extracting potential topics is conducive to further analysis, such as sentiment analysis, text classification, and recommendation systems. However, it is difficult for us to directly extract information from short text data due to the small number of characters and random writing characteristics of short text data. [0003] Chinese Patent Publication No. CN107798043A, date of publication June 28...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to overcome the disadvantages of slow training speed, high time complexity, poor scalability, and sparse short text features in the prior art. The present invention provides a short text topic recognition method based on Dirichlet variational autoencoder , speed up model training, solve the problem of sparse features of the short text topic model, and enhance the classification and clustering effect of short text while performing short text topic recognition

Images