Method for determining optimal topic number of LDA topic model based on vocabulary similarity

Abstract

The invention discloses a method for determining optimal topic number of an LDA topic model based on vocabulary similarity, which comprises the steps of extracting topic words by utilizing an LDA model, and searching an optimal topic number based on similarity between word vectors. The method comprises the following steps: firstly, carrying out word segmentation and other preprocessing on text data, and carrying out topic modeling on a text by applying an LDA topic model to obtain corresponding word distribution under each topic; converting word distribution into word vector distribution, andanalyzing topic quality and determining an optimal topic number by utilizing similarity among vectors and based on LDA semantic association. According to the method provided by the invention, the optimal topic number can be automatically determined, the limitation of manual setting is avoided, and the method better serves the clustering analysis of the microblog text data.

Description

technical field [0001] The invention belongs to the technical field of natural language processing, and relates to a natural language processing model, in particular to a method for determining the optimal topic number of an LDA topic model based on lexical similarity. Background technique [0002] With the rapid development of the Internet, Weibo, as an open platform for user communication and information dissemination, is becoming more and more popular. Mining user interests and preferences and analyzing user preference behavior characteristics play a very important role in public opinion monitoring, network security management, and commercial value promotion. However, each user browses thousands of microblogs every day, and the massive amount of microblog information increases the difficulty for users to obtain the information they need, which affects user experience. Accurately obtaining user preferences is the key to proactively pushing content of interest to users on ...

AI Technical Summary

Problems solved by technology

At present, it is generally believed that the biggest problem of the LDA topic model sampled by Gibbs is that the optimal number of topics cannot be determined. In most cases, the number of topics is artificially set through experience. The number of topics is very important to the iterative process and results. Less will have a great impact on the model, resulting in accuracy errors in the final document distribution

Images