A hot topic discovery method based on btm and single-pass

Abstract

The present invention provides a hot topic discovery method based on BTM and Single-pass. The method first uses the BTM topic model to carry out topic modeling, obtains the topic distribution of the corpus data set, then uses VSM to vectorize, and then uses the improved The Single-pass algorithm clusters the results obtained above, sorts out the clustering results to obtain new clustering results, and secondly, parallelizes the above-mentioned hot topic discovery method to improve its performance in the case of large amounts of data. The speed of topic mining. The present invention can well solve the problems of microblog data sparseness and the ability to process massive data, and the improved Single-pass algorithm can well reduce the computational complexity, maintain the stability of the algorithm, and effectively process new data It has a good calculation and analysis of the continuous influence of hot topics, and the data set can still maintain the quality of topic discovery on the basis of improving data processing efficiency through the MapReduce framework.

Description

technical field [0001] The invention relates to a hot topic discovery method based on BTM and Single-pass, belonging to text clustering in the field of data mining. Background technique [0002] With the popularity of smart phones and the Internet, people can always pay attention to the latest major events in some countries and societies through the Weibo APP. Discovering and researching hot topics on Weibo is of great value in the fields of business and scientific research. More and more scholars are doing relevant research on Weibo. [0003] In traditional hot topic discovery, algorithms such as LDA topic model and K-Means are generally used for research. However, the traditional LDA model mainly solves the problem of long text, and the processing effect on short text data such as Weibo At the same time, microblog data has the characteristics of sparse data and strong contextual relevance, which is difficult for the LDA model to solve. [0004] In order to handle massive...

AI Technical Summary

Problems solved by technology

First of all, the amount of data to be processed in the process of hot topic discovery is huge, and it takes time and effort for a single host and processor to process

Secondly, the pure BTM model is too slow for topic mining of data. Finally, after using the BTM topic model for modeling, the characteristics of Weibo streaming data are not considered, so its classification effect needs to be improved

Images