Online internet topic mining method based on improved LDA model

Abstract

The invention discloses an online internet topic mining method based on an improved LDA model. The method corresponds to a continuous and streaming type topic mining process conducted in a segmented mode, n web pages are processed each time, the web pages are usually acquired by web crawlers from the internet in an online and real-time mode, and the mining results of the contents of the web pages generate k topics. After the current n topics are processed, the newly acquired n web pages are continuously processed through the mentioned process. The process mainly includes initialization of On-LDA model hyper-parameters, dynamic updating of the On-LDA model hyper-parameters, internet topic mining based on the On-LDA model and the like. By means of the method, the assignment way and use effect in respect to the hyper-parameters and of a traditional LDA model in the topic mining process are radically changed, the classified information to which the web page contents belong is fully utilized to assign initial values to the model hyper-parameters and , the initial values of the hyper-parameters completely depend on the web page contents to be mined, and the computing process is simplified while reasonability is achieved.

Description

technical field [0001] The invention belongs to the technical field of the Internet, and specifically relates to an online mining method for Internet topics based on an improved LDA model. Topics are detected and mined online. Background technique [0002] The rapid development and widespread popularization of the Internet has gradually become an important medium for people to quickly obtain, publish and transmit information. Especially in recent years, the mobile Internet has been greatly developed. It fully combines the advantages of both mobile communication and the Internet, making it more convenient for people to obtain information. A large number of information resources from many sources and with different positions are constantly emerging on the Internet, and some hot and sensitive topics reflected by them are often disseminated and diffused at an extremely fast speed with the help of the Internet, which has a major impact on society. Therefore, how to detect and m...

AI Technical Summary

Problems solved by technology

In addition, the LDA model uses the same hyperparameter when generating the probability distribution of all topics for each word This approach is not reasonable

Therefore, topic mining and detection models such as PLSA and LDA are generally suitable for offline topic mining environments where the corpus is relatively static. For the real-time and streaming online mining requirements of Internet topics, in terms of rationality, timeliness, computing efficiency and accuracy, etc. greatly discounted

Images