Chinese microblog topic detection method and system based on semanteme, time and social relation

Abstract

The invention provides a Chinese microblog topic detection method and system based on semantics, time and social relations, which is used for solving the problem that in topic detection, microblog data is poor in topic detection effect due to the defects of short text, spoken language, polysemy and the like. The method comprises the steps of collecting the microblog data of related topics at a certain time interval, performing pre-training on acquired microblog data by using a pre-training language model BERT (Binary Encoder Transformers), and performing pre-training on the acquired microblogdata by using the pre-training language model BERT to obtain pre-trained microblog data; conducting vectorization representation on the microblog text through a pre-trained BERT model, and acquiring microblog semantic representation based on context semantics; proposing a text clustering algorithm comprehensively considering a time factor and a forwarding relationship between microblogs so that the problem that the traditional microblog topic detection only considers text semantic similarity is solved. The invention is mainly used for microblog search tasks, and the topic detection results ofrelated microblogs are used for improving the microblog search hit rate.

Description

technical field [0001] The invention belongs to the field of natural language processing and information retrieval, relates to topic detection and tracking technology, and is mainly aimed at topic detection of Chinese microblog data. [0002] technical background [0003] In recent years, due to the widespread popularization and rapid development of network technology, the speed of information dissemination on the network and the scale of the amount of information in the network are unprecedentedly huge. As a new social network media, Weibo has gradually become an important source of information for people. Because the content of microblog is very short, and microblog information can be released on various terminals, a large amount of microblog data will be generated in a short period of time on the microblog platform. If we only manually process these huge and disorganized information content on Weibo, it will not only greatly increase the workload, but also it is difficult...

AI Technical Summary

Problems solved by technology

[0006] The technical problem to be solved by the present invention is how to solve the problems of dimension disaster, sparse features, polysemy, etc. and inaccurate topic detection results caused by the messy and short text content of Weibo.

Images