Topic clustering method and device, electronic equipment and storage medium

Abstract

The invention provides a topic clustering method and device, electronic equipment and a storage medium. According to the method, regression analysis can be carried out on the text data set based on aBERT model to obtain a base dataset, paragraph information of each text can be better expressed, the accuracy of text representation is improved, a configuration quantity of data are selected from thebasic data set for labeling, a small amount of annotation information is used for assisting overall unsupervised clustering, by adopting an Agglomerative Clustering model, clustering is carried out by combining a first inter-class distance and a second inter-class distance, and a similarity model trained based on a BERT algorithm is further adopted to obtain the target clustering result, so thatthe clustering result under the large inter-class distance is adopted as guidance, the clustering results under the small inter-class distance are combined, meanwhile, the recall rate and accuracy areensured, the dependence on the clustering distance and category is reduced, and the clustering effect is improved. The invention further relates to a block chain technology. The BERT model, the Agglomerative Clustering model and the similarity model can be stored on the block chain.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a topic clustering method, device, electronic equipment and storage medium. Background technique [0002] With the rapid development of Internet technology, major news portals have emerged as the times require, and these websites have become the main channels for news media to release news and people to obtain information. However, due to the complexity, redundancy, and rapidity of updates and dissemination of online news, it is difficult for users to quickly and accurately obtain the key information they need, and there are many difficulties in fine-grained topic clustering for news. . [0003] In the existing technical solutions, the clustering method often needs to set the number of clusters or the distance, and this information cannot be known in advance, which brings a huge challenge to the clustering task. Since it is an unsupervised task, it is impossible...

AI Technical Summary

Problems solved by technology

However, due to the complexity, redundancy, and rapidity of updates and dissemination of online news, it is difficult for users to quickly and accurately obtain the key information they need, and there are many difficulties in fine-grained topic clustering for news.

[0003] In the existing technical solutions, the clustering method often needs to set the number of clusters or the distance, and this information cannot be known in advance, which brings a huge challenge to the clustering task

Since it is an unsupervised task, it is impossible to determine the number of clusters or distances of the test data by obtaining the training data

[0004] In addition, when clustering, there are also problems that the text cannot be fully understood and the features are sparse, which will lead to inaccurate clustering results, so that the same topic is divided into different categories, or the same category is divided into different categories. topic

Images