Text clustering method, electronic device and storage medium

Abstract

The invention discloses a text clustering method. The method comprises the steps of receiving a text clustering instruction sent by a user; pre-training a pre-determined initial language model by utilizing the to-be-clustered corpus to obtain a target language model; sequentially inputting each text in the to-be-clustered corpus into the target language model for feature extraction, obtaining a sentence vector of each text in the to-be-clustered corpus according to a model output result, and generating a to-be-clustered sentence vector set; and, by utilizing a preset clustering algorithm, clustering the to-be-clustered corpora based on the to-be-clustered sentence vector set to obtain sentence vectors corresponding to each category, and determining a clustering result of the to-be-clustered corpora. The invention further discloses an electronic device and a computer storage medium. By utilizing the method and the device, the text clustering accuracy and efficiency can be improved.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a text clustering method, an electronic device and a computer-readable storage medium. Background technique [0002] With the popularization of artificial intelligence in daily life applications, the development of natural language processing is becoming more and more important. Since most of the corpus has no labels and the high cost of labeling, unsupervised clustering of text is particularly important. [0003] However, for texts within the category of professional domain corpus, the existing techniques are not effective in clustering such texts. Taking insurance common questions as an example, users’ inquiries about insurance belong to the category of insurance-specific corpus, which has the characteristics of small data size, diverse expression methods, professional difficulty in interpreting, and noisy data (for example, advertisements). For this type of text , the ex...

AI Technical Summary

Problems solved by technology

[0004] 1) In the current mainstream method of generating sentence vectors, using the existing word vector summation and averaging will ignore the context information of the text, while models such as sent2vec and doc2vec require large-scale, high-quality training corpus, and user insurance consulting The corpus does not meet the conditions;

[0005] 2) For clustering algorithms, due to the complexity and diversity of insurance problems, it is difficult to define the number of categories, and it is impossible to accurately select parameters;

[0006] In view of the above defects, the efficiency and accuracy of text clustering are greatly reduced

Images