Semantic similarity analysis method based on text clustering

Abstract

The invention discloses a semantic similarity analysis method based on text clustering. The method comprises the following steps: taking unprocessed text data as input; performing word frequency statistics on texts subjected to data preprocessing, adding word frequency statistics information serving as priori knowledge into text clustering, proposing a posteriori judgment criterion, and performingan unsupervised clustering method on the basis of taking the word frequency statistics as a classifier to improve the accuracy and timeliness of a text clustering result; carrying out synonym ambiguity elimination on the processed text, and carrying out semantic role labeling; and generating a semantic vector fused with the context features, processing the text sequence by adopting two LSTMs withcompletely same structures and parameters, adding the product and variance of the results, amplifying the same points and differences of the texts, and calculating to obtain a final result of similarity analysis. The method can be applied to actual scenes of text similarity analysis in various different fields, and text data of different types can be well processed.

Description

Technical field [0001] The invention belongs to the field of natural language processing, and relates to a semantic similarity analysis method based on text clustering. Background technique [0002] Text clustering and semantic similarity detection have always been an important research topic in the field of natural language processing. It can automatically and accurately determine the text category, semantic extraction and similarity comparison in text data. Handling and application are crucial. In recent years, the maturity of the research field has always been accompanied by a sharp increase in the number of reports and scientific research results. It is important to summarize and provide an overview. The similarity analysis method based on text clustering has received more and more attention. This method can It is divided into two stages of text clustering and text similarity analysis. The current methods basically pay more attention to word frequency information, while igno...

AI Technical Summary

Problems solved by technology

[0003] The problem of efficient similarity analysis and detection of text in specific fields has been basically solved, but it is difficult to quickly and accurately obtain similarity in a multi-field and large text database. Problems such as high dimensionality of word vectors, sparse data, ignorance of low-frequency words, and lack of semantic information, etc., and professional vocabulary in the text and synonyms that may cause ambiguity in different fields will also affect the similarity analysis results

Although the semantic similarity analysis method based on deep learning can reduce this impact and improve the accuracy, it has the problem of too long detection time. Therefore, how to perform text similarity analysis as quickly and efficiently as possible in a wide range of fields has become an issue. a burning problem

Images