Text similarity computing method and system based on improved LDA topic model

Abstract

The invention relates to a text similarity computing method and system based on an improved LDA topic model. The method comprises the following steps: acquiring a plurality of text sets in a WMF_LDA topic model; performing similarity computing on words in a preprocessed word set by virtue of a word2vec word vector model and generating a plurality of word similarity values; generating a field topicword set according to similarity between the words; obtaining a probability distribution of a document subjected to word semantic combination on different topics by virtue of the LDA topic model; anddetermining topic distribution similarity between any two texts, thus the text similarity is obtained. The method provided by the invention has the advantages that firstly screening is performed, sothat quantity of the words in the topic word set is reduced; and unified mapping is performed on synonyms and words in the same field, and then probability distributions of the texts are obtained through modeling, so that computation dimensionality is low in a process of computing the similarity of two texts, wasted space is reduced, and the problems that a word level is focused on excessively anddifferent types of texts can not be fully excavated and utilized are solved.

Description

technical field [0001] The invention relates to the technical field of language processing, in particular to a text similarity calculation method based on an improved LDA topic model. Background technique [0002] Text similarity research is an important research topic in natural language processing. The traditional VSM method uses TF-IDF as a feature to construct a vector, and uses cosine distance to calculate the similarity of text, but this method simply uses word frequency as a feature, without considering the semantic features of words and text. [0003] Text similarity is an important topic that has been widely studied in the fields of linguistics, psychology and information theory. At present, there are various methods for calculating text similarity, and many achievements have been made. The traditional calculation method of text similarity is characterized by word frequency, and the text is expressed as a vector. The dimension of the vector is the number of all wo...

AI Technical Summary

Problems solved by technology

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the existence of large dimensions, serious waste of space, too much concentration on the word level, and failure to fully tap and utilize the inherent usefulness between different types of texts in the existing text similarity calculation method. word difference problem

Images