Unsupervised text similarity calculation method

Abstract

The invention relates to an unsupervised text similarity calculation method, which comprises the following steps of: 1, pre-training an embedded layer model, and pre-training all words in a problem set to generate word vectors meeting the requirements of the model; 2, mining semantic information of sentences through a coding layer network; step 3, performing model improvement based on TFIDF fusion; the method comprises the steps that when each question sentence is input into a neural network, TFIDF calculation is conducted on each input question sentence, calculated weights are input into theneural network, final sentence vector representation is controlled, a normalized TFIDF calculation method is adopted, and the final sentence vector representation is fused into a coding layer and a representation layer. According to the method, the deep neural network model (Bi-LSTM) is used for unsupervised training of the corpus to obtain the language model, and the information of the large-scale corpus can be fully utilized in an unsupervised training mode, so that the text matching accuracy is improved, and the information retrieval precision is improved.

Description

technical field [0001] The invention relates to a communication method, in particular to an unsupervised text similarity calculation method. Background technique [0002] With the advent of the era of big data and the explosive growth of information, information retrieval and matching are playing an increasingly important role in various fields. And one of the key technologies is the text similarity calculation technology. Traditional text similarity calculation methods are mainly string-based methods and corpus-based methods. The string-based method is to compare texts at the literal level, and the co-occurrence and repetition degree of strings are used as the measure of similarity; the corpus-based method is to use the information obtained from the corpus to calculate the text similarity. The method based on the corpus is mainly based on the neural network method, and the training classifier is obtained through the algorithm of supervised learning, and the similarity sco...

AI Technical Summary

Problems solved by technology

[0003] The string-based method considers characters or words as independent knowledge units, and does not consider the meaning of the words themselves and the relationship between words, so this method cannot accurately calculate the

Although the method based on the supervised neural network can make good use of semantic information, the quality of the training classifier depends largely on the accuracy of the training samples, and the construction of label data is a time-consuming and labor-intensive work. Supervision methods are becoming more and more difficult to meet the requirements of rapid information growth

Images