Short text similarity calculating method

Abstract

The invention discloses a short text similarity calculating method. The calculating method comprises steps of acquiring corpus data, pre-processing the corpus data to achieve training corpus, achieving a key word extracting model based on the training corpus, segmenting the training corpus via a segmenting tool, and acquiring a word vector set via training of word2vec, acquiring a text input by a user and questions correctly answered by candidate answers to achieve a segmenting result and a key word extracting result, calculating the questions correctly answered by candidate answers and the word vector set of the text input by the user based on the segmenting result and the key word extracting result, acquiring a sentence vector based on the word vector set, calculating similarity of two sentence vectors, and conducting similarity correction for information contained in the user input text and the questions correctly answered by the candidate answers to achieve corrected similarity. Cosine similarity of sentence vectors of the user input text and the question sentences correctly answered by candidate answers is calculated and similarity is corrected via sentence patterns, named entity and pronouns.

Description

technical field [0001] The invention relates to the field of Internet technology, in particular to the field of intelligent man-machine dialogue. Background technique [0002] With the continuous evolution of the informatization of human society and the rising cost of human services, people increasingly hope to communicate with computers through natural language, and the human-computer intelligent chat system has become a product born under such a historical background. [0003] In the existing human-computer dialogue system, there are mainly two implementation methods, one is the retrieval model, and the other is the generation model. The retrieval model regards a round of human-computer dialogue as an information retrieval process, prepares a certain amount of data in advance, and builds an index for the questions in the question-answer pair (which consists of a question and several answers). When the user enters one or several sentences, it is regarded as a retrieval, an...

AI Technical Summary

Problems solved by technology

However, the method of using the TF*IDF of the vector to calculate the cosine similarity to find the most similar problem only considers the text similarity between sentences, that is, how many repeated word segmentations there are literally to judge the similarity between sentences, which is obviously not enough , such as "I'm sleepy" has the same semantics as "I want to sleep", but there are almost no repeated words literally, this method cannot deal with this situation

In addition, since man-machine dialogue systems usually use short sentences, TF is basically 1, which has little effect, which will also affect the effect of the method

[0005] Therefore, the defect in the prior art is that the cosine similarity between the user input and the word vector of the question of the candidate question-answer pair is calculated through the TF*IDF value of the word segmentation, and only the text similarity between sentences is considered, that is, it can only be used literally How many repeated word segmentations are used to judge the similarity of sentences, which will make the judgment of similarity very inaccurate, which directly leads to inaccurate information for replying users in the man-machine dialogue system

Images