Sentence similarity calculation method and system, electronic equipment and storage medium

Abstract

The invention provides a sentence similarity calculation method and system, electronic equipment and a storage medium. According to the technical scheme, the method comprises the steps: carrying out word segmentationon a first sentence and a second sentence with the similarity to be calculated, and training a word vector model; traversing all vocabularies in the first sentence and the second sentence, calculating word vectors of the vocabularies through the word vector model, and further calculating the similarity between the word vectors of all the vocabularies in the first sentence and the word vectors of all the vocabularies in the second sentence; determining a similarity value of each vocabulary in the first sentence relative to the second sentence based on a preset rule; averaging the similarity values of all vocabularies in the first sentence relative to the second sentence, and judging whether the first sentence and the second sentence are similar sentences or not according to the average value and based on a preset threshold value. According to the method and system, the problems that the generalization ability is limited, the data mining depth is not enough, the calculation amount is large and the response time in an online environment is long when the sentence similarity is calculated in an existing method are solved.

Description

technical field [0001] The invention belongs to the technical field of natural language processing, and in particular relates to a sentence similarity calculation method, system, electronic equipment and storage medium. Background technique [0002] In the field of natural language processing technology, especially in the construction of intelligent question answering systems, the calculation of sentence similarity is an important issue. The existing calculation methods of sentence similarity mainly include methods based on statistics and methods based on deep learning. Statistics-based methods mainly include Levenstein distance (edit distance), BM25 algorithm, TFIDF calculation method, and TextRank algorithm. Statistics-based methods have limited generalization ability and insufficient depth of data mining; deep learning-based methods mainly include: DSSM (Deep Structured Semantic Models) model, bert model, etc., the traditional deep learning-based models have a large amoun...

AI Technical Summary

Problems solved by technology

[0003] The embodiment of the present application provides a sentence similarity calculation method, system, electronic equipment and storage medium, to at least solve the problem of limited generalization ability, insufficient depth of data mining and large amount of calculation in existing methods when calculating sentence similarity , the problem of long response time in the online environment

Images