Method and device for calculating similarity between texts, storage medium and electronic equipment

Abstract

The invention relates to an inter-text similarity calculation method and device, a storage medium and electronic equipment. The method comprises the steps that word segmentation and stop word filtering processing is conducted on a first text and a second text with the similarity to be calculated, and a first word segmentation set which does not contain repeated word segmentation and corresponds tothe first text and a second word segmentation set which corresponds to the second text are obtained according to the processing result; Determining the semantic information transfer cost between thefirst text and the second text according to the information amount of each segmented word in the first segmented word set and the second segmented word set in the text and the word embedding vector corresponding to each segmented word; And determining the similarity between the first text and the second text according to the semantic information transfer cost. Therefore, the semantic influence ofeach word in the text and the context of each word on the text is fully considered, and the calculation basis of the similarity is closer to the semantics of the text, so that the calculated similarity is more accurate.

Description

technical field [0001] The present disclosure relates to the field of computer technology, in particular to a method, device, storage medium and electronic equipment for calculating similarity between texts. Background technique [0002] In the prior art, when calculating the similarity between texts, a structured processing method is generally adopted. Firstly, the text is structured, for example, two pieces of text are processed into vectors, such as word-based one-hot representation, word-based one-hot representation, word embedding and accumulation-based vectorized representation, etc. Afterwards, similarity calculations are performed on the results of text structured processing, such as calculating the Euclidean distance between text vectors, the cosine angle between vectors, etc. However, the semantics of a word in a text will be affected by the context of the word’s position in the text, thus affecting the semantic conversion between words between texts, and the abov...

AI Technical Summary

Problems solved by technology

However, the semantics of a word in a text will be affected by the context of the word’s position in the text, thus affecting the semantic conversion between words between texts, and the above-mentioned similarity calculation method focuses on the w...

Images