Text similarity calculation method and device and electronic device

Abstract

The embodiment of the invention discloses a text similarity calculation method and device and an electronic device. The method according to the embodiment of the invention comprises the following steps: obtaining an original text and a target text; calculating an editing distance between the original text and the target text; determining the longest common substring of the original text and the target text, and obtaining a starting position of the longest common substring in the original text; calculating text similarity between the original text and the target text based on the starting position of the longest common substring in the original text. The embodiment of the invention combines the editing distance of the original text and the target text and the longest common substring to calculate the text similarity, the calculated text similarity is closer to the reality, and the accuracy of the text similarity calculation is improved.

Description

technical field [0001] The invention relates to the technical field of video playback, in particular to a text similarity calculation method, device and electronic equipment. Background technique [0002] Similarity calculation is used to measure the degree of similarity between objects. In the field of natural language processing technology, similarity calculation is a basic operation, which is widely used in technical scenarios such as data mining, data classification, information retrieval and information collection. Text similarity calculation is a type of similarity calculation often involved in the field of natural language processing technology. By calculating the similarity between different texts, it is possible to perform cluster analysis, text matching or deduplication processing on large-scale text corpora. [0003] The text similarity calculation methods in the prior art mainly include cosine similarity, edit distance and neural network language model-based sim...

AI Technical Summary

Problems solved by technology

However, the existing text similarity methods, whether constructing feature vectors or calculating edit distance, only calculate the similarity of text in character content, and do not or rarely take into account the impact of word order on grammar and semantics, for example, When calculating the bullet chat similarity, the traditional Levenshtein algorithm (a kind of edit distance algorithm) is used. This method is mainly based on the edit distance from the source string to the target string. This method does not consider the common substring between them. String factor, unable to handle reversed string case

This has led to the weak processing ability of the text similarity calculation method in the prior art to the Chinese language that depends on word order, and the accuracy of calculating the similarity of Chinese text is not ideal enough.

Images