Text repeatability judgment method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a text repeatability judgment method and device, electronic equipment and a storage medium, and the method comprises the steps: recognizing the length of a target text; if the length of the target text is greater than a first preset value, segmenting the target text into a plurality of text segments of which the lengths are not greater than the first preset value; respectively utilizing a signature simhash algorithm to obtain a signature value of each text segment in the plurality of text segments; judging whether the target text and the historical text are repeated or not based on the signature values of the plurality of text segments and inverted index information of the historical text in a distributed full-text search engine ElasticSearch; wherein inverted index information of at least one historical text is stored in the ElasticSearch, and the inverted index information of the historical text comprises a signature value of the historical text and a signature value of each text segment into which the historical text is segmented when the length of the inverted index information of the historical text is greater than a first preset value. According to the embodiment of the invention, the method achieves the repeated judgment of the text contents from the sentence dimension, and can effectively recognize a problem that a part of contents in two texts are repeated.

Description

technical field [0001] The present disclosure relates to Internet technologies, in particular to a method and device for determining text repetition, electronic equipment and storage media. Background technique [0002] Text similarity calculation has a wide range of applications in information retrieval, data mining, machine translation, document duplication detection and other fields. For example, when used to control public opinion on a microblog website, the sentences that need to be restricted can be entered into the database in advance, then when a user posts a microblog, the content of the user's income can be compared with the sentences in the database first, and if it matches the database The sentence in is not allowed to be issued by the user. [0003] In the process of realizing the present disclosure, the inventors found that the existing text similarity algorithm cannot well solve the phenomenon of text judgment and partial repetition. For example, for the com...

AI Technical Summary

Problems solved by technology

Because the shingle vector space of the k-shingle algorithm is huge, especially when the value of k is large, the shingle vector space is larger, and the storage resources and computing resources consumed are high; in addition, the comparison granularity of the k-shingle algorithm is finer, Due to the limited number of words, the repetition of words in two texts does not mean that the sentences are repeated. Based on the k-shingle algorithm, as long as there are word repetitions between the two texts, it will be judged as two texts. It is easy to misjudgment, resulting in The repetition judgment of the text is not accurate

Images