Similar text retrieval method and device

Abstract

The invention relates to the field of computer retrieval, and discloses a similar text retrieval method and device. According to the similar text retrieval method and device, N indexes are preset in to-be-indexed texts, and respectively correspond to N different index algorithms, each index comprises a plurality of index values, and the index values correspond to at least one to-be-indexed text. The method comprises the following steps: respectively calculating the N index values, corresponding to the N indexes, of a text to be retrieved by using the N different index algorithms; respectively retrieving the to-be-indexed texts, with the same index value, as candidate similar documents in the corresponding indexes by using the N index values; conducting similarity match on the text to be retrieved and all candidate similar documents one by one to obtain the proximate text. According to the similar text retrieval method and device, the index values generated by a plurality of index algorithms are used to construct a plurality of indexes, the probability that the candidate similar documents obtained through the plurality of indexes contain the test similar to the text to be retrieved is greatly increased, the recall rate on retrieval of a plurality of short texts is relatively high, and missed retrieval can be prevented.

Description

technical field [0001] The invention relates to the field of computer retrieval, in particular to a retrieval method and device for similar texts. Background technique [0002] Text similarity refers to a measure of the degree of matching between two or more texts. The closer the semantics or word frequency between the texts, the higher the text similarity between them. Traditional text matching, such as edit distance calculation and vector space model (VSM), can measure the similarity between two texts very well, but for massive texts, it exposes the shortcomings of slow comparison speed and high space requirements. [0003] At present, fingerprint algorithms such as simhash are generally used in the industry to convert the text similarity problem into the edit distance calculation of simhash codes, thereby solving the indexing problem of massive texts. This method has a significant effect on the similarity calculation of large texts, but for medium Long or short text (wit...

AI Technical Summary

Problems solved by technology

Traditional text matching, such as edit distance calculation and vector space model (VSM), can measure the similarity between two texts very well, but for massive texts, it exposes the shortcomings of slow comparison speed and high space requirements

[0003] At present, fingerprint algorithms such as simhash are generally used in the industry to convert the text similarity problem into the edit distance calculation of simhash codes, thereby solving the indexing problem of massive texts. This method has a significant effect on the similarity calculation of large texts, but for medium Long or short text (within 100 characters), it will have a very high deviation (although the two texts have a high similarity, the edit distance of simhash is very large), which cannot be applied to scenarios that require high recall

Images