Text duplicate removal method and device

Abstract

The invention provides a text duplicate removal method and device. According to the technical scheme, feature word strings of case texts are sliced, and the signature values of all slices are calculated, and accordingly an incidence relation between the signature values and the case texts is established to form a case library; when duplicate removal processing needs to be conducted on the texts to be processed, the feature word strings of the texts to be processed are sliced, the signature values of all the slices are calculated, the case text corresponding to the slices are determined according to the signature values of all the slices, accordingly statistics is conducted on the number of the signature values corresponding to the same case text, the similarity between the texts to be processed and the corresponding case texts is calculated by the adoption of the number of the maximum signature values, and thus similarity judgment is conducted. According to the text duplicate removal method and device, the required calculated amount is less, and a smaller misjudgment rate can be guaranteed.

Description

technical field [0001] The present application relates to the technical field of text processing, in particular to a text deduplication method and device. Background technique [0002] The current text deduplication methods mainly include the following types: text hash, cosine similar text calculation, simhash, which are introduced below. [0003] 1) Text hash method: Calculate the hash value (for example) Murmur hash value of the text content, compare whether the hash values ​​of the two texts are the same to determine whether they are the same text, and if the hash values ​​are consistent, the text is considered to be the same. [0004] The text hash method can quickly judge whether two texts are similar, but the judgment conditions are too strict, and the text content must be exactly the same, otherwise different hash values ​​may be calculated. For example: "Let it go." and "Let it go!" are the same text, but because the final punctuation marks are different, they are m...

AI Technical Summary

Problems solved by technology

[0009] The advantage of the simhash algorithm is that it can greatly reduce the calculation workload in the case of massive texts. The disadvantage is that the implementation is more complicated, and the calculation of the Hamming distance is relatively large.

[0010] It can be seen that among the above three methods, the first method has a higher misjudgment rate, and the calculation amount of the latter two methods is too large, and both the misjudgment rate and the calculation amount cannot be considered

Images