A text deduplication method and device

Abstract

The invention provides a text duplicate removal method and device. According to the technical scheme, feature word strings of case texts are sliced, and the signature values of all slices are calculated, and accordingly an incidence relation between the signature values and the case texts is established to form a case library; when duplicate removal processing needs to be conducted on the texts to be processed, the feature word strings of the texts to be processed are sliced, the signature values of all the slices are calculated, the case text corresponding to the slices are determined according to the signature values of all the slices, accordingly statistics is conducted on the number of the signature values corresponding to the same case text, the similarity between the texts to be processed and the corresponding case texts is calculated by the adoption of the number of the maximum signature values, and thus similarity judgment is conducted. According to the text duplicate removal method and device, the required calculated amount is less, and a smaller misjudgment rate can be guaranteed.

Description

Technical field [0001] This application relates to the technical field of text processing, and in particular to a method and device for text deduplication. Background technique [0002] The current text de-duplication methods mainly include the following: text hash, cosine similar text calculation, and simhash, which will be introduced below. [0003] 1) Text hash method: Calculate the hash value of the text content (for example) the Murmur hash value, compare whether the hash values ​​of the two texts are the same to determine whether it is the same text, the hash value is the same, the text is considered the same. [0004] The text hash method can quickly judge whether two texts are similar, but the judgment conditions are too strict, and the text content must be exactly the same, otherwise it may be caused by calculating different hash values. For example: "Let it evolve." and "Let it evolve!" are originally the same text, but because the final punctuation marks are different, th...

AI Technical Summary

Problems solved by technology

[0009] The advantage of the simhash algorithm is that it can greatly reduce the calculation workload in the case of massive texts. The disadvantage is that the implementation is more complicated, and the calculation of the Hamming distance is relatively large.

[0010] It can be seen that among the above three methods, the first method has a higher misjudgment rate, and the calculation amount of the latter two methods is too large, and both the misjudgment rate and the calculation amount cannot be considered

Images