Large-scale document similarity detection method

Abstract

The invention provides a large-scale document similarity detection method. The method comprises the steps of S1, calculating the similarity of other information of documents in a document set; S2, enabling each document content to correspond to a signature S and a f-dimensional vector V; S3, performing word segmentation processing on the document content; S4, comprehensively calculating a weight of a feature word x; S5, mapping the feature word x into a signature h by using a hash function, traversing all bits of the h, and adjusting the V; S6, traversing the V, adjusting the signature S, andfinally generating a signature value, corresponding to the document content, of the signature S; S7, dividing the signature value corresponding to the document content into n blocks, mapping the blocks to a bucket by using the hash function, and judging whether double hash is performed or not; S8, taking the documents of the same bucket as a candidate pair, and calculating the similarity; and S9,judging whether the documents are similar documents or not. The method is high in detection accuracy and high in executive efficiency, and can be widely used in the internet large-scale data mining.

Description

technical field [0001] The invention relates to a detection method, in particular to a large-scale document similarity detection method, which belongs to the field of computer algorithms. Background technique [0002] With the advent of the era of big data, data-based information is growing rapidly, and data takes up more and more space. Such a large amount of data has brought about huge storage problems. The study found that the proportion of redundant data in the stored data is greater than 60%, and the proportion of redundancy will continue to increase in the future. Redundant data reduces the efficiency of user retrieval and query data, and a large amount of storage resources are wasted in storing redundant data, and users do not want to see a bunch of retrieval results with the same or similar content. On the other hand, data mining developers also face the problem of data duplication and redundancy when crawling data through the web. Therefore, document similarity de...

AI Technical Summary

Problems solved by technology

In essence, the Simhash algorithm is an improved hash algorithm designed to solve the deduplication of similar data, but after it is actually applied to document similarity detection, it is found that this algorithm still has a high accuracy rate. Not high enough, low efficiency problem

Images