System and method for distributed retrieval and deduplication of massive documents

Abstract

A massive document distribution searching duplication removing system comprises a document pre-processing module, a document feature calculating module, a distribution database building module, a storage module, a distribution searching module and a similarity calculating module. The document feature calculating module calculates document feature vectors according to importance degree of a word to a document. The distribution database building module maps the document into different storage subregions according to the document feature vectors. The distribution searching module searches a plurality of subregions where a target document belongs, the similarity calculating module calculates the similarity of the target document and all documents in the plurality of subregions, and massive document distribution duplication removing operation is achieved. By means of the system and method, a distribution system idea is adopted, massive documents are scattered into a plurality of subsets, and duplication removing calculation is conducted in one or a few of subsets, similarity calculation amount is reduced, and document duplication removing efficiency is improved.

Description

technical field [0001] The invention belongs to the technical field of information processing, and in particular relates to a system and method for distributed retrieval and deduplication of massive documents in the era of big data. Background technique [0002] With the advent of the era of big data, all kinds of information are increasing exponentially, and all walks of life and fields are facing the pressure of massive information collection, processing, and storage. Therefore, it is necessary to overcome the development of the times to check for duplicate documents or similar documents from the source. technical difficulties. For example, search results with the same or similar content in the results returned by current search engines account for about 45%. Therefore, it is necessary to judge which web pages have the same or similar content when collecting search information. [0003] There are three types of web page deduplication technologies commonly used in the fiel...

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to provide a system and method for distributed retrieval and deduplication of massive documents, which uses the idea of ​​distributed system to disperse the massive document library into dozens or even more subsets, so that the deduplication can be performed in one subset or several In order to solve the problem of low efficiency and large amount of calculation in massive document deduplication technology in the era of big data

Images