Network page efficient and accurate deduplication system based on cloud computing

Abstract

The invention provides a network page efficient and accurate deduplication system based on cloud computing, and aims to solve the problems that most of web pages searched by an existing search engineare static web pages, due to the existence of a large amount of transshipment and plagiarism, the main content of a large number of web pages is repeated, and for the search engine, the repeated web pages virtually increase the burden of index storage, and meanwhile, more retrieval time can be consumed; the webpage deduplication system based on the Hadoop cloud platform is designed and realized bycombining an open source framework, other modules of a search engine can be better connected by adopting a mode of detecting and judging duplicate in real time after a spider program captures a webpage; and in a massive webpage collection stage, the network page efficient and accurate deduplication system based on cloud computing can preprocess the web pages in advance, then web page similarity detection and discovery are carried out, repeated web pages or web pages with high similarity are removed, and therefore index quality is improved, retrieval results are optimized, and good search experience is provided for users.

Description

technical field [0001] The invention relates to a system for accurately deduplicating webpages, in particular to an efficient and accurate deduplication system for webpages based on cloud computing, and belongs to the technical field of deduplication for webpages. Background technique [0002] With the rapid development of electronic communication and computer network technology, web sites have grown rapidly, and the number of web pages has reached hundreds of billions. It has become the biggest problem for users to find the information they care about from the massive information database. Search engines solve this problem very well. Search engines collect information from the Internet, organize and process the collected information, and then provide users with an easy-to-use retrieval system. Users only need to enter the content they care about through the retrieval system. Keywords can search for the desired information. [0003] Most of the webpages collected by existin...

AI Technical Summary

Problems solved by technology

[0013] The efficient and accurate deduplication system for web pages based on cloud computing provided by the present invention can not achieve high efficiency and accuracy for the similarity detection of massive web pages by the SimHash algorithm. Considering the limitation of time and space resources, the present invention is based on cloud The calculated webpage deduplication method has excellent time complexity and space complexity. Through the improvement and optimization of the cloud-based SimHash algorithm, the cost of calculating the Hamming distance is greatly reduced. The cloud-based webpage deduplication is practical, efficient, and easy to expand , Accurate and high-speed, solves the problem of rapid deduplication of massive web pages. It is a method of deduplication with significant innovation and outstanding advantages; design and implement an efficient and accurate deduplication system based on Hadoop for web page fingerprint comparison, Significantly less time spent and significantly more accurate results

Images