Distributed network crawler URL (uniform resource locator) duplicate removal system and method

Abstract

The invention provides a distributed network crawler URL (uniform resource locator) duplicate removal system and method. The system comprises crawler collection sub-nodes, a central server and a database server. The method comprises the following steps that: each crawler collection sub-node registers to the central server; the crawler collection sub-node obtains a URL from a database waiting queue and obtains new URL information from the URL; the crawler collection sub-node performs first-level duplicate removal on the newly obtained URL; if the first-level duplicate removal fails, the URL is discarded; if the first-level duplicate removal succeeds, the newly obtained URL is added to a local URL abstract table and sent to the central server; the central server performs second-level duplicate removal on the newly obtained URL; if the second-level duplicate removal succeeds, the URL is added to a global URL abstract table; and the crawler collection sub-node adds the link of the URL to the waiting queue. According to the system and method provided by the invention, the duplicate removal task concentrated on the central node can be decomposed to the crawler collection sub-nodes through the first-level duplicate removal by a hierarchical duplicate removal mechanism, and the central server maintains a global duplicate removal table through a second-level duplicate removal mode, thus the system expansion is remarkably facilitated, and the system design, deployment and operation are very flexible and convenient.

Description

technical field [0001] The invention relates to the field of the Internet, in particular to a system and method for URL deduplication of a distributed web crawler based on a distributed architecture. Background technique [0002] At present, distributed web crawlers can be divided into three types: master-slave mode, autonomous mode and hybrid mode according to different communication methods. The master-slave mode means that one host is used as the control node to manage all the hosts running web crawlers. The crawler only needs to receive tasks from the control node and submit new tasks to the control node. In this process, there is no need to Communicating with other crawlers is simple and easy to manage. The control node needs to communicate with all crawlers, and it needs an address list to save the information of all crawlers in the system. When the number of crawlers in the system changes, the coordinator needs to update the data in the address list, and this proces...

AI Technical Summary

Problems solved by technology

The deduplication method based on memory Hash uses the Hash table in memory to perform deduplication, but this method consumes too much memory resources

The deduplication method based on the disk path is to encode the link in MD5 and create the corresponding path in the disk. This path-based deduplication method has high time efficiency, but it is not suitable for a large-scale web crawling Distributed crawler system

The last weight-removal method based on the Bloom filter is a more commonly used weight-removal scheme. The advantage is that it is very fast and saves space, but there will be a certain rate of misrecognition. It is also a memory-based weight-removal method.

[0004] Generally, distributed web crawlers will choose a relatively stable database deduplication method, but the deduplication efficiency is too low, and it is not suitable for large amounts of data.

Images