Distributed spider system and periodical increment capture method

Abstract

The invention discloses a distributed spider system. The system is configured to comprise three parts, namely a distributed service based on ZooKeeper, a system component and a database, wherein the system component comprises a system monitoring component Monitor, a coordination component Coordinator, a log collection component Logger and a basic spider component Spider, the database comprises a Redis memory database, redis is a key-value storage form, and a distributed URL task queue and a distributed BloomFilter are stored in the Redis memory database. The invention furthermore discloses a periodical increment capture method based on the system. The method comprises the steps that the coordination component Coordinator periodically imports tasks to the distributed URL task queue and awakens the Spider component in dormancy; and the Spider component performs dormancy or periodical increment capture according to execution of the current distributed URL task queue. Through the system and the method, stand-alone spiders are effectively combined, distributed spiders with high availability, high stability and a high throughput rate in a cluster environment are obtained, and periodical increment capture is realized.

Description

technical field [0001] The invention relates to the technical field of high-efficiency data collection of Internet big data, in particular to a distributed crawler system and a periodic incremental crawling method. Background technique [0002] The web crawler starts from the URL (Uniform Resource Locator, Uniform Resource Locator) of one or several initial webpages, and obtains the URL on the initial webpage. Extract new URLs and put them into the task queue until the stop condition of the system is met. [0003] With the rapid development of the Internet, network data has shown explosive growth, and network data sources are becoming more and more diversified. In the face of such a large and diverse Internet data, how to improve the crawling efficiency of web crawlers and how to implement customizable crawling strategies for different data sources is very important. [0004] Compared with the traditional stand-alone crawler, the distributed crawler can significantly impro...

AI Technical Summary

Problems solved by technology

[0004] Compared with the traditional stand-alone crawler, the distributed crawler can significantly improve the crawling efficiency of the crawler, but it also introduces new problems: multi-node task distribution in a distributed environment, load balancing problems, web page repetition problems and Periodic incremental crawling issues, etc.

Images