Distributed network crawler system and catching method thereof

Abstract

The invention discloses a distributed network crawler system and a catching method of the distributed network crawler system. The distributed network crawler system and the catching method of the distributed network crawler system comprise a parameterized control catching module for conducting parameterized allocation to resources to be caught according to catching contents, catching types and user agent in use; a vertical channel catch custom module for managing and appointing catching behaviors of each vertical channel and have the function of statistics; a unitary catch scheduling module for merging catch requests of all the vertical channels and scheduling and catching in a unified mode according to polite control and pressure condition to sites of the opposite side; general storage and calculation module for using the caught general storage request and calculation request of different vertical channels through arrangement; a catch result distributing module for sending results to the appointed place according to appointment of the catch behaviors of each vertical channel catch custom module. The distributed network crawler system and the catching method of the distributed network crawler system can achieve multi-channel catch, and effectively use the present whole broadband and other resources to serve various sub-channels inside a search engine better.

Description

technical field [0001] The invention relates to search engine technology, in particular to a distributed web crawler system and a grabbing method thereof, which can be used in a search engine web crawler module. Background technique [0002] Many current search engines cover a variety of vertical search channels, and more than 90% of the data sources of each vertical channel must be actively crawled by web crawlers. [0003] Since different vertical channel products have different types of captured data and different sizes of resource content, and each vertical channel product prefers to handle its own capture tasks independently without being interfered by other businesses, this requires The crawling and processing tasks of each vertical channel are independent, but different vertical channels may have access requirements for the same site. Therefore, it will cause a contradiction between the overall bandwidth resources and multiple vertical channels within the search en...

AI Technical Summary

Problems solved by technology

[0003] Since different vertical channel products have different types of captured data and different sizes of resource content, and each vertical channel product prefers to handle its own capture tasks independently without being interfered by other businesses, this requires The crawling and processing tasks of each vertical channel are independent, but different vertical channels may have access requirements for the same site

Therefore, it will cause a contradiction between the overall bandwidth resources and multiple vertical channels within the search engine

Images