Construction method for anti-mimic death crawler system

Abstract

The invention discloses a method for establishing an anti-halt creeper system. The method comprises the following steps: (1) detecting and processing requested web pages; (2) detecting and processing network response; (3) detecting and processing memory space; and (4) repeatedly executing the step (1), the step (2) and the step (3) until all the hyperlinks of the web pages are processed. The method can effectively prevent the generation of the halt state of the creeper system, obviously reduce the waiting time of the creeper system and improve the creeping efficiency of the creeper system, provide a general framework for the establishment of the creeper system with robustness, and effectively reduce the development cost of the system.

Description

technical field

[0001] The invention relates to a construction method of a network data collection system, in particular to a construction method of an anti-fake crawler system. Background technique

[0002] Human beings have entered the information age, and the information explosion, more and more overwhelming information makes people breathless. In this situation, in order to extract useful information quickly and improve the efficiency of work and study, search engines have been proposed and implemented. As the basis of search engines and the only source of data processed by search engines, the status and importance of crawler systems are gradually highlighted. Unlike other search engine components, crawlers are closely related to network and storage, which causes the external environment to have a profound impact on the robustness of crawlers. The current general search engine crawler system has poor robustness and cannot adapt to the diversity of the network environme...

Images