Web crawler crawling rule replacement method, scheduling end and crawling end

Abstract

The invention discloses a crawler capturing rule replacement method, a scheduling end and a capturing end. The crawler capturing rule replacement method includes the steps of sending a capturing task to the capturing end used for capturing network information, and sending a file with a rule to be switched and a website with the rule to be switched to the capturing end, wherein the capturing task includes a website to be captured and the scheduling end version number of a scheduling end capturing rule file corresponding to the website to be captured, and the file with the rule to be switched is used for enabling the capturing end capturing rule file which is stored at the capturing end and corresponds to the website with the rule to be switched to be replaced by the file with the rule to be switched through the capturing end. According to the crawler capturing rule replacement method, the scheduling end and the capturing end, the capturing rules are independently stored into rule files through the capturing end, and when replacement is conducted, only the rule files need to be replaced and the whole capturing end does not need to be restarted. Meanwhile, all the rule files are managed and stored in a unified mode through the scheduling end, therefore, the rule files do not need to be independently uploaded for each capturing end, and the management disorder is avoided.

Description

technical field [0001] The invention relates to the technical field related to webpage crawlers, in particular to a webpage crawler crawling rule replacement method, a scheduling terminal and a crawling terminal. Background technique [0002] Web crawlers are the basis for obtaining website information. In order to obtain website information, corresponding rules need to be configured for different websites. However, the page style rules of the website are not static. Once the page style rules change, the original crawling rules will inevitably become invalid. . [0003] The existing practice is to reconfigure the rules for the page after the revision, and then need to restart the crawling node to deal with the page revision. This method is feasible for the crawling of a single website and the revision of the website is infrequent. When there are a large number of websites, it will also cause frequent restarts of crawling nodes. The disadvantages are very obvious: [0004]...

AI Technical Summary

Problems solved by technology

[0007] Based on this, it is necessary to address the large-scale crawler system in the prior art, the replacement method of grabbing rules is likely to cause frequent restarts of grabbing nodes, and the management of modifying grabbing rules for grabbing nodes is likely to lead to technical problems of confusion. Web crawler crawling rule replacement method, scheduling end and grabbing end

Images