Crawler template change monitoring method and system, electronic equipment and storage medium

Abstract

The invention discloses a crawler template change monitoring method and system, electronic equipment and a storage medium. The method comprises the steps: searching for a crawler script without data in a crawler script library; storing the configuration ids corresponding to all the searched data-free crawler scripts into a data-free crawler script database; adding the configuration id in the data-crawler-free script database into a detection queue for data-free detection; obtaining code information of a crawler script corresponding to the configuration id according to the configuration id; obtaining a URL set in the crawler script according to the code information, and traversing and downloading each URL in the URL set to obtain a downloading result value of each URL; and determining whether the crawler template is changed or not according to whether the URL downloading result value is empty or not, whether the URL request response state code is equal to a first preset state code or not, whether the number of URL detail links is greater than zero or not, and whether a callback return value of a callback function located in the three-layer template has a value or not. According to the method, automatic crawler template change monitoring is achieved by monitoring the plurality of return values.

Description

technical field [0001] The invention relates to the technical field of data monitoring, in particular to a monitoring method, system, electronic equipment and storage medium for reptile template modification. Background technique [0002] In the process of data crawling, abnormalities inevitably occur, resulting in inaccurate crawler data. Monitoring the quality of crawler data can detect problems in time and ensure data reliability. [0003] Usually, the monitoring of crawler data quality includes the following steps: first, analyze whether the spider file is running normally (that is, whether there is data); if there is data, receive the data source for detection, and compare the rules; if there is no data, set the method , rule operation framework for detection, the specific monitoring flow chart is as follows figure 1 shown. The setting methods and rules in the above steps need to be found manually based on experience, which is not intelligent enough. Contents of the...

AI Technical Summary

Problems solved by technology

[0004] In view of this, the embodiment of the present invention provides a monitoring method, system, electronic device and storage medium for crawler template change, so as to solve the problem that the crawler template change monitoring in the prior art is not intelligent enough

Images