Data acquisition quantity monitoring method and system, electronic equipment and storage medium

Abstract

The invention discloses a data acquisition quantity monitoring method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring an acquisition data source comprising a plurality of pieces of acquisition data; respectively obtaining field information according to each piece of collection data, wherein the field information comprises a label value, a configuration cid and a detail page data volume; respectively judging whether the label value is equal to a preset value; if not, filtering the collected data; if so, storing the field information into a MySQL database to obtain a valid data set; traversing all configuration cid in the effective data set, and querying a corresponding channel URL data volume according to each configuration cid; adding the data volumes corresponding to the configuration cid to obtain the total data volume of the channel URL; judging whether the total data volume is greater than a preset URL number or not; if the total data volume is not greater than the preset URL number, performing data monitoring according to the detail page data volume of the configured cid; and if the total data volume is larger than the preset URL number, performing data monitoring according to the channel URL data volume configured with the cid, and achieving automatic monitoring of the data volume.

Description

technical field [0001] The invention relates to the field of Internet data processing, in particular to a monitoring method, system, electronic equipment and storage medium for data collection. Background technique [0002] With the rapid development of the Internet, it has become more and more common to collect data on the Internet through crawlers. In the data collection process of the crawler script, there is an abnormality in the collection, so it is necessary to monitor the amount of data collected by the crawler script to ensure that the collection is normal. [0003] In the prior art, the amount of collected data is usually monitored based on artificially set methods and rules. The accuracy of the detection rule algorithm is low. To determine different detection values ​​according to the real-time situation of each website, it can only be maintained manually, or Can not be fully intelligent monitoring. Contents of the invention [0004] In view of this, the embodi...

AI Technical Summary

Problems solved by technology

[0004] In view of this, the embodiment of the present invention provides a monitoring method, system, electronic equipment and storage medium of the amount of data collection, so as to solve the shortcoming that the amount of data collection cannot be automatically monitored in the prior art

Images