Data acquisition method and device and storage medium

Abstract

The invention provides a data acquisition method and device and a storage medium. The scheduling task is sent to the master control node through the scheduling node; and the master control node receives the operation state information sent by each data acquisition node, and allocates a scheduling task to the data acquisition nodes according to a preset strategy, the pre-acquired processing capability information of each data acquisition node and the operation state information of each data acquisition node, so that the data acquisition nodes execute the scheduling task. All the data acquisition nodes are managed in a unified mode through the master control node, loads are balanced for all the data acquisition nodes, the response capacity of data acquisition and the utilization rate of machine resources are improved, the data acquisition nodes can be distributed in different machine rooms, the advantages of bandwidth and multiple IP addresses of the multiple machine rooms are fully utilized, and dynamic expansion and contraction of the nodes are supported.

Description

technical field [0001] The present invention relates to the field of communication technology, in particular to a data collection method, device and storage medium. Background technique [0002] The most important resources that data collection needs to rely on are bandwidth, IP address, processor and memory. In the case of relatively cheap hardware resources, processor and memory will not become the bottleneck that limits the scale of the collection system. The real bottleneck is often bandwidth and IP address. . Large-scale downloading of web pages and other network content requires the support of sufficient network bandwidth. Websites usually limit the number of times a certain IP address can be accessed per unit time. Therefore, it is necessary to have enough IP addresses for large-scale and high-time collection of websites. support. [0003] The current open source distributed collection framework needs to deploy multiple collection nodes in the same computer room bas...

AI Technical Summary

Problems solved by technology

[0003] The current open source distributed collection framework needs to deploy multiple collection nodes in the same computer room based on the requirements of reducing public network bandwidth and transmission efficiency. Based on this deployment requirement, users with multiple computer rooms need to Unable to conveniently utilize the bandwidth of multiple computer rooms and the advantages of multiple IP addresses

At the same time, if multiple acquisition programs are deployed on one machine, it is impossible to control the consumption of system resources by each acquisition program, and only one acquisition program is deployed on one machine, which will cause machine resource utilization Problems of imbalance and waste of resources

Images