Docker-based data acquisition method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of big data, and relates to a Docker-based data acquisition method and device, computer equipment and a storage medium. The method comprises the steps of obtaining a data acquisition task, and sending a mirror image of a crawling program container to at least one cloud server according to the data acquisition task; generating at least one crawling program container in each cloud server according to the mirror image of the crawling program container, wherein a crawling program runs in the crawling program container to become a crawling node; and sending the data acquisition task to the crawling node, and executing data acquisition operation on the data acquisition task through the crawling node. According to the scheme provided by the invention,a Docker technology is adopted to send the crawling program container mirror image to at least one cloud server, a plurality of crawling nodes can be automatically deployed in the same cloud server, the occupied system resources are few, the cloud server resources can be effectively utilized, the crawling nodes can be increased as required, the isolation among the crawling nodes is high, and the system stability is good.

Description

technical field [0001] Embodiments of the present invention belong to the field of big data technology, and in particular relate to a Docker-based data collection method, device, computer equipment, and storage medium. Background technique [0002] In the era of big data, in data-based systems, it is often necessary to collect a large amount of raw data. Part of these raw data comes from the Internet. For this part of the Internet, the existing data collection process is generally through the cloud server In the face of large-scale collection requirements, the existing data collection generally adopts the method of horizontal enhancement, that is, increasing the number of cloud servers to achieve the purpose of large-scale collection by increasing the number of crawling nodes, or through crawling Multi-threaded scheduling is used on the nodes to run the crawler program concurrently to achieve the purpose of large-scale collection. [0003] However, for the method of horizon...

AI Technical Summary

Problems solved by technology

[0004] In view of this, the embodiment of the present invention provides a Docker-based data collection method to solve the problem of wasting cloud server resources by adding crawling nodes in a horizontally enhanced manner, and it is easy to be limited in scale due to limited cloud server resources, which cannot be effectively implemented The problem of increasing crawling nodes on demand, and solving the problem of poor isolation of multi-threaded crawler threads when multi-threaded scheduling is used on crawler nodes for data collection, prone to blocking and poor system stability

Images