Docker-based data acquisition method and device, computer equipment and storage medium

A data acquisition and computer technology, applied in the field of big data, can solve the problems of poor system stability, limited scale, easy to block, etc., and achieve the effect of occupying less system resources, strong isolation, and strong versatility
CN110457555AInactive Publication Date: 2019-11-15PINGAN INT SMART CITY TECH CO LTD
5 Cites 3 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
PINGAN INT SMART CITY TECH CO LTD
Publication Date
2019-11-15
Estimated Expiration
Not applicable · inactive patent

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention belongs to the technical field of big data, and relates to a Docker-based data acquisition method and device, computer equipment and a storage medium. The method comprises the steps of obtaining a data acquisition task, and sending a mirror image of a crawling program container to at least one cloud server according to the data acquisition task; generating at least one crawling program container in each cloud server according to the mirror image of the crawling program container, wherein a crawling program runs in the crawling program container to become a crawling node; and sending the data acquisition task to the crawling node, and executing data acquisition operation on the data acquisition task through the crawling node. According to the scheme provided by the invention,a Docker technology is adopted to send the crawling program container mirror image to at least one cloud server, a plurality of crawling nodes can be automatically deployed in the same cloud server, the occupied system resources are few, the cloud server resources can be effectively utilized, the crawling nodes can be increased as required, the isolation among the crawling nodes is high, and the system stability is good.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] Embodiments of the present invention belong to the field of big data technology, and in particular relate to a Docker-based data collection method, device, computer equipment, and storage medium. Background technique

[0002] In the era of big data, in data-based systems, it is often necessary to collect a large amount of raw data. Part of these raw data comes from the Internet. For this part of the Internet, the existing data collection process is generally through the cloud server In the face of large-scale collection requirements, the existing data collection generally adopts the method of horizontal enhancement, that is, increasing the number of cloud servers to achieve the purpose of large-scale collection by increasing the number of crawling nodes, or through crawling Multi-threaded scheduling is used on the nodes to run the crawler program concurrently to achieve the purpose of large-scale collection.

[0003] However, for the method of horizon...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More