A method for distributed collection of public page data

Abstract

The invention discloses a method for collecting public page data in a distributed manner, which comprises the following steps of: mirroring a crawler program, and packaging all programming environmentdependencies and software environment dependencies into a mirror image; distributing the mirror image to each machine node by using a weighted polling algorithm to ensure the overall load balance ofa crawler cluster, and managing resource allocation of a crawler task in a docker swarm command form to complete dynamic increase and decrease of cluster resources of the crawler task. The method hasthe advantages that problems of the resource preemption, incapability of dynamic resource allocation and weak cluster expandability of the distributed crawler in a cluster environment can be solved, auser only needs to set parameters, such as the starting number of the tasks in the cluster, the distribution of the tasks in the cluster nodes and blacklist machine nodes corresponding to the tasks,of each task , crawler codes and setting files are packaged and then uploaded to the cluster, and then automatic task distribution can be completed, the development cost is saved, and the problem thatcluster resource distribution is lack of cluster programming environment dependence and dependence conflicts is avoided.

Description

technical field [0001] The invention relates to office automation, relates to electrical digital data processing, and especially refers to a method for collecting public page data in a distributed manner. Background technique [0002] A web crawler (sometimes called a spider or spider bot, often shortened to crawler) is a web robot that systematically browses the World Wide Web, often for web indexing (web crawling). [0003] 1. Most of the existing crawler technologies use the http / https protocol to download data from the target website, and analyze and extract data after downloading. [0004] 2. Most distributed crawler architectures use the redis cluster as the distribution queue, import all tasks into the distribution queue, and use the list queue of the redis cluster to ensure the uniqueness of the tasks. [0005] 3. Start a specified number of crawler processes on each machine node. Each crawler process pulls urls from the distribution queue for crawling. Each crawler...

AI Technical Summary

Problems solved by technology

[0011] Aiming at the shortcomings of the prior art, the purpose of the present invention is to propose a distributed crawler process management method based on docker swarm to solve the problems of resource preemption, inability to dynamically allocate resources, and weak cluster scalability of distributed crawlers in a cluster environment. , the user does not need to deal with the resource scheduling problem of the machine nodes in the cluster, and does not need to pay attention to the cluster programming environment dependency problem, but only needs to set the parameters of each task, such as the number of tasks to start in the cluster, the distribution of tasks in the cluster nodes, and the task corresponding black List machine nodes, package the crawler code and configuration files and upload them to the cluster, then the task can be automatically distributed to save development costs and avoid cluster resource allocation problems due to lack of dependencies and dependency conflicts in the cluster programming environment

Images