Unlock instant, AI-driven research and patent intelligence for your innovation.

Distributed crawler task distribution method and system

A crawler system and distributed technology, applied in the field of distributed crawler task assignment methods and systems, can solve problems such as low efficiency, and achieve the effect of improving efficiency and reasonable task assignment

Inactive Publication Date: 2018-02-09
MAXTRON TECHSHENZHEN CO LTD
View PDF7 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

It solves the shortcoming of low efficiency of technical solutions in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed crawler task distribution method and system
  • Distributed crawler task distribution method and system
  • Distributed crawler task distribution method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0028] Please refer to figure 1 , figure 1 It is a distributed crawler task assignment method proposed in the first preferred embodiment of the present invention, the method is as follows figure 1 shown, including the following steps:

[0029] Step S101, receiving or initiating an assignment message, the assignment message is used to assign a task manager from the distributed crawler system.

[0030] Step S102, the distributed device sends N data pa...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed crawler task distribution method. The method comprises the following steps: distributed equipment receives or initiates a distribution message, wherein the distribution message is used for distributing a task manager from a distributed crawler system; the distributed equipment sequentially sends N data packets to other M equipment of the distributed equipment;the distributed equipment counts the sum of M time delays of the N data packets returned by the M equipment, and solves an average value of the sum of M time delays, and selects the equipment with minimum sum of time delay from the average value of the sum of M+1 time delays as the task manager, wherein the task manager acquires crawler tasks and acquires distance to the equipment connected withthe task manager and the number of the crawler tasks; and the task manager distributes the crawler task according to the distance and the number of the crawler tasks. The technical scheme has the advantage of high efficiency.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a distributed crawler task assignment method and system. Background technique [0002] A web crawler (also known as a web spider, a web robot, and more often referred to as a web chaser in the FOAF community) is a program or script that automatically grabs information on the World Wide Web according to certain rules. Other less commonly used names include ant, autoindex, emulator, or worm. [0003] A web crawler is actually an application program for capturing network information. Existing web crawlers capture a large amount of data, and tasks are generally distributed equally, resulting in low data search efficiency. Contents of the invention [0004] The present application provides a distributed crawler task assignment method. It solves the shortcoming of low efficiency of technical solutions in the prior art. [0005] On the one hand, a distributed crawler task assignment ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30G06F9/50
CPCG06F9/5083G06F16/951
Inventor 马岩
Owner MAXTRON TECHSHENZHEN CO LTD