Distributed multi-task scheduling web crawler device and system

A scheduling network and multi-task technology, which is applied in the field of distributed multi-task scheduling web crawler devices and systems, can solve the problems of affecting work progress and reducing work efficiency, so as to improve the processing speed, speed up the processing speed, and avoid repeated information extraction. Effect

Pending Publication Date: 2021-12-24
安徽壹零贰肆加科技有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, most of the existing web crawlers can only fetch a single link or grab a single program, and cannot perform multitasking scheduling of web links or a certain program, which greatly affects the work progress. Reduced work efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed multi-task scheduling web crawler device and system
  • Distributed multi-task scheduling web crawler device and system
  • Distributed multi-task scheduling web crawler device and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0023] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are exemplary only, and are not intended to limit the scope of the present invention. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present invention.

[0024] Such as Figure 1-4 As shown, a distributed multi-task dispatching web crawler device and system proposed by the present invention includes an information input module, an information receiving module is connected to an output end of the information input module, and an information identification module is connected to an output end of the information receiving module. The output end of the identification module i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed multi-task scheduling web crawler device and system, relates to the technical field of web crawlers, and provides the following scheme for solving the problems that an existing web crawler can only independently call one link or can only capture an independent program, the work progress is greatly influenced, and the work efficiency is reduced. The following scheme is proposed, an information input module is included, the output end of the information input module is connected with an information receiving module, and the output end of the information receiving module is connected with an information identification module. The invention is ingenious in design and excellent in conception, the information can be recognized firstly through the installed information recognition module, the information processing speed of the system is increased, the installed computer crawler modules can grab multiple themes respectively, the information grabbing speed of the system is increased, the installed link processing module can remove repeated information, the information processing speed is improved, and the device is convenient to use, high in practicability and convenient to popularize.

Description

technical field [0001] The invention relates to the technical field of web crawlers, in particular to a distributed multi-task scheduling web crawler device and system. Background technique [0002] Web crawler, also known as web spider, web robot, in the FOAF community, more often referred to as a web chaser, is a program or script that automatically grabs information on the World Wide Web according to certain rules, and others are not commonly used There are also names such as ants, automatic indexing, simulation programs, or worms. [0003] However, most of the existing web crawlers can only fetch a single link or grab a single program, and cannot perform multitasking scheduling of web links or a certain program, which greatly affects the work progress. Reduced work efficiency. Contents of the invention [0004] (1) Purpose of the invention [0005] In order to solve the technical problems existing in the background technology, the present invention proposes a distri...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/33G06F16/951G06F40/258
CPCG06F16/334G06F16/951G06F40/258
Inventor 尹娜
Owner 安徽壹零贰肆加科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap