Check patentability & draft patents in minutes with Patsnap Eureka AI!

Crawler task monitoring method and device

A crawler, task technology, applied in hardware monitoring, instrumentation, electrical digital data processing, etc.

Pending Publication Date: 2021-12-24
上海妙一生物科技有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0011] The purpose of this application is to address the deficiencies in the prior art and provide a monitoring method, device, computer equipment, and computer-readable storage medium for crawler tasks, so as to at least solve the problem that the crawler management platform in the related art only monitors crawler running effectively. Avoid the problem of data crawling failure

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Crawler task monitoring method and device
  • Crawler task monitoring method and device
  • Crawler task monitoring method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0049] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application, and those skilled in the art can also apply the present application to other similar scenarios. In addition, it can also be understood that although such development efforts may be complex and lengthy, for those of ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a crawler task monitoring method and device, and the method comprises the steps: executing at least one of the following monitoring operations after the operation of a crawler task is finished: monitoring a crawler code operation error reporting condition; monitoring the updating condition of the persistent crawling data; monitoring the CPU utilization rate and the memory occupancy rate of the server; and monitoring the crawler task according to a monitoring result of the monitoring operation. Through the method and the device, the problem that data crawling failure cannot be effectively avoided due to the fact that a crawler management platform only monitors crawler operation in the related technology is solved, and the data crawling failure is avoided as much as possible through crawler operation monitoring, data monitoring and server monitoring.

Description

technical field [0001] The present invention relates to the technical field of crawler project management, in particular to a monitoring method, device, computer equipment and computer-readable storage medium for crawler tasks. Background technique [0002] Currently, crawler project management platforms include Crawlab, a general crawler management platform, and scrapydweb, a distributed management platform only for scrapy crawler frameworks. Their basic features are as follows: [0003] (1) Master node [0004] The master node interacts with the web front-end page and distributes crawler tasks to different working nodes. The main functions in charge include the scheduling of crawler tasks and the deployment of tasks, and the configuration files of the platform are also applied on the master node. [0005] (2) Working nodes [0006] Work nodes are mainly responsible for the execution of crawler tasks and the storage of crawled data. It can communicate with the master no...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F11/30
CPCG06F11/3051G06F11/3024G06F11/3037
Inventor 杨峰张令忠
Owner 上海妙一生物科技有限公司
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com