Unlock instant, AI-driven research and patent intelligence for your innovation.

Method and device for task scheduling and method and device for computing node execution tasks

A computing node and task scheduling technology, applied in the Internet field, can solve problems such as not being able to satisfy crawling efficiency, and unable to make maximum use of machine resources

Active Publication Date: 2020-11-20
JINGDONG TECH HLDG CO LTD
View PDF10 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This method will lead to different crawler tasks need to customize the number of different threads, a thread will stop after completing its own crawler tasks, so it is impossible to maximize the use of machine resources
Furthermore, in the case of a deep hierarchy, only using different entries to set the number of threads cannot meet the needs of improving crawling efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for task scheduling and method and device for computing node execution tasks
  • Method and device for task scheduling and method and device for computing node execution tasks
  • Method and device for task scheduling and method and device for computing node execution tasks

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0057] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. It should be understood, however, that these descriptions are exemplary only, and are not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concepts of the present disclosure.

[0058] The terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting of the present disclosure. The terms "comprising", "comprising", etc. used herein indicate the presence of stated features,...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a task scheduling method, and the task scheduling method comprises the steps: obtaining task information of at least one to-be-executed task, wherein the task information comprises at least one piece of request data aiming at a page; acquiring state information distributed for the at least one to-be-executed task, wherein the state information is used for representing whether the to-be-executed task is executed or not; and storing the task information and the state information of the at least one to-be-executed task into a predetermined space according to a predetermined rule. The invention further provides a task scheduling device and a method and a device for the computing node to execute a task.

Description

technical field [0001] The present disclosure relates to the technical field of the Internet, and more specifically, to a task scheduling method and device, and a computing node execution method and device. Background technique [0002] With the rapid development of the network, the Internet has become the carrier of a large amount of information, how to effectively extract and use this information has become a huge challenge. In order to solve the above problems, the application of web crawler technology for directional crawling of relevant webpage resources was born. [0003] In the process of implementing the disclosed concept, the inventors have found at least the following problems in the prior art: web crawlers often perform multi-level page crawling in the process of crawling pages. For example, starting from the uniform resource locator (URL) entry of a website and crawling to the list page, there will be a large number of pages that need to be crawled. For example...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48G06F16/953G06Q30/06
CPCG06F9/4881G06F16/953G06Q30/0601
Inventor 张琪琦高光海
Owner JINGDONG TECH HLDG CO LTD