Job scheduling method and device and distributed system

A distributed system and job scheduling technology, applied in the field of data processing, can solve the problems of insufficient utilization of cluster resources, large amount of data in central nodes, affecting scheduling efficiency, etc., to improve resource utilization, reduce job running time, improve The effect of scheduling efficiency

Active Publication Date: 2017-01-04
ALIBABA GRP HLDG LTD
View PDF5 Cites 13 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, in this job scheduling method, the amount of data processed and maintained by the central node is huge, which affects the scheduling efficiency, and does not make full use of cluster resources, resulting in waste of resources

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Job scheduling method and device and distributed system
  • Job scheduling method and device and distributed system
  • Job scheduling method and device and distributed system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0057] The implementation of the present application will be described in detail below with reference to the accompanying drawings and examples, so as to fully understand and implement the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects.

[0058] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0059] Memory may include non-permanent storage in computer-readable media, in the form of random access memory (RAM) and / or nonvolatile memory, such as read-only memory (ROM) or flash memory (flashRAM). Memory is an example of computer readable media.

[0060] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, mod...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a job scheduling method and device and a distributed system. The distributed system at least comprises a center node, multiple control nodes connected with the center node and multiple calculation nodes connected with the control nodes respectively; the center node distributes tasks of a job to all the control nodes; the control nodes schedule all task slices to run in the calculation nodes connected with the control nodes. The method comprises the steps that the first control node scheduling the first task notifies the second control node scheduling the second task to acquire running data generated by at least one task slice of the first task when at least one task slice of the first task finishes running; the second control node acquires the running data generated by at least one task slice of the first task, distributes the running data to each task slice of the second task and schedules at least one task slice of the second task to run, and the running data is processed. According to the embodiment, scheduling efficiency is improved, and the resource utilization rate is increased.

Description

technical field [0001] The present application belongs to the technical field of data processing, and in particular relates to a job scheduling method, device and distributed system. Background technique [0002] In traditional distributed systems, distributed jobs include a limited number of tasks (tasks), and there are certain dependencies between tasks. Each task is divided into multiple task slices (instances). When the job is running, each Task slices of tasks perform data processing. [0003] Based on the traditional distributed system, the job scheduling method is usually performed by the central node (master). For example, suppose the job has two tasks, namely task1 and task2. The dependency between task1 and task2 is task2 and the input is task1. output, when the job runs, the central node will look for a task without a predecessor task, assuming that task1 runs first, and when all the task slices of task1 run, the central node will then schedule task2 to run. [...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/46
CPCG06F9/4881G06F9/5038G06F9/5066G06F9/52
Inventor 才华
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap