Unlock instant, AI-driven research and patent intelligence for your innovation.

Task scheduling method based on pre-release resource list under hadoop platform

A resource list, task scheduling technology, applied in the field of distributed computing, can solve problems such as hard to find

Active Publication Date: 2019-04-26
HUNAN UNIV
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

These two parameters are closely related to the cluster situation and the job situation. It is generally difficult to find the parameter settings suitable for the current job situation in the cluster, and it is impossible to re-schedule these two parameters when the job situation changes.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Task scheduling method based on pre-release resource list under hadoop platform
  • Task scheduling method based on pre-release resource list under hadoop platform
  • Task scheduling method based on pre-release resource list under hadoop platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] The embodiments of the present invention will be further described below in conjunction with the accompanying drawings and examples.

[0032] Such as figure 1 As shown, the present invention follows the resource three-level scheduling model shown in the figure, including:

[0033] Step 1: Select a queue. Select the queue with the highest priority according to the principle of fairness.

[0034] Step 2: Select a job. From the jobs in the selected queue, select the job with the highest priority according to the principle of fairness or FIFO.

[0035] Step 3: Select a task. A task is selected by a task scheduling method based on a list of pre-released resources.

[0036] In the figure, ①Fair principle selects the queue; ②Fair or FIFO principle selects the job; ③Selects the task based on the pre-release resource list of the job.

[0037] figure 1 Then it is a schematic flow chart of an embodiment of the present invention, and the method includes:

[0038] S101. The ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a pre-release resource list based task scheduling algorithm on a Hadoop platform. The resource scheduling is better helped by fully utilizing historical information recorded by Hadoop and cluster current status monitoring information. The algorithm does not need to set the delay waiting time manually. The contradiction between the fairness and locality is solved by resources in the pre-releasing resource list for pre-scheduling. In addition, the task scheduling algorithm provided by the invention can be applied to a fairness scheduler and a computing capacity scheduler at the same time like a delay scheduling algorithm. The scheduling algorithm provided by the invention utilizes the pre-release resource list to obtain good effects in aspects of Hadoop completion time, task locality, and average job response time through the matching schedule of the resource list and a task list.

Description

technical field [0001] The invention relates to the technical field of distributed computing, in particular to a task scheduling method based on a pre-release resource list under a Hadoop platform. Background technique [0002] Internet technology has given birth to the advent of the era of big data. At present, big data has become a hot research focus. Due to the huge amount of data, a single computer can no longer meet the storage and computing requirements, and various big data computing models and their corresponding distributed computing systems have begun to emerge. MapReduce is undoubtedly the most classic big data computing model. Apache Hadoop, as an open source implementation of MapReduce, has been widely used. Task scheduling and resource allocation have always been key technologies in the research of large-scale distributed clusters, which are especially important for improving the computing efficiency of big data clusters. [0003] At present, the commonly us...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48
CPCG06F9/4806
Inventor 李智勇陈京陈少淼杨波王尽如
Owner HUNAN UNIV