Task scheduling method and task scheduling device

Abstract

The invention discloses a task scheduling method and a task scheduling device, wherein the method comprises the following steps of obtaining request information of TaskTrackers, wherein the request information is task requesting information sent when the TaskTrackers have an idle work groove; calculating the resource consumption features and the rest task proportion of each task queue, wherein the resource construction feature of any one task queue is obtained through calculation according to the CPU (Central Processing Unit) occupation rate, the memory occupation rate and the bandwidth occupation rate of operating tasks in the task queue, and the rest task proportion of any one task queue is the ratio value of non-operating tasks to all of the tasks in the task queue; selecting the task array achieving the effect that the product of the resource consumption feature and the rest task proportion in the task queue is not smaller than that of other task queues as a target task queue; and processing the non-operating tasks in the target task queue by using the idle work groove. Therefore the non-operating tasks in the task queue in urgent need of being processed are preferably processed, and the task execution efficiency is greatly improved.

Description

technical field [0001] The present invention relates to the technical field of big data Hadoop cluster scheduling, and more specifically, to a task scheduling method and device. Background technique [0002] Hadoop is a distributed system infrastructure. With the application of Hadoop more and more widely, it faces more and more problems, including the problems faced when implementing task scheduling. [0003] Specifically, the default job scheduling algorithm under the Hadoop platform in the prior art is mainly the FIFO (FirstInputFirstOutput, first-in-first-out) algorithm, that is, when tasks need to be scheduled, the tasks are scheduled according to the first-in-first-out principle. The task of is the first existing task among the unprocessed tasks, thereby achieving fairness in terms of time in the process of scheduling tasks. However, the inventor found that scheduling tasks according to the first-in-first-out algorithm only considers the fairness of time, but does not...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to provide a task scheduling method and device to solve the problem in the prior art that the tasks that need to be processed urgently may not be processed in time, resulting in low task execution efficiency

Images