Task scheduling method for self-learning feedback in hadoop multi-job environment

Abstract

The invention relates to a task scheduling method in the field of high-performance clusters. The task scheduling method comprises the steps: after analysis of each working node of a Hadoop in a job submitting stage, acquiring an actual stage weight according with a task, after processing of a geometric mean method, establishing a reference standard of the stage weight for left tasks of the job; in a task feedback stage, adopting the reference standard for the left tasks of the job, and estimating the task residual execution time by combining with progresses of sub-stages; in a job feedback stage, solving a geometric mean of stage weights of all tasks by using a staging manner, and establishing a job name-stage weight mapping record to be used as a reference of executing subsequent jobs on the node. According to the task scheduling method, self-learning and information feedback can be respectively carried out on the task of each job in a multi-job parallel executing environment to obtain more accurate stage weight estimation, the accuracy of pre-estimating the task residual execution time is improved, the hit rate of selecting dated tasks is increased, and the optimal utilization of cluster resources is promoted.

Description

technical field [0001] The invention relates to a task scheduling method in the field of high-performance clusters, in particular to a task scheduling method based on task autonomous learning and an information feedback mechanism in a Hadoop multi-job environment. Background technique [0002] MapReduce is a parallel data processing model for large-scale data-intensive applications. As an open source implementation of MapReduce, Hadoop has been widely used in various fields. However, the existing Hadoop has great limitations, because its development principle is originally aimed at the homogeneous environment, and the default scheduling mechanism is also designed based on the assumptions of node homogeneity and task linear execution, but in actual applications , due to differences in hardware configuration, resource virtualization, etc., these assumptions cannot be satisfied. [0003] Usually, a Hadoop cluster is composed of many conventional computers. These machines are ...

AI Technical Summary

Problems solved by technology

Some methods set the time ratio of each stage to a fixed value closer to a certain task. Although the purpose of improvement has been achieved to a certain extent, the limitations are self-evident. For different tasks, this value has no reference value, even if the same Tasks, in the case of different available resources, the actual weights are also different; therefore, a method of dynamically adjusting the phase weights is produced, using the information of the previous tasks as a reference to update the phase weights, but this method is not suitable for multi-job and multi-task Larger jitter will be generated in the case of simultaneous execution

Images