Load balance mechanism-based weight priority task scheduling method

Abstract

The invention belongs to the technical field of resource allocation, and discloses a load balance mechanism-based weight priority task scheduling method, which avoids excessively high load of a singlehigh-weight worker node and improves the problem that a large task is directly discarded. The method comprises the steps of firstly classifying tasks: classifying the tasks into four types accordingto two conditions including a task reading speed and an occupied resource quantity; adding a load balance module to each worker node, thereby enabling each worker node to become a distributed load balance model; and dynamically monitoring resource usage and load conditions of each worker node, performing mapping to each worker node according to weight values on a master node and the task classification condition, and performing task scheduling. The problems of load imbalance among the high-weight worker nodes and queue congestion are solved.

Description

technical field [0001] The invention belongs to the technical field of resource allocation, and in particular relates to a weight priority Task scheduling method based on a load balancing mechanism. Background technique [0002] At present, the existing technologies commonly used in the industry are as follows: [0003] In the spark source code, the scheduling process from task tasks to worker nodes is prioritized according to the priority of the stage and the locality of the task. By default, the task uses one thread for calculation. When broadcasting a task, if the task is larger than 128mb-200mb, the task will be discarded directly. Tasks will be assigned to each worker node by the master node in the form of a queue. This will lead to unbalanced distribution among the nodes, resulting in long-term starvation of high-configuration nodes and long-term full-load status of low-configuration nodes, resulting in longer task completion time, lower performance, and lower resou...

AI Technical Summary

Problems solved by technology

[0004] In summary, the problems in the prior art are: The existing improved algorithm does not take into account that there are many waiting tasks on the worker nodes with high weights, resulting in task congestion; when the task task selects the worker nodes, only the node with the largest weight is considered, resulting in a single The situation where the number of worker node tasks is too high

Images