Weight priority task task scheduling method based on load balancing mechanism

Abstract

The invention belongs to the technical field of resource allocation, and discloses a weight-priority Task task scheduling method based on a load balancing mechanism, so as to avoid the problem that the load of a single high-weight worker node is too high, and the problem that a large task task is directly discarded is improved; firstly, the The task task is classified, and the task task is divided into four categories based on the two conditions of the task task reading speed and the amount of resources occupied; a load balancing module is added to each worker node to make it a distributed load balancing model; and dynamic Monitor the resource usage and load on each worker node, and then map to each worker node by classifying the task according to the size of the weight on the master node, and then schedule the task task. Solve the problem of load imbalance and queue congestion among high-weight worker nodes.

Description

technical field [0001] The invention belongs to the technical field of resource allocation, and in particular relates to a weight priority Task scheduling method based on a load balancing mechanism. Background technique [0002] At present, the existing technologies commonly used in the industry are as follows: [0003] In the spark source code, the scheduling process from task tasks to worker nodes is prioritized according to the priority of the stage and the locality of the task. By default, the task uses one thread for calculation. When broadcasting a task, if the task is larger than 128mb-200mb, the task will be discarded directly. Tasks will be assigned to each worker node by the master node in the form of a queue. This will lead to unbalanced distribution among the nodes, resulting in long-term starvation of high-configuration nodes and long-term full-load status of low-configuration nodes, resulting in longer task completion time, lower performance, and lower resou...

AI Technical Summary

Problems solved by technology

[0004] In summary, the problems in the prior art are: The existing improved algorithm does not take into account that there are many waiting tasks on the worker nodes with high weights, resulting in task congestion; when the task task selects the worker nodes, only the node with the largest weight is considered, resulting in a single The situation where the number of worker node tasks is too high

Images