Dispatching management method for task queue priority of large data set

Abstract

The invention discloses a dispatching management method for task queue priority of a large data set. According to the dispatching management method, a circulating thread for task acquisition is operated at each node of a task acquisition end, tasks are acquired from different queues according to priority of the queues, the tasks can be preferentially acquired from the high-priority queues, and meanwhile, the condition that the low-priority tasks cannot be blocked by the high-priority tasks is also taken into consideration. The queue service can support multiple queues of multiple types, the priority of each queue can be set, and average operating time of one queue can be acquired according to operating records; the number of the tasks acquired from each queue is dynamically adjusted according to multiple conditions including the task priority, queue length, the average operating time of each queue, the maximum number of the tasks acquired every time and the like, the tasks are dispatched reasonably, and resource waste is prevented.

Description

Technical field [0001] The invention relates to the technical field of big data processing, in particular to a scheduling management method for task queue priorities of big data sets. Background technique [0002] With the advent of the big data era, it has derived its own unique architecture, and it has also directly promoted the development of storage, network and computing software technologies. As the smallest unit in data processing, the number of tasks has also shown explosive growth. Similarly, the task queue that carries tasks faces the scheduling problem of task priority. [0003] At present, the problems faced in this situation are: when using a normal queue, adjusting the task priority in a list type queue will greatly reduce the performance of the queue; when using a simple multiple queue, you need to set up multiple queues with different priorities. , Priority scheduling is achieved by changing the queue where tasks are located, but when high priority queues accumulat...

AI Technical Summary

Problems solved by technology

[0003] At present, the problems faced in this situation are: when using ordinary queues, adjusting the task priority in the list type queue will greatly reduce the performance of the queue; while using simple multi-queue, it is necessary to set up multiple queues with different priorities , to achieve priority scheduling by changing the queue where the task is located, but when the high priority queue is piled up, other priority queues will be completely blocked, so many working nodes are not fully utilized, and some nodes will be overloaded

Images