Data task scheduling method and device, equipment and readable storage medium

Abstract

The invention discloses a data task scheduling method. The method includes the steps of firstly, selecting to-be-executed data tasks from all data tasks according to a predefined rule and then determining demanded occupied resources for each to-be-executed data task during execution; secondly, according to the occupied resources, determining a scheme with the maximum number of the to-be-executed data tasks in which cluster idle resources can run; finally, using the to-be-executed data tasks contained in the scheme as target data tasks and executing the target data tasks. Therefore, the numberof the to-be-executed data tasks which can be contained is determined according to the idle conditions of the cluster idle resources, correspondingly the utilization rate of the cluster idle resourcesis increased, and the execution efficiency of the data tasks is improved. At the same time, since the maximum number of the data tasks can be executed in the cluster idle resources, the total duration of continuous running of all the data tasks on a cluster is shortened while the degree of parallelism of all the data tasks is increased. The invention also discloses a data task scheduling device,equipment and a readable storage medium which can also achieve the technical effects.

Description

technical field [0001] The invention relates to the field of big data, in particular to a data task scheduling method, device, and device-readable storage medium. Background technique [0002] With the rapid development of Internet technology, the era of big data is also coming. The data volume of the big data centers in the background of many applications reaches tens of billions. Facing such a huge amount of data to be analyzed, it is particularly important to analyze the data effectively and quickly. important. [0003] The traditional data task of the data to be analyzed is to sort the data task through the priority strategy, and execute the data task in order from high priority to low priority. However, although this method ensures that the task with high priority is executed first, However, since the priority strategy is adopted, it will be executed sequentially from high-priority data tasks to low-priority data tasks. When there are multiple high-priority data tasks,...

AI Technical Summary

Problems solved by technology

At this time, whether to run three 40MB data tasks or to run 50MB and 40MB data tasks is randomly determined. When the cluster idle resources choose to run 50MB and 40MB data tasks, there are still 30MB idle, which will lead to cluster idle resources The utilization rate of the data task is low and the execution efficiency of the data task is low. Moreover, due to the small number of running data tasks, that is, the low parallelism of the running data tasks, all data tasks continue to run on the cluster for a long time

Images