High-performance cluster task scheduling method and device, electronic equipment and storage medium

Abstract

The invention discloses a high-performance cluster task scheduling method and device, electronic equipment and a readable storage medium. The method comprises the steps: constructing a task prediction model in advance based on a deep learning algorithm, wherein the task prediction model is used for predicting the application running time of each user task in the simultaneous running process of different jobs; training a task scheduling model in advance, wherein the task scheduling model is used for dynamically optimizing a task scheduling process based on trial and error and delay reward characteristics of reinforcement learning; inputting current task operation parameters and calculation parameters of high-performance cluster task scheduling into the task prediction model to obtain application operation prediction time; and calling the task scheduling model to obtain a task scheduling result based on the application running prediction time. According to the method, a user does not need to design computing resources, and the utilization rate of a high-performance cluster and the task scheduling accuracy can be effectively improved.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a high-performance cluster task scheduling method, device, electronic equipment and readable storage medium. Background technique [0002] With the rapid expansion of computing needs in different industries, the development of commercial CPU (central processing unit, central processing unit) and high-speed interconnection network equipment, high-performance computing (HPC) clusters have achieved unprecedented development in the past 20 years. By aggregating a large number of microprocessor units, the high-performance computing cluster has excellent scalability and high cost performance, and can quickly solve complex problems. Therefore, it is widely used in nuclear explosion simulation, weather forecast, engineering calculation and other technical fields . According to Moore's Law, the computing power of high-performance cluster platforms increases exponentially ever...

AI Technical Summary

Problems solved by technology

However, since the high-performance cluster scheduling system cannot predict the running time of the application, during the task allocation process, the user needs to customize the resource application size when submitting tasks, and cannot adaptively adjust the allocated resources according to the actual status of the cluster

Due to the competition among different applications for resources such as memory bandwidth, the running time will be affected. The existing technology cannot consider running multiple applications on a single node of the cluster at the same time, resulting in inaccurate task scheduling.

In the traditional application prediction model, the regression method is relatively simple and cannot describe the mutual interference of multiple applications running at the same time, so it cannot provide accurate results in such complex situations

In addition, the regression model cannot take into account the resource constraints in the application running process, such as the number of cores, memory capacity, storage bandwidth, etc., so the obtained model is usually only applicable to the situation where all the resources of a single node are occupied, and the accuracy of the running time is not good when there are constraints. high

In addition, there are shortcomings in the traditional heuristic scheduling module. For example, the Min-Min algorithm will allocate tasks to nodes with faster processing speeds as much as possible, so that nodes with slower processing speeds will always be in a hungry state, resulting in low node utilization.

Images