Deep learning training task scheduling system for gpu cluster

Abstract

A GPU cluster-oriented deep learning training task scheduling system comprises an offline performance characterization module and an online task scheduling deployment module. The offline performance characterization module respectively predicts the similarity of different training tasks to determine the hardware index set to be collected for a newly submitted task and the performance degradation degree, i.e., the performance interference degree, generated after mixed deployment of two training tasks, and collects the calculation time of the training task under different batch sizes in the exclusive GPU mode to fit the batch size-calculation time relationship. The online task scheduling deployment module takes out a task to be scheduled from a task queue and generates the GPU affinity and batch size distribution strategy of the task to be scheduled according to the batch size-calculation time relationship and the performance interference degree, and divides the training task submitted by a user into appropriate GPUs according to the GPU number specified by the user, i.e., the GPUs with the minimum performance interference after mixed deployment, to minimize the average job completion time and improve the cluster resource utilization. The problem of maximizing the overall system throughput and improving the cluster resource utilization under the condition that part of the training tasks cannot be fully utilized by the GPU resources alone and need to comply with the group scheduling principle in the GPU cluster is completely solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of neural network applications, and particularly relates to a deep learning training task scheduling system for a GPU cluster. BACKGROUND

[0002] Many existing data centers still use general cluster schedulers such as Yarn, Borg and the like, and these scheduling systems are designed for traditional big data applications such as MapReduce. In these scheduling systems, a DNN training task is simply regarded as another type of big data application, and the scheduling system selects the task according to a certain scheduling algorithm, allocates a group of GPUs required by the task, and runs the task exclusively on the group of GPUs until the end. SUMMARY

[0003] The application is directed to the problems that the existing scheduling system cannot perceive the model and workload of a training task, does not support sharing of GPUs, cannot fully utilize the computing capacity of the GPUs, and is prone to load imbalance and dragging of a global task by a subtask, and proposes a deep learning training task scheduling system for a GPU cluster, which completely solves the problems of maximizing the overall throughput of the system and improving the utilization rate of cluster resources under the condition that some training tasks cannot fully utilize GPU resources alone and need to comply with the group scheduling principle in the GPU cluster.

[0004] The application is implemented by the following technical scheme:

[0005] The application relates to a deep learning training task scheduling system for a GPU cluster, which comprises an offline performance characterization module and an online task scheduling and deployment module, wherein the offline performance characterization module predicts the similarity of different training tasks to determine the hardware index set to be collected for a newly submitted task and the performance interference degree caused by mixed deployment of two training tasks, that is, the performance interference degree, and collects the calculation time of the training task at different batch sizes in the exclusive GPU mode to fit the batch size-calculation time relationship; the online task scheduling and deployment module takes out a task to be scheduled from a task queue and generates the GPU affinity and batch size distribution strategy of the task to be scheduled according to the batch size-calculation time relationship and the performance interference degree, and allocates appropriate GPUs to the training task according to the training task submitted by a user and the number of GPUs specified by the user, that is, the GPUs with the minimum performance interference after mixed deployment, so as to minimize the average job completion time and improve the utilization rate of cluster resources.

[0006] The GPU affinity refers to the calculation speed of the task to be scheduled after being deployed on the GPU, and the faster the calculation speed, the higher the affinity.

[0007] The offline performance characterization module deploys a training task submitted by a user on a single reserved special GPU, collects model structure feature information of the training task, determines the similarity of the training task to existing tasks in a system database by using a task similarity prediction model to determine a set of hardware indicators to be collected, and then collects hardware indicator data of the task during runtime through a few iterations.

[0008] Since the online task scheduling and deployment module allows two training processes to share the same GPU, this dynamic hybrid deployment mode will result in a situation where some GPUs are shared by some training processes and some GPUs are exclusively used by some training processes. In this case, the processes on the shared GPUs become laggards due to the influence of resource competition on the calculation speed, and other training processes need to be synchronized and waited for, thereby prolonging the task completion time. By dynamically changing the batch size of each process of the training task to maintain a relatively balanced load state to eliminate the influence of the laggards, that is, the offline performance characterization module first samples the <batch size, training time> of the training task to obtain the performance curve of the training task under exclusive GPU, that is, the batch size-computation time relationship, and derives the batch size-computation time relationship of the training task running on any GPU according to the exclusive performance curve and the performance interference prediction model, thereby reallocating the global batch size of the training task to ensure that the computation time of each training process under the new batch size distribution is basically the same.

[0009] The performance interference prediction model is used to estimate the performance interference degree of the training task on the GPU. Wherein, T is the job completion time of the training task when the GPU is exclusively used, and T' is the job completion time in the shared GPU mode.

[0010] The present application relates to a GPU cluster-oriented deep learning training task scheduling method based on the above-mentioned system, which adds the training task to the task scheduling queue and performs online resource allocation after offline collection and performance characterization of the training task to be trained, determines the target GPU of the training task to be processed and the training data volume on each GPU.

[0011] The offline collection refers to running the training task to be trained on a special reserved GPU, modifying the number of samples processed by a single iteration calculation of the training task, that is, the batch size, during data collection, and collecting data samples in the format of <batch size, computation time> and calculating the estimated runtime of the training task.

[0012] The performance characterization refers to: according to the predicted running time, classifying the to-be-trained task as a long job and a short job, when it is a long job, collecting all to-be-collected hardware index data of the long job and adding it to the task scheduling queue, when it is a short job, collecting structural feature information of the short job for predicting the similarity of the training task and a running task, using non-core hardware index data corresponding to the most similar running training task to replace the corresponding hardware index data of the new training task, and then adding it to the task scheduling queue.

[0013] The online resource allocation refers to: according to the scheduling algorithm based on the minimum principle of performance interference degree, determining the target GPU of the to-be-processed training task and the training data quantity on each GPU through the batch size-computing time relationship and the predicted performance interference degree.

[0014] Technical effects

[0015] The model structure feature information data is used to predict the similarity of different training tasks, and compared with the prior art, the time required for offline collection of hardware index data of a short job can be significantly shortened, and the offline performance characterization overhead is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is a flowchart of the present application;

[0017] Figure 2 It is a schematic diagram for determining a to-be-collected hardware index set using task similarity;

[0018] Figure 3 It is a schematic diagram of the execution process of the scheduler based on GPU affinity;

[0019] Figure 4 It is a flowchart of the embodiment. DETAILED DESCRIPTION

[0020] The embodiment relates to a GPU cluster-oriented deep learning training task scheduling system, which comprises an offline performance characterization module and an online task scheduling and deployment module, wherein: the offline performance characterization module respectively predicts the similarity of different training tasks to determine a to-be-collected hardware index set of a newly submitted task and the performance decline degree, i.e. performance interference degree, generated after mixed deployment of two training tasks, and offline collects the computing time of different batch sizes of the training task in the exclusive GPU mode to fit the batch size-computing time relationship; the online task scheduling and deployment module takes out a to-be-scheduled task from a task queue and generates GPU affinity and batch size distribution strategies of the to-be-scheduled task according to the batch size-computing time relationship and the performance interference degree, and according to the number of GPUs specified by a user, the to-be-scheduled task is distributed to appropriate GPUs, i.e. GPUs with the minimum performance interference after mixed deployment, so as to minimize the average job completion time and improve the cluster resource utilization rate.

[0021] The offline performance characterization module comprises a task similarity prediction unit, a hardware information collection unit, a batch size calculation time characterization unit and a performance interference degree prediction unit, wherein: the task similarity prediction unit calculates the computational graph information submitted by the user, performs computational graph traversal, obtains the structural feature information of the training task, inputs the structural feature information of each completed task into a task similarity prediction model, obtains the similarity of the to-be-processed task and other tasks to determine the hardware collection index set; the hardware information collection unit deploys the to-be-processed task on a special GPU according to the hardware collection index set determined by the task similarity prediction unit, and collects the corresponding hardware indicators by using NSIGHT tools; the batch size calculation time characterization unit performs data fitting according to the calculation time corresponding to different batch sizes sampled when the to-be-processed task runs on the special GPU, and obtains the batch size-computation time relationship of the task; the performance interference degree prediction unit inputs the hardware indicator data of the to-be-processed training task and the hardware indicator data of the task running on the GPU into a performance interference degree prediction model, and obtains the performance degradation degree of each of the two training tasks after sharing the GPU.

[0022] The online task scheduling and deployment module comprises a GPU affinity-based task scheduler, an online task batch size allocation unit and a mixed deployment interference conduction effect correction unit, wherein: the GPU affinity-based task scheduler performs greedy algorithm processing according to a nonlinear integer programming problem to obtain the target deployment GPU of the to-be-processed task, the online task batch size allocation unit reallocates the batch size of the to-be-scheduled task, and the target is to minimize the maximum iteration calculation time of all training processes The mixed deployment interference conduction effect correction unit traverses all the target deployment GPUs of the to-be-processed tasks according to the known deployment decision, reallocates the batch size of the running task that causes serious load imbalance phenomenon due to the deployment of a new training task, and obtains the global balanced load state.

[0023] The task similarity refers to the weighted sum of the performance interference degree distribution trend similarity and the performance interference degree value similarity when two tasks share the GPU with each training task in the benchmark test set, SI=w1SI1+w2SI2, wherein: w1, w2 are the trend similarity and value similarity weight coefficients, the trend similarity The value similarity <sl 11 , sl 12 ,..., sl 1n > represents the performance interference degree vector of model 1 when sharing the GPU with n models in the benchmark test set, and <sl 11 , sl 12 ,..., sl 1n > represents the performance interference degree vector of model 2.

[0024] The computing time of 100 iterations in exclusive GPU and shared GPU states is recorded respectively as t solo_a , t solo_b , t colo_a , t colo_b for each mixed deployment combination <a, b> of each pair of training tasks a and training task b deployed on the same GPU, and the performance interference degree corresponding to the mixed deployment combination is: After the training data set is established, the structure feature information of the model is used as the input feature, and the similarity is used as the label to train the prediction model, and 10-fold cross-validation is used to select a suitable prediction model.

[0025] In this embodiment, the task similarity prediction model uses an XGBOOST model.

[0026] This embodiment relates to a GPU cluster-oriented deep learning training task scheduling method based on the above system. After the to-be-trained tasks are collected and characterized offline, the training tasks are added to the task scheduling queue and online resource allocation is performed, the target GPU of the to-be-processed training task and the training data volume on each GPU are determined.

[0027] This embodiment summarizes representative deep learning micro benchmarks and uses Spearman correlation coefficients to select 14 hardware index data with strong correlation as shown in Table 1.

[0028] Table 1 Correlation between DNN training task mixed deployment interference intensity and running index

[0029] Indicator Spearman correlation coefficient Indicator Spearman correlation coefficient SM occupancy -0.85 Grid size -0.77 SM throughput -0.8 Shared memory per Block -0.78 Kernel runtime -0.76 Registers per thread -0.76 Memory throughput -0.78 SharedMemory configuration size -0.77 SMActiveCycles -0.77 PCIe read data volume - L2 cache throughput -0.82 PCIe write data volume - DRAM throughput -0.81 Block limit registers 0.72

[0030] This embodiment predicts the performance interference degree faced by different tasks by changing the relative position of the hardware index data of the two mixed deployment tasks in the prediction model input data: the runtime hardware index data corresponding to task a and task b is spliced as the input of the performance interference degree prediction model, and the performance interference degree of task a after mixed deployment is output; the runtime hardware index data corresponding to task b and task a is spliced as the input of the performance interference degree prediction model, and the performance interference degree of task b after mixed deployment is output. Based on the 528 mixed deployment instances collected, this embodiment establishes a training data set for the performance interference degree prediction model, and uses 10-fold cross-validation to select a suitable prediction model.

[0031] As shown in Table 2, the mean absolute error of using linear regression (LR), K nearest neighbor regression, random forest regression (RFR) and XGBOOST is shown in Table 2: XGBOOST has the best performance, so the performance interference degree prediction model of the present application selects XGBOOST.

[0032] Table 2 average absolute error of performance interference under different prediction methods

[0033] Linear regression K-Nearest Neighbors regression Random Forest regression XGBOOST Mean Absolute Error 0.158 0.176 0.077 0.051

[0034] Due to the large number of neural network model operators, the time cost of performance characterization for all 14 hardware indicators is large. Therefore, all training tasks are divided into long and short jobs, and the classification standard is the predicted running time threshold t0. For each new training task, if it is a long job (predicted training time greater than t0), all 14 hardware indicators shown in Table 2 will be collected, denoted as S0; if it is a short job, only 5 most relevant hardware indicators (SM occupancy, L2 cache throughput, DRAM throughput, PCIe read data volume, and PCIe write data volume) are collected, denoted as S1, and the remaining indicator values are obtained from the corresponding data of the most similar running task. Based on this, the similarity of different training tasks is mined using model structure feature information.

[0035] The structure feature information includes: floating point operation number, model parameter number, convolution layer number, normalization layer number, activation layer number, and input data size.

[0036] The nonlinear integer programming problem refers to: for distributed DNN training tasks, due to the synchronization requirement of parameter update, the time of each iteration depends on the longest calculation time in all working processes, that is: Wherein: is the calculation time of one iteration of DNN training task j, P j is the parallelism of task j, that is, the number of GPUs required, is the calculation time of one iteration of the i-th training process of training task j.

[0037] The calculation time of one iteration of the training process is related to the batch size and task placement, that is: Wherein: bs i is the batch size allocated to the i-th training process of training task j, g ji is the placement of the i-th training task of task j on the g ji GPU. Further, the predicted calculation time of the training process when it exclusively uses the GPU at different batch sizes can be obtained by fitting the data collected by the offline performance characterization module.

[0038] At the same time, the calculation time of one iteration according to the mixed performance prediction model is Wherein: SL(g ji ) is the performance interference generated by placing the training task process on the g ji GPU, and SL(g jiIf the gth GPU is idle, the interference is 1, otherwise the gth GPU is collected. ji The hardware information of the tasks running on each GPU is predicted using the offline trained XGBOOST model; Tb(bs ji ) is the workload of training process i in exclusive mode is bs i When the scheduling system needs to deploy a new DNN training task, in order to maximize the system throughput, the scheduling decision is: Among them TP k is the training throughput of the task to be deployed, the nonlinear function ∑ j∈J TP j is the sum of the training throughput of tasks in the system; constraints: in: Ensure that the sum of the loads of all worker processes for training task j equals the specified workload for that training task. The GPU-affinity-based task scheduler not only aims to minimize the performance disruption experienced by newly deployed tasks, thereby improving their throughput, but also minimizes disruption experienced by running tasks. Therefore, the overall goal is to maximize the sum of the throughput of all tasks.

[0039] like Figure 3 As shown, the GPU affinity-based task scheduler uses a greedy algorithm to find an approximate optimal solution, specifically: taking out the task to be deployed and the runtime hardware indicator data of the task collected by the offline performance characterization module from the task queue, then using the runtime hardware indicator data of the task and the corresponding data of the tasks running on each GPU as input, using the performance interference predictor model to predict the respective performance interference of the task to be scheduled and the shared task after the task to be scheduled is deployed on each GPU and selecting the required GPU with the smallest sum of the two, then adjusting the batch size distribution of the task to be deployed through the online task batch size allocation unit to balance the workload, and then correcting the interference conduction effect caused by the batch size distribution adjustment of the task to be deployed through the mixed interference conduction effect correction unit, and finally deploying and running the task according to the generated GPU and batch size decision.

[0040] The adjustment of the batch size distribution of tasks to be deployed specifically includes:

[0041] a) Obtain the iterative computation time vector t of each training process that exclusively uses the GPU under the evenly distributed batch size by using the batch size-computation time relationship obtained by the offline performance characterization module. j ={t j1 , t j2 ,...,t jp}, the calculation time multiplied by the corresponding performance interference is the expected calculation time t′ in the mixed deployment mode j =tj • SL j where: J is the training task to be deployed, BS is the total batch size, p is the parallelism, i.e. the number of GPUs needed, t j is the vector of iteration computation time of training task J in exclusive GPU mode under current configuration, SL j is the vector of predicted performance interference of each training process on the GPU to be deployed, SL j = {sl j1 , sl j2 ,..., sl jp}

[0042] b) Calculate the extreme value range of iteration computation time of all training processes When the extreme value range is greater than the computation time difference threshold, perform load balancing operation, specifically: transfer batch size adjustment value samples from the training process with the longest computation time to the training process with the shortest computation time and gradually reduce the batch size adjustment value until the adjusted extreme value range value is less than the time difference threshold.

[0043] c) After determining the deployment scheme (GPU allocation and batch size allocation) of the task to be scheduled J, since the deployment strategy will also interfere with the performance of the running tasks sharing the GPU, thus breaking the existing balanced load state of the running tasks. Therefore, this embodiment needs to re-allocate the batch size of the affected running tasks, but re-allocating the batch size of the running tasks will again affect the balanced load state of other tasks, i.e. the interference propagation effect of mixed deployment.

[0044] The interference propagation effect caused by the batch size distribution adjustment of the task to be deployed is corrected, which means that all target deployment GPUs of all tasks to be scheduled are traversed, and the batch size of the running tasks that have a serious load imbalance phenomenon due to the deployment of new training tasks is re-allocated to obtain a global balanced load state. To prevent frequent batch size re-allocation, an adjustment threshold η sl is set, and when the performance interference of the affected task changes, i.e. the change difference of the performance interference of the running task due to the deployment of the task to be scheduled exceeds the threshold, the batch size re-allocation algorithm is called to re-allocate the batch size of the task.

[0045] Through specific actual experiments, the basic parameters of the deep neural network model tested are as follows

[0046] Model name Parallelism Dataset Training epochs ResNet-50 [1,2,4] CIFAR-10 Random(3, 10) ResNet-152 [1,2,4] CIFAR-10 Random(3, 10) Vgg11 [1,2,4] CIFAR-10 Random(3, 10) Vgg19 [1,2,4] CIFAR-10 Random(3, 10) mobilenet_v3_small [1,2,4] CIFAR-10 Random(3, 10) mobilenet_v3_large [1,2,4] CIFAR-10 Random(3, 10) mnasnet0_5 [1,2,4] CIFAR-10 Random(3, 10) shufflenet_v2_x1_0 [1,2,4] CIFAR-10 Random(3, 10) squeezenet1_0 [1,2,4] CIFAR-10 Random(3, 10)

[0047] According to the above neural network model specifications, 42 training tasks are randomly generated, and the submission interval of two training tasks is 1 to 10 minutes.

[0048] The embodiment adopts First In First Out (FIFO) for performance comparison, FIFO refers to selecting the task with the earliest submission time from the scheduling queue and adopting exclusive GPU mode for resource allocation.

[0049] Performance comparison of different scheduling algorithms for 42 training tasks

[0050]

[0051] Compared with the exclusive GPU training mode, the application can reduce the average task completion time by 27.91%.

[0052] Compared with the prior art, the application utilizes the parallelism of training task data processing and the computing power characteristics of GPU to obtain that the relationship between the running time of a single iteration of a training task and the work load can be accurately predicted by sampling a small number of sample data, so as to establish a corresponding time model; secondly, a performance interference model of mixed deployment of training tasks is established, the interference degree of each training task is accurately predicted by using the runtime hardware information of all training tasks on the GPU; finally, the physical location of the required resources is determined based on the minimum interference principle during task deployment. The application considers the load balancing problem of distributed training tasks while scheduling based on the minimum interference principle, eliminates the lag problem caused by resource heterogeneity and dynamic mixed deployment in the training process, can effectively improve the GPU utilization, reduce the queuing time of high parallelism tasks, and improve the system throughput.

[0053] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the application, the protection scope of the application is subject to the claims and is not limited by the above specific embodiments, each implementation scheme within the scope is subject to the constraints of the application.

Claims

1. A deep learning training task scheduling system for GPU clusters, characterized by: include: An offline performance characterization module and an online task scheduling and deployment module, wherein: the offline performance characterization module predicts the similarity of different training tasks to determine the set of hardware indicators to be collected for the newly submitted task and the degree of performance degradation caused by the mixed deployment of the two training tasks, namely, the performance interference, and collects the computing time of different batch sizes of the training task in the exclusive GPU mode offline to fit its batch size-computation time relationship; the online task scheduling and deployment module takes the tasks to be scheduled from the task queue and generates the GPU affinity and batch size distribution strategy for the tasks to be scheduled based on the batch size-computation time relationship and performance interference. It allocates the appropriate GPU to the training task submitted by the user and the number of GPUs specified by the user, namely, the GPU with the least performance interference after mixed deployment to minimize the average job completion time and improve cluster resource utilization; The offline collection refers to running the task to be trained on a dedicated reserved GPU, modifying the number of samples processed by a single iteration of the training task during the data collection process, i.e., the batch size, collecting data samples in the format of <batch size, calculation time>, and calculating the estimated running time of the training task; The performance profiling method includes: classifying the training task into long jobs and short jobs based on the expected running time; collecting all the hardware indicator data to be collected for the long job and adding it to the task scheduling queue; collecting the structural feature information of the short job to predict the similarity between the training task and the running task; replacing the corresponding hardware indicator data of the new training task with the non-core hardware indicator data corresponding to the most similar running training task, and adding it to the task scheduling queue; The performance interference , where: T is the job completion time when the training task exclusively occupies the GPU, and T' is the job completion time in shared GPU mode.

2. The deep learning training task scheduling system for GPU clusters according to claim 1 is characterized in that: The batch size distribution strategy is to maintain a relatively balanced load by dynamically changing the batch size of each process of the training task to eliminate the influence of the lagging ones. That is, the offline performance characterization module first samples the <batch size, training time> of the training task to obtain the performance curve of the training task under the exclusive GPU, that is, the batch size-computation time relationship. Based on the exclusive performance curve and the performance interference prediction model, the batch size-computation time relationship when the training task runs on any GPU is derived, thereby redistributing the global batch size of the training task to ensure that the computing time of each training process is basically the same under the new batch size distribution.

3. The deep learning training task scheduling system for GPU clusters according to claim 1 is characterized in that: The offline performance characterization module deploys the training task submitted by the user on a single reserved dedicated GPU, collects the model structure feature information of the training task, uses the task similarity prediction model to judge the similarity between the training task and the existing tasks in the system database to determine the hardware indicator set to be collected, and then collects the hardware indicator data of the task during operation through a few iterative calculations; then, based on the hardware indicator data of the new training task during operation and the corresponding hardware indicator data of the training tasks on each GPU in the cluster, inputs them into the performance interference prediction model, estimates the degree of performance degradation caused by each of the two training tasks sharing the GPU, and provides guidance for the online task scheduling and deployment module to make GPU allocation decisions for new training tasks.

4. The deep learning training task scheduling system for GPU clusters according to claim 1 is characterized in that: The online task scheduling and deployment module includes: a task scheduler based on GPU affinity, an online task batch size allocation unit, and a mixed interference conduction effect correction unit, wherein: the task scheduler based on GPU affinity performs a greedy algorithm based on a nonlinear integer programming problem to obtain the target deployment GPU for the task to be processed, and the online task batch size allocation unit reallocates the batch size of the task to be scheduled, with the goal of minimizing the maximum iterative calculation time of all training processes The mixed-deployment interference conduction effect correction unit traverses the target deployment GPUs of all pending tasks based on known deployment decisions, and redistributes the batch size of running tasks that have serious load imbalance caused by deploying new training tasks to obtain a global balanced load state. The parallelism of task j, that is, the number of GPUs required, The time it takes for the i-th training process of training task j to perform one iterative calculation.

5. The deep learning training task scheduling system for GPU clusters according to claim 3 is characterized in that: The offline performance characterization module includes: a task similarity prediction unit, a hardware information collection unit, a batch size calculation time characterization unit and a performance interference prediction unit, wherein: the task similarity prediction unit traverses the calculation graph according to the training task calculation graph information submitted by the user, obtains the structural feature information of the training task, and inputs it together with the corresponding structural feature information of each completed task into the task similarity prediction model, obtains the similarity between the task to be processed and other tasks to determine the hardware collection indicator set; the hardware information collection unit deploys the task to be processed on a dedicated GPU according to the hardware collection indicator set determined by the task similarity prediction unit, and uses the NSIGHT tool to collect corresponding hardware indicators; the batch size calculation time characterization unit performs data fitting based on the corresponding calculation time of different batch sizes sampled when the task to be processed is running on the dedicated GPU, and obtains the batch size-calculation time relationship of the task; the performance interference prediction unit inputs the hardware indicator data of the training task to be processed and the hardware indicator data of the task running on the GPU into the performance interference prediction model, and obtains the degree of performance degradation of each of the two training tasks after sharing the GPU.

6. The deep learning training task scheduling system for GPU clusters according to claim 3 or 5, characterized in that: The hardware indicator data includes: SM occupancy, SM throughput, kernel running time, device memory throughput, SM active cycles, L2 cache throughput, DRAM throughput, grid size, shared memory size per block, number of registers per thread, shared memory configuration size, PCIe read data volume, PCIe write data volume, and block limit register number; The structural feature information includes: the number of floating-point operations, the number of model parameters, the number of convolutional layers, the number of normalization layers, the number of activation layers, and the input data size; The task similarity is the weighted sum of the similarity of the performance interference distribution trend and the similarity of the performance interference value when two tasks and each training task in the benchmark test set share the GPU. , is the weight coefficient of trend similarity and value similarity, trend similarity , value similarity , It represents the performance interference vector generated when model 1 shares the GPU runtime with n models in the benchmark set. Represents the performance interference vector of model 2.

7. The deep learning training task scheduling system for GPU clusters according to claim 2 or 4, characterized in that: The redistribution of the global batch size of the training task refers to: a) Obtain the iterative computation time vector of each training process using the GPU exclusive at an evenly distributed batch size using the batch size-computation time relationship obtained by the offline performance characterization module. The calculation time multiplied by the corresponding performance interference is the expected calculation time in the mixed deployment mode. , where: J is the training task to be deployed, BS is the total batch size, and p is the degree of parallelism, that is, the number of GPUs required. is the iterative computation time vector of the training task J in the current configuration using the exclusive GPU mode, is the predicted performance interference vector of each training process on the GPU to be deployed, ; b) Calculate the extreme range of iteration time for all training processes , when the extreme value range is greater than the computing time difference threshold 𝜂, the load balancing operation is performed, specifically: the batch size adjustment value samples are transferred from the training process with the longest computing time to the training process with the shortest computing time and the batch size adjustment value is gradually reduced until the adjusted extreme value range is less than the time difference threshold; c) After the deployment plan of the scheduled task J, i.e., GPU allocation and batch size allocation, is determined, the batch sizes of the affected running tasks are reallocated.

8. The deep learning training task scheduling system for GPU clusters according to claim 4 is characterized in that: The correction mentioned above refers to setting an adjustment threshold When the performance interference degree of a training process of the affected task changes, that is, the change difference of the performance interference degree of the running task caused by the deployment of the task to be scheduled exceeds the threshold, the batch size reallocation algorithm is called to reallocate the batch size of the task.

9. A method for scheduling deep learning training tasks for a GPU cluster based on the system of any one of claims 1 to 8, characterized in that: After offline data collection and performance characterization of the training tasks, the training tasks are added to the task scheduling queue and online resource allocation is performed to determine the target GPU for the training tasks to be processed and the amount of training data on each GPU.

10. The deep learning training task scheduling method according to claim 9, characterized in that: The online resource allocation refers to determining the target GPU for the training task to be processed and the amount of training data on each GPU based on the batch size-computation time relationship and the predicted performance interference, according to a scheduling algorithm based on the principle of minimizing performance interference.

Images