A method for GPU resource allocation for deep learning tasks

Abstract

The application provides a GPU resource allocation method for a deep learning task, comprising: obtaining network structure features, execution features and an available resource set corresponding to the deep learning task; predicting resource utilization of the deep learning task according to the network structure features and the execution features based on a utilization prediction model to obtain a resource utilization set; determining all work nodes satisfying resource requirements of the deep learning task from the available resource set based on the resource utilization set, selecting a work node from all the work nodes according to a predetermined selection rule to allocate resources to the deep learning task, and sending the deep learning task and its resource requirements to the selected work node. The technical scheme of the application predicts the resource requirement amount of the deep learning task, allocates the GPU resources in the container cloud cluster reasonably according to the predicted resource requirement amount, thereby realizing GPU resource sharing in the container cloud cluster and improving GPU utilization in the container cloud cluster.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of container cloud platform, in particular to a GPU resource allocation method, more particularly to a GPU resource allocation method for deep learning tasks. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technology, the computing resources of traditional CPUs (graphics processing units) have been difficult to meet the computing power requirements of deep learning application loads. Because of its stronger computing resources and higher memory bandwidth, GPU has become the mainstream accelerator for deep learning applications. One of the key problems in GPU clusters is how to schedule multiple deep learning application tasks to achieve optimal system performance. Currently, with the rapid development of cloud computing technology, cloud platforms based on Docker containerization have become one of the main infrastructure platforms for running deep learning applications. Kubernetes, as a container cluster scheduling system and resource management platform, can centrally manage CPU, memory, network and other resources in the cluster, and also supports the addition of other heterogeneous hardware resources including GPU through a device plugin framework. In the GPU scheduling and management plugin implemented by NVIDIA, Kubernetes can manage GPU resources, but can only allocate one GPU to one container group and cannot share resources among multiple container groups.

[0003] To solve the above technical problems, current GPU resources are shared among tasks based on virtual Compute Unified Device Architecture (vCUDA), that is, vCUDA cuts the GPU resources in a fine-grained manner, lets users specify the GPU usage share to limit the use of a single GPU by a container group, and thus realizes the sharing of the same GPU resource by multiple container groups. However, in task scheduling, because vCUDA uses a simple Binpack method, it ignores the characteristics of the tasks themselves, and thus easily causes over-allocation of resources, making it difficult to fully utilize GPU resources. SUMMARY

[0004] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide a GPU resource allocation method for deep learning tasks.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] According to a first aspect of the present application, a GPU resource allocation method for a deep learning task is provided, which is applied to a resource scheduling system of a multi-tenant container cloud platform, the resource scheduling system comprising a control node and a plurality of worker nodes, and the method comprises the following steps performed at the control node:

[0007] obtaining network structure features corresponding to the deep learning task, execution features corresponding to the deep learning task, and an available resource set, the available resource set comprising remaining resources of specified computing power GPUs in the plurality of worker nodes;

[0008] predicting resource utilization of the deep learning task based on a utilization prediction model according to the network structure features and the execution features, to obtain a resource utilization set, the resource utilization set comprising resource utilization of the deep learning task on a plurality of specified computing power GPUs;

[0009] determining all worker nodes satisfying resource requirements of the deep learning task from the available resource set based on the resource utilization set, and selecting a worker node from the all worker nodes satisfying the resource requirements of the deep learning task according to a predetermined selection rule to allocate resources to the deep learning task;

[0010] sending the deep learning task and its resource requirements to the selected worker node, the resource requirements indicating resource utilization of the deep learning task on a plurality of specified computing power GPUs.

[0011] In some embodiments of the present application, the predetermined selection rule is to select a worker node having the most GPU remaining resources from the all worker nodes satisfying the resource requirements of the deep learning task to allocate resources to the deep learning task.

[0012] In some embodiments of the present application, the utilization prediction model is trained in the following manner:

[0013] obtaining a first training sample set comprising a plurality of training samples, each training sample comprising hardware features corresponding to a specified computing power GPU, network structure features corresponding to a deep learning task, execution features corresponding to the deep learning task, and a label, the label comprising an indication of resource utilization of the deep learning task on the specified computing power GPU in the training sample;

[0014] training the utilization prediction model to output a resource utilization set using the first training sample set, and updating parameters of the utilization prediction model according to a loss value calculated based on the output resource utilization set and the corresponding label.

[0015] In some embodiments of the present application, the network structure features corresponding to the deep learning task include input data sizes corresponding to various graph nodes, and when there are repeated graph nodes in the network computation graph features, a graph node with the largest input data size is selected from the repeated graph nodes as the input data size corresponding to the repeated graph nodes in the training sample.

[0016] In some embodiments of the present application, the execution features include batch size, iteration number, input size, or a combination thereof.

[0017] In some embodiments of the present application, the method further comprises:

[0018] When all the current working nodes cannot meet the resource requirement of the deep learning task and the deep learning task is a delay-sensitive task, it is determined whether there is releasable resource and the releasable resource is not less than the resource allocated to the working node of the deep learning task,

[0019] If yes, the releasable resource is released and the deep learning task is allocated to the working node;

[0020] If no, the deep learning task is stored in a delay-sensitive task waiting queue, and the releasable resource is a resource allocated to a non-delay-sensitive task.

[0021] In some embodiments of the present application, the method further comprises:

[0022] When all the current working nodes cannot meet the resource requirement of the deep learning task and the deep learning task is a non-delay-sensitive task, the deep learning task is stored in a non-delay-sensitive task waiting queue.

[0023] In some embodiments of the present application, the method further comprises:

[0024] Based on the execution time prediction model, the execution time of the deep learning task on the specified computing power GPU is predicted according to the network structure features corresponding to the deep learning task, the execution features corresponding to the deep learning task, the resource utilization rate corresponding to the deep learning task on the specified computing power GPU, and the hardware features corresponding to the specified computing power GPU, to obtain a prediction execution time set for detecting whether the execution of the non-delay-sensitive task meets the preset service quality, and the prediction execution time set is transmitted to the working node allocated with the resource of the deep learning task, wherein the prediction execution time set includes the predicted execution time of the deep learning task on multiple specified computing power GPUs.

[0025] In some embodiments of the present application, the execution time prediction model is trained in the following manner:

[0026] obtaining a second training sample set, which comprises a plurality of training samples, each training sample comprising hardware features corresponding to a specified computing power GPU, network structure features corresponding to a deep learning task, execution features corresponding to the deep learning task, resource utilization corresponding to the deep learning task on the specified computing power GPU, and a label, the label comprising an indication of execution time of the deep learning task on the specified computing power GPU in the training sample;

[0027] training the execution time prediction model to output a predicted execution time using the second training sample set, and updating parameters of the execution time prediction model according to a loss value calculated based on the output predicted execution time and the corresponding label.

[0028] According to the second aspect of the present application, a GPU resource allocation method for a deep learning task is provided, which is applied to a resource scheduling system of a multi-tenant container cloud platform, the resource scheduling system comprising a control node and a plurality of worker nodes, each worker node comprising a plurality of GPUs, and the method comprising the following steps performed at each worker node:

[0029] receiving a deep learning task and resource requirements thereof sent by the control node to the current worker node according to the method of the first aspect, the resource requirements indicating resource utilization of the deep learning task on a plurality of specified computing power GPUs;

[0030] allocating a GPU and corresponding GPU resources for the deep learning task based on remaining resources of the plurality of GPUs in the current worker node and the resource requirements.

[0031] In some embodiments of the present application, when the deep learning task is a latency-sensitive task, the deep learning task is allocated to a GPU with the most remaining resources.

[0032] In some embodiments of the present application, when the deep learning task is a non-latency-sensitive task, the deep learning task is allocated to a GPU with the shortest execution time, wherein the execution time is predicted by an execution time prediction model based on network structure features corresponding to the deep learning task, resource utilization corresponding to the deep learning task on a specified computing power GPU, and hardware features corresponding to the specified computing power GPU.

[0033] In some embodiments of the present application, the method further comprises:

[0034] when the deep learning task is a latency-sensitive task, obtaining a predicted execution time of the deep learning task on the current GPU from a set of predicted execution times sent by the control node, and monitoring whether the execution of the deep learning task meets a preset quality of service based on the predicted execution time.

[0035] In some embodiments of the present application, the monitoring whether the execution of the deep learning task meets the preset service quality comprises:

[0036] obtaining the executed time and the execution progress of the deep learning task;

[0037] updating the execution feature and the network structure feature of the deep learning task according to the execution progress;

[0038] predicting the remaining execution time of the deep learning task based on the execution time prediction model according to the corresponding resource utilization of the deep learning task on the specified computing power GPU, the corresponding hardware feature of the specified computing power GPU, the updated execution feature and the network structure feature;

[0039] when the sum of the executed time and the remaining execution time is greater than the preset execution time, judging that the execution of the delay-sensitive task does not meet the preset service quality.

[0040] In some embodiments of the present application, when it is monitored that the execution of the deep learning task does not meet the preset service quality, the GPU resource of the current GPU allocated to the deep learning task is increased; if the GPU resource is insufficient when the resource allocated to the deep learning task is increased, the resource allocation to the non-delay-sensitive task on the current GPU is reduced; if the GPU resource of the deep learning task is increased, the execution resource of the non-delay-sensitive task is insufficient, the execution state of the non-delay-sensitive task is saved and the execution of the non-delay-sensitive task is ended.

[0041] In some embodiments of the present application, the method further comprises:

[0042] when it is monitored that the execution of the deep learning task meets the preset service quality, the resource utilization of the deep learning task on the current GPU is queried; if the resource utilization exceeds the predicted resource utilization of the deep learning task on the current GPU sent by the control node, the execution of the deep learning task on the current GPU is paused; when the resource utilization decreases to the predicted resource utilization, the deep learning task is executed on the current GPU again.

[0043] According to the third aspect of the present application, a resource scheduling system supporting GPU sharing is provided, comprising:

[0044] a control node configured to transmit a deep learning task and its resource requirement to a working node according to the method of the first aspect, the resource requirement indicating the resource utilization of the deep learning task on a plurality of specified computing power GPUs;

[0045] a plurality of worker nodes, each worker node configured to allocate GPUs and corresponding GPU resources for the deep learning task according to the method of the second aspect.

[0046] In some embodiments of the present application, the control node comprises:

[0047] a first prediction unit configured to predict resource utilization of the deep learning task based on the utilization prediction model according to the network structure features and the execution features, to obtain a set of resource utilizations, the set of resource utilizations comprising resource utilizations of the deep learning task on a plurality of specified computing power GPUs.

[0048] a first scheduling unit configured to determine all worker nodes satisfying resource requirements of the deep learning task from the set of available resources based on the set of resource utilizations, and select a worker node from all worker nodes satisfying the resource requirements of the deep learning task according to a predetermined selection rule to allocate resources for the deep learning task, and send the deep learning task and its resource requirements to the selected worker node, the resource requirements indicating resource utilizations of the deep learning task on a plurality of specified computing power GPUs.

[0049] In some embodiments of the present application, the control node further comprises:

[0050] a latency-sensitive task waiting queue configured to store latency-sensitive tasks when all current worker nodes cannot satisfy resource requirements of the latency-sensitive tasks;

[0051] a non-latency-sensitive task waiting queue configured to store non-latency-sensitive tasks when all current worker nodes cannot satisfy resource requirements of the non-latency-sensitive tasks;

[0052] wherein when the latency-sensitive task waiting queue is not empty, GPU resources are preferentially allocated to latency-sensitive tasks in the latency-sensitive task waiting queue.

[0053] In some embodiments of the present application, the first prediction unit is further configured to predict execution time of the deep learning task on a specified computing power GPU based on an execution time prediction model according to network structure features corresponding to the deep learning task, execution features corresponding to the deep learning task, resource utilization corresponding to the deep learning task on the specified computing power GPU, and hardware features corresponding to the specified computing power GPU, to obtain a set of predicted execution times for detecting whether execution of the non-latency-sensitive task meets a preset quality of service.

[0054] In some embodiments of the present application, the worker node comprises:

[0055] a topology awareness unit configured to obtain remaining resources of a plurality of GPUs in the current worker node;

[0056] The second prediction unit is configured to predict the execution time of the non-delay-sensitive task on the specified computing power GPU based on the execution time prediction model and the network structure features, the execution features, the resource utilization of the non-delay-sensitive task on the specified computing power GPU, and the hardware features of the specified computing power GPU, to obtain the execution time of the non-delay-sensitive task on multiple specified computing power GPUs.

[0057] The second scheduling unit is configured to allocate the GPU and the corresponding GPU resource for the deep learning task based on the remaining resources of the multiple GPUs in the current working node and the resource demand, and to allocate the GPU with the shortest execution time for the non-delay-sensitive task based on the execution time of the non-delay-sensitive task on the multiple specified computing power GPUs.

[0058] In some embodiments of the present application, the working node further comprises:

[0059] The elastic adjustment unit is configured to monitor whether the execution of the delay-sensitive task meets the preset service quality, and if the execution of the delay-sensitive task does not meet the preset service quality, increase the GPU resource of the current GPU allocated to the deep learning task, reduce the resource allocation of the non-delay-sensitive task on the current GPU if the GPU resource is insufficient when the resource allocated to the deep learning task is increased, save the execution state of the non-delay-sensitive task and end the execution of the non-delay-sensitive task if the execution resource of the non-delay-sensitive task is insufficient after the GPU resource allocated to the deep learning task is increased.

[0060] The resource limiting unit is configured to query the resource utilization of the delay-sensitive task on the current GPU if the execution of the delay-sensitive task meets the preset service quality, suspend the execution of the delay-sensitive task on the current GPU if the resource utilization of the delay-sensitive task on the current GPU exceeds the predicted resource utilization of the delay-sensitive task on the current GPU sent by the control node, and resume the execution of the delay-sensitive task on the current GPU when the resource utilization decreases to the predicted resource utilization.

[0061] Compared with the prior art, the present application has the following advantages:

[0062] By predicting the resource demand of the deep learning task, the GPU resources in the container cloud cluster are reasonably allocated according to the predicted resource demand, so as to realize the sharing of the GPU resources in the container cloud cluster and improve the GPU utilization in the container cloud cluster. BRIEF DESCRIPTION OF DRAWINGS

[0063] The embodiments of the present application will be further described below with reference to the drawings, in which:

[0064] Figure 1 A flowchart of a GPU resource allocation method for a deep learning task according to an embodiment of the present application implemented on a control node;

[0065] Figure 2 A DAG graph of a network structure of a deep learning task according to an embodiment of the present application;

[0066] Figure 3 A flowchart of a GPU resource allocation method for a deep learning task according to an embodiment of the present application implemented on a worker node;

[0067] Figure 4 A schematic diagram of a resource scheduling system based on a Kubernetes platform according to an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose of the present application clearer, the technical solutions and advantages are described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] As mentioned in the background section, in order to realize the sharing of GPU resources in multiple container groups, the current GPU resource sharing between tasks is based on virtual Compute Unified Device Architecture (vCUDA), but due to the simple Binpack method used in task scheduling, the characteristics of the tasks are ignored, resulting in over-allocation of resources and the problem that GPU resources cannot be fully utilized. To solve the above technical problems, the present application provides a GPU resource allocation method for deep learning tasks, which predicts the resource demand of deep learning tasks to allocate GPU resources in a container cloud cluster according to the predicted resource demand, thereby realizing the sharing of GPU resources in the container cloud cluster and improving the GPU utilization in the container cloud cluster.

[0070] In order to realize the present application, the inventors have made corresponding improvements to the control node and the worker node of the resource scheduling system applied to the Kubernetes container cloud platform, which will be described below from three aspects: (1) improvement of the control node, (2) improvement of the worker node, and (3) resource scheduling system.

[0071] (1) Improvement of the control node

[0072] According to one embodiment of the present application, the present application provides a GPU resource allocation method for a deep learning task, applied to a resource scheduling system of a multi-tenant container cloud platform, wherein the resource scheduling system comprises one control node and a plurality of working nodes, and the method comprises executing steps A1, A2, A3 and A4 shown in the following table at the control node. Figure 1 In order to better understand the present application, each step will be described in detail below in combination with specific embodiments.

[0073] Step A1: obtaining network structure features corresponding to the deep learning task, execution features corresponding to the deep learning task, and a set of available resources;

[0074] The network structure features corresponding to the deep learning task include various computation graph nodes and input data sizes (number of input data of the computation graph nodes) corresponding to the various computation graph nodes, and the computation graph nodes at least include AddV2, BiasAdd, BiasAddGrad, Mul, Conv2D, Concat, SoftMax, ReLU, MaxPool, AvgPool, or a combination thereof. Since each deep learning task is different, the computation graph nodes and the corresponding input data sizes covered by each deep learning task are also different. In order to enable the utilization rate prediction model to perform corresponding resource prediction according to the network structure features of different deep learning tasks, according to one embodiment of the present application, a network structure feature table is constructed in advance according to the computation graph nodes, and then when the network structure features corresponding to the deep learning task are subsequently acquired, the input data sizes acquired are filled into the corresponding computation graph nodes in the network structure feature table. For example, the preset network structure feature table is: {AddV2(0), Conv2D(0), MaxPool(0), ReLU(0), MatMul(0), AvgPool(0), Mul(0), Concat(0), SoftMax(0)}, wherein the content before the parentheses is the name of the computation graph node, and the content in the parentheses is the input data size of the computation graph node. If the graph nodes included in the deep learning task a are AddV2, Conv2D, MaxPool, MatMul, and SoftMax, and the input data sizes corresponding to the various computation graph nodes are 2000, 4800, 3200, 4200, and 100 respectively, then the input data sizes are filled into the corresponding computation graph nodes to obtain the network structure features of the deep learning task a: {AddV2(2000), Conv2D(4800), MaxPool(3200), ReLU(0), MatMul(4200), AvgPool(0), Mul(0), Concat(0), SoftMax(100)}. Further, considering that the repeated computation graph node with the largest input data size has a greater impact on the utilization rate prediction, when a repeated computation graph node appears in a deep learning task, the largest input data size in the repeated computation graph node is filled into the corresponding computation graph node in the network structure feature table.For example, the deep learning task b includes three repeated Conv2D nodes, and the input data sizes are 2400, 2800 and 4800 respectively; the remaining nodes are AddV2, MaxPool, ReLU and Concat, and the input data sizes are 2400, 4800, 1000 and 100 respectively. Since the maximum input data size in the Conv2D node is 4800, 4800 is filled in the Conv2D of the network structure feature table, and the network structure feature corresponding to the deep learning task b is {AddV2(2400), Conv2D(4800), MaxPool(4800), ReLU(1000), MatMul(0), AvgPool(0), Mul(0), Concat(100), SoftMax(100)}.

[0075] The execution features include batch size, iteration number, input size or a combination thereof. Since each deep learning task is different, the data corresponding to the batch size, iteration number and input size is also different. In order to enable the utilization prediction model to perform corresponding resource prediction according to the execution features of different deep learning tasks, according to an embodiment of the present application, an execution feature table is constructed in advance according to the execution features. When the execution features corresponding to the deep learning task are obtained subsequently, the obtained data can be filled in the corresponding nodes of the execution feature table. For example, the preset execution feature table is {batchsize(0), iteration(0), input(0)}, where the contents before the brackets are batch size, iteration number and input size respectively, and the contents in the brackets are corresponding parameters. If the parameters of the batch size, iteration number and input size of the deep learning task a are 32, 2000 and 1000 respectively, the execution features of the deep learning task a are represented as {batchsize(32), iteration(2000), input(1000)}.

[0076] According to an embodiment of the present application, considering that the resource scheduling system has multiple worker nodes, each worker node has multiple GPUs, and the GPUs have different computing power due to different specification parameters, in order to facilitate subsequent allocation of appropriate GPU resources for the deep learning task, when the available resource set is obtained, in addition to the remaining resources of each GPU in the worker node, the computing power of each GPU also needs to be counted. In the embodiment of the present application, the available resource set includes the remaining resources of the specified computing power GPU in the multiple worker nodes.

[0077] Step A2: predicting the resource utilization of the deep learning task according to the network structure features and the execution features based on the utilization prediction model to obtain a resource utilization set; according to an embodiment of the present application, considering that the computing power of the GPU has differences, the resource utilization of the same deep learning task on different computing power GPUs is inconsistent, therefore, in order to reasonably allocate the corresponding GPU resources for the deep learning task, the deep learning task needs to be predicted on all computing power GPUs to obtain the resource utilization of the deep learning task on all computing power GPUs, that is, in the embodiment of the present application, the resource utilization set includes the resource utilization of the deep learning task on multiple specified computing power GPUs.

[0078] According to one embodiment of the present application, the utilization prediction model comprises a GPU computing resource utilization prediction model and a storage resource utilization prediction model, wherein the GPU computing resource utilization prediction model is used to predict the computing resource utilization of a deep learning task on a plurality of specified computing power GPUs, so that the GPUs allocate corresponding computing resources to the deep learning task; the storage resource utilization prediction model is used to predict the storage resource utilization of a deep learning task on a plurality of specified computing power GPUs, so that the GPUs allocate corresponding storage resources to the deep learning task. According to one embodiment of the present application, the GPU computing resource utilization prediction model and the storage resource utilization prediction model are consistent in training samples, training models and training methods except that the labels are different (the label of the GPU computing resource utilization prediction model is an indication of the computing resource utilization of a deep learning task on a specified computing power GPU in a training sample; the label of the storage resource utilization prediction model is an indication of the storage resource utilization of a deep learning task on a specified computing power GPU in a training sample), therefore, the training process of the two models is described below taking the GPU computing resource utilization prediction model as an example. According to one embodiment of the present application, the GPU computing resource utilization prediction model is trained in the following manner: a first training sample set is obtained, which comprises a plurality of training samples, each training sample comprising hardware features corresponding to a specified computing power GPU, network structure features corresponding to a deep learning task, execution features corresponding to the deep learning task, and a label, the label comprising an indication of the computing resource utilization of the deep learning task on the specified computing power GPU in the training sample; the utilization prediction model is trained to output a computing resource utilization set using the first training sample set, and the parameters of the GPU computing resource utilization prediction model are updated according to the loss value calculated based on the output computing resource utilization set and the corresponding label. According to one embodiment of the present application, the GPU computing resource utilization prediction model is trained based on a multi-layer perceptron using a back propagation algorithm, and the cost function is set to mean square error or mean absolute error. According to one embodiment of the present application, the hardware features in the training sample comprise the computing power of the GPU, the memory size of the GPU, and the number of computing cores of the GPU, which can be directly obtained by checking the performance parameters of the GPU. In order to facilitate the model to identify the hardware features of the GPU, the hardware features of the GPU in the training sample are represented in a preset format, for example, it can be represented as: {capability(), memory(), cores()}, wherein the content before the parentheses represents the computing power of the GPU, the memory size (in gigabytes (Giga Byte, GB)) of the GPU, and the number of computing cores of the GPU, and the content in the parentheses is the corresponding performance parameter, for example, the hardware features of a GPU can be represented as: {capability(7), memory(32), cores(2880)}.According to one embodiment of the present application, the network structure features corresponding to the deep learning task in the training sample are described by a directed acyclic graph (DAG) through a TensorFlow program, as shown in the following. Figure 2As shown, the DAG node structure is used as the network model feature of the deep learning task. Considering that each deep learning task is different, the computational graph nodes covered by each deep learning task and the corresponding input data size are also different, in order to facilitate the model to identify the network structure feature, the network structure feature in the training sample is expressed in a predetermined format, for example, it can be expressed as: {AddV2(0), Conv2D(0), MaxPool(0), ReLU(0), MatMul(0), AvgPool(0), Mul(0), Concat(0), SoftMax(0)}, wherein the content before the bracket is the name of the computational graph node, and the content in the bracket is the input data size corresponding to the computational graph node. If the deep learning task a includes the computational graph nodes AddV2, Conv2D, MaxPool, MatMul and SoftMax, and the input data sizes corresponding to each computational graph node are 2000, 4800, 3200, 4200 and 100 respectively, then the network structure feature of the deep learning task a is expressed as: {AddV2(2000), Conv2D(4800), MaxPool(3200), ReLU(0), MatMul(4200), AvgPool(0), Mul(0), Concat(0), SoftMax(100)}. Further, considering that there are repeated computational graph nodes in the network structure feature, and the repeated node with the largest input data size has a greater impact on the utilization prediction, therefore, when a repeated computational graph node appears in a deep learning task, the largest input data size in the repeated computational graph node is filled into the corresponding computational graph node in the network structure feature table. For example: the deep learning task b includes 3 repeated computational graph nodes Conv2D, and the input data sizes are 2400, 2800 and 4800 respectively; the remaining computational graph nodes are AddV2, MaxPool, ReLU and Concat, and the input data sizes are 2400, 4800, 1000 and 100 respectively. Since the largest input data size in the computational graph node Conv2D is 4800, 4800 is filled into the Conv2D in the network structure feature table, and the network structure feature corresponding to the deep learning task b is: {AddV2(2400), Conv2D(4800), MaxPool(4800), ReLU(1000), MatMul(0), AvgPool(0), Mul(0), Concat(100), SoftMax(100)}.According to an embodiment of the present application, the execution features include batch size, iteration number and input size, in order to facilitate the model to identify the execution features, the execution features in the training sample are expressed in a preset format, for example, it can be expressed as: {batchsize(0), iteration(0), input(0)}, wherein the content before the bracket represents batch size, iteration number and input size respectively, and the content in the bracket is the corresponding parameter. For example, the parameters of batch size, iteration number and input size of a deep learning task a are 32, 2000 and 1000 respectively, and the execution features of the deep learning task a are expressed as: {batchsize(32), iteration(2000), input(1000)}. In summary, in an embodiment of the present application, each training sample is represented by data in three dimensions of network structure features, execution features and GPU hardware features, that is, the training sample is represented as:

[0079] P = {i_dag, i_exec, i_gpu};

[0080] Wherein idag, iexec and igpu represent the key information of the network computation graph features, the execution features and the hardware features respectively. For example, the network computation graph features of a training sample are expressed as: i_dag = {AddV2(2000), Conv2D(4800), MaxPool(3200), ReLU(1000), MatMul(4200), AvgPool(0), Mul(0), Concat(100), SoftMax(100)}; the execution features are expressed as: idag = {batchsize(32), iteration(2000), input(1000)}; the hardware features are expressed as igpu = {capability(7), memory(32), cores(2880)}, and P is expressed as: P = {AddV2(2000), Conv2D(4800), MaxPool(3200), ReLU(1000), MatMul(4200), AvgPool(0), Mul(0), Concat(100), SoftMax(100), batchsize(32), iteration(2000), input(1000), capability(7), memory(32), cores(2880)}.

[0081] It is worth noting that the order of each computing graph node in the network computing graph feature, the order of each feature in the execution feature, the order of each feature in the hardware feature, and the order of each parameter in the training sample are only an adaptive description, and the implementer can make corresponding adjustments according to the specific circumstances of actual application, and the present application does not make any limitation.

[0082] Step A3: determining all working nodes satisfying the resource requirement of the deep learning task from the available resource set based on the resource utilization set, and selecting a working node from all working nodes satisfying the resource requirement of the deep learning task according to a predetermined selection rule to allocate resources for the deep learning task;

[0083] Since the available resource set includes the remaining resources of each GPU in each worker node, and the resource utilization set includes the resource utilization of the deep learning task on various computing power GPUs, when the resource utilization of the deep learning task on various computing power GPUs is used to match the worker nodes that meet the corresponding resource requirements, there may be multiple worker nodes that match. When allocating resources for the deep learning task, only one worker node is needed to allocate GPU resources for the deep learning task. Therefore, a predetermined selection rule is needed to select one worker node from all worker nodes that meet the resource requirements of the deep learning task to allocate resources for the deep learning task. Considering that the GPU resources allocated to the deep learning task can be increased according to the execution of the deep learning task, according to an embodiment of the present application, the predetermined selection rule is set as follows: from all worker nodes that meet the resource requirements of the deep learning task, select the worker node with the most GPU remaining resources to allocate resources for the deep learning task. For example, the resource utilization set corresponding to the deep learning task d is: {40% of the resources of A1 computing power GPU, 50% of the resources of A2 computing power GPU, 70% of the resources of A3 computing power GPU}, and the available resource set is: {worker node a (90% of the remaining resources of A1 computing power GPU, 100% of the remaining resources of A1 computing power GPU, 30% of the remaining resources of A1 computing power GPU), worker node b (90% of the remaining resources of A1 computing power GPU, 40% of the remaining resources of A2 computing power GPU, 80% of the remaining resources of A3 computing power GPU), worker node c (10% of the remaining resources of A1 computing power GPU, 20% of the remaining resources of A2 computing power GPU, 50% of the remaining resources of A3 computing power GPU)}, thus it can be known that both worker node a and worker node b can allocate corresponding GPU resources to the deep learning task d. However, since there is one A1 computing power GPU with 100% remaining resources in worker node a, which is more than the remaining resources of A1 computing power GPU and A3 computing power GPU in worker node b, the deep learning task d is sent to worker node a to facilitate the subsequent increase of GPU resources allocated to the deep learning task d according to the execution of the deep learning task d. In addition, if the remaining resources of the GPU used to allocate resources for the deep learning task in multiple worker nodes are the same, a worker node is randomly selected to allocate the deep learning task or the allocation is performed according to the distribution order of the worker nodes in the resource scheduling system.For example, the resource utilization set corresponding to the deep learning task e is: {A1, 30% of the resources of the GPU, A2, 50% of the resources of the GPU, A3, 90% of the resources of the GPU}, the available resource set is: {work node a (A1, 0% of the remaining resources of the GPU, A1, 20% of the remaining resources of the GPU, A1, 80% of the remaining resources of the GPU), work node b (A1, 0% of the remaining resources of the GPU, A1, 30% of the remaining resources of the GPU, A3, 80% of the remaining resources of the GPU), work node c (A1, 20% of the remaining resources of the GPU, A2, 80% of the remaining resources of the GPU, A3, 80% of the remaining resources of the GPU)}, it can be known that the three work nodes can all allocate GPU resources for the deep learning task and the GPU that can allocate GPU resources for the deep learning task has the same maximum remaining resources, which is 80%, therefore the deep learning task is randomly allocated to one of the work nodes or the deep learning task is allocated to work node a according to the distribution order of the work nodes in the resource scheduling system.

[0084] It should be explained that the selection rule in the scheme is not unique, the embodiment of the present application is only an illustrative description, and the implementer can make corresponding adjustments according to the specific situation of the actual application, for example, all work nodes that meet the resource requirements of the deep learning task can be scored based on a predetermined scoring rule, the work node with the highest score is selected from all the work nodes to allocate resources for the deep learning task, and the present application does not make any limitation thereto.

[0085] According to one embodiment of the present application, if all the working nodes cannot meet the resource requirement of the deep learning task in the allocation process, and the deep learning task is a delay-sensitive task, it is determined whether there is a releasable resource which is not less than the resource allocated to the working node of the deep learning task, if yes, the releasable resource is released and the deep learning task is allocated to the working node; if no, the deep learning task is stored in the delay-sensitive task waiting queue, wherein the releasable resource is the resource allocated to the non-delay-sensitive task. Since the delay-sensitive task has a higher requirement for timeliness and cannot tolerate long delay, and the non-delay-sensitive task has no requirement for timeliness and does not need to guarantee its execution speed, when there is no GPU resource allocated to the delay-sensitive task, by checking whether the GPU resource is occupied by the non-delay-sensitive task and releasing the resource of the non-delay-sensitive task when the GPU resource is occupied by the non-delay-sensitive task, the resource is preferentially supplied to the delay-sensitive task to improve the service quality of the delay-sensitive task. For example: the resource utilization rate set corresponding to the deep learning task f is: {A1 GPU resource of 35%, A2 GPU resource of 45%, A3 GPU resource of 65%}, the available resource set is: {working node a (A1 GPU remaining resource 0%, A1 GPU remaining resource 0%, A1 GPU remaining resource 0%), working node b (A1 GPU remaining resource 0%, A1 GPU remaining resource 30%, A3 GPU remaining resource 50%), working node c (A1 GPU remaining resource 20%, A2 GPU remaining resource 20%, A3 GPU remaining resource 40%)}, it can be seen that working node a, working node b and working node c cannot allocate corresponding resources to deep learning task c, but A1 GPU of working node c allocates 40% of the resource to the non-delay-sensitive task, after recycling the resource allocated to the non-delay-sensitive task, the available resource of A1 GPU of working node c becomes 60%, so the deep learning task can be sent to working node c after recycling the resource allocated to the non-delay-sensitive task. According to one embodiment of the present application, if all the working nodes cannot meet the resource requirement of the deep learning task, and the deep learning task is a non-delay-sensitive task, the deep learning task is stored in the non-delay-sensitive task waiting queue in the control node, and is rescheduled when there is an idle resource.

[0086] Step A4: sending the deep learning task and its resource requirement to the selected working node, the resource requirement indicating the resource utilization rate of the deep learning task on the specified multiple power GPUs.

[0087] Further, considering that concurrently executing multiple services on a single GPU increases the latency of a single task, affecting the execution of the task. In deep learning tasks, many latency-sensitive tasks have quality of service requirements, which have high requirements for timeliness and cannot tolerate long delays. Therefore, in resource allocation, not only the resource utilization of the task needs to be considered, but also the quality of service of the task needs to be met. According to an embodiment of the present application, in order to monitor the quality of service of the latency-sensitive task, the execution time of the latency-sensitive task is also predicted, so as to subsequently increase the GPU resources allocated to the latency-sensitive task according to the execution time of the latency-sensitive task, preventing the problem of low service quality caused by insufficient GPU resource allocation. According to an embodiment of the present application, based on the execution time prediction model, the execution time of the latency-sensitive task on the specified computing power GPU is predicted according to the network structure feature corresponding to the latency-sensitive task, the execution feature corresponding to the latency-sensitive task, the resource utilization of the latency-sensitive task on the specified computing power GPU, and the hardware feature corresponding to the specified computing power GPU, to obtain a set of predicted execution times, and the set of predicted execution times is transmitted to the worker node for allocating resources to the latency-sensitive task. The set of predicted execution times includes predicted execution times of the latency-sensitive task on multiple specified computing power GPUs. According to an embodiment of the present application, the execution time prediction model is trained in the following manner: a second training sample set is obtained; it includes multiple training samples, each training sample includes the hardware feature corresponding to the specified computing power GPU, the network structure feature corresponding to the latency-sensitive task, the execution feature corresponding to the latency-sensitive task, the resource utilization of the latency-sensitive task on the specified computing power GPU, and a label, the label includes an indication of the execution time of the latency-sensitive task on the specified computing power GPU in the training sample; the execution time prediction model is trained using the second training sample set to output a predicted execution time, and the parameters of the execution time prediction model are updated according to the loss value calculated from the output predicted execution time and the corresponding label. The processing method of the hardware feature corresponding to the GPU, the network structure feature corresponding to the latency-sensitive task, and the execution feature corresponding to the latency-sensitive task is described in the utilization prediction model, so this embodiment will not be described.

[0088] (II) Improvement of the worker node

[0089] According to an embodiment of the present application, the present application provides a GPU resource allocation method for deep learning tasks, applied to a resource scheduling system of a multi-tenant container cloud platform. The resource scheduling system includes a control node and multiple worker nodes, each worker node including multiple GPUs. The method includes executing steps B1 and B2 as shown in the following figure at each worker node. Figure 3 In order to better understand the present application, each step will be described in detail below in conjunction with specific embodiments.

[0090] Step B1: the receiving control node sends a deep learning task and its resource requirement to the current working node according to the method of the foregoing embodiment, wherein the resource requirement indicates the resource utilization of the deep learning task on multiple specified computing power GPUs;

[0091] Step B2: based on the remaining resources of the multiple GPUs in the current working node and the resource requirement, a GPU and corresponding GPU resources are allocated to the deep learning task.

[0092] Since each GPU in a single working node has different computing power and remaining resources, the working node also needs to allocate a corresponding GPU and GPU resources to the deep learning task according to the remaining resources of the GPU. For example, the resource requirement of the deep learning task g is 20% of the resources of an A1 computing power GPU, and the working node has four A1 computing power GPUs, each of which has remaining resources of 50%, 10%, 0%, and 0%. The working node then needs to allocate the deep learning task g to the GPU with 50% remaining resources and allocate 20% of the resources to the deep learning task g, thereby achieving reasonable allocation of resources. When there are multiple GPUs in the current working node that can allocate resources to the deep learning task, in order to facilitate subsequent dynamic adjustment of the amount of resources allocated to the latency-sensitive task, the deep learning task is preferably allocated to the GPU with the most remaining resources. For example, the resource requirement of the deep learning task is 30% of the resources of an A1 computing power GPU, and the working node has four A1 computing power GPUs, each of which has remaining resources of 80%, 40%, 100%, and 10%. The working node then allocates the deep learning task to the GPU with 100% remaining resources and allocates 30% of the resources to the deep learning task.

[0093] According to an embodiment of the present application, in order to ensure that there are enough GPU resources that can be allocated to the latency-sensitive task, and also to reduce the waiting time of the latency-sensitive task in the latency-sensitive waiting queue, and to ensure the execution efficiency of the latency-sensitive task, when the deep learning task is a non-latency-sensitive task, the deep learning task is allocated to the GPU with the shortest execution time. The execution time is predicted based on the execution time prediction model according to the network structure characteristics corresponding to the non-latency-sensitive task, the execution characteristics corresponding to the non-latency-sensitive task, the resource utilization of the non-latency-sensitive task on the specified computing power GPU, and the hardware characteristics corresponding to the specified computing power GPU. Since the execution time prediction model in the present embodiment is the same as the execution time prediction model in the control node, the relevant training or processing process can be referred to the training or processing process of the execution time prediction model in the control node, which will not be described herein.

[0094] In view of the fact that many delay-sensitive tasks have quality of service requirements in deep learning tasks, in order to meet the quality of service requirements of the delay-sensitive tasks, it is also necessary to monitor whether the execution of the delay-sensitive tasks meets the preset quality of service. According to an embodiment of the present application, monitoring whether the execution of the deep learning task meets the preset quality of service comprises:

[0095] obtaining the executed time and the execution progress of the delay-sensitive task;

[0096] updating the execution feature and the network structure feature corresponding to the delay-sensitive task according to the execution progress;

[0097] For example, the total iteration number of a delay-sensitive task is 100, that is, iteration(100), after executing for a period of time, the delay-sensitive task has executed 60 times, and there are still 40 times to be executed, then the corresponding iteration number is updated, that is, iteration(100) is updated to iteration(40). It should be noted that updating the execution feature and the network structure feature corresponding to the delay-sensitive task according to the execution progress is prior art, and the present application does not involve improvement thereof, so the specific updating process and principle will not be described in detail.

[0098] predicting the remaining execution time of the deep learning task based on the execution time prediction model according to the resource utilization rate corresponding to the deep learning task on the specified computing power GPU, the hardware features corresponding to the specified computing power GPU, the updated execution feature and the network structure feature;

[0099] When the sum of the executed time and the remaining execution time is greater than the preset execution time, it is judged that the execution of the delay-sensitive task does not meet the preset quality of service. According to an embodiment of the present application, the preset execution time is the predicted execution time of the deep learning task on the current GPU sent by the control node.

[0100] For example, the preset execution time of the time-sensitive task is 2 min, and the task has been executed for 1 min 30 s. The execution time prediction model predicts that it will take another 1 min to complete the execution according to the current execution condition, that is, the actual execution time of the task is 2 min 30 s under the current execution condition, which exceeds the preset execution time 2 min, indicating that the execution state of the task is abnormal and the preset service quality is not met. Resources allocated to the task need to be increased to shorten the unexecuted time to meet the preset service quality. It is worth noting that when the resources allocated to the task are increased, the increase in resources can be achieved by pre-setting an increase step, for example, the preset increase step is 10%. If the preset service quality is not met, 10% of the GPU resources of the current GPU are allocated to the task. If the preset service quality is still not met after increasing 10% of the GPU resources, 10% of the GPU resources of the current GPU are allocated to the task until there is no remaining resource of the current GPU to allocate to the task. According to an embodiment of the present application, if the GPU resource is insufficient when the resource allocated to the deep learning task is increased, the resource allocation to the non-time-sensitive task on the GPU is reduced. Since the non-time-sensitive task can tolerate a longer delay, more resources can be allocated to the time-sensitive task by releasing the resources of the non-time-sensitive task. If the execution resource of the non-time-sensitive task is insufficient after the resource allocated to the deep learning task is increased, the execution state of the non-time-sensitive task is saved and the execution of the non-time-sensitive task is ended, and the non-time-sensitive task is put into the non-sensitive waiting queue. When there is a free GPU resource on the current GPU, the non-time-sensitive task is rescheduled for execution. According to an embodiment of the present application, when it is monitored that the execution of the deep learning task meets the preset service quality, the resource utilization rate of the deep learning task on the current GPU is queried. If the resource utilization rate exceeds the predicted resource utilization rate of the deep learning task on the current GPU sent by the control node, the execution of the deep learning task on the current GPU is paused. When the resource utilization rate decreases to the predicted resource utilization rate, the deep learning task is executed on the current GPU again.

[0101] (III) Resource scheduling system

[0102] According to an embodiment of the present application, the present application provides a resource scheduling system based on a Kubernetes platform, as shown in Figure 4 , comprising:

[0103] The control node is configured to transmit the deep learning task and its resource requirement to the worker node according to the method of the corresponding embodiment in the improvement of (I) the control node. The resource requirement indicates the resource utilization rate of the deep learning task on the specified computing power GPU.

[0104] a plurality of worker nodes, each worker node configured to allocate GPUs and corresponding GPU resources for a deep learning task according to the method of the corresponding embodiment in (ii) improvement of the worker node.

[0105] According to an embodiment of the present application, the control node comprises:

[0106] a first prediction unit configured to predict resource utilization of the deep learning task according to network structure features and execution features based on a utilization prediction model, to obtain a resource utilization set, the resource utilization set comprising resource utilization of the deep learning task on a plurality of specified computing power GPUs;

[0107] a first scheduling unit configured to determine all worker nodes satisfying resource requirements of the deep learning task from the available resource set based on the resource utilization set, and select a worker node from all worker nodes satisfying the resource requirements of the deep learning task according to a predetermined selection rule to allocate resources for the deep learning task, and send the deep learning task and its resource requirements to the selected worker node, the resource requirements indicating resource utilization of the deep learning task on a plurality of specified computing power GPUs.

[0108] According to an embodiment of the present application, the control node further comprises:

[0109] a latency-sensitive task waiting queue for storing the latency-sensitive task when all current worker nodes cannot satisfy the resource requirements of the latency-sensitive task;

[0110] a non-latency-sensitive task waiting queue for storing the non-latency-sensitive task when all current worker nodes cannot satisfy the resource requirements of the non-latency-sensitive task;

[0111] wherein when the latency-sensitive task waiting queue is not empty, the GPU resources are preferentially allocated to the latency-sensitive task in the latency-sensitive task waiting queue.

[0112] According to an embodiment of the present application, the first prediction unit is further configured to:

[0113] predict execution time of the deep learning task on the specified computing power GPU based on the network structure features corresponding to the deep learning task, the execution features corresponding to the deep learning task, the resource utilization corresponding to the deep learning task on the specified computing power GPU, and the hardware features corresponding to the specified computing power GPU based on an execution time prediction model, to obtain a prediction execution time set for detecting whether the execution of the non-latency-sensitive task meets the preset quality of service.

[0114] According to an embodiment of the present application, the worker node comprises:

[0115] a topology awareness unit configured to obtain remaining resources of a plurality of GPUs in the current worker node;

[0116] a second prediction unit configured to predict execution time of the non-delay-sensitive task on the specified computing power GPU based on the execution time prediction model and the network structure feature corresponding to the non-delay-sensitive task, the execution feature corresponding to the non-delay-sensitive task, the resource utilization rate of the non-delay-sensitive task on the specified computing power GPU, and the hardware feature corresponding to the specified computing power GPU, to obtain execution time of the non-delay-sensitive task on a plurality of specified computing power GPUs;

[0117] a second scheduling unit configured to allocate GPUs and corresponding GPU resources for the deep learning task based on the remaining resources and resource requirements of the plurality of GPUs in the current working node, and to allocate a GPU with the shortest execution time for the non-delay-sensitive task based on the execution time of the non-delay-sensitive task on a plurality of specified computing power GPUs.

[0118] According to an embodiment of the present application, the working node further comprises:

[0119] an elastic adjustment unit configured to monitor whether the execution of the delay-sensitive task meets the preset service quality, and to increase GPU resources of the current GPU allocated to the deep learning task when the execution of the delay-sensitive task does not meet the preset service quality; to reduce resource allocation to the non-delay-sensitive task on the current GPU when GPU resources are insufficient after increasing the resource allocation to the deep learning task; and to save the execution state of the non-delay-sensitive task and end the execution of the non-delay-sensitive task when the non-delay-sensitive task is insufficient in execution resources after increasing the GPU resources allocated to the deep learning task.

[0120] a resource limiting unit configured to query resource utilization rate of the delay-sensitive task on the current GPU when the execution of the delay-sensitive task meets the preset service quality, to suspend execution of the delay-sensitive task on the current GPU when the resource utilization rate of the delay-sensitive task on the current GPU exceeds the predicted resource utilization rate of the delay-sensitive task on the current GPU sent by the control node, and to resume execution of the delay-sensitive task on the current GPU when the resource utilization rate drops to the predicted resource utilization rate.

[0121] The prior art specifies the use share of the task in the GPU by the user, ignores the characteristics of the task itself, and has the problem that the GPU resources cannot be fully utilized due to over-allocation of resources. Based on this, in order to overcome the above technical problems, the resource demand of the deep learning task is predicted, and the GPU resources in the container cloud cluster are reasonably allocated according to the predicted resource demand, so that the GPU resources are reasonably allocated according to the characteristics of different tasks, the user does not need to specify the resources required by the task, the problem of over-allocation of resources is avoided, and the GPU utilization in the container cloud cluster is improved while realizing the sharing of the GPU resources in the container cloud cluster. In addition, the execution time of the task is predicted through the execution time prediction module, so that the execution state of the delay-sensitive task can be monitored in real time, and the resource allocation of the delay-sensitive task is adjusted in time according to the execution state, so as to ensure the service quality of the user's submitted task.

[0122] It should be noted that although the above describes the steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of the steps can be performed concurrently, or even in a different order, as long as the desired function can be achieved.

[0123] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0124] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or punched tape, and any suitable combination of the foregoing. A non-transitory, computer-readable storage medium does not include a signal.

[0125] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments covered by the claims. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the spirit and scope of the described embodiments. It is intended that the scope of the application should only be limited by the appended claims.

Claims

1. A GPU resource allocation method for a deep learning task, applied to a resource scheduling system of a multi-tenant container cloud platform, the resource scheduling system comprising a control node and a plurality of worker nodes, characterized in that, The method comprises the following steps performed at a control node: obtaining network structure features corresponding to a deep learning task, execution features corresponding to the deep learning task, and a set of available resources, the set of available resources comprising remaining resources of GPUs in a plurality of worker nodes, the network structure features corresponding to the deep learning task comprising various computation graph nodes and input data sizes corresponding to the various computation graph nodes, the execution features comprising batch size, iteration number, input size, or a combination thereof; predicting resource utilization of the deep learning task based on a utilization prediction model according to the network structure features and the execution features to obtain a set of resource utilizations, the set of resource utilizations comprising resource utilizations of the deep learning task on a plurality of specified GPUs, wherein the utilization prediction model is trained in the following manner: obtaining a first set of training samples comprising a plurality of training samples, each training sample comprising hardware features corresponding to a specified GPU, network structure features corresponding to a deep learning task, execution features corresponding to the deep learning task, and a label, the label comprising an indication of resource utilization of the deep learning task on the specified GPU in the training sample; training the utilization prediction model to output a set of resource utilizations using the first set of training samples, updating parameters of the utilization prediction model according to a loss value calculated based on the output set of resource utilizations and the corresponding labels; determining all worker nodes satisfying resource requirements of the deep learning task from the set of available resources based on the set of resource utilizations, and selecting a worker node from the all worker nodes satisfying the resource requirements of the deep learning task according to a predetermined selection rule to allocate resources to the deep learning task; sending the deep learning task and its resource requirements to the selected worker node, the resource requirements indicating resource utilizations of the deep learning task on a plurality of specified GPUs.

2. The method of claim 1, wherein, The predetermined selection rule is to select a worker node having the most remaining GPU resources from the all worker nodes satisfying the resource requirements of the deep learning task to allocate resources to the deep learning task.

3. The method of claim 1, wherein, When there are repeated computation graph nodes in the network structure features, selecting a computation graph node having the largest input data size from the repeated computation graph nodes as an input data size corresponding to the repeated computation graph node in the training sample.

4. The method according to one of claims 1 to 3, characterized in that, The method further comprises: when none of the current all worker nodes can satisfy the resource requirements of the deep learning task and the deep learning task is a latency-sensitive task, determining whether there is releasable resource and the releasable resource is not less than the resource allocated to the worker node for the deep learning task, if yes, releasing the releasable resource and allocating the deep learning task to the worker node; if no, storing the deep learning task in a latency-sensitive task waiting queue, the releasable resource being a resource allocated to a non-latency-sensitive task.

5. The method according to one of claims 1 to 3, characterized in that The method further comprises: When all the current worker nodes cannot meet the resource requirement of the deep learning task and the deep learning task is a non-delay-sensitive task, the deep learning task is stored in a non-delay-sensitive task waiting queue.

6. The method of claim 4, wherein, The method further comprises: predicting, based on an execution time prediction model, the execution time of the deep learning task on the specified computing power GPU according to the network structure feature corresponding to the deep learning task, the execution feature corresponding to the deep learning task, the resource utilization rate corresponding to the deep learning task on the specified computing power GPU, and the hardware feature corresponding to the specified computing power GPU, to obtain a predicted execution time set for detecting whether the execution of the non-delay-sensitive task meets the preset service quality, and transmitting the predicted execution time set to the worker node allocated with the resource of the deep learning task, wherein the predicted execution time set comprises predicted execution times of the deep learning task on multiple specified computing power GPUs.

7. The method of claim 6, wherein, The execution time prediction model is trained in the following manner: obtaining a second training sample set, which comprises multiple training samples, each training sample comprising a hardware feature corresponding to a specified computing power GPU, a network structure feature corresponding to a deep learning task, an execution feature corresponding to the deep learning task, a resource utilization rate corresponding to the deep learning task on the specified computing power GPU, and a label, wherein the label comprises an indication of the execution time of the deep learning task on the specified computing power GPU in the training sample; training the execution time prediction model to output a predicted execution time using the second training sample set, and updating the parameters of the execution time prediction model according to the loss value calculated based on the output predicted execution time and the corresponding label. 8.A GPU resource allocation method for a deep learning task, applied to a resource scheduling system of a multi-tenant container cloud platform, the resource scheduling system comprising one control node and a plurality of worker nodes, each worker node comprising a plurality of GPUs, characterized in that, The method comprises executing the following steps at each worker node: receiving the deep learning task and its resource requirement sent by the control node to the current worker node according to the method of any one of claims 1-7, wherein the resource requirement indicates the resource utilization rate of the deep learning task on multiple specified computing power GPUs; allocating a GPU and corresponding GPU resource for the deep learning task based on the remaining resources of multiple GPUs in the current worker node and the resource requirement.

9. The method of claim 8, wherein, When the deep learning task is a delay-sensitive task, the deep learning task is allocated to the GPU with the most remaining resources.

10. The method of claim 8, wherein, When the deep learning task is a non-delay-sensitive task, the deep learning task is allocated to the GPU with the shortest execution time, wherein the execution time is predicted based on an execution time prediction model according to the network structure feature corresponding to the deep learning task, the resource utilization rate corresponding to the deep learning task on the specified computing power GPU, and the hardware feature corresponding to the specified computing power GPU.

11. The method of claim 8, wherein, The method further comprises: When the deep learning task is a delay-sensitive task, obtaining the predicted execution time of the deep learning task on the current GPU from the predicted execution time set sent by the control node, and monitoring whether the execution of the deep learning task meets the preset service quality based on the predicted execution time.

12. The method of claim 11, wherein, The monitoring whether the execution of the deep learning task meets the preset service quality comprises: obtaining an executed time and an execution progress of the deep learning task; updating execution features and network structure features of the deep learning task according to the execution progress; predicting a remaining execution time of the deep learning task based on the execution time prediction model according to the corresponding resource utilization of the deep learning task on a specified computing power GPU, the corresponding hardware features of the specified computing power GPU, the updated execution features, and the network structure features; when the sum of the executed time and the remaining execution time is greater than a preset execution time, determining that the execution of the delay-sensitive task does not meet the preset service quality.

13. The method of claim 11, wherein, When it is monitored that the execution of the deep learning task does not meet the preset service quality, increasing GPU resources of a current GPU allocated to the deep learning task; if GPU resources are insufficient when the resources allocated to the deep learning task are increased, reducing resource allocation to non-delay-sensitive tasks on the current GPU; if GPU resources are insufficient after the GPU resources allocated to the deep learning task are increased, saving the execution state of the non-delay-sensitive task and ending the execution of the non-delay-sensitive task.

14. The method of claim 11, wherein, The method further comprises: When it is monitored that the execution of the deep learning task meets the preset service quality, querying resource utilization of the deep learning task on a current GPU, and if the resource utilization exceeds the predicted resource utilization of the deep learning task on the current GPU sent by the control node, pausing the execution of the deep learning task on the current GPU, and when the resource utilization decreases to the predicted resource utilization, re-executing the deep learning task on the current GPU.

15. A resource scheduling system supporting GPU sharing, the system comprising: comprises: a control node configured to transmit a deep learning task and resource requirements thereof to a worker node according to the method of any one of claims 1-7, the resource requirements indicating resource utilization of the deep learning task on multiple specified computing power GPUs; a plurality of worker nodes, each worker node being configured to allocate a GPU and corresponding GPU resources to the deep learning task according to the method of any one of claims 8-14.

16. The resource scheduling system of claim 15, wherein, The control node comprises: a first prediction unit configured to predict resource utilization of the deep learning task based on a utilization prediction model according to the network structure features and the execution features, to obtain a resource utilization set, the resource utilization set including resource utilization of the deep learning task on multiple specified computing power GPUs; a first scheduling unit configured to determine all worker nodes satisfying resource requirements of the deep learning task from the available resource set based on the resource utilization set, and select a worker node from all worker nodes satisfying the resource requirements of the deep learning task according to a predetermined selection rule to allocate resources to the deep learning task, and transmit the deep learning task and the resource requirements thereof to the selected worker node, the resource requirements indicating resource utilization of the deep learning task on multiple specified computing power GPUs.

17. The resource scheduling system of claim 16, wherein, The control node further comprises: a latency-sensitive task waiting queue for storing the latency-sensitive task when all the current working nodes cannot meet the resource requirement of the latency-sensitive task; a non-latency-sensitive task waiting queue for storing the non-latency-sensitive task when all the current working nodes cannot meet the resource requirement of the non-latency-sensitive task; wherein, when the latency-sensitive task waiting queue is not empty, GPU resources are preferentially allocated to the latency-sensitive task in the latency-sensitive task waiting queue.

18. The resource scheduling system of claim 16, wherein, The first prediction unit is further configured to predict, based on an execution time prediction model, execution time of the deep learning task on a specified computing power GPU according to network structure features corresponding to the deep learning task, execution features corresponding to the deep learning task, resource utilization corresponding to the deep learning task on the specified computing power GPU, and hardware features corresponding to the specified computing power GPU, to obtain a set of predicted execution times for detecting whether execution of the non-latency-sensitive task meets a preset quality of service.

19. The resource scheduling system of claim 15, wherein, The working node comprises: a topology awareness unit configured to obtain remaining resources of a plurality of GPUs in the current working node; a second prediction unit configured to predict, based on an execution time prediction model, execution time of the non-latency-sensitive task on a specified computing power GPU according to network structure features corresponding to the non-latency-sensitive task, execution features corresponding to the non-latency-sensitive task, resource utilization corresponding to the non-latency-sensitive task on the specified computing power GPU, and hardware features corresponding to the specified computing power GPU, to obtain execution time of the non-latency-sensitive task on a plurality of specified computing power GPUs; a second scheduling unit configured to allocate, based on the remaining resources of the plurality of GPUs in the current working node and the resource requirement, a GPU and corresponding GPU resources for the deep learning task, and to allocate, according to the execution time of the non-latency-sensitive task on the plurality of specified computing power GPUs, a GPU with the shortest execution time for the non-latency-sensitive task.

20. The resource scheduling system of claim 19, wherein, The working node further comprises: an elastic adjustment unit configured to monitor whether execution of the latency-sensitive task meets a preset quality of service, and to increase GPU resources allocated to the deep learning task on the current GPU when execution of the latency-sensitive task does not meet the preset quality of service, to reduce resource allocation to the non-latency-sensitive task on the current GPU when GPU resources are insufficient after the increase in resource allocation to the deep learning task, and to save an execution state of the non-latency-sensitive task and end execution of the non-latency-sensitive task when the non-latency-sensitive task is insufficient in resources after the increase in GPU resources allocated to the deep learning task; a resource limitation unit configured to, when execution of the latency-sensitive task meets the preset quality of service, query resource utilization of the latency-sensitive task on the current GPU, and to suspend execution of the latency-sensitive task on the current GPU when the resource utilization of the latency-sensitive task on the current GPU exceeds predicted resource utilization of the latency-sensitive task on the current GPU sent by the control node, and to resume execution of the latency-sensitive task on the current GPU when the resource utilization falls to the predicted resource utilization.

Images