Gpu space-time sharing method for deep learning services

Abstract

The embodiment of the present disclosure provides a GPU space-time sharing method for deep learning service, which belongs to the technical field of data processing, and specifically comprises the following steps: step 1, reading a DNN model list of to-be-allocated GPU resources in a system initialization state; step 2, estimating a GPU computing resource demand quantity; step 3, sorting the DNN models in the DNN model list according to the GPU computing resource demand quantity from large to small, and sequentially allocating GPU computing resources to each DNN model in a time and space dimension in a fine-grained manner according to the sorting result; step 4, updating the residual load size of each DNN model in the DNN model list, and removing the DNN model with a residual load size of 0 from the DNN model list; step 5, detecting whether the DNN model list is empty, if yes, ending the resource allocation process of the GPU space-time sharing, and if not, executing step 2. Through the scheme of the present disclosure, the scheduling efficiency and real-time performance are improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to a GPU spatiotemporal sharing method for deep learning services. Background Technology

[0002] Currently, with the rapid development of deep learning, deep neural network (DNN) models have been widely applied in various fields. Inference tasks based on DNN models often consume significant computing resources, making it impossible for resource-constrained devices to support deep learning applications with low latency and high accuracy. To address this, many cloud service providers deploy DNN inference computation as a service in cloud data centers (referred to as deep learning service systems), enabling resource-constrained devices to support deep learning applications through this service. To meet users' real-time requirements, cloud service providers typically deploy deep learning service systems on large-scale graphics processing unit (GPU) clusters. Whenever the system receives an inference request from a user, it performs inference computation on the requested DNN model and returns the corresponding prediction result.

[0003] The GPU spatiotemporal sharing strategy fully combines the characteristics of time-sharing and spatial sharing. It not only enables the GPU to support the execution of multiple tasks at any given time but also allocates GPU resources to different DNN model instances over time. Compared to single time-sharing or spatial sharing strategies, the GPU spatiotemporal sharing strategy is more effective in improving GPU resource utilization and reducing the hardware cost of the system. However, current research on GPU spatiotemporal sharing for deep learning service systems still faces the following challenges:

[0004] 1) The execution time of a DNN task depends on performance interference from other tasks executed in parallel. However, in GPU time-space sharing mode, parallel DNN tasks overlap arbitrarily in the time dimension, increasing the uncertainty of the start and end times of task execution. That is, the combination of tasks executed in parallel on the GPU varies greatly at different times. Because the degree of interference between different combinations of parallel tasks often differs, it is difficult for deep learning service systems to predict the execution time of each DNN task in GPU time-space sharing mode, thus making it difficult to guarantee the user's real-time requirements.

[0005] 2) The GPU resource requirements of DNN models dynamically change with metrics such as batch request size and service level objective (SLO) latency. These dynamic factors increase the difficulty of improving GPU resource utilization using spatiotemporal sharing models. Current GPU spatiotemporal sharing methods for deep learning services cannot effectively and flexibly allocate GPU space based on the resource requirements of DNN tasks. Furthermore, a trade-off must be made between user real-time requirements and GPU resource utilization, increasing the complexity of GPU resource management and scheduling.

[0006] It is evident that there is an urgent need for a GPU spatiotemporal sharing method with high scheduling efficiency and real-time performance for deep learning services. Summary of the Invention

[0007] In view of this, the present disclosure provides a GPU spatiotemporal sharing method for deep learning services, which at least partially solves the problems of poor scheduling efficiency and real-time performance in the prior art.

[0008] In a first aspect, embodiments of this disclosure provide a GPU spatiotemporal sharing method for deep learning services, including:

[0009] Step 1: During system initialization, read the list of DNN models to be allocated GPU resources. Each element in the list is a triple, where each element includes a DNN model, its remaining load size, and latency requirements.

[0010] Step 2: Estimate the GPU computing resource requirements for each DNN model in the DNN model list based on its load size.

[0011] Step 3: Sort the DNN models in the DNN model list from largest to smallest according to their GPU computing resource requirements, and allocate GPU computing resources to each DNN model in a fine-grained manner according to the sorting results, from the time and space dimensions.

[0012] Step 4: Based on the GPU resource allocation results, update the remaining load size of each DNN model in the DNN model list, and remove the DNN models with a remaining load size of 0 from the DNN model list.

[0013] Step 5: Check if the DNN model list is empty. If it is, end the GPU time-space sharing resource allocation process. If not, proceed to step 2.

[0014] According to a specific implementation of an embodiment of this disclosure, step 2 specifically includes:

[0015] Step 2.1, set the variable i to an initial value of 1;

[0016] Step 2.2: Read the i-th element in the DNN model list and obtain the saturation throughput of the DNN model corresponding to the i-th element when it is independently deployed on a single GPU and allocated with GPU partitions of different sizes.

[0017] Step 2.3: If the remaining load size of the DNN model corresponding to the i-th element is not less than the saturation throughput corresponding to the most efficient partition, then mark the DNN model corresponding to the i-th element as saturated load; otherwise, find the smallest GPU partition between 10% and the size of the most efficient partition that has a batch size b that satisfies the preset formula, and mark the DNN model corresponding to the i-th element as unsaturated load.

[0018] Step 2.4: Execute i = i + 1, and determine whether i is greater than the number of DNN models in the DNN model list. If yes, end the GPU computing resource requirement estimation process; otherwise, execute step 2.2.

[0019] According to a specific implementation of an embodiment of this disclosure, the preset formula is as follows:

[0020]

[0021] Where slo is the latency SLO of the DNN model corresponding to the i-th element, and l(b,p) is the inference latency of the DNN model corresponding to the i-th element when it is independently deployed in a GPU partition of size p and the batch size is b.

[0022] According to a specific implementation of an embodiment of this disclosure, step 3 specifically includes:

[0023] Step 3.1: Set the variable i to an initial value of 1;

[0024] Step 3.2: Read the i-th element in the sorted list of DNN models. If the DNN model corresponding to the i-th element is marked as saturated load, then execute step 3.3; otherwise, execute step 3.4.

[0025] Step 3.3: Select a GPU partition that meets the first preset requirement from the existing GPU partitions in the system. If a GPU partition that meets the first preset requirement exists, calculate the maximum batch size that meets the latency requirement of the DNN model corresponding to the i-th element, and allocate GPU computing resources to the DNN model corresponding to the i-th element in a fine-grained manner. Then execute step 3.6. Otherwise, add a GPU to the system and then re-execute step 3.3.

[0026] Step 3.4: Select a GPU partition that meets the second preset requirement from the existing GPU partitions in the system. If a GPU partition that meets the second preset requirement exists, calculate the maximum batch size that meets the latency requirement of the DNN model corresponding to the i-th element, and allocate GPU computing resources to the DNN model corresponding to the i-th element in a fine-grained manner. Then execute step 3.6. Otherwise, execute step 3.5.

[0027] Step 3.5: Select a GPU partition that meets the third preset requirement from the existing GPU partitions in the system. If a GPU partition that meets the third preset requirement exists, calculate the maximum batch size that meets the latency requirement of the DNN model corresponding to the i-th element, and allocate GPU computing resources to the DNN model corresponding to the i-th element in a fine-grained manner. Then execute step 3.6. Otherwise, add a GPU to the system and then re-execute step 3.5.

[0028] Step 3.6: Execute i = i + 1, and determine whether i is greater than the number of DNN models in the sorted DNN model list. If yes, end the GPU resource dynamic allocation process; otherwise, execute step 3.2.

[0029] According to a specific implementation of an embodiment of this disclosure, the first preset requirement includes: the partition has not been allocated any workload and is not less than the GPU computing resource requirement of the DNN model corresponding to the i-th element; and, a GPU partition that meets the GPU computing resource requirement of the DNN model corresponding to the i-th element is divided from the GPU partition, which can meet the latency requirement of the DNN model corresponding to the i-th element and does not reduce the throughput of other loads on the same GPU.

[0030] The second preset requirement includes: the partition has been allocated workload, and its size is not less than the GPU computing resource requirement of the DNN model corresponding to the i-th element; and allocating the DNN model corresponding to the i-th element to the GPU partition can meet the latency requirements of all inference requests.

[0031] The third preset requirement includes: the partition has not been assigned any workload, and its size is not less than the GPU computing resource requirement of the DNN model corresponding to the i-th element; and, a GPU partition that meets the GPU computing resource requirement of the DNN model corresponding to the i-th element is divided from the GPU partition, which can meet the latency requirement of the DNN model corresponding to the i-th element, and does not reduce the throughput of other loads on the same GPU.

[0032] The GPU spatiotemporal sharing scheme for deep learning services in this embodiment includes: Step 1, reading a list of DNN models to be allocated GPU resources during system initialization, where each element of the list is a triplet, and each element includes a DNN model, its remaining load size, and latency requirements; Step 2, estimating the GPU computing resource requirements for each DNN model in the list based on its load size; Step 3, sorting the DNN models in the list according to their GPU computing resource requirements from largest to smallest, and allocating GPU computing resources to each DNN model in a fine-grained manner based on the sorting results in terms of time and space; Step 4, updating the remaining load size of each DNN model in the list based on the GPU resource allocation results, and removing DNN models with a remaining load size of 0 from the list; Step 5, checking whether the DNN model list is empty. If it is, the GPU spatiotemporal sharing resource allocation process ends; otherwise, step 2 is executed.

[0033] The beneficial effects of the embodiments of this disclosure are as follows: By predicting the performance of DNN inference tasks under diverse parallel conditions, the scheme of this disclosure provides guidance for allocating GPU resources for multiple model instances; based on the diverse factors such as the request load and request latency (SLO) of each model instance, the GPU resource requirements of the DNN task are estimated, and GPU computing resources are dynamically and flexibly allocated in fine granular terms from the time and space dimensions, maximizing the utilization of GPU resources while ensuring the quality of deep learning services. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A flowchart illustrating a GPU spatiotemporal sharing method for deep learning services provided in this embodiment of the disclosure;

[0036] Figure 2 A GPU spatiotemporal sharing framework diagram for deep learning services is provided in this embodiment of the disclosure;

[0037] Figure 3 This is an example diagram illustrating concurrent execution of DNN tasks in a GPU spatiotemporal sharing environment, provided by an embodiment of this disclosure.

[0038] Figure 4 A schematic diagram illustrating the most efficient GPU partitioning of an instance model, GoogleNet, provided as an embodiment of this disclosure.

[0039] Figure 5 This diagram illustrates a performance comparison between a method of this disclosure and existing technologies, provided as an embodiment of this disclosure. Detailed Implementation

[0040] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0041] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0042] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0043] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0044] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0045] With the rapid development of deep learning, deep neural network (DNN) models have been widely applied in various fields. Inference tasks based on DNN models often consume significant computing resources, making it difficult for resource-constrained devices to support deep learning applications with low latency and high accuracy. To address this, many cloud service providers deploy DNN inference computation as a service in cloud data centers (referred to as deep learning service systems), enabling resource-constrained devices to support deep learning applications. To meet users' real-time requirements, cloud service providers typically deploy deep learning service systems on large-scale graphics processing unit (GPU) clusters. Whenever the system receives an inference request from a user, it performs inference computation on the requested DNN model and returns the corresponding prediction result.

[0046] Ideally, deep learning service systems should meet key performance indicators such as low latency, low cost, and high throughput. However, in real-world production scenarios, a single DNN inference request often fails to fully utilize expensive GPU computing resources. To improve GPU resource utilization, mainstream deep learning service systems employ a batching strategy to process inference requests from the same model in batches, returning the results data uniformly after obtaining the inference results for all requests. This approach effectively improves GPU resource utilization and overall system throughput, but it increases the response latency of a single inference request. Since online inference tasks of DNN models typically have strict Service Level Objectives (SLOs), meaning that inference requests must receive a response within a certain timeframe, the batching strategy increases the risk of violating the inference request SLO.

[0047] Furthermore, GPU sharing is another effective method to improve GPU resource utilization, which involves deploying multiple DNN models on the same GPU server. Existing research mainly employs time-sharing, spatial sharing, and spatiotemporal sharing strategies to manage and schedule multiple DNN model instances deployed on the same GPU. Given input data, the completion time of a DNN task running independently on a GPU has deterministic characteristics. Under the time-sharing strategy, the GPU runs only a single DNN task within any scheduling unit, naturally preserving the predictability of DNN task execution time and simplifying the scheduling process to meet latency SLO. However, DNN tasks supported by the time-sharing strategy run independently on the GPU sequentially, failing to fully utilize the parallel execution characteristics of the GPU kernel, and still resulting in low GPU resource utilization. Spatial sharing technologies, represented by NVIDIA MPS (Multi Process Service), CUDA (Compute Unified Device Architecture) streams, and MIG (Multi Instance GPU), allow the GPU to run multiple tasks at any given time, with each task using only a portion of the GPU's computing resources, which helps improve the GPU resource utilization of deep learning service systems. However, under the space-sharing strategy, concurrently running tasks will suffer varying degrees of performance degradation, posing a challenge to the stability of inference request response time.

[0048] The GPU spatiotemporal sharing strategy fully combines the characteristics of time-sharing and spatial sharing. It not only enables the GPU to support the execution of multiple tasks at any given time but also allocates GPU resources to different DNN model instances over time. Compared to single time-sharing or spatial sharing strategies, the GPU spatiotemporal sharing strategy is more effective in improving GPU resource utilization and reducing the hardware cost of the system. However, current research on GPU spatiotemporal sharing for deep learning service systems still faces the following challenges:

[0049] 1) The execution time of a DNN task depends on performance interference from other tasks executed in parallel. However, in GPU time-space sharing mode, parallel DNN tasks overlap arbitrarily in the time dimension, increasing the uncertainty of the start and end times of task execution. That is, the combination of tasks executed in parallel on the GPU varies greatly at different times. Because the degree of interference between different combinations of parallel tasks often differs, it is difficult for deep learning service systems to predict the execution time of each DNN task in GPU time-space sharing mode, thus making it difficult to guarantee the user's real-time requirements.

[0050] 2) The GPU resource requirements of DNN models dynamically change with the size of batch requests and the time-of-flight (SLO) latency. These dynamic factors increase the difficulty of improving GPU resource utilization using spatiotemporal sharing models. Current GPU spatiotemporal sharing methods for deep learning services cannot effectively and flexibly allocate GPU space according to the resource requirements of DNN tasks. Furthermore, a trade-off must be made between user real-time requirements and GPU resource utilization, increasing the complexity of GPU resource management and scheduling.

[0051] This disclosure provides a GPU spatiotemporal sharing method for deep learning services, which can be applied to GPU resource scheduling in cloud service scenarios.

[0052] See Figure 1 This is a flowchart illustrating a GPU spatiotemporal sharing method for deep learning services provided in an embodiment of this disclosure. Figure 1 As shown, the method mainly includes the following steps:

[0053] Step 1: During system initialization, read the list of DNN models to be allocated GPU resources. Each element in the list is a triple, where each element includes a DNN model, its remaining load size, and latency requirements.

[0054] In specific implementation, such as Figure 2 The diagram illustrates a GPU spatiotemporal sharing framework for deep learning services. It primarily consists of four parts: a performance analysis module, an interference-aware latency predictor, a global scheduler, and a backend executor. These four parts interact to jointly complete the GPU spatiotemporal sharing resource allocation process. The global scheduler receives inference requests from users and estimates the GPU resource requirements of each DNN model instance based on its load (inference requests per second). The performance analysis module collects performance data for each DNN model under diverse parallel execution conditions and provides this data to the interference-aware latency predictor to calculate the parallel inference latency of the DNN model. Parallel inference latency refers to the latency when multiple DNN models simultaneously perform inference computations on the GPU. Figure 3As shown, ad represents different DNN model instances, and the GPU is divided into three parts, i.e., three GPU partitions. At any given time, each partition runs DNN inference computation tasks, so tasks on different partitions can run simultaneously, i.e., spatial GPU sharing. The latency when multiple DNN inference tasks are executed simultaneously on the GPU is called parallel inference latency. Parallel DNN tasks can overlap arbitrarily in the time dimension, meaning the combination of tasks executed simultaneously on the GPU varies greatly at different times, and the degree of mutual interference between different tasks is completely different. The interference-aware latency predictor builds a performance prediction model for DNN tasks based on sampled data and provides guidance for the latency SLO-aware GPU partitioning and scheduling strategy. The latency SLO-aware GPU partitioning and scheduling module allocates GPU computing resources to each DNN model based on the GPU resource requirements of each DNN model and the information provided by the latency predictor, and sends the guidance information to the backend executor. The backend executor, based on this guidance information, places each DNN model on a GPU that meets the requirements and allocates the corresponding number of GPU computing resources to the relevant model. In addition, whenever the global scheduler detects a significant change in the overall load, it will re-execute the GPU space-time shared resource allocation process.

[0055] The resource allocation process for GPU space-time sharing is as follows:

[0056] The system initialization state includes one GPU, and reads a list of DNN models to be allocated GPU resources, where each element of the list is a triplet. <model i r i ,slo i >, where model i r represents the DNN model instance corresponding to the i-th element in the list. i For model i The remaining load size, i.e., the amount of data currently being sent to the model per second as monitored. i The number of inference requests, slo i For model i The corresponding delay SLO.

[0057] Step 2: Estimate the GPU computing resource requirements for each DNN model in the DNN model list based on its load size.

[0058] Based on the above embodiments, step 2 specifically includes:

[0059] Step 2.1, set the variable i to an initial value of 1;

[0060] Step 2.2: Read the i-th element in the DNN model list and obtain the saturation throughput of the DNN model corresponding to the i-th element when it is independently deployed on a single GPU and allocated with GPU partitions of different sizes.

[0061] Step 2.3: If the remaining load size of the DNN model corresponding to the i-th element is not less than the saturation throughput corresponding to the most efficient partition, then mark the DNN model corresponding to the i-th element as saturated load; otherwise, find the smallest GPU partition between 10% and the size of the most efficient partition that has a batch size b that satisfies the preset formula, and mark the DNN model corresponding to the i-th element as unsaturated load.

[0062] Step 2.4: Execute i = i + 1, and determine whether i is greater than the number of DNN models in the DNN model list. If yes, end the GPU computing resource requirement estimation process; otherwise, execute step 2.2.

[0063] Furthermore, the preset formula is as follows:

[0064]

[0065] Where slo is the latency SLO of the DNN model corresponding to the i-th element, and l(b,p) is the inference latency of the DNN model corresponding to the i-th element when it is independently deployed in a GPU partition of size p and the batch size is b.

[0066] In practice, the GPU computing resource requirements are estimated for each DNN model in the list of models, based on its workload. The specific steps are as follows:

[0067] 2.1 Set a variable i with an initial value of 1 to index the elements in the list models;

[0068] 2.2 Read the i-th element from the list models <model i r i ,slo i >, Get model i The saturation throughput when deployed independently on a single GPU and allocated GPU partitions of varying sizes; where a GPU partition refers to a sub-part of the GPU, not the entire GPU, and its size is a percentage of the total GPU computing resources allocated to it; GPU partitioning can be implemented using techniques such as MPS and MIG; for example, MPS can be used to allocate 30% of the GPU computing resources to the inference task of a DNN model, i.e., allocate a GPU partition of 30% size to that DNN model; test model i The method for achieving saturated throughput on a given input data and GPU partition is to first find the maximum batch size b that satisfies the following formula:

[0069] 2l(b,p)≤slo

[0070] Where slo is model i The delay SLO, l(b,p) is the model i Inference latency when deployed independently on GPU partitions of size p with batch size b. After finding the maximum batch size b that satisfies this condition, the model can be calculated as b / l(b,p). i The saturation throughput on a GPU partition of size p. The model is obtained sequentially using this method. i The saturation throughput was measured on GPU partitions of sizes 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, and 100%. The GPU partition with the largest increase in saturation throughput was identified as the model. i The most efficient GPU partition. For example... Figure 4 The diagram shows the most efficient GPU partition for the GoogleNet model. The horizontal axis represents the size of the GPU partition, and the vertical axis represents the saturation throughput of the GoogleNet model measured on NVIDIA A100 GPU partitions of the corresponding size. The saturation throughput growth rate of a 20% GPU partition is 2.13 times that of a 10% GPU partition. After 20%, the saturation throughput growth rate becomes slower and slower. Therefore, the 20% GPU partition is the most efficient GPU partition for the GoogleNet model.

[0071] 2.3, if model i Remaining load size r i The throughput should not be less than the saturation throughput corresponding to the most efficient partition, which is the most efficient GPU partition, i.e., the model. i The GPU computing resource requirements, and the model i Mark it as saturated load; otherwise, find the smallest GPU partition between 10% and the most efficient partition size that has a batch size b that satisfies the following formula; this partition is the model. i The GPU computing resource requirements, and the model i Marked as non-saturated load:

[0072]

[0073] Where slo is model i The delay SLO, l(b,p) is the model i Inference latency when deployed independently in GPU partitions of size p and in batches of size b.

[0074] 2.4 Execute i = i + 1, and determine whether i is greater than the number of DNN models in the list models. If yes, end the estimation process of GPU computing resource requirements; otherwise, execute step 2.2.

[0075] Step 3: Sort the DNN models in the DNN model list from largest to smallest according to their GPU computing resource requirements, and allocate GPU computing resources to each DNN model in a fine-grained manner according to the sorting results, from the time and space dimensions.

[0076] Based on the above embodiments, step 3 specifically includes:

[0077] Step 3.1: Set the variable i to an initial value of 1;

[0078] Step 3.2: Read the i-th element in the sorted list of DNN models. If the DNN model corresponding to the i-th element is marked as saturated load, then execute step 3.3; otherwise, execute step 3.4.

[0079] Step 3.3: Select a GPU partition that meets the first preset requirement from the existing GPU partitions in the system. If a GPU partition that meets the first preset requirement exists, calculate the maximum batch size that meets the latency requirement of the DNN model corresponding to the i-th element, and allocate GPU computing resources to the DNN model corresponding to the i-th element in a fine-grained manner. Then execute step 3.6. Otherwise, add a GPU to the system and then re-execute step 3.3.

[0080] Step 3.4: Select a GPU partition that meets the second preset requirement from the existing GPU partitions in the system. If a GPU partition that meets the second preset requirement exists, calculate the maximum batch size that meets the latency requirement of the DNN model corresponding to the i-th element, and allocate GPU computing resources to the DNN model corresponding to the i-th element in a fine-grained manner. Then execute step 3.6. Otherwise, execute step 3.5.

[0081] Step 3.5: Select a GPU partition that meets the third preset requirement from the existing GPU partitions in the system. If a GPU partition that meets the third preset requirement exists, calculate the maximum batch size that meets the latency requirement of the DNN model corresponding to the i-th element, and allocate GPU computing resources to the DNN model corresponding to the i-th element in a fine-grained manner. Then execute step 3.6. Otherwise, add a GPU to the system and then re-execute step 3.5.

[0082] Step 3.6: Execute i = i + 1, and determine whether i is greater than the number of DNN models in the sorted DNN model list. If yes, end the GPU resource dynamic allocation process; otherwise, execute step 3.2.

[0083] Furthermore, the first preset requirement includes: the partition has not been allocated any workload, and the GPU computing resource requirement of the DNN model corresponding to the i-th element is not less than the GPU computing resource requirement of the DNN model corresponding to the i-th element; and, a GPU partition that meets the GPU computing resource requirement of the DNN model corresponding to the i-th element is divided from the GPU partition, and the latency requirement of the DNN model corresponding to the i-th element is met, without losing the throughput of other loads on the same GPU.

[0084] The second preset requirement includes: the partition has been allocated workload, and its size is not less than the GPU computing resource requirement of the DNN model corresponding to the i-th element; and allocating the DNN model corresponding to the i-th element to the GPU partition can meet the latency requirements of all inference requests.

[0085] The third preset requirement includes: the partition has not been assigned any workload, and its size is not less than the GPU computing resource requirement of the DNN model corresponding to the i-th element; and, a GPU partition that meets the GPU computing resource requirement of the DNN model corresponding to the i-th element is divided from the GPU partition, which can meet the latency requirement of the DNN model corresponding to the i-th element, and does not reduce the throughput of other loads on the same GPU.

[0086] In practice, the DNN models in the list are sorted from largest to smallest according to their estimated GPU computing resource requirements. Then, according to the sorting results, GPU computing resources are allocated to each DNN model in a fine-grained manner. The specific steps are as follows:

[0087] 3.1 Set a variable i with an initial value of 1 to index the elements in the sorted list models;

[0088] 3.2 Read the i-th element from the list models <model i r i ,slo i >; if model i If the load is marked as saturated, proceed to step 3.3; otherwise, proceed to step 3.4.

[0089] 3.3 Select a GPU partition from the existing GPU partitions in the system that meets the following requirements: 1) The partition has not been assigned any workload and is not smaller than the model. i 1) The GPU computing resource requirements; 2) If the GPU partition is divided to just match the model i GPU partitioning to meet the GPU computing resource requirements of the model. iThe latency SLO is minimized without sacrificing throughput for other loads on the same GPU. If a GPU partition exists that satisfies both requirements, the maximum batch size b that meets the requirements is calculated as: 2l(b,p)≤slo, where slo is the model's latency SLO. i The delay SLO, l(b,p) is the model i Parallel inference latency when deployed on a GPU partition of size p and with a batch size of b, where p is the model size. i The GPU computing resource requirements, and then the model i The allocation results for this round are set as follows: <l i ,b i > Execute step 3.6, where b i l represents the maximum batch size that meets the requirements in the above formula. i For b i The corresponding parallel inference latency; otherwise, add a GPU to the system and then re-execute step 3.3;

[0090] 3.4 Select a GPU partition from the existing GPU partitions in the system that meets the following requirements: 1) The partition has been allocated workloads and its size is not less than the model size. i 1) GPU computing resource requirements; 2) Model i The GPU partition allocated to this system can satisfy the latency SLO (Standard Loop) for all inference requests. If a GPU partition exists that meets both of these requirements, assuming the scheduling period of this GPU partition is d, calculate the maximum batch size b that meets the requirements: d + l(b,p) ≤ slo, where slo is the model size. i The delay SLO, l(b,p) is the model i Parallel inference latency when deployed on GPUs of size p and batch size b, where p is the model size. i The GPU computing resource requirements, and then the model i The allocation results for this round are set as follows: <l i ,b i > Execute step 3.6, where b i l represents the maximum batch size that meets the requirements in the above formula. i If it equals d; otherwise, proceed to step 3.5.

[0091] 3.5 Select a GPU partition from the existing GPU partitions in the system that meets the following requirements: 1) The partition has not been assigned any workload, and its size is not less than the model size. i 1) The GPU computing resource requirements; 2) If the GPU partition is divided to just match the model iGPU partitioning to meet the GPU computing resource requirements of the model. i The latency SLO must be minimized without sacrificing throughput for other loads on the same GPU. If a GPU partition exists that satisfies both requirements, calculate the maximum batch size b that meets the requirements: Where slo is model i The delay SLO, l(b,p) is the model i Parallel inference latency when deployed on a GPU partition of size p and with a batch size of b, where p is the model size. i GPU computing resource requirements, r i For model i The remaining load size, then the model i The allocation results for this round are set as follows: <l i ,b i > Execute step 3.6, where b i l represents the maximum batch size that meets the requirements in the above formula. i equal Otherwise, add a GPU to the system and then repeat step 3.5;

[0092] 3.6 Execute i = i + 1, and determine whether i is greater than the number of DNN models in the list models. If yes, end the dynamic allocation process of GPU resources; otherwise, execute step 3.2.

[0093] Step 4: Based on the GPU resource allocation results, update the remaining load size of each DNN model in the DNN model list, and remove the DNN models with a remaining load size of 0 from the DNN model list.

[0094] In practice, based on the GPU resource allocation results, the remaining load size of each DNN model in the list `models` is updated; assuming model i The corresponding allocation result is <l i ,b i >, then its remaining load size r i Updated to If model i The corresponding remaining load size r i If it is 0, then the model i Remove from the list of models.

[0095] Step 5: Check if the DNN model list is empty. If it is, end the GPU time-space sharing resource allocation process. If not, proceed to step 2.

[0096] In practice, it is possible to detect in real time whether the DNN model list is empty. If it is, it can be determined that the resource scheduling has been completed and the GPU time-space sharing resource allocation process can be ended. If not, it is assumed that there is still a workload of GPU resources to be allocated and the process can return to step 2 to continue calculating the resource requirements.

[0097] The GPU spatiotemporal sharing method for deep learning services provided in this embodiment guides the allocation of GPU resources for multiple model instances by predicting the performance of DNN inference tasks under diverse parallel conditions. Based on diverse factors such as the request load and request latency (SLO) of each model instance, it estimates the GPU resource requirements of the DNN task and dynamically and flexibly allocates GPU computing resources in a fine-grained manner from the time and space dimensions, maximizing the utilization of GPU resources while ensuring the quality of deep learning services.

[0098] The GPU spatiotemporal sharing method for deep learning services provided in this disclosure can dynamically allocate and schedule GPU resources while ensuring the latency SLO requirement of DNN inference tasks. This strategy is highly efficient and scalable, minimizing the hardware cost of investing in deep learning service systems. The beneficial effects are as follows: 1) It can guarantee the real-time requirements of deep learning service systems in GPU spatiotemporal sharing mode by predicting the performance of DNN inference tasks under diverse parallel conditions; 2) By dynamically and finely allocating GPU resources from both temporal and spatial dimensions based on workload resource requirements, it improves GPU resource utilization and provides a more flexible solution for resource management of deep learning service systems.

[0099] Figure 5 This diagram illustrates a performance comparison between the present invention and existing technologies on a server composed of eight NVIDIA A100 GPUs. The system configuration is shown in the table below:

[0100] Table 1

[0101]

[0102]

[0103] Among them, Gpulet is the most advanced GPU time-space sharing scheduling system in the world. Compared with Gpulet, this invention can use fewer GPUs while meeting the latency SLO of more than 99% of requests.

[0104] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0105] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A GPU spatiotemporal sharing method for deep learning services, characterized in that, include: Step 1: During system initialization, read the list of DNN models to be allocated GPU resources. Each element in the list is a triple, where each element includes a DNN model, its remaining load size, and latency requirements. Step 2: Estimate the GPU computing resource requirements for each DNN model in the DNN model list based on its load size. Step 3: Sort the DNN models in the DNN model list from largest to smallest according to their GPU computing resource requirements, and allocate GPU computing resources to each DNN model in terms of time and space according to the sorting results. Step 3 specifically includes: Step 3.1: Set the variable i to an initial value of 1; Step 3.2: Read the i-th element in the sorted list of DNN models. If the DNN model corresponding to the i-th element is marked as saturated load, then execute step 3.3; otherwise, execute step 3.

4. Step 3.3: Select a GPU partition that meets the first preset requirement from the existing GPU partitions in the system. If a GPU partition that meets the first preset requirement exists, calculate the maximum batch size that meets the latency requirement of the DNN model corresponding to the i-th element, allocate GPU computing resources to the DNN model corresponding to the i-th element, and then execute step 3.

6. Otherwise, add a GPU to the system and then execute step 3.3 again. Step 3.4: Select a GPU partition that meets the second preset requirement from the existing GPU partitions in the system. If a GPU partition that meets the second preset requirement exists, calculate the maximum batch size that meets the latency requirement of the DNN model corresponding to the i-th element, allocate GPU computing resources to the DNN model corresponding to the i-th element, and then execute step 3.

6. Otherwise, execute step 3.

5. Step 3.5: Select a GPU partition that meets the third preset requirement from the existing GPU partitions in the system. If a GPU partition that meets the third preset requirement exists, calculate the maximum batch size that meets the latency requirement of the DNN model corresponding to the i-th element, allocate GPU computing resources to the DNN model corresponding to the i-th element, and then execute step 3.

6. Otherwise, add a GPU to the system and then execute step 3.5 again. Step 3.6: Execute i=i+1, and determine whether i is greater than the number of DNN models in the sorted DNN model list. If yes, end the dynamic allocation process of GPU resources; otherwise, execute step 3.

2. Step 4: Based on the GPU resource allocation results, update the remaining load size of each DNN model in the DNN model list, and remove the DNN models with a remaining load size of 0 from the DNN model list. Step 5: Check if the DNN model list is empty. If it is, end the GPU time-space sharing resource allocation process. If not, proceed to step 2.

2. The method according to claim 1, characterized in that... Step 2 specifically includes: Step 2.1, set the variable i to an initial value of 1; Step 2.2: Read the i-th element in the DNN model list and obtain the saturation throughput of the DNN model corresponding to the i-th element when it is independently deployed on a single GPU and allocated with GPU partitions of different sizes. Step 2.3: If the remaining load size of the DNN model corresponding to the i-th element is not less than the saturation throughput corresponding to the most efficient partition, then mark the DNN model corresponding to the i-th element as saturated load; otherwise, find the smallest GPU partition between 10% and the size of the most efficient partition that has a batch size b that satisfies the preset formula, and mark the DNN model corresponding to the i-th element as unsaturated load. Step 2.4: Execute i=i+1 and determine whether i is greater than the number of DNN models in the DNN model list. If yes, end the GPU computing resource requirement estimation process; otherwise, execute step 2.

2.

3. The method according to claim 2, characterized in that... The preset formula is: in, Let SLO be the delay of the DNN model corresponding to the i-th element. Let be the inference latency of the DNN model corresponding to the i-th element when it is deployed independently on a GPU partition of size p with a batch size of b. This represents the load size of the DNN model corresponding to the i-th element.

4. The method according to claim 3, characterized in that... The first preset requirement includes: the partition has not been assigned any workload and is not less than the GPU computing resource requirement of the DNN model corresponding to the i-th element; and, a GPU partition that meets the GPU computing resource requirement of the DNN model corresponding to the i-th element is split from the GPU partition, and can meet the latency requirement of the DNN model corresponding to the i-th element, without losing the throughput of other loads on the same GPU. The second preset requirement includes: the partition has been allocated workload, and its size is not less than the GPU computing resource requirement of the DNN model corresponding to the i-th element; and allocating the DNN model corresponding to the i-th element to the GPU partition can meet the latency requirements of all inference requests. The third preset requirement includes: the partition has not been assigned any workload, and its size is not less than the GPU computing resource requirement of the DNN model corresponding to the i-th element; and, a GPU partition that meets the GPU computing resource requirement of the DNN model corresponding to the i-th element is divided from the GPU partition, which can meet the latency requirement of the DNN model corresponding to the i-th element, and does not reduce the throughput of other loads on the same GPU.

Images