Multi-level GPU resource management and scheduling system for deep learning task

Abstract

The invention discloses a multi-level GPU resource management and scheduling system for a deep learning task, and the system comprises an MIG hardware isolation layer which is used for segmenting and isolating GPU physical resources to form independent sub-examples; the MPS elastic sharing layer is used for dynamically adjusting SM quotas according to task loads and realizing elastic allocation of computing resources; the Kernel-level fine scheduling layer is used for monitoring a Kernel-level resource contention state in real time and ensuring that an online task is executed preferentially; the performance predictor is used for outputting task performance data under different MIG and MPS configurations; and the Hybrid Partition Scheduler is used for selecting an optimal MIG partition combination and MPS quota strategy based on the performance data, and realizing the unification of resource isolation, elastic sharing and real-time scheduling of online tasks through cross-layer collaboration of the above components. According to the method, the problem that resource isolation, elastic sharing and real-time performance are difficult to consider in the prior art is solved, resource fragmentation and inter-task interference are reduced, and the method is suitable for large-scale deep learning training and reasoning services.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and deep learning technology, specifically to a multi-level GPU resource management and scheduling system for deep learning tasks. Background Technology

[0002] With the widespread application of deep neural network (DNN) models in computer vision, natural language processing, and large-scale recommendation systems, graphics processing units (GPUs) have become the core computing resource for modern deep learning training and inference tasks. To further improve GPU utilization, various GPU sharing mechanisms have been proposed in recent years, enabling multiple deep learning tasks to be executed in parallel on a single GPU. Among these, NVIDIA's Multi-Process Service (MPS) and Multi-Instance GPU (MIG) are two of the most representative technologies.

[0003] MPS primarily enables multiple processes to share computing resources on the same GPU by partitioning and reusing Streaming Multiprocessors (SMs). It alleviates the problem of low GPU resource utilization by scheduling the proportion of active threads in each SM to achieve coarse-grained resource sharing between different tasks. However, due to the lack of hardware-level isolation in MPS, tasks may still compete for critical resources such as memory bandwidth, L2 cache, and registers, leading to uncontrollable latency and unstable task performance, especially in scenarios where online inference and offline training are mixed.

[0004] Unlike the software-level sharing mechanism of MPS, MIG provides GPUs with hardware-level resource partitioning capabilities. By dividing GPU physical resources, such as SM, L2 cache, and Dynamic Random Access Memory (DRAM) bandwidth, into several independent sub-instances, MIG can provide strongly isolated runtime environments for different tasks, effectively avoiding interference caused by multi-task contention. Although MIG improves resource conflict issues, its fixed partition granularity leads to limited resource configuration space, and reconfiguration of MIG partitions often requires terminating all active tasks, resulting in high system overhead. In addition, static partitioning is prone to resource fragmentation, preventing GPU utilization from reaching its optimal level.

[0005] To combine the advantages of both, existing research has attempted to integrate MIG and MPS, enabling tasks to be shared within MIG sub-partitions via MPS while maintaining isolation between sub-partitions. However, this hybrid approach still faces several challenges in practical deployment: Firstly, MIG partitioning patterns are limited and complex to combine, while MPS thread ratio configuration space is enormous, making it extremely difficult to find the optimal configuration when dynamic tasks arrive; secondly, even in a MIG+MPS environment, fine-grained resource competition may still occur between tasks during kernel execution, such as bandwidth conflicts, L2 cache jitter, and uneven SM allocation, while existing technologies lack an effective scheduling mechanism for kernel-level resource behavior.

[0006] Meanwhile, deep learning service platforms in real-world production environments often involve a large number of online inference tasks and offline training tasks running in combination. These mixed deployment scenarios not only require GPU systems to have high throughput capabilities but also demand stable and predictable Quality of Service (QoS) for latency-sensitive tasks (such as real-time inference). However, existing MIG, MPS, or simple combinations thereof cannot provide real-time priority scheduling capabilities for online tasks, and traditional methods struggle to dynamically adjust GPU resource allocation based on actual resource utilization, easily leading to performance fluctuations or resource waste when facing dynamic load changes.

[0007] Existing works such as MIGER and MISO have attempted to improve GPU resource allocation through modeling methods, but they still mainly focus on partition-level or task-level scheduling, lacking fine-grained control over the kernel execution level. Since DNN model training and inference tasks have different resource behavior patterns (such as alternating computationally intensive and bandwidth-intensive phases), ignoring kernel-level resource contention will inevitably lead to problems such as long-tail latency, reduced throughput, and difficulty in guaranteeing QoS.

[0008] Therefore, in multi-tasking, multi-tenant GPU environments, how to strike a balance between the hardware isolation advantages of MIG and the resource elasticity of MPS, while introducing a more granular kernel-level scheduling mechanism to alleviate resource contention and improve GPU utilization, is a key technical problem that urgently needs to be solved in the field of GPU sharing and scheduling. Summary of the Invention

[0009] In view of the above-mentioned deficiencies of the prior art, the present invention at least solves the following technical problems: Existing GPU sharing technologies struggle to simultaneously meet the collaborative requirements of resource isolation, elastic sharing, and real-time online task performance, leading to performance instability and low resource utilization in multi-tasking scenarios. 1. MPS can only achieve software-level SM resource partitioning, lacking hardware-level isolation of key resources such as memory bandwidth and L2 cache, which can easily lead to resource contention between tasks and cause significant fluctuations in latency of online inference tasks; 2. Although MIG provides hardware-level resource partitioning, the partition granularity is fixed, reconfiguration requires terminating all active tasks, which is extremely costly. In addition, static partitioning is prone to resource fragmentation and is difficult to adapt to dynamic load. 3. The “MIG+MPS” hybrid solution has configuration optimization challenges. The MIG partition combination is complex, the MPS quota configuration space is huge, it is difficult to select the optimal solution when dynamic tasks arrive, and it lacks a kernel-level fine-grained scheduling mechanism, which cannot alleviate fine-grained resource contention such as bandwidth conflicts and L2 cache jitter. 4. Existing technologies cannot provide real-time priority scheduling capabilities for online inference tasks, making it difficult to balance the high throughput requirements in mixed deployment scenarios with the QoS stability of latency-sensitive tasks, and performance fluctuations or resource waste are likely to occur under dynamic loads. Current GPU multitasking scheduling lacks accurate performance prediction capabilities, leading to inefficient resource allocation. 1. Scheduling decisions rely on experience or fixed strategies, making it difficult to effectively assess the performance gains of tasks under different MIG and MPS configurations, and also making it difficult to predict the degree of resource contention brought about by multi-task concurrency, which can easily lead to performance degradation due to improper configuration. 2. Traditional scheduling methods frequently trigger MIG reconfiguration, resulting in additional system overhead and further reducing GPU resource utilization and operational stability.

[0010] This invention discloses a multi-layered GPU resource management and scheduling system for deep learning tasks, including a MIG hardware isolation layer, an MPS elastic sharing layer, a kernel-level fine-grained scheduling layer, a performance predictor, and a Hybrid PartitionScheduler. The MIG hardware isolation layer is used to divide and isolate GPU physical resources to form independent sub-instances; The MPS elastic sharing layer is used to dynamically adjust SM quotas according to task load to achieve elastic allocation of computing resources; The Kernel-level fine-grained scheduling layer is used to monitor Kernel-level resource contention status in real time and ensure that online tasks are executed with priority. The performance predictor is used to output task performance data under different MIG and MPS configurations; The Hybrid Partition Scheduler is used to select the optimal MIG partition combination and MPS quota strategy based on the performance data. Through cross-layer collaboration of the above components, the unification of resource isolation, elastic sharing, and real-time online task scheduling is achieved; Furthermore, the performance predictor is built based on the Random Forest algorithm and is configured with a task feature acquisition module; Furthermore, the task feature acquisition module is used to acquire task type, model size, memory requirements, SM usage trend, L2 cache access characteristics, and bandwidth usage pattern. Furthermore, the Kernel-level fine-grained scheduling layer includes a Kernel Scheduler, which is deployed in the GPU Runtime layer and is configured with a resource monitoring unit and a slack time evaluation model. Furthermore, the resource monitoring unit is used to collect data on SM occupancy ratio, L2 cache usage, memory bandwidth requirements, memory access conflict counts, and execution time estimates. Furthermore, the task performance data output by the performance predictor includes latency, throughput, and resource pressure indicators, providing data support for the configuration decisions of the Hybrid Partition Scheduler; Furthermore, the physical resource partitioning objects of the MIG hardware isolation layer include SM, L2 cache and DRAM bandwidth, and MIG partition instances are created through the GPU driver interface; Furthermore, when the Kernel Scheduler detects bandwidth or L2 conflicts or when online tasks are blocked by offline tasks, it evaluates the delay window of offline tasks through the slack time evaluation model and performs preemptive scheduling. Furthermore, the preemptive scheduling includes pausing offline task kernels, increasing the priority of online tasks, or slicing the execution of offline kernels; Furthermore, the MPS elastic sharing layer adopts a resource allocation logic that prioritizes adjusting SM quotas and reduces MIG reconfiguration, thereby reducing the proportion of MIG reconstruction in the task lifecycle.

[0011] The three-tier GPU shared architecture (MMK) proposed in this invention, which integrates MIG, MPS, and Kernel-level fine-grained scheduling, has significant technical advantages, superior performance, strong production feasibility, and broad prospects for industrial application compared to existing technologies, as detailed below: In terms of technical advantages, this invention provides a novel and efficient resource management solution for GPU multi-tasking concurrent environments. By combining the hardware isolation capabilities of MIG, the elastic resource sharing characteristics of MPS, and the fine-grained scheduling mechanism of the Kernel Scheduler, it precisely solves key pain points in existing GPU scheduling, such as severe resource contention, uncontrollable online task latency, and rigid MIG configuration. Simultaneously, by integrating a performance predictor based on Random Forest, it achieves a shift in scheduling decisions from experience-driven to data-driven, making the combined optimization of MIG and MPS more accurate and reliable. It can dynamically adjust resource allocation strategies according to the different characteristics of online inference and offline training tasks, maximizing global throughput while ensuring the real-time performance of online tasks, and significantly improving the overall operating performance of multi-tasking GPU systems. In terms of performance metrics, experimental verification on data center-grade GPU platforms such as NVIDIA A100 shows that the performance of this invention is significantly better than existing GPU resource scheduling methods: system throughput can be increased by about 35%, average task completion time is reduced by about 28%, and overall task execution cycle (Makespan) is reduced by about 32%; through fine-grained resource contention control of the Kernel Scheduler, latency jitter of online inference tasks is significantly reduced, QoS stability is significantly improved, and online task latency can be reduced by more than 40% and latency fluctuation by more than 60%; the prediction error of the performance predictor under different MIG / MPS configurations is only about 3.5%, which is far better than traditional empirical configuration strategies; the operating overhead of the scheduling system is extremely low, with MIG reconstruction accounting for only about 1.5% of the task lifecycle and the monitoring overhead of the Kernel Scheduler being about 0.08%, achieving a balance between scheduling accuracy and operating efficiency; In terms of production implementation, this invention demonstrates excellent feasibility and engineering scalability: the core mechanism adopts a hierarchical scheduling approach, enabling seamless integration with existing deep learning platforms such as Kubernetes, Ray, and Slurm, as well as various cloud-based GPU training services; the performance predictor and scheduler modules are highly portable, adaptable to multiple GPU architectures such as A100 and H100, and have a wide range of applications; the scheduling process is fully automated, requiring no manual adjustment of MIG / MPS parameters, and automatically configuring the optimal resource solution based on task characteristics, significantly reducing the operational burden on deep learning engineers. The overall framework is clear, easily extensible, and suitable for large-scale GPU cluster scheduling and deep learning service deployment in real-world production scenarios. In terms of industrial applications, thanks to its superior performance enhancement capabilities and highly automated resource management design, this invention has significant industrial application value: it can be widely applied to high-concurrency, multi-task deep learning scenarios such as image recognition, speech recognition, recommendation systems, and natural language processing, effectively improving the latency stability of inference services and the throughput efficiency of training tasks; it can serve as a core scheduling component for GPU cloud services, providing refined multi-tenant resource isolation and scheduling services for enterprise-level AI clusters, cloud-native AI platforms, and large-scale model training infrastructure; its data-driven scheduling and automated resource configuration capabilities significantly reduce operational complexity, helping enterprises achieve higher GPU resource utilization under controllable costs, and has the potential for large-scale application in fields such as AI cloud computing, intelligent service platforms, large-scale model infrastructure, and edge inference services. Attached Figure Description

[0012] Figure 1 This is a flowchart of the multi-level GPU resource management and scheduling method for deep learning tasks according to the present invention; Figure 2 This is a schematic diagram of the fine-grained GPU resource scheduling mechanism architecture based on Kernel Scheduler of this invention. Detailed Implementation

[0013] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0014] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0015] This invention discloses a multi-layered GPU resource management and scheduling system for deep learning tasks, including a MIG hardware isolation layer, an MPS elastic sharing layer, a kernel-level fine-grained scheduling layer, a performance predictor, and a Hybrid PartitionScheduler. The MIG hardware isolation layer is used to divide and isolate GPU physical resources to form independent sub-instances; The MPS elastic sharing layer is used to dynamically adjust SM quotas according to task load, so as to realize elastic allocation of computing resources; The Kernel-level fine-grained scheduling layer is used to monitor Kernel-level resource contention status in real time and ensure that online tasks are executed with priority. The performance predictor is used to output task performance data under different MIG and MPS configurations; Hybrid Partition Scheduler is used to select the optimal MIG partition combination and MPS quota strategy based on the above performance data; Through cross-layer collaboration among the aforementioned components, a unified approach to resource isolation, elastic sharing, and real-time online task scheduling is achieved.

[0016] The specific implementations of each embodiment will be described in detail below with reference to the accompanying drawings: Example 1 In this embodiment, the disclosed GPU MIG and MPS collaborative scheduling method driven by a performance predictor is a concrete implementation of the core collaborative logic of the "MIG hardware isolation layer, MPS elastic sharing layer, and Hybrid Partition Scheduler" in the multi-layer GPU resource management and scheduling system for deep learning tasks of this invention. It is used to automatically select the optimal GPU resource configuration in multi-task deep learning scenarios, such as... Figure 1 As shown.

[0017] The system first acquires basic task information through a task feature acquisition module, including task type, model size, memory requirements, SM usage trends, L2 cache access characteristics, and bandwidth usage patterns. This information is then organized into task feature vectors and input into the performance predictor. The performance predictor of this invention employs a random forest model, learning the performance of different tasks under different MIG partitioning structures and MPS quota settings through a large number of historical samples. It can output key indicators such as latency, throughput, and resource pressure under various configurations, transforming the scheduling process from experience-based decision-making to data-driven decision-making. Figure 1 The collaborative logic between the "performance predictor" and the "random forest model" is shown in the diagram.

[0018] The Hybrid Partition Scheduler calculates the overall benefit of combining candidate MIGs and MPS based on the prediction results, including the degree to which online task latency is satisfied, the improvement in offline task throughput, and configuration costs, and selects the optimal solution. Subsequently, the scheduler calls the MIG management module to create actual MIG partition instances through the GPU driver interface and uses the MPS control module to allocate different SM quotas to each task to achieve optimal resource allocation. Figure 1 The execution flow of "Creating MIG Partition Instance" and "MPS Control" is shown in the figure.

[0019] The innovation of this embodiment lies in the collaborative operation of the performance predictor and the Hybrid Partition Scheduler, which constructs a data-driven automated resource allocation mechanism. This not only solves the blindness of traditional experience-based configuration but also controls the performance prediction error of different MIG and MPS combinations to a low level of approximately 3.5%, providing core technical support for a 35% increase in overall system throughput and a 28% reduction in average task completion time. Simultaneously, by accurately matching task requirements with resource allocation, this method effectively reduces resource fragmentation caused by static MIG partitioning, directly improving the accuracy and efficiency of system resource allocation in multi-task mixed deployment environments.

[0020] Example 2

[0021] In this embodiment, the disclosed fine-grained GPU resource scheduling mechanism based on the Kernel Scheduler is the core implementation method of the "Kernel-level fine-grained scheduling layer" in the multi-layered GPU resource management and scheduling system for deep learning tasks of this invention. It is used to solve the resource contention problem caused by concurrent execution of multiple tasks on the GPU, ensuring the real-time performance of online inference tasks, such as... Figure 2 As shown.

[0022] The Kernel Scheduler is deployed at the GPU Runtime layer. It intercepts CUDA Kernel startup instructions to obtain real-time hardware resource usage information for each Kernel, including SM usage ratio, L2 cache usage, memory bandwidth requirements, memory access conflict counts, and execution time estimates. Based on this data, it determines in real-time whether resource contention exists. Figure 2 The linkage logic between the "Kernel Scheduler" and the "Resource Monitoring Unit" is shown in the diagram.

[0023] When the system detects that the kernel for the online inference task is blocked by the offline training task, it first evaluates the deferable time window of the training task using the SlackTime model. Then, it executes the online task priority mechanism, which allows the online task to preempt GPU resources by pausing the training task kernel, increasing the execution priority of the inference task kernel, or sliced ​​execution of the training kernel. Figure 2 The scheduling path is shown in the diagram, which includes "pause training," "increase inference priority," and "slice execution." Once the online task is completed or resource constraints are alleviated, the system resumes the execution of the training task.

[0024] The innovation of this embodiment lies in the fact that a fine-grained real-time resource contention solution is built through the Kernel Scheduler. This not only achieves the QoS guarantee goal of reducing online task latency by more than 40% and latency fluctuation by more than 60%, but also ensures that the overall throughput of offline training tasks is basically unaffected through offline task delay window evaluation and sliced ​​execution. This provides key technical support for the system to balance "high real-time performance and high throughput", and perfectly adapts to the core requirements of multi-task mixed deployment scenarios.

[0025] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multi-level GPU resource management and scheduling system for deep learning tasks, characterized in that, It includes the MIG hardware isolation layer, the MPS elastic sharing layer, the Kernel-level fine-grained scheduling layer, as well as the performance predictor and the Hybrid PartitionScheduler; The MIG hardware isolation layer is used to divide and isolate GPU physical resources to form independent sub-instances; The MPS elastic sharing layer is used to dynamically adjust SM quotas according to task load to achieve elastic allocation of computing resources; The Kernel-level fine-grained scheduling layer is used to monitor Kernel-level resource contention status in real time and ensure that online tasks are executed with priority. The performance predictor is used to output task performance data under different MIG and MPS configurations; The Hybrid Partition Scheduler is used to select the optimal MIG partition combination and MPS quota strategy based on the performance data. Through cross-layer collaboration among the aforementioned components, a unified approach to resource isolation, elastic sharing, and real-time online task scheduling is achieved.

2. The multi-level GPU resource management and scheduling system for deep learning tasks as described in claim 1, characterized in that, The performance predictor is built based on the Random Forest algorithm and is equipped with a task feature acquisition module.

3. The multi-level GPU resource management and scheduling system for deep learning tasks as described in claim 2, characterized in that, The task feature acquisition module is used to obtain task type, model size, memory requirements, SM usage trend, L2 cache access characteristics, and bandwidth usage pattern.

4. The multi-level GPU resource management and scheduling system for deep learning tasks as described in claim 1, characterized in that, The Kernel-level fine-grained scheduling layer includes a Kernel Scheduler, which is deployed in the GPURuntime layer and is configured with a resource monitoring unit and a slack time evaluation model.

5. The multi-level GPU resource management and scheduling system for deep learning tasks as described in claim 4, characterized in that, The resource monitoring unit is used to collect data on SM occupancy rate, L2 cache usage, memory bandwidth requirements, memory access conflict count, and execution time estimation.

6. The multi-level GPU resource management and scheduling system for deep learning tasks as described in claim 1, characterized in that, The task performance data output by the performance predictor includes latency, throughput, and resource pressure metrics, providing data support for the configuration decisions of the HybridPartition Scheduler.

7. The multi-level GPU resource management and scheduling system for deep learning tasks as described in claim 1, characterized in that, The physical resource partitioning objects of the MIG hardware isolation layer include SM, L2 cache and DRAM bandwidth, and MIG partition instances are created through the GPU driver interface.

8. The multi-level GPU resource management and scheduling system for deep learning tasks as described in claim 4, characterized in that, When the Kernel Scheduler detects bandwidth or L2 conflicts, or when online tasks are blocked by offline tasks, it evaluates the delay window of the offline tasks using the slack time evaluation model and performs preemptive scheduling.

9. The multi-level GPU resource management and scheduling system for deep learning tasks as described in claim 8, characterized in that, The preemptive scheduling includes pausing offline kernel tasks, increasing the priority of online tasks, or slicing the execution of offline kernel tasks.

10. The multi-level GPU resource management and scheduling system for deep learning tasks as described in claim 1, characterized in that, The MPS elastic sharing layer adopts a resource allocation logic that prioritizes adjusting SM quotas and reduces MIG reconfiguration, thereby reducing the proportion of MIG reconstruction in the task lifecycle.

Images