A method and system for quality of service oriented GPU space-time adaptive resource scheduling

Abstract

The application discloses a quality-of-service oriented GPU space-time adaptive resource scheduling method and system, belongs to the field of heterogeneous resource scheduling and management under deep learning inference, and comprises the following steps: determining a minimum SM subset satisfying a quality-of-service constraint and corresponding SM mask configuration in an offline stage; collecting multi-task co-location operation data offline, and constructing an interference-aware delay prediction model by using the multi-task co-location operation data; in an online scheduling stage, when a new task arrives, it is judged whether the GPU is in an idle state according to a current GPU load state; the constructed interference-aware delay prediction model is used for prediction, and it is judged whether the predicted inference delay and the predicted delay both satisfy preset quality-of-service constraints; a time multiplexing strategy is adopted to schedule and execute tasks by intercepting a kernel-level command buffer, and it is ensured that only one task occupies all resources of the GPU at the same time. The application can improve system throughput and guarantee the quality of service of delay-sensitive tasks.

Description

Technical Field

[0001] This invention belongs to the field of heterogeneous resource scheduling and management under deep learning inference, specifically involving a GPU spatiotemporal adaptive resource scheduling method and system oriented towards quality of service assurance. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially deep neural networks, computing tasks place extremely high demands on throughput and storage bandwidth. GPUs, with their powerful parallel computing capabilities, have become the mainstream computing platform. However, in latency-sensitive applications, inference tasks often employ small batch sizes to meet stringent Quality of Service (QoS) requirements. This makes it difficult for a single task to fully utilize the GPU's streaming processing units (SMs) and other computing and storage resources, resulting in significant waste. Therefore, co-location of multiple computing tasks running simultaneously on the same GPU has become a key means to improve the overall throughput and resource utilization of the GPU.

[0003] In multi-task co-location scenarios, existing GPU multi-task management solutions mainly fall into two categories: time reuse strategies and space partitioning strategies. Time reuse controls the execution order of tasks in the time dimension, allowing multiple tasks to take turns exclusively using GPU resources to ensure the stability of a single task. Space partitioning (such as multi-process service (MPS) and multi-instance GPU technologies) divides GPU hardware resources in the spatial dimension, allocating different sub-resources to different tasks in order to achieve high-concurrency parallel execution of tasks.

[0004] However, current GPU scheduling methods all have significant shortcomings in practical applications. First, time-reuse schemes are essentially serial executions, which not only severely limit concurrency and easily lead to queue head blocking, but also fail to promptly reclaim idle computing resources during execution, significantly limiting the overall system throughput. Second, existing space partitioning schemes have coarse-grained partitioning and static configurations, lacking runtime dynamic adjustment capabilities; more importantly, since different tasks still share underlying physical resources such as L2 cache and GPU memory bandwidth under space partitioning, high-intensity concurrent access can easily cause severe performance interference and latency escalation, resulting in very limited QoS guarantees. In summary, existing solutions cannot achieve a dynamic balance between maximizing GPU resource utilization and strictly guaranteeing service quality. Therefore, there is an urgent need to design a new method that can adaptively perform fine-grained resource scheduling and isolation in both time and space dimensions based on task characteristics and interference risks at runtime. Summary of the Invention To address the limitations of existing technologies, this invention provides a GPU spatiotemporal adaptive resource scheduling method and system for quality of service assurance. The aim is to provide a GPU resource management scheme that can adaptively schedule and isolate resources in the time and space dimensions based on task characteristics and interference risks at runtime, thereby simultaneously improving system throughput and ensuring the quality of service for latency-sensitive tasks.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A GPU spatiotemporal adaptive resource scheduling method for quality of service assurance includes: S1. In the offline phase, perform performance profiling on the deep neural network task to determine the minimum subset of service quality constraints and the corresponding SM mask configuration. S2. Collect multi-task co-location operation data offline, and use the multi-task co-location operation data to build an interference perception delay prediction model; S3. During the online scheduling phase, when a new task arrives, determine whether the GPU is idle based on the current GPU load status. If so, start the task directly through the space partitioning mode using the determined minimum SM subset and the corresponding SM mask configuration. If not, proceed to the next step. S4. Based on the constructed interference-aware delay prediction model, predict the inference delay of the new task in the co-location environment with the current running task, and the prediction delay of the current running task after the addition of the new task; determine whether the predicted inference delay and prediction delay both meet the preset service quality constraints; if so, execute the elastic space partitioning strategy, establish a set of non-overlapping SM masks, and allow tasks to co-locate and execute in parallel; if not, proceed to the next step. S5. Based on kernel-level command buffer interception, a time-reuse strategy is used to schedule and execute tasks, ensuring that only one task can exclusively occupy all GPU resources at any given time, in order to prevent resource interference from causing service quality defaults; after the task finishes running, the elastic space partitioning strategy is restored.

[0006] A further improvement of this invention is that the determination of the minimum subset of service quality constraints and the corresponding service quality mask in step S1 adopts a binary search-based method, specifically including the following steps: S11, Baseline latency of the measurement model under full GPU resources And set an allowable performance loss threshold. S12. Perform a binary search within the range of 1 to the total number of global SMs, calculate the median number of SMs in each search, and generate the corresponding SM mask; S13. Run the target DNN task on the allocated median number of SMs and measure the inference latency. Calculate the performance degradation rate S14. Determine the performance degradation rate. Is it less than or equal to the performance loss threshold? If yes, record the current optimal number of SMs and its corresponding SM mask, and search for a smaller range of SMs; if no, search for a larger range; S15, output the minimum number of SMs that finally satisfies the QoS constraints and its corresponding SM mask configuration.

[0007] A further improvement of this invention is that the construction of the interference-aware delay prediction model using multi-task co-location operation data in step S2 specifically includes the following steps: S21. Key performance characteristics of offline data collection tasks: Combine the resource characteristics of the task to be run with those of the currently running task to form a feature vector. ;in and S22 represents the task's demand intensity for L2 cache bandwidth and video memory bandwidth, respectively; S22 represents the task inference latency observed under co-location execution conditions. As a supervisory signal, constructing regression learning problems S23. A gradient boosting decision tree-based regression model, XGBoost, is used for training. Multiple weak regressors are combined using an additive model. By minimizing the objective function, a balance is achieved between prediction accuracy and model complexity, thus obtaining an interference-aware delay prediction model.

[0008] A further improvement of this invention is that, in step S3, the task is started directly through the space partitioning mode using a determined minimum subset of SMs and the corresponding SM mask configuration, specifically including the following steps: The system identifies that no other tasks are running on the current GPU, extracts the optimal number of task managers (SMs) and SM mask configuration for the corresponding new task, dynamically intercepts the kernel of the corresponding task, and injects task metadata containing the optimal number of SMs and SM mask configuration into the current task before the kernel is launched to the GPU. The kernel with the injected task metadata is launched to the GPU, so that the kernel of the corresponding task starts running on the specified minimum subset of SMs.

[0009] A further improvement of this invention is that the execution of the flexible space partitioning strategy in step S4, which establishes a set of non-overlapping SM masks and allows tasks to be co-located and executed in parallel, specifically includes the following steps: S41. The centralized scheduler calculates the non-overlapping SM mask configuration for each task and injects task metadata containing SM resource configuration parameters into the current task before the GPU kernel is launched. S42. The kernel of the corresponding task is dynamically intercepted, and before the kernel is launched to the GPU, the calculated SM mask configuration is dynamically bound to the corresponding CUDA stream or kernel execution context. S43. During kernel execution, constraints are applied by the GPU hardware scheduler to allow only the thread blocks of the task to be allocated to the subset of SMs specified by the SM mask to run, thereby achieving physical SM-level resource isolation.

[0010] A further improvement of this invention is that the kernel-level command buffer interception described in step S5 employs a time-reuse strategy to schedule and execute tasks, ensuring that only one task exclusively occupies all GPU resources at any given time. Specifically, this includes the following steps: S51. During application execution, the user-mode transparent proxy component intercepts the GPU interaction interface, identifies and records the virtual address range of the command buffer used to submit GPU instructions; S52. Utilizes the memory protection mechanism provided by the operating system to lock write permissions and sets the memory page corresponding to the command buffer to a read-only state; S53. Captures the page-level write protection exception triggered when the application writes instructions to the command buffer, suspends the current thread, and initiates a GPU time slice request to the centralized scheduler; S54. After obtaining execution permission authorization, temporarily restores the write permission of the command buffer to allow instructions to be written and submitted, and sets the command buffer to a read-only state again after the time slice ends to reclaim GPU execution rights.

[0011] A GPU spatiotemporal adaptive resource scheduling system for quality of service assurance includes: Offline processing unit: In the offline stage, it performs performance profiling on deep neural network tasks and determines the minimum subset of service quality constraints and the corresponding service quality mask configuration. Interference-aware delay prediction model building unit: offline collection of multi-task co-location operation data, and use of multi-task co-location operation data to build an interference-aware delay prediction model; Online scheduling phase processing unit: During the online scheduling phase, when a new task arrives, it determines whether the GPU is idle based on the current GPU load status; if so, it directly starts the task through the space partitioning mode using a determined minimum SM subset and the corresponding SM mask configuration; if not, it executes the next unit. Judgment and prediction unit: Based on the constructed interference-aware delay prediction model, predict the inference delay of the new task in the co-location environment with the current running task, and the prediction delay of the current running task after the addition of the new task; determine whether the predicted inference delay and prediction delay both meet the preset service quality constraints; if so, execute the flexible space partitioning strategy, establish a set of non-overlapping SM masks, and allow tasks to co-locate and execute in parallel; if not, execute the next unit. Execution Unit: Based on kernel-level command buffer interception, a time-reuse strategy is used to schedule and execute tasks, ensuring that only one task can exclusively occupy all GPU resources at any given time, in order to prevent resource interference from causing service quality defaults; after the task finishes running, the elastic space partitioning strategy is restored.

[0012] A further improvement of this invention is that the determination of the minimum subset of service quality constraints and the corresponding service quality mask in the offline processing unit adopts a binary search-based method, specifically including the following steps: S11, Baseline latency of the measurement model under full GPU resources And set an allowable performance loss threshold. S12. Perform a binary search within the range of 1 to the total number of global SMs, calculate the median number of SMs in each search, and generate the corresponding SM mask; S13. Run the target DNN task on the allocated median number of SMs and measure the inference latency. Calculate the performance degradation rate S14. Determine the performance degradation rate. Is it less than or equal to the performance loss threshold? If yes, record the current optimal number of SMs and its corresponding SM mask, and search for a smaller range of SMs; if no, search for a larger range; S15, output the minimum number of SMs that finally satisfies the QoS constraints and its corresponding SM mask configuration.

[0013] A further improvement of this invention is that the method of constructing the interference-aware delay prediction model using multi-task co-location running data in the interference-aware delay prediction model construction unit specifically includes the following steps: S21. Key performance characteristics of offline data collection tasks: Combine the resource characteristics of the task to be run with those of the currently running task to form a feature vector. ;in and S22 represents the task's demand intensity for L2 cache bandwidth and video memory bandwidth, respectively; S22 represents the task inference latency observed under co-location execution conditions. As a supervisory signal, constructing regression learning problems S23. A gradient boosting decision tree-based regression model, XGBoost, is used for training. Multiple weak regressors are combined using an additive model. By minimizing the objective function, a balance is achieved between prediction accuracy and model complexity, thus obtaining an interference-aware delay prediction model.

[0014] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a GPU spatiotemporal adaptive resource scheduling method for quality of service assurance.

[0015] The present invention has at least the following beneficial technical effects: Compared with existing GPU multitasking scheduling and resource management technologies, the GPU spatiotemporal adaptive resource scheduling method and system proposed in this invention, which is oriented towards quality of service assurance, has significant advantages in many aspects, such as architecture design, scheduling strategy flexibility, quality of service assurance capability, and engineering deployability. It can effectively solve the fundamental problem that existing technologies cannot balance resource utilization and QoS assurance. The specific advantages are reflected in the following aspects.

[0016] (i) Achieving a dynamic balance between GPU resource utilization and QoS guarantee Existing GPU scheduling schemes generally present a significant contradiction between maximizing resource utilization and strictly guaranteeing quality of service (QoS): while time reuse schemes offer strong isolation, they sacrifice internal GPU parallelism; spatial partitioning schemes, although improving concurrency, suffer from static resource configuration and uncontrollable interference, easily leading to latency jitter. This invention introduces a spatiotemporal adaptive scheduling mechanism that dynamically selects between spatial parallel execution and time isolation execution strategies based on task characteristics, system load status, and interference risk predictions during runtime, thereby achieving optimal scheduling decisions in different operating scenarios. When the interference risk is low, the system prioritizes spatial partitioning to run multiple tasks in parallel, maximizing GPU throughput; when a potential QoS breach risk is predicted, the system automatically switches to the time isolation mechanism to ensure that the QoS of latency-sensitive tasks is not affected. This dynamic balancing mechanism breaks through the traditional "either / or" scheduling paradigm, making GPU resource utilization and QoS guarantees no longer mutually exclusive, significantly improving the overall system efficiency and stability. II. Utilizing page protection mechanisms to implement kernel-mode mandatory access control and construct physical-level resource isolation boundaries.

[0017] (ii) Supports fine-grained spatial isolation at the SM level, with high precision in resource allocation. To address the issues of coarse-grained partitioning and uneven resource utilization in existing spatial partitioning techniques (such as MPS and MIG), this invention proposes a flexible spatial partitioning mechanism based on Streaming Multiprocessor (SM) masks, achieving fine-grained isolation of GPU computing resources at the Streaming Multiprocessor (SM) level. This mechanism can impose precise SM usage constraints on tasks during the kernel launch phase, ensuring that different tasks execute only on their respective authorized SM subsets, thus achieving true mutual exclusion of computing resources at the hardware level. Compared to existing schemes that partition resources using fixed ratios or fixed hardware units, this invention offers higher resource allocation precision and better matches the actual computing needs of different tasks at different stages. Through this mechanism, this invention effectively avoids the waste caused by over-allocation of resources, while also preventing performance degradation caused by insufficient resources, providing a more reasonable resource allocation method for high-concurrency and diverse GPU workloads.

[0018] (iii) Possesses dynamic adjustment capability during operation and strong adaptability to load changes. Most existing GPU space allocation techniques configure resources before task startup. Once a task is running, it's difficult to adjust according to load changes, and reconfiguration often requires interrupting the task or restarting the device, resulting in significant system overhead. This invention achieves the ability to dynamically adjust GPU resource allocation at runtime through a centralized resource management and scheduling mechanism. When a task arrives, completes, or its load characteristics change, the system can instantly recalculate the resource allocation scheme and dynamically update the SM mask or scheduling policy for each task without terminating the currently executing task. This dynamic adjustment capability enables this invention to effectively handle situations with frequent load changes and inconsistent task lifecycles in real-world deployment environments, significantly improving the system's adaptability and robustness in real production environments.

[0019] (iv) Introduce interference-aware delay prediction to achieve pre-emptive control of QoS risks. Existing GPU multitasking scheduling schemes generally lack effective modeling of performance interference, typically only addressing it passively after QoS violations have occurred, leading to poor system stability. This invention innovatively introduces an interference-aware latency prediction mechanism. By modeling the resource characteristics of tasks, it can predict potential performance interference risks before tasks are actually co-located and executed. This prediction mechanism can not only assess the execution latency of new tasks in co-location scenarios but also predict their performance impact on existing tasks, thus achieving bidirectional QoS risk assessment. By completing risk assessment before scheduling decisions are made, this invention achieves proactive prevention of QoS violations, avoiding the cascading amplification effect of performance jitter and significantly improving system stability and predictability in multitasking scenarios.

[0020] (v) The system is transparent, highly versatile, and has low engineering deployment costs. This invention employs a user-space transparent proxy implementation, controlling GPU scheduling behavior through a dynamic linking injection mechanism without requiring any modification to the application source code or relying on specific upper-layer programming interfaces. This design significantly reduces system deployment and maintenance costs, enabling existing GPU applications to directly benefit from the scheduling capabilities of this invention without modification. Furthermore, the core mechanism of this invention focuses on the critical path of GPU driver-hardware interaction, exhibiting strong versatility and scalability. It can adapt to different GPU platforms and other command buffer-based acceleration devices, demonstrating good engineering feasibility and promotional value. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is the overall architecture diagram of a GPU spatiotemporal adaptive resource scheduling method for quality of service assurance.

[0023] Figure 2 This is a flowchart of the SM allocation algorithm based on binary search.

[0024] Figure 3 Flowchart of a QoS-aware GPU multitasking scheduling algorithm.

[0025] Figure 4 This is a block diagram of a GPU spatiotemporal adaptive resource scheduling system for quality of service assurance. Detailed Implementation

[0026] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0027] In the description of this invention, it should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0029] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0031] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] Example 1 This invention provides a GPU spatiotemporal adaptive resource scheduling method for quality of service assurance, comprising: S1. In the offline phase, perform performance profiling on the deep neural network task to determine the minimum subset of service quality constraints and the corresponding SM mask configuration. S2. Collect multi-task co-location operation data offline, and use the multi-task co-location operation data to build an interference perception delay prediction model; S3. During the online scheduling phase, when a new task arrives, determine whether the GPU is idle based on the current GPU load status. If so, start the task directly through the space partitioning mode using the determined minimum SM subset and the corresponding SM mask configuration. If not, proceed to the next step. S4. Based on the constructed interference-aware delay prediction model, predict the inference delay of the new task in the co-location environment with the current running task, and the prediction delay of the current running task after the addition of the new task; determine whether the predicted inference delay and prediction delay both meet the preset service quality constraints; if so, execute the elastic space partitioning strategy, establish a set of non-overlapping SM masks, and allow tasks to co-locate and execute in parallel; if not, proceed to the next step. S5. Based on kernel-level command buffer interception, a time-reuse strategy is used to schedule and execute tasks, ensuring that only one task can exclusively occupy all GPU resources at any given time, in order to prevent resource interference from causing service quality defaults; after the task finishes running, the elastic space partitioning strategy is restored.

[0033] In step S1 of this embodiment, determining the minimum subset of service quality constraints (SMs) and their corresponding SM masks is achieved using a binary search-based method, specifically including the following steps: S11, Baseline latency of the measurement model under full GPU resources And set an allowable performance loss threshold. S12. Perform a binary search within the range of 1 to the total number of global SMs, calculate the median number of SMs in each search, and generate the corresponding SM mask; S13. Run the target DNN task on the allocated median number of SMs and measure the inference latency. Calculate the performance degradation rate S14. Determine the performance degradation rate. Is it less than or equal to the performance loss threshold? If yes, record the current optimal number of SMs and its corresponding SM mask, and search for a smaller range of SMs; if no, search for a larger range; S15, output the minimum number of SMs that finally satisfies the QoS constraints and its corresponding SM mask configuration.

[0034] In step S2 of this embodiment, the construction of the interference-aware delay prediction model using multi-task co-location operation data specifically includes the following steps: S21. Key performance characteristics of offline data collection tasks: Combine the resource characteristics of the task to be run with those of the currently running task to form a feature vector. ;in and S22 represents the task's demand intensity for L2 cache bandwidth and video memory bandwidth, respectively; S22 represents the task inference latency observed under co-location execution conditions. As a supervisory signal, constructing regression learning problems S23. A gradient boosting decision tree-based regression model, XGBoost, is used for training. Multiple weak regressors are combined using an additive model. By minimizing the objective function, a balance is achieved between prediction accuracy and model complexity, thus obtaining an interference-aware delay prediction model.

[0035] In step S3 of this embodiment, the task is started directly through the space partitioning mode using a determined minimum SM subset and the corresponding SM mask configuration. This specifically includes the following steps: The system identifies that no other tasks are running on the current GPU, extracts the optimal number of task managers (SMs) and SM mask configuration for the corresponding new task, dynamically intercepts the kernel of the corresponding task, and injects task metadata containing the optimal number of SMs and SM mask configuration into the current task before the kernel is launched to the GPU. The kernel with the injected task metadata is launched to the GPU, so that the kernel of the corresponding task starts running on the specified minimum subset of SMs.

[0036] In step S4 of this embodiment, the execution of the flexible space partitioning strategy, establishing a set of non-overlapping SM masks and allowing tasks to be co-located and executed in parallel, specifically includes the following steps: S41. The centralized scheduler calculates the non-overlapping SM mask configuration for each task and injects task metadata containing SM resource configuration parameters into the current task before the GPU kernel is launched. S42. The kernel of the corresponding task is dynamically intercepted, and before the kernel is launched to the GPU, the calculated SM mask configuration is dynamically bound to the corresponding CUDA stream or kernel execution context. S43. During kernel execution, constraints are applied by the GPU hardware scheduler to allow only the thread blocks of the task to be allocated to the subset of SMs specified by the SM mask to run, thereby achieving physical SM-level resource isolation.

[0037] In step S5 of this embodiment, the kernel-level command buffer interception and time-reuse strategy for scheduling and executing tasks ensures that only one task exclusively occupies all GPU resources at any given time. This specifically includes the following steps: S51. During application execution, the user-mode transparent proxy component intercepts the GPU interaction interface, identifies and records the virtual address range of the command buffer used to submit GPU instructions; S52. Utilizes the memory protection mechanism provided by the operating system to lock write permissions and sets the memory page corresponding to the command buffer to a read-only state; S53. Captures the page-level write protection exception triggered when the application writes instructions to the command buffer, suspends the current thread, and initiates a GPU time slice request to the centralized scheduler; S54. After obtaining execution permission authorization, temporarily restores the write permission of the command buffer to allow instructions to be written and submitted, and sets the command buffer to a read-only state again after the time slice ends to reclaim GPU execution rights.

[0038] Example 2 This invention provides a GPU spatiotemporal adaptive resource scheduling method and system for quality of service assurance. By combining task offline feature analysis, interference-aware latency prediction, and spatiotemporal collaborative scheduling mechanisms at runtime, it achieves a dynamic balance between maximizing GPU resource utilization and the quality of service (QoS) of latency-sensitive tasks.

[0039] (I) Overview of the overall technical solution The overall system adopts a two-stage architecture that separates offline analysis and online scheduling, including: Offline performance profiling phase: Performance modeling is performed on different deep neural network (DNN) tasks to obtain their key performance characteristics under different resource configurations and co-location scenarios, which are used to guide runtime resource allocation and interference prediction; Online Spatiotemporal Adaptive Scheduling Phase: During task execution, the scheduling strategy is adaptively switched between the spatial dimension (SM partitioning) and the temporal dimension (time-division multiplexing) based on the real-time load status and predicted performance interference risks, thereby improving the concurrent execution capability of the GPU while ensuring QoS.

[0040] The entire system is centered around a user-space transparent proxy component, which allows for fine-grained control over GPU resource scheduling without requiring any modifications to the application source code.

[0041] (II) System Architecture Composition like Figure 1 As shown, the system of the present invention includes the following functional modules: Offline performance profiling module: used to perform performance modeling for different deep neural network tasks, execute a binary search-based SM allocation algorithm to determine the minimum SM subset that meets QoS and the corresponding SM mask, and train an interference-aware delay prediction model; Elastic Partitioning: Used to perform fine-grained spatial isolation and dynamic resource binding for different tasks based on SM masks, according to scheduling decisions when the risk of interference is low. Kernel-level command buffer access interception module: Used to initiate time slice scheduling when there is a high risk of interference leading to QoS violation, and to achieve strict time isolation by intercepting GPU command buffer writes and setting memory read and write protection; Interference-aware delay prediction and QoS-aware scheduling decision module: Before task scheduling, it uses the prediction model to evaluate the inference delay of newly arriving tasks and currently running tasks under co-location conditions, and performs adaptive scheduling decisions under QoS constraints. User-space transparent agent component (Agent).

[0042] The above modules work together to achieve dynamic scheduling and isolation of GPU resources in both time and space dimensions.

[0043] (III) Offline performance analysis and method for determining minimum SM requirements During the offline phase, the system performs the following processing for each DNN inference task: 1. Performance Indicator Collection The target DNN model was run on the GPU in both exclusive execution and multi-task co-location execution modes, and its key performance characteristics were collected, including but not limited to: inference latency; SM utilization; L2 cache bandwidth utilization; and video memory (DRAM) bandwidth utilization.

[0044] 2. Determining the minimum SM subset To avoid over- or under-allocation of runtime resources, we employ a binary search-based method for determining the number of task manager (SM). The core idea is to use the execution latency of the task under full GPU resources as a benchmark, and within an allowable performance loss threshold (within 5% or a custom threshold). The number of available SMs is gradually reduced through binary search; the smallest subset of SMs that meets the QoS constraints is found, and the corresponding SM mask is generated.

[0045] First, the baseline latency of the model under full GPU resources was measured. .

[0046] Set the allowable performance loss threshold .

[0047] By adjusting the size of the SM mask, in [1, Perform a binary search within the range of ] .

[0048] For the median number of SMs in each search, measure the model latency. Calculate the performance degradation rate .like If the number of SMs is sufficient, search for a smaller range of SMs; otherwise, search for a larger range.

[0049] The final output is the minimum number of SMs that satisfy the QoS constraints. The corresponding SM mask configuration serves as the baseline resource requirement for this model in spatial isolation mode.

[0050] The detailed flowchart is as follows: Figure 2 As shown, this method significantly reduces offline analysis overhead compared to linear search, while providing a high-precision resource allocation basis for dynamic space partitioning at runtime. Subsequently, based on the acquired data, we construct a performance profile library and train an interference prediction model before the system goes online.

[0051] 3. Training of Interference-Aware Delay Prediction Model An interference-aware latency prediction model is constructed using multi-task co-location operation data collected offline. The co-location operation data includes, but is not limited to, L2 cache bandwidth usage, DRAM bandwidth usage, SM usage characteristics, and corresponding inference latency changes of different tasks under co-location execution conditions.

[0052] In this embodiment, the resource characteristics of the task to be run and the currently running task are combined to form a feature vector:

[0053] in and These represent the task's demand intensity for L2 cache bandwidth and GPU memory bandwidth, respectively; other dimensions can be expanded according to actual needs. The inference latency of the task is based on the observed latency under co-location execution conditions. As a supervisory signal, construct the regression learning problem:

[0054] in This represents the interference-aware delay prediction model.

[0055] In this embodiment, a gradient boosting decision tree-based regression model (XGBoost) is preferably used, which combines multiple weak regressors in an additive model manner:

[0056] in This represents the set of regression tree functions. The model is trained by minimizing an objective function that includes a prediction error term and a model complexity term, in order to achieve a balance between prediction accuracy and model complexity.

[0057] The XGBoost model was chosen for two reasons: firstly, it effectively characterizes the nonlinear relationship between GPU resource characteristics and inference latency, and exhibits strong fitting ability for interference caused by different task resource access patterns; secondly, its inference computation overhead is low, with single prediction time controllable in the microsecond to sub-millisecond range, making it suitable for high-frequency calls during online scheduling without significantly impacting overall system performance. The trained interference-aware latency prediction model is used in the online phase to quickly predict the inference latency of newly arriving tasks and currently running tasks under co-location execution conditions based on their resource characteristics, and to assess the existence of service quality breach risk, providing a basis for subsequent spatiotemporal adaptive scheduling decisions.

[0058] (iv) Flexible spatial partitioning and SM fine-grained isolation mechanism To address the issues of coarse-grained partitioning of existing GPU resource space, lack of runtime elastic adjustment capabilities, and difficulty in meeting the service quality assurance requirements in multi-task co-location scenarios, we propose an elastic space partitioning and fine-grained isolation mechanism based on streaming multiprocessor (SM) masks to achieve controllable parallel sharing of GPU computing resources without introducing high-overhead context switching.

[0059] 1. SM-level hardware resource isolation The system is based on the SM selection mechanism supported by the GPU hardware scheduling unit. It achieves SM-level physical computational power isolation by applying SM mask constraints during the GPU kernel's emission phase. In its implementation, this mechanism includes the following steps: Task Metadata Injection: Before the GPU kernel launch, the user-space transparent proxy component injects task metadata (TMD) into the current task by intercepting kernel loading-related interfaces. The task metadata includes at least task identification information, priority information, and corresponding SM resource configuration parameters.

[0060] SM Mask Binding: The centralized scheduler calculates the SM mask configuration for each task based on the minimum SM requirements determined during the offline performance profiling phase and the current global GPU load state. The proxy component then dynamically binds the SM mask to the corresponding CUDA stream or kernel execution context through callbacks or equivalent interfaces provided by the GPU runtime.

[0061] Hardware scheduling constraints take effect: During kernel execution, the GPU hardware scheduler only allows thread blocks of a task to be allocated to a subset of SMs specified by the SM mask for execution, thereby limiting the range of computing resources available to the task at the hardware level. Different tasks are allocated to non-overlapping subsets of SMs, avoiding computational power contention at the SM level.

[0062] In this way, the system can achieve fine-grained and controllable isolation of GPU computing resources without modifying the application's computing logic.

[0063] 2. Runtime elastic adjustment for multi-task co-location To adapt to dynamically changing multi-tasking environments, the system's elastic space partitioning mechanism supports online adjustments to SM resource configurations during task execution. The centralized scheduler triggers a resource reconfiguration process when one of the following events occurs: a new GPU task arrives → an existing task completes execution and releases resources → the load characteristics or QoS constraints of the running task change.

[0064] In the above scenario, the scheduler recalculates the SM mask allocation scheme for each task based on the latest global resource view, and distributes the updated mask configuration to the corresponding tasks through the agent component. Since the adjustment of the SM mask applies to the kernel launch phase or subsequent kernel execution batches, this system can achieve smooth reallocation of GPU resources in most cases without interrupting or terminating the currently executing tasks, avoiding the performance loss caused by frequent context switching.

[0065] (v) Kernel-level command buffer access interception and time reuse mechanism In scenarios involving multiple tasks running concurrently with significant resource interference risks, simply relying on space partitioning may not meet the quality of service requirements of latency-sensitive tasks. Therefore, this invention proposes a GPU time reuse mechanism based on kernel-level command buffer access interception as a key module of the system, used to achieve strict time isolation of GPU execution rights when necessary.

[0066] 1. Core Idea GPU instructions are first written to the command buffer in the user-mode driver (UMD), and then submitted to the GPU hardware for execution by the kernel-mode driver (KMD). This command buffer is typically exposed to the user process in a memory-mapped manner. Based on these characteristics, this invention does not parse the specific GPU instruction content, nor does it introduce traditional GPU context switching. Instead, it fine-tunes the GPU execution timing during the instruction submission phase by controlling the write access permissions of the command buffer, thereby achieving low-overhead time reuse.

[0067] 2. Command Buffer Identification and Mapping During application runtime, the user-mode transparent proxy component intercepts key interfaces interacting with the GPU driver, identifying and recording the virtual address range of the command buffer used to submit GPU instructions. This process includes, but is not limited to: resolving the memory mapping relationship between the user-mode driver and the kernel-mode driver; tracking the creation, remapping, and release of the command buffer; and establishing access monitoring capabilities for the command buffer without interfering with the normal application logic. This step lays the foundation for subsequent access control and time isolation.

[0068] 3. Memory-protected write access interception mechanism When time-reuse scheduling needs to be enabled, the agent component imposes write access restrictions on the memory pages where the identified command buffer resides. The core steps are as follows: Write permission locking: Utilize the memory protection mechanism provided by the operating system to set the memory page corresponding to the command buffer to a read-only state, prohibiting application threads from directly writing GPU instructions into it.

[0069] Access Triggering and Exception Handling: When the application attempts to write instructions to the command buffer, a page-level write-protection exception is triggered. This exception is caught by the signal handling logic registered by the proxy component.

[0070] Time slice request and scheduling arbitration: After catching an exception, the agent component pauses the current thread and initiates a GPU time slice request to the centralized scheduler. The scheduler decides whether to grant execution permission based on the global scheduling policy and QoS constraints.

[0071] Authorization and Permission Restoration: Once authorization is granted, the agent component temporarily restores write permissions to the command buffer, allowing the application to complete the writing and submission of instructions.

[0072] Reclaiming after the time slice ends: When the instruction batch is submitted or the time slice expires, the agent component sets the command buffer to read-only again and reclaims the GPU execution rights.

[0073] This time-reuse mechanism based on command buffer access interception is not dependent on specific GPU instruction formats and has strong versatility; it avoids the high latency and state saving overhead caused by traditional GPU context switching; it can implement scheduling control at millisecond or even finer-grained time scales; and it complements the spatial partitioning mechanism, providing a fallback solution for QoS assurance.

[0074] (vi) Spatiotemporal adaptive scheduling method based on interference-aware delay prediction and QoS awareness To achieve a dynamic balance between GPU resource utilization and quality of service (QoS) assurance in a multi-task co-location environment, a QoS-aware spatiotemporal adaptive scheduling method based on interference-aware latency prediction is proposed. This method avoids QoS breaches by performing a "pre-assessment" of potential performance interference before task scheduling and dynamically switching between spatial parallel execution and temporal isolated execution based on the prediction results.

[0075] 1. Interference sensing delay prediction mechanism During the online scheduling phase, the system calls the interference-aware delay prediction model trained offline to predict the following two types of delays: the prediction inference delay of a newly arriving task under the condition of co-executing with the currently running task; and the possible delay changes of the currently running task after the introduction of a new task.

[0076] The prediction model takes as input a resource feature profile built during the offline phase of the task, including but not limited to L2 cache bandwidth requirements, GPU memory bandwidth requirements, and SM usage characteristics; the output is the predicted latency value under co-existing interference conditions. Through this prediction mechanism, the system can assess potential QoS risks before the task actually starts.

[0077] 2. QoS-aware scheduling decision-making process When new tasks Upon arrival, the central scheduler determines the GPU's load status. The following QoS-aware scheduling algorithm is executed, and the specific flowchart is as follows: Figure 3 As shown: Idle state handling: If the GPU is currently not running any tasks ( The minimum subset of SMs and the corresponding SM mask configuration determined in step S1 are used directly. Tasks are launched using a spatial partitioning mode to minimize startup latency.

[0078] Co-location risk assessment: If a task is currently running, call the interference prediction model trained in step S1 to predict the prediction latency of the new task in the co-location environment. Prediction delay of existing tasks after adding new tasks .

[0079] Scheduling mode decision: Determine QoS constraints and check whether the predicted latency of all tasks is lower than their set QoS threshold (i.e., and ).

[0080] 1) Branch 1 (Spatial Partitioning Mode): If the QoS constraints are met, the interference risk is low. The system calculates a set of non-overlapping SM masks to ensure that new and old tasks are mutually exclusive on the physical SMs, and performs flexible spatial partitioning.

[0081] 2) Branch Two (Time Multiplexing Mode): If the QoS constraints of any task may be violated, it indicates a high risk of interference. The system falls back to strict time-slice round-robin scheduling, performs kernel-level command buffer interception, and ensures that only one task exclusively uses all GPU resources at any given time to guarantee high-priority QoS.

[0082] By combining interference-aware latency prediction with spatiotemporal collaborative scheduling, potential QoS risks can be identified before task initiation, avoiding post-event remediation; the spatial parallelism of GPUs can be fully utilized in low-interference scenarios; and temporal isolation can provide strong QoS guarantees in high-interference scenarios, achieving a dynamic adaptive balance between GPU resource utilization and service quality.

[0083] (vii) Transparent Proxy and System Implementation The overall system utilizes a user-space transparent agent component to uniformly intercept and control GPU scheduling behavior during runtime. This agent component is provided as a dynamic link library and is automatically loaded via environment variables during application startup. This allows for the takeover and scheduling of GPU-related operations without modifying the application source code or altering the original programming model. The main features are as follows: 1. Loading and running mechanism of proxy components In its implementation, the agent component is configured to be loaded preferentially when the application process starts and inserted into the call path between the application and the GPU driver at runtime. After initialization, the agent component automatically performs the following operations: establishes a communication channel with the centralized resource scheduler for reporting task status and receiving scheduling instructions; initializes internal state information related to GPU resource control, including task identifiers, resource configuration parameters, and scheduling policy context; and registers necessary interception logic to intervene in the execution flow when the application triggers GPU-related operations. Since the agent component's loading process is completely independent of the application itself, users do not need to modify or recompile existing programs, thus ensuring high system transparency and good compatibility. For example, when the DNN inference application starts, the system injects a transparent user-space agent (Agent DLL) into the application process through the LD_PRELOAD mechanism. This agent does not modify the application source code but directly intercepts CUDA driver layer APIs (such as cuStreamCreate and cuLaunchKernel) and underlying system calls (such as ioctl). Afterward, the agent establishes communication with the centralized resource management orchestrator, reports task information, and receives scheduling instructions.

[0084] 2. Proxy implementation method of the flexible space partitioning module The elastic space partitioning module, as a functional submodule of the proxy component, resides in the same user-mode process space as the application and is loaded and unloaded along with the application process's lifecycle. When the application creates a GPU execution context, creates a computation flow, or initiates a kernel execution request, the proxy component intercepts the relevant GPU driver interfaces and, before kernel launch, dynamically binds the corresponding SM mask or equivalent computational resource constraint information to the current task according to the scheduling policy issued by the centralized resource scheduler. Specifically, during each kernel launch, the proxy component: queries the global resource allocation status maintained by the centralized resource scheduler; obtains the SM subset or space isolation parameters allocated to the current task; and injects the aforementioned resource constraint information into the kernel execution path, ensuring that the kernel can only execute within the authorized computational resource scope. By encapsulating the elastic space partitioning logic as part of the proxy component, this invention can dynamically adjust the space resource configuration of each task at runtime without the application needing to be aware of or participate in the resource management process.

[0085] 3. Proxy-based implementation of kernel-level command buffer access interception in modules When the system determines that a time-reuse scheduling strategy is required, the agent component internally activates a kernel-level command buffer access interception module to achieve precise control over GPU execution timing. This module also runs as an agent within a user-space process and intercepts command submission behavior at critical paths of application-GPU driver interaction. Its basic workflow includes: dynamically identifying and tracking command buffer memory areas related to GPU command submission in the application process; applying access control policies to these command buffers, restricting write access without scheduling authorization; when the application attempts to write instructions to the command buffer, the agent module captures this behavior and synchronously initiates a time slice request to the centralized resource scheduler; after obtaining scheduling authorization, it temporarily opens the command buffer write permission, allowing instruction submission, and then revokes access after the time slice ends. In this way, the kernel-level command buffer access interception module can achieve strict time isolation of GPU execution rights without parsing the specific GPU instruction content or affecting the CPU-side execution logic of the process.

[0086] 4. Collaborative working mechanism between the agent component and the centralized scheduler The proxy component is primarily responsible for local interception and execution control, while the centralized resource scheduler is responsible for maintaining the global resource view and generating scheduling decisions. The two work together through inter-process communication mechanisms, and their responsibilities are divided as follows: Proxy component: responsible for performing specific resource control actions within a single application process, including space partitioning constraint injection, command submission interception, and scheduling strategy execution; The centralized resource scheduler is responsible for maintaining the global resource usage status of the GPU, executing interference-aware latency prediction and QoS-aware scheduling algorithms, and issuing specific resource configuration and scheduling instructions to each agent component. Through this "centralized decision-making, distributed execution" architecture, this invention can achieve consistent and controllable GPU resource scheduling behavior in a multi-process, multi-task co-location environment.

[0087] By unifying the elastic space partitioning module and the kernel-level command buffer access interception module into a user-space transparent proxy component and dynamically loading it into each GPU application process, it has the following significant advantages: it does not intrude on the application and achieves true application transparency; it does not depend on specific upper-layer programming interfaces and has strong compatibility; it supports fine-grained space and time resource control; and it significantly reduces system deployment and maintenance costs.

[0088] Therefore, the system can achieve efficient and controllable spatiotemporal adaptive resource scheduling on existing GPU computing platforms with low engineering costs.

[0089] Example 3 like Figure 4As shown, the present invention provides a GPU spatiotemporal adaptive resource scheduling system for quality of service assurance, comprising: Offline processing unit: In the offline stage, it performs performance profiling on deep neural network tasks and determines the minimum subset of service quality constraints and the corresponding service quality mask configuration. Interference-aware delay prediction model building unit: offline collection of multi-task co-location operation data, and use of multi-task co-location operation data to build an interference-aware delay prediction model; Online scheduling phase processing unit: During the online scheduling phase, when a new task arrives, it determines whether the GPU is idle based on the current GPU load status; if so, it directly starts the task through the space partitioning mode using a determined minimum SM subset and the corresponding SM mask configuration; if not, it executes the next unit. Judgment and prediction unit: Based on the constructed interference-aware delay prediction model, predict the inference delay of the new task in the co-location environment with the current running task, and the prediction delay of the current running task after the addition of the new task; determine whether the predicted inference delay and prediction delay both meet the preset service quality constraints; if so, execute the flexible space partitioning strategy, establish a set of non-overlapping SM masks, and allow tasks to co-locate and execute in parallel; if not, execute the next unit. Execution Unit: Based on kernel-level command buffer interception, a time-reuse strategy is used to schedule and execute tasks, ensuring that only one task can exclusively occupy all GPU resources at any given time, in order to prevent resource interference from causing service quality defaults; after the task finishes running, the elastic space partitioning strategy is restored.

[0090] In the offline processing unit of this embodiment, determining the minimum subset of service quality constraints (SMs) and their corresponding SM masks is achieved using a binary search-based method, specifically including the following steps: S11, Baseline latency of the measurement model under full GPU resources And set an allowable performance loss threshold. S12. Perform a binary search within the range of 1 to the total number of global SMs, calculate the median number of SMs in each search, and generate the corresponding SM mask; S13. Run the target DNN task on the allocated median number of SMs and measure the inference latency. Calculate the performance degradation rate S14. Determine the performance degradation rate. Is it less than or equal to the performance loss threshold? If yes, record the current optimal number of SMs and its corresponding SM mask, and search for a smaller range of SMs; if no, search for a larger range; S15, output the minimum number of SMs that finally satisfies the QoS constraints and its corresponding SM mask configuration.

[0091] The method of constructing an interference-aware delay prediction model using multi-task co-location running data in the interference-aware delay prediction model construction unit of this embodiment specifically includes the following steps: S21. Key performance characteristics of offline data collection tasks: Combine the resource characteristics of the task to be run with those of the currently running task to form a feature vector. ;in and S22 represents the task's demand intensity for L2 cache bandwidth and video memory bandwidth, respectively; S22 represents the task inference latency observed under co-location execution conditions. As a supervisory signal, constructing regression learning problems S23. A gradient boosting decision tree-based regression model, XGBoost, is used for training. Multiple weak regressors are combined using an additive model. By minimizing the objective function, a balance is achieved between prediction accuracy and model complexity, thus obtaining an interference-aware delay prediction model.

[0092] Example 4 The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a GPU spatiotemporal adaptive resource scheduling method for quality of service assurance.

[0093] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This application is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] The inventive points protected by this invention are as follows: This invention proposes a GPU spatiotemporal adaptive resource scheduling method and system for quality of service (QoS) assurance. Addressing the technical challenges of low GPU resource utilization, uncontrollable interference, and difficulty in guaranteeing QoS in multi-task deep learning inference scenarios, it constructs a unified resource orchestration mechanism that integrates offline profiling, online prediction, spatial elastic partitioning, and temporal isolation. Its core innovations are summarized below: Based on an offline performance profile-driven elastic space partitioning mechanism, this invention proposes an elastic computing resource partitioning method. Addressing the non-linear performance scaling characteristics of individual deep learning tasks on GPUs, this method automatically determines the smallest subset of computing units that meet given quality of service constraints by systematically measuring task latency under different computing resource scales during the offline phase. This mechanism avoids the static over-allocation or resource insufficiency problems commonly found in traditional GPU resource allocation, enabling runtime scheduling decisions based on the "minimum sufficient resources" as the basic unit. This releases a large amount of reusable computing resources while ensuring task QoS, providing a spatial foundation for multi-task parallelism.

[0098] Fine-grained dynamic spatial isolation technology based on computation unit mask: Addressing the issues of coarse granularity and non-adjustable runtime characteristics in existing GPU spatial division technology, this invention introduces a dynamic spatial isolation method based on computation unit mask. This method applies variable computation unit access constraints to the task execution path, ensuring that different tasks running on the same physical GPU can only use their respective authorized subsets of computation units, thereby achieving mutually exclusive execution space between tasks at the hardware level. This spatial isolation mechanism has the following characteristics: (1) fine-grained isolation, accurate to a single or small number of computation units; (2) supports dynamic adjustment at runtime without interrupting currently executing tasks; (3) completely transparent to upper-layer applications, independent of application code modifications. This solves the inherent deficiencies of existing GPU spatial division schemes in terms of flexibility and scalability.

[0099] Predictive Scheduling Decision Model for Shared Bandwidth Interference: This invention proposes a scheduling decision method based on shared resource interference prediction. Addressing the issue of insufficient physical isolation between L2 cache bandwidth and global memory bandwidth in GPU multi-task co-location scenarios, it constructs task-level resource access feature vectors to predict and assess potential performance interference and latency escalation risks before task co-location. This prediction model, based on a lightweight regression algorithm, completes QoS risk assessment within microseconds, providing the scheduling system with prior information on whether to allow spatial co-location execution. By introducing a prediction-driven decision mechanism, this invention avoids the QoS violation problems common in traditional "post-intervention" scheduling strategies, achieving a shift from passive control to proactive avoidance.

[0100] A Spatiotemporal Adaptive Hybrid Scheduling Framework Based on Prediction Results: Unlike existing GPU scheduling schemes that only support time reuse or spatial reuse, this invention constructs a prediction-aware spatiotemporal adaptive hybrid scheduling framework. When a task arrives, this framework dynamically switches between two scheduling modes based on the interference risk results output by the prediction model: (1) In low-interference scenarios, multiple tasks are executed in parallel using spatial isolation to maximize GPU resource utilization; (2) In high-interference scenarios, it automatically degrades to time isolation to ensure that latency-sensitive tasks exclusively occupy the GPU execution window. This mechanism achieves a dynamic balance between system throughput and strict QoS guarantees, avoiding the performance instability of a single scheduling strategy under complex load scenarios.

[0101] A non-intrusive time isolation mechanism based on command buffer access control: This invention proposes a GPU time isolation method based on command buffer access control. By precisely controlling command submission behavior at the user-mode and driver-mode interaction boundary, it achieves forced arbitration of GPU execution flow. This method does not require parsing specific instruction content; instead, it dynamically controls the access permissions of the memory region where the command buffer resides, blocking tasks from submitting execution requests to the GPU when they are not authorized. Command writing and submission are only allowed after the scheduler grants a time slice. Thus, strict task-level time reuse and isolation are achieved without relying on hardware preemption capabilities or pausing other CPU computations.

[0102] Application-Transparent User-Space Proxy and Centralized Resource Orchestration Architecture: To reduce deployment and migration costs, this invention designs a GPU resource management architecture that is completely transparent to the application. By injecting a user-space proxy module during the task startup phase, the underlying driver interface is uniformly intercepted and controlled. Simultaneously, a centralized resource orchestrator maintains a global resource view and issues scheduling policies. This architecture achieves unified management and dynamic scheduling of multi-task GPU resources without requiring any code modifications to user applications, significantly reducing system integration complexity and enhancing adaptability to different GPU vendors and driver versions.

[0103] A QoS-priority resource management mechanism for multi-task inference scenarios: This invention takes service quality as the core constraint objective, explicitly introduces task-level QoS thresholds during the scheduling decision-making process, and uses latency constraints as the primary basis for resource allocation and scheduling mode selection. By combining offline profiling, online prediction, and spatiotemporal hybrid scheduling, this invention achieves stable protection for QoS-sensitive tasks in high-concurrency inference scenarios, while significantly improving overall GPU resource utilization efficiency.

[0104] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0105] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A GPU spatiotemporal adaptive resource scheduling method for quality of service assurance, characterized in that, include: S1. In the offline phase, perform performance profiling on the deep neural network task to determine the minimum subset of service quality constraints and the corresponding SM mask configuration. S2. Collect multi-task co-location operation data offline, and use the multi-task co-location operation data to build an interference perception delay prediction model; S3. During the online scheduling phase, when a new task arrives, determine whether the GPU is idle based on the current GPU load status. If yes, start the task directly through space partitioning mode using the determined minimum subset of SMs and the corresponding SM mask configuration; otherwise, proceed to the next step. S4. Based on the constructed interference-aware delay prediction model, predict the inference delay of the new task in the co-location environment with the current running task, and the prediction delay of the current running task after the addition of the new task; determine whether the predicted inference delay and prediction delay both meet the preset service quality constraints; if so, execute the elastic space partitioning strategy, establish a set of non-overlapping SM masks, and allow tasks to co-locate and execute in parallel; if not, proceed to the next step. S5. Based on kernel-level command buffer interception, a time-reuse strategy is used to schedule and execute tasks, ensuring that only one task can exclusively occupy all GPU resources at any given time, in order to prevent resource interference from causing service quality defaults; after the task finishes running, the elastic space partitioning strategy is restored.

2. The GPU spatiotemporal adaptive resource scheduling method for quality of service assurance according to claim 1, characterized in that, Step S1, which involves determining the minimum subset of service quality constraints (SMs) and their corresponding SM masks, employs a binary search-based method and includes the following steps: S11, Baseline latency of the measurement model under full GPU resources And set an allowable performance loss threshold. S12. Perform a binary search within the range of 1 to the total number of global SMs, calculate the median number of SMs in each search, and generate the corresponding SM mask; S13. Run the target DNN task on the allocated median number of SMs and measure the inference latency. Calculate the performance degradation rate S14. Determine the performance degradation rate. Is it less than or equal to the performance loss threshold? If yes, record the current optimal number of SMs and its corresponding SM mask, and search for a smaller range of SMs; if no, search for a larger range; S15, output the minimum number of SMs that finally satisfies the QoS constraints and its corresponding SM mask configuration.

3. The GPU spatiotemporal adaptive resource scheduling method for quality of service assurance according to claim 2, characterized in that, The step S2, which involves constructing an interference-aware delay prediction model using multi-task co-location operation data, specifically includes the following steps: S21. Key performance characteristics of offline data collection tasks: Combine the resource characteristics of the task to be run with those of the currently running task to form a feature vector. ;in and S22 represents the task's demand intensity for L2 cache bandwidth and video memory bandwidth, respectively; S22 represents the task inference latency observed under co-location execution conditions. As a supervisory signal, constructing regression learning problems S23. A gradient boosting decision tree-based regression model, XGBoost, is used for training. Multiple weak regressors are combined using an additive model. By minimizing the objective function, a balance is achieved between prediction accuracy and model complexity, thus obtaining an interference-aware delay prediction model.

4. The GPU spatiotemporal adaptive resource scheduling method for quality of service assurance according to claim 3, characterized in that, Step S3, which involves starting the task directly through the space partitioning mode using a determined minimum subset of SMs and the corresponding SM mask configuration, specifically includes the following steps: The system identifies that no other tasks are running on the current GPU, extracts the optimal number of task managers (SMs) and SM mask configuration for the corresponding new task, dynamically intercepts the kernel of the corresponding task, and injects task metadata containing the optimal number of SMs and SM mask configuration into the current task before the kernel is launched to the GPU. The kernel with the injected task metadata is launched to the GPU, so that the kernel of the corresponding task starts running on the specified minimum subset of SMs.

5. A GPU spatiotemporal adaptive resource scheduling method for quality of service assurance according to claim 4, characterized in that, The execution of the flexible space partitioning strategy described in step S4, which establishes a set of non-overlapping SM masks and allows tasks to be co-located and executed in parallel, specifically includes the following steps: S41. The centralized scheduler calculates the non-overlapping SM mask configuration for each task and injects task metadata containing SM resource configuration parameters into the current task before the GPU kernel is launched. S42. The kernel of the corresponding task is dynamically intercepted, and before the kernel is launched to the GPU, the calculated SM mask configuration is dynamically bound to the corresponding CUDA stream or kernel execution context. S43. During kernel execution, constraints are applied by the GPU hardware scheduler to allow only the thread blocks of the task to be allocated to the subset of SMs specified by the SM mask to run, thereby achieving physical SM-level resource isolation.

6. The GPU spatiotemporal adaptive resource scheduling method for quality of service assurance according to claim 5, characterized in that, Step S5, which involves using a time-reuse strategy to schedule and execute tasks based on kernel-level command buffer interception to ensure that only one task exclusively uses all GPU resources at any given time, specifically includes the following steps: S51. During application execution, the user-mode transparent proxy component intercepts the GPU interaction interface, identifies and records the virtual address range of the command buffer used to submit GPU instructions; S52. Utilizes the memory protection mechanism provided by the operating system to lock write permissions and sets the memory page corresponding to the command buffer to a read-only state; S53. Captures the page-level write protection exception triggered when the application writes instructions to the command buffer, suspends the current thread, and initiates a GPU time slice request to the centralized scheduler; S54. After obtaining execution permission authorization, temporarily restores the write permission of the command buffer to allow instructions to be written and submitted, and sets the command buffer to a read-only state again after the time slice ends to reclaim GPU execution rights.

7. A GPU spatiotemporal adaptive resource scheduling system for quality of service assurance, characterized in that, include: Offline processing unit: In the offline stage, it performs performance profiling on deep neural network tasks and determines the minimum subset of service quality constraints and the corresponding service quality mask configuration. Interference-aware delay prediction model building unit: offline collection of multi-task co-location operation data, and use of multi-task co-location operation data to build an interference-aware delay prediction model; Online scheduling phase processing unit: During the online scheduling phase, when a new task arrives, it determines whether the GPU is idle based on the current GPU load status; If yes, start the task directly through the space partitioning mode using the determined minimum subset of SMs and the corresponding SM mask configuration; otherwise, execute the next unit. Judgment and prediction unit: Based on the constructed interference-aware delay prediction model, predict the inference delay of the new task in the co-location environment with the current running task, and the prediction delay of the current running task after the addition of the new task; determine whether the predicted inference delay and prediction delay both meet the preset service quality constraints; if so, execute the flexible space partitioning strategy, establish a set of non-overlapping SM masks, and allow tasks to co-locate and execute in parallel; if not, execute the next unit. Execution Unit: Based on kernel-level command buffer interception, a time-reuse strategy is used to schedule and execute tasks, ensuring that only one task can exclusively occupy all GPU resources at any given time, in order to prevent resource interference from causing service quality defaults; after the task finishes running, the elastic space partitioning strategy is restored.

8. A GPU spatiotemporal adaptive resource scheduling system for quality of service assurance according to claim 7, characterized in that, The determination of the minimum subset of service quality constraints and the corresponding service quality mask in the offline processing unit adopts a binary search-based method, specifically including the following steps: S11, Baseline latency of the measurement model under full GPU resources And set an allowable performance loss threshold. ; S12. Perform a binary search within the range of 1 to the total number of global SMs, calculate the median number of SMs in each search, and generate the corresponding SM mask; S13. Run the target DNN task on the allocated median number of SMs and measure the inference latency. Calculate the performance degradation rate S14. Determine the performance degradation rate. Is it less than or equal to the performance loss threshold? If yes, record the current optimal number of SMs and its corresponding SM mask, and search for a smaller range of SMs; if no, search for a larger range; S15, output the minimum number of SMs that finally satisfies the QoS constraints and its corresponding SM mask configuration.

9. A GPU spatiotemporal adaptive resource scheduling system for quality of service assurance according to claim 8, characterized in that, The method for constructing an interference-aware delay prediction model using multi-task co-location runtime data, as described in the interference-aware delay prediction model construction unit, specifically includes the following steps: S21. Key performance characteristics of offline data collection tasks: Combine the resource characteristics of the task to be run with those of the currently running task to form a feature vector. ;in and S22 represents the task's demand intensity for L2 cache bandwidth and video memory bandwidth, respectively; S22 represents the task inference latency observed under co-location execution conditions. As a supervisory signal, constructing regression learning problems S23. A gradient boosting decision tree-based regression model, XGBoost, is used for training. Multiple weak regressors are combined using an additive model. By minimizing the objective function, a balance is achieved between prediction accuracy and model complexity, thus obtaining an interference-aware delay prediction model.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a GPU spatiotemporal adaptive resource scheduling method for quality of service assurance as described in any one of claims 1-6.

Images