Qos adaptive fine-grained model scheduling method and system under edge GPU

Abstract

The application relates to the field of electronic information technology and discloses a QoS adaptive fine-grained model scheduling method and system under an edge GPU, which comprises the following steps: acquiring a current model configuration, estimating the completion time of a request and estimating the delay QoS margin of the request, calculating the effective accuracy rate of current queue backlog according to the delay QoS margin of the request; selecting a corresponding candidate model configuration set according to whether the effective accuracy rate of the current queue backlog meets the QoS of a QC task, calculating the corresponding proportion of fusion strength; combining the fusion strength to calculate the effective accuracy rate of each candidate model configuration in the candidate model configuration set after fusion, executing fusion when the effective accuracy rate after fusion can meet the QoS of the QC task, and selecting the optimal model configuration according to the improvement rate after kernel fusion of different candidate model configurations. The application can meet the QoS constraint of a QC task in a resource-restricted edge GPU environment.

Description

Technical Field

[0001] This invention relates to the field of electronic information technology, and in particular to a QoS adaptive fine-grained model scheduling method and system under edge GPU. Background Technology

[0002] With the widespread deployment of deep neural networks in edge intelligence applications, in order to make full use of graphics processing unit (GPU) resources, quality of service (QoS) critical (QC) tasks are often co-located with best-effort (BE) tasks.

[0003] In existing technologies, traditional edge computing nodes mostly adopt a single and fixed-precision model architecture. When faced with sudden traffic surges, they can only improve the overall throughput and latency of the system by mechanically adjusting the batch size, without being able to adjust the accuracy metric. This adjustment method ignores the potential capabilities of model loading and switching, cannot dynamically adapt to the strict Service Level Objectives (SLOs) of different requests, and cannot achieve an effective balance between accuracy, latency, and throughput.

[0004] Existing edge computing frameworks, when dynamically adjusting model configurations, mostly only statically evaluate theoretical latency and accuracy, ignoring the switching overhead caused by model switching. Due to the lack of quantification of this overhead and prediction of request backlog, system switching triggered by sudden traffic surges can generate significant latency spikes. This scheduling method leads to a large backlog of requests, not only failing to optimize throughput but also causing a surge in end-to-end latency and severe QoS violations.

[0005] When performing AI inference tasks, existing edge nodes often fail to fully and collaboratively schedule the underlying computing units and memory resources of the GPU due to the lack of fine-grained resource management mechanisms that penetrate deep into the Streaming Multiprocessor (SM). This resource allocation method results in extremely low GPU resource utilization, making it difficult to meet the requirements of enhanced computing power coordination performance in edge scenarios.

[0006] Traditional edge node scheduling mostly relies on First-Come, First-Served (FCFS) or static heuristic rules based on experience for task allocation. These simple strategies cannot dynamically sense and adaptively adjust based on real-time system queuing status, load characteristics (computation preferences or memory access preferences), and QoS margins, making it difficult to predict QoS compliance under complex heterogeneous loads.

[0007] In summary, existing technologies struggle to simultaneously balance the accuracy and latency of inference results. Furthermore, traditional preemptive co-location strategies lack a low-overhead kernel preemption mechanism, resulting in the underutilization of fine-grained resources within the GPU (such as tensor cores and computing units). This can easily lead to performance interference and quality of service (QoS) violations in QC tasks. Summary of the Invention

[0008] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a QoS adaptive fine-grained model scheduling method and system under edge GPU, which can meet the QoS constraints of QC tasks in resource-constrained edge GPU environments.

[0009] To address the aforementioned technical problems, this invention provides a QoS adaptive fine-grained model scheduling method for edge GPUs, comprising: Get the current model configuration, estimate the completion time of requests under the current model configuration, estimate the latency QoS margin of requests based on the completion time of requests, and calculate the effective accuracy of the current queue backlog based on the latency QoS margin of requests. Based on whether the effective accuracy of the current queue backlog meets the QoS of the QC task, select the corresponding candidate model configuration set and calculate the corresponding proportion of fusion strength. The effective accuracy of each candidate model configuration after fusion is calculated based on the fusion strength. When the effective accuracy after fusion can meet the QoS of the QC task, fusion is performed. The optimal model configuration is selected based on the improvement rate after kernel fusion of different candidate model configurations.

[0010] Furthermore, the method for calculating the requested delay QoS margin is as follows: , In the formula, This represents the latency QoS margin for the k-th request. To preset the delay threshold, Configure the waiting time in the queue for the k-th request under the current model. For the quantity of the k-th request, This represents the current throughput.

[0011] Furthermore, the effective accuracy of the current queue backlog is calculated as follows: , In the formula, The effective accuracy rate of the current queue backlog. This represents the latency QoS margin for the k-th request. To address the current queue backlog, Indicates satisfaction The value is 1 if the time condition is met, and 0 otherwise.

[0012] Furthermore, the step of calculating the corresponding proportion of fusion strength based on whether the effective accuracy of the current queue backlog meets the QoS of the QC task for selecting the corresponding candidate model configuration set is as follows: If the effective accuracy of the current queue backlog is less than the effective accuracy threshold, then stop the BE task and predict the set of candidate model configurations that can meet the current request rate, with the corresponding proportion of fusion strength being 0. If the effective accuracy of the current queue backlog is greater than or equal to the effective accuracy threshold, then predict the set of candidate model configurations that can meet the current request rate and select different BE tasks according to the batch size, and calculate the corresponding proportion of fusion strength through the persistent thread block mechanism.

[0013] Furthermore, the step of stopping the BE task and predicting the set of candidate model configurations that can satisfy the current request rate specifically includes: Temporarily stop all BE tasks, search for new model configurations that make the effective accuracy of the queue backlog corresponding to the new model configuration greater than the effective accuracy threshold, and use all the new model configurations found as a candidate model configuration set. When searching for a new model configuration, priority is given to model configurations with a batch size smaller than the current batch size. If the condition cannot be met even after finding the smallest batch size, then a model configuration with lower computational complexity is selected, provided that the effective accuracy of the current queue backlog is greater than or equal to the effective accuracy threshold.

[0014] Furthermore, the prediction can satisfy the current request rate set of candidate model configurations and select different BE tasks according to the batch size, specifically: Find new model configurations that make the effective accuracy of the queue backlog corresponding to the new model configuration greater than the effective accuracy threshold, and use all the new model configurations found as a set of candidate model configurations. For each candidate model configuration in the candidate model configuration set, if the batch processing of the candidate model configuration is in a low-load area, a memory-intensive BE task is selected on a first-come, first-served basis; otherwise, a compute-intensive BE task is selected.

[0015] Furthermore, the effective accuracy of each candidate model configuration after fusion is calculated based on the fusion strength. Fusion is performed when the effective accuracy after fusion meets the QoS requirements of the QC task. Specifically: The execution time of the fusion kernel is predicted using a two-stage linear regression model, with the fusion strength as input and the execution time of the fusion kernel as output. The latency variation for each request is calculated as follows: , In the formula, For the latency variation of each request, To reduce the execution time of the fusion kernel, Indicates the execution time of the original QC task. The batch size configured for the i-th candidate model; The end-to-end latency for computing the fusion kernel is: , In the formula, To reduce end-to-end latency in the fusion kernel, For the first in the queue End-to-end latency of a request; The number of invalid requests for the fused kernel is calculated as follows: , In the formula, To account for the number of illegal requests in the fusion kernel, As a preset delay threshold, 1 Indicates satisfaction The value is 1 if the condition is met, and 0 otherwise. The accumulated number of requests in the queue; The violation rate for computing fusion kernels is: , In the formula, To reduce the violation rate of the fusion kernel; Computational fusion The effective accuracy rate for each request is: , In the formula, For the fusion The effective accuracy of each request. Configure the corresponding effective accuracy of the queue backlog for the candidate models; if , If the effective accuracy threshold is set, then the effective accuracy after fusion is considered to meet the QoS of the QC task, and fusion is performed in this case.

[0016] Furthermore, the aforementioned The calculation method is as follows: , In the formula, Configure the waiting time in the queue for the k-th request under the current model. For model switching time, For the k-th request, This indicates the processing rate configured for the candidate model. This represents the number of requests currently in the queue. For the current throughput, In order to be in Request arrival rate when initiating model switching at any time.

[0017] Furthermore, the method for calculating the improvement rate after kernel fusion of different candidate model configurations is as follows: , In the formula, For the improvement rate, This refers to the execution time of the original QC task. This refers to the execution time of the original BE task. For fusion strength, This refers to the execution time of the fusion kernel.

[0018] The present invention also provides a QoS adaptive fine-grained model scheduling system under edge GPU, including a model switching scheduling module, an offline compilation module and an adaptive kernel fusion scheduling module, wherein the adaptive kernel fusion scheduling module includes a kernel selection component, a QoS prediction component and a kernel fusion control component; The model switching scheduling module obtains the current model configuration, estimates the completion time of requests under the current model configuration, estimates the latency QoS margin of requests based on the completion time of requests, and calculates the effective accuracy of the current queue backlog based on the latency QoS margin of requests. The offline compilation module performs kernel code refactoring and pre-compilation in the offline phase for all BE applications. The adaptive kernel fusion scheduling module sets each QC task as a QC application kernel sequence and sets the BE application as a BE application kernel sequence. The fusion process takes the kernel pair of QC application kernel and BE application kernel as the basic unit. The kernel selection component observes the current state of the QC application kernel and selects complementary candidate kernel pairs from the BE application kernel sequence. Based on whether the effective accuracy of the current queue backlog meets the QoS of the QC task, it selects the corresponding candidate model configuration set and calculates the corresponding proportion of fusion strength. The QoS prediction component performs an online feasibility check on the candidate kernel pairs, calculates the effective accuracy rate to determine whether a QoS violation occurs, and calculates the effective accuracy rate of each candidate model configuration after fusion in the candidate model configuration set in combination with the fusion strength. When the effective accuracy after fusion can meet the QoS of the QC task, the kernel fusion control component performs fusion; The model switching and scheduling module selects the optimal model configuration based on the improvement rate after kernel fusion of different candidate model configurations.

[0019] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This invention establishes QoS constraints and combines the load characteristics of QC tasks with QoS constraints to trigger model switching on demand. This ensures long-term effective accuracy requirements while making full use of remaining GPU resources, and can meet the QoS constraints of QC tasks in resource-constrained edge GPU environments. Attached Figure Description

[0020] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0021] Figure 1 This is a flowchart of a method in a preferred embodiment of the present invention.

[0022] Figure 2 This is a framework diagram of the system in a preferred embodiment of the present invention.

[0023] Figure 3 The following is a comparison chart of the throughput improvement of multiple BE application sets relative to Aker when SLO=0.66 in the simulation experiment of the preferred embodiment of the present invention.

[0024] Figure 4 The following is a comparison chart of the throughput improvement of multiple BE application sets relative to Aker when SLO=0.72 in the simulation experiment of the preferred embodiment of the present invention.

[0025] Figure 5 This is an effective accuracy tracking graph under dynamic load in a simulation experiment of a preferred embodiment of the present invention.

[0026] Figure 6 This is an error graph showing the effective accuracy estimate and the actual value under a mixed BE load in a simulation experiment in a preferred embodiment of the present invention.

[0027] Figure 7 This is a comparison chart of P95 latency in a simulation experiment of a preferred embodiment of the present invention, showing the QC application with and without online QoS prediction enabled.

[0028] Figure 8 This is a comparison chart of throughput improvement under the MobileViT model in a simulation experiment of a preferred embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0030] As AI services develop, users are concerned not only with latency but also with the percentage of correct predictions returned within the deadline. While larger batch sizes can increase throughput per unit time, execution time also increases for all requests within a batch size. Therefore, under dynamic workloads, determining how to switch to an appropriate model parameter size and its corresponding batch size to meet user needs is a pressing issue.

[0031] Besides the impact of different model parameter sizes and batch sizes on accuracy and latency, it also affects fine-grained resources. Adjusting the parameter architecture of different models and their corresponding processing sizes will change the parallelism patterns of tensor cores, CUDA cores, and computationally intensive and memory-intensive tasks. As the batch size increases, both computational throughput and memory throughput generally show an upward trend, exhibiting obvious stage characteristics: in the small batch range, throughput increases more rapidly, indicating that parallelism and hardware utilization are more sensitive to batch growth; when the batch size further increases into the medium-to-large batch range, the increase is approximately linear, indicating that the system gradually enters a relatively stable load state. On the other hand, even at larger batch sizes, the highest computational throughput is only about 62%, indicating that some GPU resources are still not fully utilized during inference. Furthermore, the utilization of computational and memory resources is not balanced: overall, memory-oriented tasks are more prevalent, while computational tasks are relatively less likely to reach the same level of saturation.

[0032] As deep learning workloads become increasingly complex, modern GPU architectures have undergone significant evolution at the hardware level. In addition to traditional general-purpose computing kernels, highly specialized tensor computing kernels have been introduced. Because general-purpose computing kernels and tensor computing kernels are physically independent and parallel functional units, in practical GPU inference or training pipelines, it is difficult for a kernel with a single resource preference to simultaneously fully utilize both types of computing resources.

[0033] Therefore, there is a huge optimization space for scheduling different types of kernels to run in parallel within the same streaming multiprocessor at the same time. A typical complementary scenario is the parallelism between general-purpose computing kernels and tensor computing kernels: if they are run in parallel within the same time window, the overall hardware utilization can be significantly improved. In addition to the physical complementarity within computing units, there are also significant parallel opportunities between compute-intensive kernels and memory-intensive kernels. The GPU hardware scheduler can achieve instruction-level interleaving of computation warp and memory access warp.

[0034] To illustrate the overhead of model switching, inference was performed using both ONNX Runtime and NNFUSION, and the end-to-end inference latency for each request was recorded. Experimental results show that once model switching occurs, inference latency fluctuates significantly and generates transient peaks, indicating that the switching process introduces substantial additional overhead. Therefore, when performing model switching in dynamic workload scenarios, the overhead of model switching must be considered in system design and scheduling strategies.

[0035] Reference Figure 1 As shown, this invention discloses a QoS adaptive fine-grained model scheduling method under edge GPUs, including the following steps: S1: Obtain the current model configuration, estimate the completion time of requests under the current model configuration, estimate the latency QoS margin of requests based on the completion time of requests, and calculate the effective accuracy of the current queue backlog based on the latency QoS margin of requests.

[0036] S1-1: Obtain the current model configuration and estimate the completion time of requests under the current model configuration: , For the quantity of the k-th request, This represents the current throughput.

[0037] S1-2: Periodically collect the current model configuration (denoted as...) The waiting time of each request in the queue and the current queue backlog are calculated, and the time to complete the current request requirement is calculated based on the current throughput. The method for calculating the latency QoS margin of the request is as follows: , In the formula, This represents the latency QoS margin for the k-th request. To preset the delay threshold, Adjustments will be made based on actual circumstances. Configure the waiting time in the queue for the k-th request under the current model. For the quantity of the k-th request, This represents the current throughput.

[0038] S1-3: The method for calculating the effective accuracy of the current queue backlog is as follows: , In the formula, The effective accuracy rate of the current queue backlog. This represents the latency QoS margin for the k-th request. The current queue backlog, i.e., the number of requests currently in the queue. Indicates satisfaction The value is 1 if the condition is met, and 0 otherwise. This indicates that the current request has timed out.

[0039] S2: Select the corresponding candidate model configuration set based on whether the effective accuracy of the current queue backlog meets the QoS of the QC task, and calculate the corresponding proportion of fusion strength.

[0040] S2-1: If the effective accuracy rate of the current queue backlog is less than the effective accuracy rate threshold (i.e.) , The effective accuracy threshold is the effective accuracy that satisfies the QoS of the QC task. If the value is adjusted according to the actual situation, then execute S2-2; if the effective accuracy rate of the current queue backlog is greater than or equal to the effective accuracy rate threshold (i.e. ), execute S2-3.

[0041] S2-2: Stop the BE task and predict the set of candidate model configurations that can satisfy the current request rate (denoted as ). Specifically, this involves temporarily halting all BE tasks and searching for a new model configuration (denoted as...). This results in the effective accuracy of the queue backlog corresponding to the new model configuration being greater than the effective accuracy threshold (i.e., , Configure the effective accuracy of the corresponding queue backlog for the new model, and the calculation method and (Same), all new model configurations found will be used as a candidate model configuration set; when searching for new model configurations, priority will be given to those with a batch size smaller than the current batch size (i.e., ... , The batch size configured for the i-th candidate model. The model configuration (based on the current batch size) is used to reduce inference latency for individual requests; if the condition cannot be met until the smallest batch found is reached, then the effective accuracy of the current queue backlog is greater than or equal to the effective accuracy threshold (i.e., ...). Under the premise of selecting a model configuration with lower computational complexity, the computational complexity threshold for lower computational complexity should be adjusted according to the actual situation.

[0042] The corresponding fusion strength is 0, so execute S3.

[0043] S2-3: Predict the set of candidate model configurations that can meet the current request rate and select different BE tasks based on the batch size. Specifically, find new model configurations such that the effective accuracy of the queue backlog corresponding to the new model configuration is greater than the effective accuracy threshold (i.e., ...). To improve the computational efficiency of the QC task itself, all newly found model configurations are used as a candidate model configuration set. To achieve kernel fusion between the QC and BE tasks during dynamic inference, the fusion space is divided into two adaptive intervals. Within these intervals, the kernels of the BE task are classified and sorted: in the low-load interval (e.g., ... When the batch size is smaller, memory-intensive BE tasks will be selected with larger selections; in memory-restricted regions (e.g., When the batch size is small, the computationally intensive BE task is selected to achieve fine-grained fusion and prediction. In this embodiment, for each candidate model configuration in the candidate model configuration set, if the batch of the candidate model configuration is in a low-load region (in this embodiment, ...), the computationally intensive BE task is selected to achieve fine-grained fusion and prediction. If both computational and memory throughput can be improved simultaneously, then memory-intensive BE tasks are selected on a first-come, first-served basis; otherwise (i.e.) Select computationally intensive BE tasks.

[0044] To eliminate the impact of differences in the number of thread blocks and grid dimensions between the QC kernel and the BE kernel on the fusion strategy, a Persistent Thread Block (PTB) mechanism is introduced to uniformly convert the kernel into a fixed-block execution format. The basic idea of ​​PTB is to treat each thread block issued to the GPU as a "worker" residing on the SM. Unlike a regular kernel, in PTB mode, each persistent block repeatedly retrieves and executes the work corresponding to multiple "logical blocks" in a loop; once it has covered the set of allocated logical blocks, the persistent block exits.

[0045] The specific implementation uses source-to-source compilation to generate a PTB version of the kernel. The conversion method involves introducing a for loop outside the original kernel, and in each iteration, recalculating the logical block ID based on the persistent block index and the loop variable, thus mapping different blocks in the original grid to different iterations in the loop. Simultaneously, the total number of blocks in the original grid is used as the input parameter for the PTB kernel, allowing the PTB version kernel to cover all the work of the original grid while maintaining semantic consistency. In this way, even if the number of blocks in the original kernel dynamically changes with the input data, the PTB version still runs with a fixed number of physical blocks and completes all logical block tasks through a loop. This provides a foundation for subsequent QC-BE kernel fusion within a unified scheduling window and enables fine-grained, controllable adjustment.

[0046] This invention uses PTB to convert all BE kernels into an execution form with a fixed number of physical blocks. Specifically, the calculation of the corresponding proportion of fusion strength through the persistent thread block mechanism is as follows: , In the formula, For fusion strength, The larger the value, the more BE work is injected, and the stronger the resource consumption. The number of execution blocks injected in a single fusion. This represents the total number of blocks corresponding to the original BE task kernel under the current input. Execute S3.

[0047] S3: Calculate the effective accuracy of each candidate model configuration after fusion in the candidate model configuration set based on the fusion strength. When the effective accuracy after fusion can meet the QoS of the QC task, perform fusion.

[0048] S3-1: Predict the execution time of the fusion kernel using a two-stage linear regression model, using the fusion strength as input, and outputting the execution time of the fusion kernel (denoted as ). In this embodiment, in For intervals, a quadratic polynomial model is used for prediction. ;exist At that time, a linear regression model was used for prediction. In these two phases, runtime inputs and corresponding performance data are collected to train an execution time prediction model for each kernel.

[0049] S3-2: Since this invention also needs to guarantee the QoS of the QC application after adding the BE task, the latency change for each request is calculated as follows: , In the formula, For the latency variation of each request, To reduce the execution time of the fusion kernel, Indicates the execution time of the original QC task. The batch size configured for the i-th candidate model.

[0050] S3-3: The end-to-end latency of the fused kernel is calculated as follows: , In the formula, To reduce end-to-end latency in the fusion kernel, For the first in the queue End-to-end latency for each request.

[0051] The calculation method is as follows: , In the formula, Configure the waiting time in the queue for the k-th request under the current model. For model switching time, , The time it takes to load the model onto the GPU. For model readiness time, For the k-th request, This indicates the processing rate configured for the candidate model. This represents the number of requests currently in the queue. For the current throughput, In order to be in The request arrival rate when a model switch is initiated at the current time.

[0052] S3-4: The number of invalid requests for the fused kernel is calculated as follows: , In the formula, To account for the number of illegal requests in the fusion kernel, As a preset delay threshold, 1 Indicates satisfaction The value is 1 if the condition is met, and 0 otherwise. This represents the accumulated number of requests in the queue.

[0053] Because in During this period, the GPU is unable to process any QC requests, resulting in a service downtime; during the downtime (i.e. At the end, the accumulated request queue backlog is: There are a total of One request.

[0054] S3-5: The violation rate of the fusion kernel is calculated as follows: , In the formula, To calculate the violation rate of the fusion kernel.

[0055] S3-6: Calculation of the fused result The effective accuracy rate for each request is: , In the formula, For the fusion The effective accuracy of each request. Effective accuracy of the queue backlog corresponding to the new model configuration (i.e., candidate model configuration).

[0056] S3-7: If , If the effective accuracy threshold is set, then the effective accuracy after fusion is considered to meet the QoS of the QC task, and fusion is performed in this case; otherwise, fusion is canceled.

[0057] S4: Select the optimal model configuration based on the improvement rate after kernel fusion according to different candidate model configurations.

[0058] S4-1: The method for calculating the improvement rate is as follows: , In the formula, For the improvement rate, This refers to the execution time of the original QC task. This refers to the execution time of the original BE task. For fusion strength, For the execution time of the fusion kernel (i.e. and (Time of fusion).

[0059] S4-2: Selection The largest model configuration is used as the optimal model configuration. The optimal model configuration is used to predict the execution results and ensure QoS compliance.

[0060] This invention employs a hierarchical control strategy: at the upper layer, an offline table is used to select coarse-grained QC task models to meet hard QoS constraints; at the lower layer, the QoS margin is utilized to maximize resource utilization by finding suitable BE tasks and the optimal fusion strength.

[0061] This invention also discloses a QoS adaptive fine-grained model scheduling system for edge GPUs, such as... Figure 2 As shown, it includes a model switching scheduling module, an offline compilation module, and an adaptive kernel fusion scheduling module. The adaptive kernel fusion scheduling module includes a kernel selection component, a QoS prediction component, and a kernel fusion control component.

[0062] The model switching scheduling module obtains the current model configuration, estimates the completion time of requests under the current model configuration, estimates the latency QoS margin of requests based on the completion time of requests, and calculates the effective accuracy of the current queue backlog based on the latency QoS margin of requests.

[0063] The offline compilation module performs kernel code refactoring and pre-compilation for all BE applications during the offline phase. Simultaneously, during the offline phase, the kernel of the BE application is merged with a kernel in the QC application through a persistent thread block mechanism and transformed into a static fused kernel.

[0064] The adaptive kernel fusion scheduling module sets each QC task as a QC application kernel sequence and the BE application as a BE application kernel sequence. The fusion process takes the kernel pair of QC application kernels and BE application kernels as the basic unit.

[0065] The kernel selection component observes the current state of the QC application kernel and selects complementary candidate kernel pairs from the BE application kernel sequence. Based on whether the effective accuracy of the current queue backlog meets the QoS of the QC task, it selects the corresponding candidate model configuration set and calculates the corresponding proportion of fusion strength.

[0066] The QoS prediction component performs a rigorous online feasibility check on complementary candidate kernel pairs, uses a two-stage regression model to predict the total execution duration of the QC application after the kernel pair is fused and co-located, and calculates whether the effective accuracy results in a QoS violation; it also calculates the effective accuracy of each candidate model configuration in the candidate model configuration set after fusion based on the fusion strength.

[0067] The kernel fusion control component adaptively switches between two operating states—QoS violation state and throughput optimization state—based on QoS margin. Kernel fusion is only executed when the QoS prediction component assesses the effective accuracy of the fused result to be within a safe margin. The kernel fusion control component executes fusion only when the effective accuracy of the fused result meets the QoS requirements of the QC task.

[0068] The model switching and scheduling module selects the optimal model configuration based on the improvement rate after kernel fusion of different candidate model configurations.

[0069] The present invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements a QoS adaptive fine-grained model scheduling method under an edge GPU.

[0070] The present invention also discloses an apparatus including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a QoS adaptive fine-grained model scheduling method under an edge GPU.

[0071] This invention establishes QoS constraints, combines the "two-stage" load characteristics of QC tasks with on-demand model switching triggered by QoS constraints and fine-grained kernel fusion, thereby fully utilizing remaining GPU resources while ensuring long-term effective accuracy requirements. This allows for meeting QoS constraints of QC tasks and maximizing the throughput of BE tasks in resource-constrained edge GPU environments. Compared with existing technologies, this invention has the following advantages: 1. Effective and accurate modeling and adaptive model switching. This invention uses effective accuracy as a QoS metric. By quantifying the model switching time (the sum of model loading time and warm-up time), the invention analyzes the latency violations and effective accuracy loss caused by request backlog during the switching process. This switching overhead is hard-constrained and incorporated into the model selection decision. Under QoS pressure, models with smaller batch sizes or lower computational complexity are prioritized to ensure QoS under dynamic burst workloads.

[0072] 2. Constructing a two-stage load interval coordination mechanism based on resource characteristics. This invention observes the variation patterns of computation and memory throughput under different batch sizes of QC tasks, discovering two key characteristics in the inference process: "unsaturated computational resources" and "persistent memory preference." The fusion space is adaptively divided into two typical intervals: a "low-load zone" and a "memory-constrained zone." Based on the resource bottleneck of the current interval, computationally intensive or memory-intensive BE tasks are selectively combined to maximize overall hardware utilization.

[0073] 3. Achieve PTB-based adaptive fine-grained kernel fusion. This invention uses source-to-source compilation to generate a PTB version of the kernel. A loop mechanism is introduced outside the original kernel, enabling the PTB version kernel to map different blocks of the original mesh to loop iterations while maintaining semantic consistency, running with a fixed number of physical blocks. Based on this, a "fusion strength" (i.e., the ratio of the number of injected BE execution blocks to the total number of original blocks) is defined to achieve fine-grained and controllable fusion of BE task and QC task kernels within a unified scheduling window.

[0074] 4. Implementing a two-stage execution time prediction model to prevent QoS violations. This invention addresses performance interference easily generated under shared load by introducing a two-stage execution time prediction model, capable of predicting the impact of kernel fusion on QoS in real time. In low-load and memory-restricted regions, a quadratic polynomial model and a linear regression model are used respectively, with the fusion strength as input, to accurately predict the execution time of the fused kernel. By predicting latency changes and re-evaluating the effective accuracy, physical-level kernel fusion is only executed within a safe margin, thereby preventing QoS violations.

[0075] To further demonstrate the beneficial effects of the present invention, simulation experiments were conducted in the environment shown in Table 1 in this embodiment. The simulations were performed on an NVIDIA DGX Spark high-performance computing node. The hardware platform ran the Ubuntu 22.04 operating system, was equipped with a 20-core ARM architecture CPU, and its GPU had 6144 CUDA Cores and fifth-generation Tensor Cores, along with 128GB of unified video memory, providing ample physical resources for fine-grained kernel isolation, tensor computation, and concurrent scheduling.

[0076] Table 1 NVIDIA DGX Spark Hardware and Software Environment Configuration Table

[0077] To realistically simulate a hybrid computing scenario involving both cloud and edge computing, the experiment selected two representative types of QC and BE applications for testing. The QC application uses the mainstream ResNet series models (including ResNet-18, ResNet-34, ResNet-50, etc.) in the deep vision field, as shown in Table 2, as benchmarks for image classification inference. All models maintain a consistent overall network structure, differing only in depth and width scaling factors to cover different model capacities and computational complexity ranges. Furthermore, to achieve the fusion of the QC application kernel and the BE application kernel, the QC application is generated by the DNN compiler Rammer. The underlying convolution operators are modified into matrix multiplication using persistent thread blocks (PTB), primarily using Tensor Cores for computation. Finally, the NVCC compilation toolchain is used to complete the source-to-source reconstruction and pre-compilation of the kernel.

[0078] Table 2. Relevant parameters of ResNet series models

[0079] The BE application uses the Parboil and Rodinia benchmark suites, which are widely used to evaluate the performance of GPU microarchitectures. All benchmarks in Parboil use CUDA Cores. Based on the two-stage load characteristic theory, the test kernels in Parboil are divided into two categories: compute-intensive kernels and memory-intensive kernels, to match the resource idleness of QC tasks under different batch sizes.

[0080] Computationally intensive kernels, such as cp, cutcp, fft, and mrif. These tasks primarily consume ALU arithmetic logic units, with relatively low memory bus pressure, making them suitable for complementary fusion when QC tasks are in a "memory-restricted region".

[0081] Memory-intensive kernels, such as Stencil, LBM, and Hot3D. These tasks have relatively light computational loads but frequently access global memory, consuming a great deal of GPU memory bandwidth. They are only allowed by the scheduler when QC tasks are in the "low load zone".

[0082] Workload Request Settings: On the request generation side, for QC tasks, all requests use the ImageNet-2012 dataset, arriving according to a Poisson distribution to simulate uneven user request traffic in a real-world environment, while also introducing artificially generated sudden surges in requests. For image classification tasks using ResNet series models, based on ResNet50, a batch size of 64 is set with a latency of 87ms, and a fixed end-to-end response deadline of 100ms is set. A request is considered a breach when the sum of its queuing and execution time exceeds this threshold. Simultaneously, the correctness of the result is checked for each request. The system's effective accuracy is calculated by periodically statistically analyzing the breach request rate and result accuracy over a certain time interval. Based on the minimum ResNet18 model's Top-1 accuracy of 0.698, and assuming a 95% on-time completion rate, the QoS must be strictly greater than 0.66 throughout the entire runtime. For the BE task, three load environments were set up: compute-biased load, memory-biased load, and hybrid heterogeneous load. The compute-biased load only includes cp, cutcp, fft, and mrif, the memory-biased load only includes stencil, lbm, and hot3d, and the hybrid heterogeneous load includes a cross combination of the above loads.

[0083] In a dynamic, multi-task co-location environment, different BE tasks have different computational complexities and single-iteration times. Therefore, directly comparing their completion rates per unit time is inaccurate. To better measure the improvement effect of adaptive kernel fusion on the overall system throughput, the experiment adopted... As an evaluation indicator. A higher value indicates more efficient utilization of complementary hardware resources within the same time window, resulting in a more significant improvement in the system's actual throughput. Within the entire dynamic load test window, for each request across all scheduling cycles... Take the average value to assess the overall throughput gain.

[0084] This invention is compared with Aker and Baymax. Baymax improves GPU utilization while maintaining QoS by reordering kernels, while Aker maintains QoS and improves GPU utilization through kernel reordering and kernel fusion. Since Baymax does not involve kernel fusion, The value is always 0, therefore this invention can also be considered as... This is an improvement in throughput on Baymax.

[0085] When SLO=0.66, the throughput improvement of multiple BE application sets relative to Aker is as follows: Figure 3As shown, cp is a single kernel for preconditioned conjugate gradient, fft is a single kernel for Fast Fourier Transform, mrif is a single kernel for Multi-Rate Implicit Filter, cutcp is a single kernel for Cut-Cell Conjugate Gradient, lbm is a single kernel for Lattice Boltzmann Method, stencil is a single kernel for Finite Difference, and hot3d is a single kernel for 3D hotspot heat transfer. Hotspot is a single kernel; cp-lbm is a kernel combining cp and lbm; cp-stencil is a kernel combining cp and stencil; cp-hot3d is a kernel combining cp and hot3d; fft-lbm is a kernel combining fft and lbm; fft-stencil is a kernel combining fft and stencil; fft-hot3d is a kernel combining fft and hot3d; mrif-lbm is a kernel combining mrif and lbm; mrif-stencil is a kernel combining mrif and stencil; mrif-hot3d is a kernel combining mrif and hot3d; cutcp-lbm is a kernel combining cutcp and lbm; cutcp-stencil is a kernel combining cutcp and stencil; cutcp-hot3d is a kernel combining cutcp and hot3d. Figure 3The results show that this invention outperforms Aker on multiple single and hybrid BE task sets. Overall, this invention achieves an average throughput improvement of 55.4% (maximum 68.6%), while Aker's average improvement is 32.7% (maximum 37.3%); compared to Aker, this invention can further improve throughput by an average of approximately 20%, with a maximum improvement of 23.9%. Both methods exploit parallelism within the SM and GPU idle time through kernel fusion and kernel reordering, but Aker can only stop kernel fusion with BE applications when it detects QoS violation risks, while this invention can actively release some GPU memory bandwidth and computing resources by dynamically adjusting batch size or switching to a smaller model, thereby continuing to create conditions for BE kernel fusion while ensuring QoS. This advantage is even more pronounced in more complex hybrid BE load scenarios: in all hybrid combinations, the throughput improvement of this invention outperforms Aker, especially in combinations containing high-computation-demand kernels such as fft-lbm, where it can further improve throughput by up to 31.3% compared to Aker. The reason is that the present invention adopts a two-stage scheduling strategy, which can adaptively select BE kernels with complementary resource preferences according to the resource bottlenecks of QC applications at different stages. For example, when the QC application enters the memory-constrained stage, a compute-type BE kernel is inserted, and a memory-type BE kernel is inserted in the low-load stage, thereby making more efficient parallel use of computing units and memory access bandwidth, and obtaining higher throughput gains under heterogeneous mixed loads.

[0086] In addition, to verify the impact of different SLOs on the throughput of this invention, based on the accuracy of the medium-range ResNet50 model (75.1%), and assuming a 95% completion rate, the SLO was set to 0.72. When the QoS threshold is tightened to 0.72, as... Figure 4 As shown, the throughput improvement of both methods decreased; this is because the stricter QoS requirements reduced the space left for fusion and concurrent execution. Nevertheless, under most workload combinations, the performance of this invention still outperforms Aker, fully demonstrating the effectiveness and robustness of the invention framework under stricter QoS constraints.

[0087] Figure 5The evaluation results are presented on a workload segment extracted from the Azure Functions 2021 call trace. This invention plots a 300-second dynamic workload segment and its request arrival rate per second (RPS). Under a 300-second dynamic burst workload, the effective accuracy adaptively adjusts with load changes: during low-load phases, the effective accuracy remains stable at a high level (approximately 0.72), significantly higher than the QoS threshold of 0.66; when the QC task request arrival rate enters a small burst increase range (e.g., 80–120 seconds), the effective accuracy decreases slightly but remains relatively stable at around 0.70; however, when the QC task request arrival rate suddenly increases from 50 RPS to 150 or higher, the effective accuracy decreases with increasing resource pressure and converges to near the threshold, indicating that operations such as model switching are underway. Finally, after the burst workload ends, the effective accuracy generally recovers to a high stable range, demonstrating that this invention can maintain QoS constraints overall during long-term dynamic loads.

[0088] Figure 6 This demonstrates the error between the estimated and actual effective accuracy values ​​after kernel fusion. From Figure 6 As can be seen, the mean of the effective accuracy prediction error is strictly controlled within 0.05 in all 12 mixed load combinations. Among them, the error distribution of the MRIF-related combinations is the most concentrated, showing extremely high prediction stability, while the FFT and CUEP series combinations show a slight increase in error dispersion due to frequent dynamic model switching. This may be because MRIF and other tasks have regular resource requirements, resulting in minimal error, while FFT and other tasks, while significantly improving throughput, introduce more frequent kernel concurrency preemption, leading to instantaneous latency fluctuations and thus a slightly larger variance in prediction error. Even under the most complex heterogeneous load, the upper limit of error does not exceed 0.08, which also demonstrates that this invention can accurately estimate the impact of model switching and kernel fusion on QoS under dynamic load conditions.

[0089] Figure 7The P95 latency of the QC application was compared under three load phases with and without online QoS prediction enabled. The results show that as the request arrival rate increases from 0–50 RPS to 100–150 RPS, the QC application latency gradually increases in both settings. This is mainly due to the simultaneous increase in queuing time and execution pressure as the system load increases. In contrast, the additional latency introduced by the online QoS prediction module remains relatively small: in the 0–50 RPS, 50–100 RPS, and 100–150 RPS phases, the additional P95 latency is only 0.7 ms, 1.2 ms, and 1.3 ms, respectively, with corresponding increases of only about 0.9%, 1.4%, and 1.4%. Furthermore, even in the burst load phase of 100–150 RPS, where the QC application's P95 latency is close to 100 ms, the additional overhead introduced by online QoS prediction remains at around 1 ms, far less than the system average latency. This indicates that the operation of this prediction module does not affect end-to-end latency. Based on the previous results of effective accuracy tracking, it can be seen that the online QoS prediction mechanism proposed in this invention can promptly detect QoS risks under dynamic load, provide a scheduling basis for model switching and kernel fusion, and at the same time bring only minimal runtime overhead, thus having good practicality.

[0090] After validating the ResNet series of CNN architectures, the MobileViT series models were introduced for extended evaluation to verify the present invention's support for Transformer architectures with significant accuracy gradients. MobileViT combines the spatial inductive bias of CNNs with the global processing capabilities of Transformers, providing an ideal scalability verification environment for the dynamic model switching of the present invention. The relevant parameters of the MobileViT series models are shown in Table 3.

[0091] Table 3. Relevant parameters of MobileViT series models

[0092] The throughput improvement comparison results under the MobileViT model are as follows: Figure 8As shown, this invention significantly outperforms Aker across all BE application combinations, especially exhibiting strong resource complementarity in computationally intensive combinations such as FFT: leveraging the large number of idle computing units generated during MobileViT execution, this invention achieves a throughput gain of up to 48.2% in the FFT-LBM combination, which is 19.8% higher than Aker's throughput improvement. Simultaneously, even in combinations with intense memory access contention, such as MRIF, it can still reduce bandwidth pressure through dynamic model switching based on two-stage load characteristics, prompting BE applications to continue kernel fusion. This demonstrates that this invention can achieve adaptive fine-grained model scheduling in QC applications under both CNN and Transformer architectures.

[0093] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A QoS adaptive fine-grained model scheduling method under edge GPUs, characterized in that, include: Get the current model configuration, estimate the completion time of requests under the current model configuration, estimate the latency QoS margin of requests based on the completion time of requests, and calculate the effective accuracy of the current queue backlog based on the latency QoS margin of requests. Based on whether the effective accuracy of the current queue backlog meets the QoS of the QC task, select the corresponding candidate model configuration set and calculate the corresponding proportion of fusion strength. The effective accuracy of each candidate model configuration after fusion is calculated based on the fusion strength. When the effective accuracy after fusion can meet the QoS of the QC task, fusion is performed. The optimal model configuration is selected based on the improvement rate after kernel fusion of different candidate model configurations.

2. The QoS adaptive fine-grained model scheduling method under edge GPUs according to claim 1, characterized in that: The method for calculating the requested delay QoS margin is as follows: ， In the formula, This represents the latency QoS margin for the k-th request. To preset the delay threshold, Configure the waiting time in the queue for the k-th request under the current model. For the quantity of the k-th request, This represents the current throughput.

3. The QoS adaptive fine-grained model scheduling method under edge GPUs according to claim 1, characterized in that: The effective accuracy rate of the current queue backlog is calculated as follows: ， In the formula, The effective accuracy rate of the current queue backlog. This represents the latency QoS margin for the k-th request. To address the current queue backlog, Indicates satisfaction The value is 1 if it is active, and 0 otherwise.

4. The QoS adaptive fine-grained model scheduling method under edge GPUs according to claim 1, characterized in that: The step of selecting the corresponding candidate model configuration set based on whether the effective accuracy of the current queue backlog meets the QoS of the QC task, and calculating the corresponding proportion of fusion strength, is as follows: If the effective accuracy of the current queue backlog is less than the effective accuracy threshold, then stop the BE task and predict the set of candidate model configurations that can meet the current request rate, with the corresponding proportion of fusion strength being 0. If the effective accuracy of the current queue backlog is greater than or equal to the effective accuracy threshold, then predict the set of candidate model configurations that can meet the current request rate and select different BE tasks according to the batch size, and calculate the corresponding proportion of fusion strength through the persistent thread block mechanism.

5. The QoS adaptive fine-grained model scheduling method under edge GPUs according to claim 4, characterized in that: The set of candidate model configurations for stopping the BE task and predicting that can meet the current request rate is specifically as follows: Temporarily stop all BE tasks, search for new model configurations that make the effective accuracy of the queue backlog corresponding to the new model configuration greater than the effective accuracy threshold, and use all the new model configurations found as a candidate model configuration set. When searching for a new model configuration, priority is given to model configurations with a batch size smaller than the current batch size. If the condition cannot be met even after finding the smallest batch size, then a model configuration with lower computational complexity is selected, provided that the effective accuracy of the current queue backlog is greater than or equal to the effective accuracy threshold.

6. The QoS adaptive fine-grained model scheduling method under edge GPUs according to claim 4, characterized in that: The prediction can satisfy the current request rate set of candidate model configurations and select different BE tasks according to the batch size, specifically: Find new model configurations that make the effective accuracy of the queue backlog corresponding to the new model configuration greater than the effective accuracy threshold, and use all the new model configurations found as a set of candidate model configurations. For each candidate model configuration in the candidate model configuration set, if the batch processing of the candidate model configuration is in a low-load area, then a memory-intensive BE task is selected on a first-come, first-served basis. Otherwise, select a computationally intensive BE task.

7. The QoS adaptive fine-grained model scheduling method under edge GPUs according to claim 1, characterized in that: The effective accuracy of each candidate model configuration in the candidate model configuration set after fusion is calculated based on the fusion strength. Fusion is performed when the effective accuracy after fusion can meet the QoS of the QC task. Specifically: The execution time of the fusion kernel is predicted using a two-stage linear regression model, with the fusion strength as input and the execution time of the fusion kernel as output. The latency variation for each request is calculated as follows: ， In the formula, For the latency variation of each request, To reduce the execution time of the fusion kernel, This indicates the execution time of the original QC task. The batch size configured for the i-th candidate model; The end-to-end latency for computing the fusion kernel is: ， In the formula, To reduce end-to-end latency in the fusion kernel, For the first in the queue End-to-end latency of a request; The number of invalid requests for the fused kernel is calculated as follows: ， In the formula, To account for the number of illegal requests in the fusion kernel, As a preset delay threshold, 1 Indicates satisfaction The value is 1 if the time condition is met, and 0 otherwise. The cumulative number of requests in the queue; The violation rate for computing fusion kernels is: ， In the formula, To reduce the violation rate of the fusion kernel; Computational fusion The effective accuracy rate for each request is: ， In the formula, For the fusion The effective accuracy of each request. Configure the corresponding effective accuracy of the queue backlog for the candidate models; if , If the effective accuracy threshold is set, then the effective accuracy after fusion is considered to meet the QoS of the QC task, and fusion is performed in this case.

8. The QoS adaptive fine-grained model scheduling method under edge GPUs according to claim 7, characterized in that: The The calculation method is as follows: ， In the formula, Configure the waiting time in the queue for the k-th request under the current model. For model switching time, For the k-th request, This indicates the processing rate configured for the candidate model. This represents the number of requests currently in the queue. For the current throughput, In order to be in Request arrival rate when initiating model switching at any time.

9. The QoS adaptive fine-grained model scheduling method under edge GPUs according to claim 1, characterized in that: The method for calculating the improvement rate is as follows: ， In the formula, For the improvement rate, This refers to the execution time of the original QC task. This refers to the execution time of the original BE task. For fusion strength, This refers to the execution time of the fusion kernel.

10. A QoS adaptive fine-grained model scheduling system under edge GPUs, characterized in that: It includes a model switching scheduling module, an offline compilation module, and an adaptive kernel fusion scheduling module. The adaptive kernel fusion scheduling module includes a kernel selection component, a QoS prediction component, and a kernel fusion control component. The model switching scheduling module obtains the current model configuration, estimates the completion time of requests under the current model configuration, estimates the latency QoS margin of requests based on the completion time of requests, and calculates the effective accuracy of the current queue backlog based on the latency QoS margin of requests. The offline compilation module performs kernel code refactoring and pre-compilation in the offline phase for all BE applications. The adaptive kernel fusion scheduling module sets each QC task as a QC application kernel sequence and sets the BE application as a BE application kernel sequence. The fusion process takes the kernel pair of QC application kernel and BE application kernel as the basic unit. The kernel selection component observes the current state of the QC application kernel and selects complementary candidate kernel pairs from the BE application kernel sequence. Based on whether the effective accuracy of the current queue backlog meets the QoS of the QC task, it selects the corresponding candidate model configuration set and calculates the corresponding proportion of fusion strength. The QoS prediction component performs an online feasibility check on the candidate kernel pairs, calculates the effective accuracy rate to determine whether a QoS violation occurs, and calculates the effective accuracy rate of each candidate model configuration after fusion in the candidate model configuration set in combination with the fusion strength. When the effective accuracy after fusion can meet the QoS of the QC task, the kernel fusion control component performs fusion; The model switching and scheduling module selects the optimal model configuration based on the improvement rate after kernel fusion of different candidate model configurations.

Images