LLM offline-online hybrid load scheduling optimization method
By employing an LLM offline-online hybrid load scheduling optimization method, and utilizing heterogeneous execution engines and a progressive task migration strategy, the problem of resource sharing and contention in LLM inference services is solved, achieving online request latency guarantees and high throughput for offline tasks, thereby improving resource utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING INFORMATION HIGH-SPEED RAILWAY RES INST OF SCI AND TECH
- Filing Date
- 2026-04-22
- Publication Date
- 2026-07-24
AI Technical Summary
In existing technologies for LLM inference services, resource sharing and contention in hybrid scheduling scenarios lead to online request latency failing to meet SLO, unstable offline task throughput, and low resource utilization, making it difficult to achieve efficient scheduling.
The LLM offline-online hybrid load scheduling optimization method is adopted. Through request classification, heterogeneous execution engines and progressive task migration strategy, it ensures that online requests reside in the KV cache in the GPU execution engine, and the Hybrid execution engine offloads memory-intensive computations to the CPU. Combined with adaptive batch size adjustment and asynchronous synchronization mechanism, resource isolation and efficient utilization are achieved.
It guarantees the latency SLO for online requests, provides a high-throughput and stable execution environment for offline tasks, improves resource utilization, and reduces performance jitter and heavy computation overhead.
Smart Images
Figure CN122086630B_ABST
Abstract
Description
Technical Field
[0001] This invention mainly relates to the field of computer software and large language model inference service technology, and in particular to an LLM offline-online hybrid load scheduling optimization method. Background Technology
[0002] Inference services for large language models (LLMs) have become a core infrastructure supporting modern artificial intelligence applications. In actual production, service loads are typically heterogeneous and can be divided into two categories based on Service Quality Objectives (SLOs): interactive online requests and batch processing offline requests. The former (such as intelligent dialogue and real-time assistance) is extremely sensitive to latency, and its performance is usually defined by first-to-first latency (TTFT) and inter-to-token latency (TPOT), requiring strict long-tail latency guarantees (e.g., P99 TTFT < 500ms). The latter (such as content summarization and data generation) has less latency constraints, with maximizing throughput as the core objective. To meet the drastically different SLO requirements of these two types of requests, the industry commonly adopts a resource isolation deployment paradigm, allocating independent computing clusters to each type of request. While this approach simplifies the complexity of scheduling and assurance, it inevitably leads to severely low resource utilization. Especially considering that online request traffic typically exhibits significant day-night patterns and sudden fluctuations (peak demand can reach 3-6 times the average within minutes), resource scaling at the minute level is often difficult to achieve due to the complexity of the infrastructure and engineering costs. Service providers are forced to over-provision resources based on peak loads, resulting in expensive GPU computing power being idle most of the time, significantly increasing operating costs.
[0003] Co-scheduling online and offline requests across shared GPU instances is considered a key approach to improving resource efficiency. The core idea is to utilize periods of low online request load to execute offline tasks, thereby improving the long-term average utilization of the GPU. However, achieving this goal faces three interrelated fundamental challenges:
[0004] (1) Scheduling interference and head-of-line blocking: If a simple first-come, first-served (FCFS) strategy is adopted, long-running offline requests will block subsequent online requests, directly causing them to fail to meet the strict latency SLO.
[0005] (2) Increased competition for GPU memory resources: Autoregressive LLM inference requires a dynamically maintained and growing key-value cache (KV Cache) for each request. The KV Caches of online and offline requests directly compete for the fixed GPU memory space. When high-priority online requests cannot be scheduled due to insufficient GPU memory, the system has to evict or replace the KV Cache of low-priority offline requests, thereby introducing high recomputation overhead or data transfer latency between the host and GPU devices, which seriously impairs offline throughput and overall system efficiency.
[0006] (3) Scheduling optimization challenges under uncertainty: The generation length of LLM requests cannot be known in advance, making its total computational cost uncertain. This makes it difficult to accurately predict the request completion time, and it becomes extremely difficult to design a dynamic scheduling strategy for online requests that satisfies both TTFT and TPOT, while also taking into account the optimal offline throughput.
[0007] To address these challenges, existing research has made some progress. For example, systems like vLLM alleviate head-of-line congestion by introducing a basic preemption mechanism; Sarathi-Serve et al. utilize block pre-filling techniques to coordinate the execution of the pre-filling and decoding phases. More advanced works, such as HyGen and Echo, have begun to formalize hybrid scheduling as a constrained optimization problem that maximizes offline throughput while ensuring online SLO constraints, and have designed corresponding prediction and scheduling algorithms.
[0008] However, these advanced solutions still share a common structural limitation: they all perform hybrid scheduling within a single, homogeneous execution engine, lacking fundamental isolation between the two types of tasks in their competition for critical resources (especially KV cache memory). Therefore, when online load fluctuates drastically, the execution of offline tasks suffers severe performance instability and throughput degradation due to frequent swapping out and in of the KV cache. Existing systems also often fail to adequately quantify the significant performance overhead caused by large-scale KV cache migration or recomputation when making scheduling decisions.
[0009] Therefore, the current field urgently needs a new system design paradigm that can achieve isolation of computational paths and lightweight state transitions, thereby providing deterministic performance guarantees for online requests under dynamic loads, while providing a stable, efficient and high-throughput execution environment for offline tasks. Summary of the Invention
[0010] Addressing the core challenges arising from resource sharing and competition in hybrid scheduling scenarios, this invention aims to propose an LLM offline-online hybrid load scheduling optimization method. This method transcends the limitations of existing homogeneous scheduling frameworks by employing systematic hardware and software co-design. It ensures that online requests strictly meet latency SLOs while providing a high-throughput, stable execution environment for offline tasks, thereby achieving simultaneous optimization of resource utilization and service quality.
[0011] To achieve the above objectives, this invention provides an LLM offline-online hybrid load scheduling optimization method, comprising the following steps:
[0012] Step 1: Use the LLM offline-online hybrid load scheduling optimization system. The system includes a request classifier, a GPU execution engine, a Hybrid execution engine, and a task migration and resource control module. The request classifier is set at the system entry point. The GPU execution engine and the Hybrid execution engine provide the execution and isolation environment for online and offline request tasks. The task migration and resource control module is responsible for request task scheduling and control.
[0013] Step 2: The request classifier dynamically classifies each arriving request into a high-priority online request or a low-priority offline request according to a preset strategy, and routes the request to the high-priority online request queue or the low-priority offline request queue respectively.
[0014] Step 3: The GPU execution engine serves the high-priority online request queue, keeping the key-value cache of all high-priority online requests fully resident in GPU memory;
[0015] The Hybrid execution engine serves the low-priority offline request queue. It adopts a heterogeneous computing paradigm, offloading the memory-intensive Attention layer computation in the GPU to the CPU for execution, and offloading the corresponding key-value cache management from GPU memory to CPU memory.
[0016] Step 4: The task migration and resource control module adopts a progressive task migration strategy and an adaptive batch size adjustment strategy to dynamically allocate computing resources based on the real-time queue status of the GPU execution engine and the Hybrid execution engine.
[0017] Furthermore, in step 1, the GPU execution engine and the Hybrid execution engine share the same memory in the GPU, share the same model weights, use the CUDA Stream mechanism and concurrently execute the same MLP layer computations in the forward propagation, thus avoiding the overhead of repeated storage and loading of model weight parameters.
[0018] Furthermore, in step 3, the Hybrid execution engine uses vLLM-like paging memory management technology to organize the key-value cache on the CPU memory side.
[0019] The Hybrid execution engine divides a computation batch into several micro-batches for pipelined execution to mask the overhead of PCIe data transfer.
[0020] Furthermore, in step 4, the incremental task migration strategy adopts an incremental asynchronous key-value cache pre-synchronization mechanism. For all offline tasks executed on the GPU execution engine, the newly generated key-value cache will be continuously and asynchronously synchronized to the CPU memory in the background in an incremental and batch manner.
[0021] Incremental asynchronous synchronous transmission only transmits newly added key-value cache data and uses batch merging and packaging DMA transmission.
[0022] Furthermore, in step 4, the adaptive batch size adjustment strategy uses a feedback-based adaptive batch adjuster, and the decision logic includes the following steps:
[0023] Step 4.1: Status Monitoring: Before the start of each scheduling iteration, collected metrics include the length of the high-priority waiting queue. GPU core utilization ;
[0024] Step 4.2: Decision-making and adjustment: When detected and When the pre-set congestion threshold is exceeded, it means that high-priority online requests are queued. The system will then maximize the batch size of the GPU execution engine and reduce the batch size of the Hybrid execution engine.
[0025] When detected and If the congestion threshold is less than the preset threshold, it means that high-priority online requests are not queued, and the batch size of the Hybrid execution engine is restored or increased.
[0026] Beneficial Effects: This invention provides an LLM offline-online hybrid load scheduling optimization method with the following technical effects: (1) Ensuring strict latency (SLO) for online requests: Ensuring that the first-to-first-to-the ...
[0027] In summary, this invention, through systematic hardware and software co-design, ensures that online requests strictly meet latency SLOs while providing a high-throughput, stable execution environment for offline tasks, thereby achieving optimization of both resource utilization and service quality. Attached Figure Description
[0028] Figure 1 This is a schematic diagram of the Attention computation execution flow of the Hybrid execution engine involved in the embodiments of the present invention;
[0029] Figure 2 This is a diagram of the LLM offline-online hybrid load scheduling optimization method and system architecture involved in the embodiments of the present invention. Detailed Implementation
[0030] like Figures 1-2 As shown, this invention provides an LLM offline-online hybrid load scheduling optimization method.
[0031] Example 1: The system involved in this embodiment of the invention consists of four parts: a classification requester, a GPU execution engine, a Hybrid execution engine, and a task migration and resource control module, as follows: Figure 2 As shown in the diagram. The GPU execution engine and the Hybrid execution engine provide the execution and isolation environment for online and offline tasks, while the task migration and resource control module is responsible for task scheduling and control at other levels.
[0032] I. Classification Requester
[0033] The system's entry point includes a request classifier. This component dynamically categorizes each arriving request into high-priority or low-priority requests based on preset strategies (such as user identity, request type, or explicit service level agreements). After categorization, requests are routed to two independent priority queues (high-priority request queue or low-priority request queue). The high-priority request queue ensures that online interactive requests can be responded to rapidly by the GPU execution engine; the low-priority request queue serves offline batch processing tasks, handled by a dedicated Hybrid execution engine.
[0034] II. Heterogeneous Dual-Path Execution Architecture
[0035] This architecture includes two fully functional execution engines with different design goals, which work together to achieve the overall system objectives.
[0036] The GPU execution engine prioritizes extremely low latency and is dedicated to serving high-priority online requests, with the core design goal of achieving the lowest possible response delay. To ensure performance, the GPU execution engine resides entirely in the GPU memory for all online request key-value caches, avoiding any cross-device data movement that could introduce latency. Its scheduling strategy is highly optimized, prioritizing the rapid scheduling and execution of online requests.
[0037] Hybrid Execution Engine: Achieving High Throughput and Preemptibility. The hybrid execution engine is the core innovation of this design, specifically built to handle low-priority, high-throughput offline requests. Its performance directly determines the overall system throughput. This engine adopts a heterogeneous computing paradigm, offloading memory-intensive Attention layer computations to the CPU and offloading the corresponding KV cache management to larger CPU memory, such as... Figure 1As shown, this design achieves the key stateless preemptive characteristic: low-priority tasks can be interrupted at any time without triggering expensive GPU memory data migration. Specifically, we replace the Attention operator in the model with a more efficient CPU version and employ vLLM-like paging memory management techniques to organize the CPU-side KVCache. To further mask the PCIe data transfer overhead, the engine divides a computation batch (Mini-Batch) into multiple micro-batches for pipelined execution.
[0038] The two execution engines are not entirely independent. They share the same model weights stored in the GPU HBM memory, use the CUDA Stream mechanism, and concurrently execute the same MLP layer computations during forward propagation, avoiding the overhead of redundant storage and loading of model parameters. Meanwhile, their key-value (KV) caches are completely isolated in terms of physical storage and logical management: the GPU execution engine's KV cache resides in GPU memory, while the Hybrid execution engine's cache resides in host memory. This design not only eliminates cache contention but also leverages the fact that CPU memory capacity is far greater than GPU memory, enabling offline tasks to support almost infinitely expandable context lengths.
[0039] III. Load-Aware Task Migration and Resource Control Module
[0040] The online service request load exhibits significant diurnal cyclical fluctuations and sudden peak load characteristics, with daytime request rates reaching several times that of nighttime. To simultaneously achieve efficient resource utilization during off-peak periods and ensure service targets are met during peak periods, this system employs a load-aware elastic resource management mechanism. This mechanism's core comprises two collaborative components: a gradual task migration strategy and an adaptive batch size adjustment strategy, jointly ensuring the system maintains an optimal balance between performance and efficiency under dynamic loads.
[0041] The system's migration decisions are based on real-time monitoring of online request load. When the online load is consistently below a set congestion threshold, indicating that the system is in a resource-rich period, the scheduler intelligently migrates some offline tasks from the Hybrid execution engine to the GPU execution engine to fully utilize idle GPU computing power and significantly improve offline throughput. The core trade-off in migration lies in the recomputation overhead versus the KV cache relocation overhead. To address this, the system designs a progressive, asynchronous KV cache pre-synchronization mechanism. For all offline tasks executed on the GPU engine, their newly generated KV cache is continuously and asynchronously synchronized to CPU memory in the background in an incremental, batch manner. That is, offline tasks maximize CPU utilization using the GPU execution engine when the GPU is idle, and quickly switch to the Hybrid execution engine when a sudden surge in online task requests arrives through the "key-value cache pre-synchronization mechanism." This design offers two key advantages: First, it avoids the long-term blocking that may result from traditional one-time full migration, ensuring millisecond-level resource recovery when online tasks suddenly occur; second, incremental synchronization only transmits newly added data and uses batch-merged DMA transmission, effectively reducing the number of PCIe communication calls and overhead.
[0042] Although the two engines isolate memory resources, they share pre-attention and MLP layer computations, potentially competing for GPU streaming multiprocessor (SM) resources. To address this, a feedback-based adaptive batch adjuster is introduced. This adjuster dynamically allocates computational resources based on the real-time queue state of the two engines, with the following decision logic:
[0043] (1) Status monitoring: Before the start of each scheduling iteration, the collected metrics include the length of the high-priority waiting queue. GPU core utilization ;
[0044] (2) Decision-making and adjustment: When a detection is made and When the pre-set congestion threshold is exceeded, it means that high-priority online requests are queued. The system will then maximize the batch size of the GPU execution engine and reduce the batch size of the Hybrid execution engine, allocating more SM resources to high-priority online requests.
[0045] When detected and If the congestion threshold is less than the preset threshold, it means that high-priority online requests are not queued. In this case, the batch size of the Hybrid execution engine will be restored or increased to improve overall throughput.
[0046] The aforementioned strategy is coordinated and executed by a unified central scheduler. Before each inference iteration, the scheduler sequentially executes a closed-loop process of "monitoring-decision-adjustment" to achieve joint dynamic optimization of task location and computing resources. This design enables the system to automatically adjust the resource allocation of each engine based on external load pressure, continuously maximizing global resource utilization while ensuring strict SLOs.
[0047] IV. Example of Task Migration and Resource Control Operation Mechanism
[0048] The following example illustrates the mechanism of load-aware task migration, using the task migration process from a nighttime slump to a daytime peak. Initially, the system is in a low-load nighttime period with sparse online requests. Assume a server with an 80GB A100 GPU and 512GB of RAM (host) is running. The load monitor continuously reports extremely low online request rates. A long-context, low-priority offline document summarization task (assuming it needs to generate 2000 tokens) is migrated to the GPU execution engine to utilize idle GPU computing power. Its KV cache has grown to approximately 500MB and is stored in GPU memory. The scheduler determines that the current period is "resource-rich" and does not immediately evict the offline task, but instead starts a background asynchronous backup task. Whenever this low-priority offline request task generates a new token (e.g., 16KB of KV data) on the GPU, the newly added KV cache block is not immediately removed, but is added to a transfer queue and incrementally and in batches copied to a reserved area in host memory via the PCIe bus by a background thread.
[0049] If, at dawn, a sudden surge of high-priority online chat requests arrives, instantly putting the system under high load. The scheduler detects the immediate backlog in the high-priority online request queue and the imminent filling of GPU memory with new online requests. Without waiting or initiating a large-scale KV cache relocation, the system directly sends an instruction to the GPU execution engine: immediately pause the low-priority offline request task and directly release its 500MB KV cache space in GPU memory. The paused offline task and its complete context state (already backed up to host memory) are seamlessly transferred to the Hybrid execution engine. The Hybrid execution engine directly loads its KV cache from host memory, resumes Attention calculations on the CPU, and continues subsequent token generation.
[0050] Following the sudden surge, the arrival rate of online requests remained high, and the length of the high-priority waiting queue ( The queue not only failed to clear but continued to grow, exceeding the preset congestion threshold (e.g., more than 5 waiting requests in the queue). Simultaneously, GPU core utilization... Monitoring revealed that SM (Streaming Multiprocessor) resources were completely saturated, becoming a bottleneck. At this point, even though offline tasks had migrated to the Hybrid execution engine, competition for computational resources at the FFN layer began to emerge, leading to a longer overall iteration time and a risk of worsening TPOT (Term-to-Telemetry Latency). The scheduler then implemented a simple optimization decision: increasing the batch size of the GPU execution engine and decreasing the batch size of the Hybrid execution engine, for example, increasing the GPU batch size from 8 to 16 and drastically reducing the Hybrid batch size from 32 to 8. Although the absolute time of a single iteration for the GPU execution engine might increase slightly, the average waiting time for each online request was significantly reduced, and the system's ability to clear high-priority queues was significantly enhanced, effectively curbing further deterioration of TTFT and TPOT. The decrease in the Hybrid execution engine's batch size resulted in a decrease in its instantaneous throughput; however, since offline tasks themselves have no hard latency requirements, this performance fluctuation was acceptable.
[0051] As this strategy is implemented, the length of the high-priority waiting queue will... It began to gradually decrease. When When the load falls below the safe threshold, the regulator will gradually reverse the operation, slowly restoring the batch size of the Hybrid execution engine to improve overall system throughput until a new equilibrium is reached.
[0052] This invention proposes an LLM offline-online hybrid load scheduling optimization method that transcends the limitations of existing homogeneous scheduling frameworks. Through systematic hardware and software co-design, it ensures that online requests strictly meet latency SLOs while providing a high-throughput and stable execution environment for offline tasks, thereby achieving joint optimization of resource utilization and service quality.
[0053] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. However, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A hybrid offline-online load balancing optimization method for LLM, characterized in that, Includes the following steps: Step 1: Use the LLM offline-online hybrid load scheduling optimization system. The system includes a request classifier, a GPU execution engine, a Hybrid execution engine, and a task migration and resource control module. The request classifier is set at the system entry point. The GPU execution engine and the Hybrid execution engine provide execution and isolation environments for online and offline request tasks. The task migration and resource control module is responsible for request task scheduling and control. Step 2: The request classifier dynamically classifies each arriving request into a high-priority online request or a low-priority offline request according to a preset strategy, and routes the request to the high-priority online request queue or the low-priority offline request queue respectively. Step 3: The GPU execution engine serves the high-priority online request queue, keeping the key-value cache of all high-priority online requests entirely in GPU memory; The Hybrid execution engine serves the low-priority offline request queue. It adopts a heterogeneous computing paradigm, offloading the memory-intensive Attention layer computation in the GPU to the CPU for execution, and offloading the corresponding key-value cache management from GPU memory to CPU memory. The Hybrid execution engine uses vLLM-like paging memory management technology to organize the key-value cache in CPU memory. The Hybrid execution engine divides a computation batch into several micro-batches for pipelined execution to mask the overhead of PCIe data transfer. Step 4: The task migration and resource control module adopts a progressive task migration strategy and an adaptive batch size adjustment strategy to dynamically allocate computing resources based on the real-time queue status of the GPU execution engine and the Hybrid execution engine.
2. The LLM offline-online hybrid load scheduling optimization method according to claim 1, characterized in that, In step 1, the GPU execution engine and the Hybrid execution engine share the same memory in the GPU and use the same set of model weights. They use the CUDA Stream mechanism and concurrently execute the same MLP layer computations during forward propagation, thus avoiding the overhead of repeatedly storing and loading model weight parameters.
3. The LLM offline-online hybrid load scheduling optimization method according to claim 1, characterized in that, In step 4, the incremental task migration strategy adopts an incremental asynchronous key-value cache pre-synchronization mechanism. For all offline tasks executed on the GPU execution engine, the newly generated key-value cache will be continuously and asynchronously synchronized to the CPU memory in the background in an incremental and batch manner. Incremental asynchronous synchronous transmission only transmits newly added key-value cache data and uses batch merging and packaging DMA transmission.
4. The LLM offline-online hybrid load scheduling optimization method according to claim 1, characterized in that, In step 4, the adaptive batch size adjustment strategy uses a feedback-based adaptive batch adjuster, and the decision logic includes the following steps: Step 4.1: Status Monitoring: Before the start of each scheduling iteration, collected metrics include the length of the high-priority waiting queue. GPU core utilization ; Step 4.2: Decision-making and adjustment: When detected and When the pre-set congestion threshold is exceeded, it means that high-priority online requests are queued. The system will then maximize the batch size of the GPU execution engine and reduce the batch size of the Hybrid execution engine. When detected and If the congestion threshold is less than the preset threshold, it means that high-priority online requests are not queued, and the batch size of the Hybrid execution engine is restored or increased.