Large model reasoning framework scheduler based on time delay and cache awareness

Abstract

The invention discloses a large model reasoning framework scheduler based on time delay and cache awareness, and belongs to the technical field of artificial intelligence and distributed systems. According to the method, a self-adaptive double-level collaborative framework is adopted and comprises a batch processing resource allocation layer and a cache planning scheduling layer based on a prefix tree. The batch processing resource distribution layer dynamically calculates a time budget and restrains token selection in each scheduling, so that the stable generation rate of a decoding request is guaranteed, the pre-filling request processing is accelerated, and the first token delay is optimized. The cache planning scheduling layer periodically constructs a waiting queue request into a prefix tree, operates a multi-stage greedy algorithm and resorts a request execution sequence to maximize the utilization rate of key value cache and the prefix reuse rate between the requests, so that the overall throughput and resource efficiency of the system are improved. The two layers of schedulers work cooperatively, the problem that service quality and system throughput are difficult to consider in a high-concurrency scene is effectively solved, and better scheduling fairness and resource utilization rate are achieved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and distributed systems technology, specifically to the optimization of request scheduling and resource allocation in large language model (LLM) inference services, and particularly to a scheduler for a large model inference framework based on latency and cache awareness. Background Technology

[0002] Large Language Models (LLMs) based on the Transformer architecture have demonstrated outstanding performance in tasks such as text generation and code completion, and their large-scale deployment has become crucial for industrial applications. However, with the dramatic increase in model parameters and the explosive growth in user requests, the challenges faced by inference systems have evolved from simple computational bottlenecks to the complex problem of coordinating and optimizing multi-dimensional resources such as computing, memory, scheduling, and communication.

[0003] To address the high-performance demands of real-time applications, open-source inference frameworks such as vLLM and SGLang have developed a multi-layered optimization system: at the computation layer, they fully leverage the potential of hardware computing power through operator fusion, kernel optimization, mixed precision, and quantization techniques; at the memory layer, they effectively overcome memory capacity limitations by designing intelligent key-value cache (KV Cache) management (Pageattention) and dynamic memory loading and unloading strategies (LM Cache, MoonCake); at the parallelism and scheduling layer, they achieve efficient resource utilization at both the request and model levels using techniques such as continuous batch processing and chunked prefill. These optimization methods significantly improve the speed of single inference iterations and enhance resource utilization and multi-user concurrent service capabilities at the system level, making inference frameworks a core hub connecting large-scale model algorithm innovation and large-scale application deployment.

[0004] LLM inference typically consists of two phases: a prefill phase and a decoding phase. The prefill phase processes the entire input prompt in parallel and generates the corresponding initial key-value cache. The decoding phase generates output tokens sequentially using an autoregressive approach. To maximize the parallel computing efficiency of GPUs and distribute memory access overhead, mainstream inference frameworks generally employ continuous batch processing and block prefilling techniques. This merges newly arriving prefilling tasks with ongoing decoding tasks, forming mixed batches, thereby significantly improving the overall system throughput.

[0005] However, efficiently scheduling user requests and mixed batches remains a highly challenging problem. The system must minimize First Token Latency (TTFT) and maintain a high and stable Token Generation Rate (TPOT) during the decoding phase while ensuring throughput efficiency. Existing advanced schedulers (such as Sarathi, based on "pause-free batch processing") aim to balance TTFT and TPOT. Their core is a decoding priority strategy and a fixed token budget to prevent decoding pauses. However, this design can lead to severe resource allocation unfairness and efficiency losses under dynamic loads, specifically manifested in: 1. Static resource allocation strategies cannot adapt to dynamic loads, causing service target conflicts. The shortcomings of existing solutions stem from their lack of adaptability to workload dynamics. Real-world inference request traffic does not arrive at a uniform rate, but rather exhibits significant alternations between "quiet periods" and "burst periods." The system's use of fixed time budgets or resource quotas to handle mixed batches leads to severe service quality imbalances in dynamic scenarios.

[0006] During the quiet period: Since no new pre-filled tasks arrive, all computational resources are allocated to ongoing decoding requests. This causes decoding requests to continuously receive far more computational resources than they need to maintain a stable output rate of time (TPOT), resulting in an "overabundance" of resource allocation.

[0007] During periods of high demand: When a large number of pre-fill requests arrive simultaneously, the system still adheres to its original static resource allocation strategy, reserving a fixed proportion of computation time or batch space for decoding tasks. This results in computational resources not being allocated promptly and flexibly to pre-fill requests that urgently need to complete initial computations, causing them to wait in the queue for extended periods. This leads to a sharp deterioration in the First Token Delay (TTFT) and frequent violations of Service Level Objectives (SLOs).

[0008] The fundamental contradiction lies in the fact that existing solutions solidify the technical means of "ensuring uninterrupted decoding" to achieve stable TPOT as a rigid resource allocation rule. This leads to rigid resource allocation—decoding requests "over-consume" computing resources during quiet periods, and "crowd out" critical resources that should be prioritized for reducing TTFT during burst periods, making it impossible for the system to simultaneously guarantee the two core service objectives of TTFT and TPOT under dynamic load.

[0009] 2. A crude queuing mechanism ignores request heterogeneity and wastes critical resources. In real-world business scenarios, user requests vary significantly in terms of input length, output length, and service level objectives. However, mainstream inference engines typically employ default queuing strategies such as "first-come, first-served" or simple "shortest task first," failing to adequately consider the heterogeneous nature of requests. For example, when key-value cache resources are limited, the FCFS strategy might mix a long context request with multiple short requests in the same batch. This results in valuable key-value cache space being heavily consumed by long requests, while only sharing computing resources with a few short requests, significantly reducing cache memory utilization efficiency. Research shows that in scenarios with severe memory constraints, the contention ratio of such simple strategies deteriorates linearly with memory limitations, causing severe performance losses.

[0010] To address the aforementioned issues, a novel scheduling framework is urgently needed to achieve intelligent request management and resource allocation. Summary of the Invention The purpose of this invention is to propose a latency- and cache-aware large-model inference framework scheduler to address the following problems raised in the background art: (1) Insufficient resource planning capability: unable to perform refined resource planning based on dynamic load and request characteristics, making it difficult to fully utilize the potential of the system and hardware.

[0011] (2) Inefficient process management: Under dynamic business load, it is difficult to provide stable and predictable service quality assurance.

[0012] To address the aforementioned issues, this invention designs an innovative two-layer scheduling framework based on latency constraints and key-value cache awareness to optimize the collaborative execution of multiple requests and the allocation of system resources, thereby improving service quality and overall resource utilization in high-concurrency scenarios.

[0013] To achieve the above objectives, the present invention adopts the following technical solution: A large-scale model inference framework scheduler based on latency and cache awareness, the scheduler includes a tightly cooperating batch resource allocation layer and a cache planning and scheduling layer based on a prefix tree; The batch processing resource allocation layer is used to dynamically calculate the batch processing time budget based on the service level target requested in the run queue in each scheduling iteration, and under the constraint of the budget, select decoded tokens and pre-filled tokens from the requests in the run queue to form a mixed batch and send it into the model for calculation, so as to accelerate the completion of pre-filled requests while stabilizing the token generation rate. The prefix tree-based cache planning and scheduling layer is used to periodically select requests from the waiting queue and add them to the running queue. By constructing the requests in the waiting queue into a prefix tree and executing an optimization algorithm, a request scheduling sequence that maximizes the utilization of the key-value cache and the prefix reuse rate between requests is generated to improve the overall throughput of the system.

[0014] Preferably, the batch processing resource allocation layer specifically performs the following operations: Dynamic budget calculation: Calculate the tightest time budget for this batch processing based on the arrival time of each request in the run queue, the initial token delay target, the time target per output token, and the current system time; Token selection under budget constraints: Within the tightest time budget, computing resources are allocated to requests in the decoding phase to generate their next token, and the remaining budget is then allocated to requests in the pre-population phase. Pre-fill concurrency control: When the computational cost of a pre-fill request exceeds the remaining budget, the pre-fill request is processed in blocks, and only the tokens that are completed within the budget are added to the current batch.

[0015] Preferably, the prefix tree-based cache planning and scheduling layer specifically performs the following operations: Prefix tree construction: For all requests in the waiting queue, construct one or more prefix trees based on their input token sequences, where nodes in the tree represent prefix paths of the token sequences and leaf nodes represent requests; Multi-stage sequence selection: Execute a multi-stage greedy algorithm that includes prefix reuse optimization and cache pool filling stages. Under the constraint of key-value cache capacity, select a set of request sequences that maximizes the reuse of common cache nodes and minimizes the cost of adding new caches. Polling and starvation prevention: The multi-stage greedy algorithm is run in rounds. After each round of scheduling, the running queue is cleared before a new round of planning begins, ensuring that all requests are processed.

[0016] Preferably, the multi-stage greedy algorithm includes: Phase 1, Prefix Selection and Expansion: Select the request with the greatest benefit from the common prefix with other requests as the seed, and add requests that share the prefix with the seed to the scheduling sequence to reuse the allocated cache nodes; Phase 2, Cache Pool Filling: For the remaining no-shared or low-shared prefix requests, add them to the scheduling sequence in ascending order of new cache cost until the cache capacity is exhausted.

[0017] Preferably, the batch processing resource allocation layer relies on a single token computation time model in the pre-filling and decoding stages for budget estimation. The time model is established based on the computational characteristics of the Transformer architecture, and the model parameters are adaptively adjusted by collecting actual execution time online to match specific hardware and runtime environments.

[0018] This invention further protects the application of the above scheduler, used to optimize request scheduling and resource allocation in different hardware card types, different user request types, and mainstream large language models based on the Transformer architecture.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) Service quality assurance: This invention stabilizes the token generation rate by constraining the batch time budget and effectively reduces the initial token delay by adaptive pre-filling scheduling, thus systematically ensuring user experience.

[0020] (2) Significantly improved resource utilization: The request sorting mechanism based on the prefix tree in this invention fully explores the common prefixes between requests, realizes efficient reuse of KV Cache and optimizes memory utilization, and improves the overall throughput of the system.

[0021] (3) Guarantee of scheduling fairness: The present invention adopts a periodic polling mechanism to avoid starvation caused by over-optimization of specific types of requests, and ensures that all types of requests can get a processing opportunity.

[0022] (4) Strong architecture versatility: The present invention is designed to be compatible with different hardware platforms, diverse request types and mainstream large language models, and has good scalability and adaptability.

[0023] (5) Balance between system efficiency and service quality: This invention achieves the optimal balance between system throughput and service latency in high-concurrency scenarios through a two-layer collaborative scheduling mechanism, providing effective technical support for the large-scale deployment of large model inference services.

[0024] In summary, this invention achieves the optimal balance between efficiency and quality in large-model inference services through two-layer collaborative scheduling, providing an effective solution to the inference scheduling problem in high-concurrency scenarios. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings involved in the embodiments are now briefly described. Obviously, the drawings in the following description are merely illustrative of some embodiments of the present invention. For those skilled in the art, other forms of drawings can be constructed based on these drawings without creative effort.

[0026] Figure 1 is a schematic diagram of the overall structure of the latency- and cache-aware inference framework scheduler mentioned in Embodiment 1 of the present invention; Figure 2 is a schematic diagram of the actual operation process of the cache planning and scheduling layer based on the prefix tree mentioned in Embodiment 1 of the present invention; Figure 3 is a schematic diagram of the actual operation process of the latency-aware batch processing resource allocation layer mentioned in Embodiment 1 of the present invention. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0028] To address the problems of existing large-scale model inference scheduling schemes, such as imprecise request processing order planning, difficulty in balancing throughput efficiency and key latency indicators, and failure to fully utilize inter-request features (such as common prefixes) to optimize resource utilization, this invention proposes a latency- and cache-aware large-scale model inference framework scheduler. By working collaboratively between a batch processing resource allocation layer and a cache planning and scheduling layer based on a prefix tree, it achieves efficient collaborative execution of multiple requests, significantly improving system throughput and resource utilization while ensuring service level objectives.

[0029] The technical solution of this invention is composed of the following two closely cooperating structural layers: 1. Batch processing resource planning layer This layer is responsible for selecting an appropriate number of tokens from the current running queue to form a batch in each scheduling iteration and feeding them into the model for computation. Its core objective is to maintain a stable token generation rate during the decoding phase while strictly ensuring the processing time budget for each batch, and simultaneously accelerate the completion of pre-filled requests as much as possible within this constraint, thereby reducing the initial token latency. This layer achieves this through the following mechanism: Dynamic time budget calculation: Based on the service level target of each request in the run queue and the real-time system status, the maximum allowed time for this batch processing is dynamically determined; Token scheduling under budget constraints: Within a given time budget, decoding tokens are prioritized to maintain output fluency, and the remaining budget is then allocated to pre-filled requests; Pre-fill concurrency control: Pre-fill requests exceeding the current budget are processed in chunks to achieve gradual filling and resource adaptation for large requests.

[0030] 2. Prefix Tree-Based Cache Planning and Scheduling Layer This layer is responsible for selecting requests from the waiting queue to add to the execution queue. Its core objective is to maximize the utilization and reuse rate of the key-value cache, thereby accommodating more concurrent requests within the limited GPU memory and indirectly reducing queuing latency and first token latency. The decision-making process considers the following factors: 1) Request arrival sequence; 2) Request length feature; 3) Prefix matching degree between requests.

[0031] This layer models the request selection problem as an optimization problem of maximizing throughput under the constraint of KV Cache capacity, and uses a multi-stage greedy algorithm based on prefix trees for efficient solution.

[0032] The following description, in conjunction with relevant accompanying drawings and specific examples, illustrates a large-scale model inference framework scheduler based on latency and cache awareness proposed in this invention.

[0033] Example 1: Please see Figure 1 This invention provides a large-scale model inference framework scheduler based on latency and cache awareness. Its core components include a cache planning and scheduling layer based on a prefix tree and a latency-aware batch processing resource allocation layer. The scheduler also includes an inference framework comprising a front-end module (AsyncLLM), a scheduling module (Scheduler), an execution module (Executor), and a memory management module (KV Cache Manager). Specifically, it includes the following: 1. System architecture and module functions User requests are initiated via HTTP or RPC, and follow the OPENAI standard by accessing the Fast API service of the front-end module (AsyncLLM). After being processed by the Tokenizer, the request is converted into a token sequence and transmitted to the prefix tree-based cache planning and scheduling layer via Socket cross-process communication.

[0034] The cache planning and scheduling layer accepts a token sequence via a socket and temporarily stores requests in a waiting queue. Based on a prefix tree, the layer periodically initiates a scheduling "round" (starting a new round when the run queue is empty), incorporating all requests in the current waiting queue into the current round's planning. This module performs optimization analysis based on prefix sharing information between requests and the system's KV cache capacity, generating an optimal request scheduling sequence (i.e., the cache planning queue) designed to maximize cache utilization. The run queue then pulls requests from the cache planning queue sequentially according to this sequence.

[0035] In each iteration, the queue is checked for each request: if the request is in the decoding phase and an end marker was output in the most recent computation... <eos>If the request is successful, it is marked as complete and removed from the system; otherwise, it is put back into the run queue. The run queue then checks whether it has reached its concurrency limit and whether there is remaining space in the KV cache. If both conditions are met, the next request is retrieved from the cache planning queue and added to the run queue.

[0036] The latency-aware batch resource allocation layer is triggered before each model execution. It dynamically selects a portion of tokens from the current running queue to form a computation batch based on the status of each request (pre-filling / decoding), the preset service level target (TTFT / TPOT), and real-time estimation based on hardware performance, and sends it to the execution module (Executor).

[0037] The Executor sends the batch data to the GPU, performs parallel computation with the model weights already loaded into GPU memory, generates a new KV Cache or the next output token, and returns the result to the Scheduler.

[0038] The scheduling module (Scheduler) evaluates the results; for those already generated... <eos>The request is processed via Socket and the front-end API, and the complete result is returned to the user.

[0039] 2. Specific Implementation of the Prefix Tree-Based Cache Planning and Scheduling Layer This module is one of the core components of this invention, designed to maximize the utilization of the KV Cache and the prefix reuse rate between user requests. Its workflow is as follows: Figure 2 As shown, the specific steps are as follows: (1) Prefix tree construction Retrieve all requests from the current waiting queue and temporarily store them in the cache planning queue. Construct one or more prefix trees based on the token sequence of all requests. Each leaf node in the tree represents a request, and the path from the root node to the leaf node is the token sequence of that request. If two leaf nodes share a part of the path, it indicates that the corresponding two requests have a common prefix. Each node pre-records the node ID sequence path_ids[i] from the root node to itself. A group of requests that share the same common prefix (i.e., have the same parent node) constitutes a "prefix group".

[0040] (2) Multi-stage sequence selection algorithm The core objective of this algorithm is to rearrange the request order in the traditional First-Come, First-Served (FCFS) model to maximize KV cache utilization and prefix reuse during parallel inference. The specific process is as follows: Phase 1: Prefix Reuse Optimization 1) Prefix group selection: For each request Ri, calculate the common prefix length (number of shared nodes) p_{i,j} of Ri and all other requests Rj (j≠i) in the queue, and sum them to obtain the "common benefit" B_i = Σ_{j≠i} p_{i,j} of the request. Select the request with the largest B_i value as the "seed" request R_seed and add it to the current round's scheduling set S. Add the path nodes path_ids[seed] of R_seed to the common cache set C, and update the used cache capacity used = Σ|len(C_i)|.

[0041] 2) Prefix group expansion: For each request Ri that has not yet been added to S, calculate its maximum sharing degree with the current public cache set C, s_i = max|path_ids[i] ∩ C_j|, and the additional cache cost required to add it, cost_i = |path_ids[i]| - s_i.

[0042] 2.1) If s_i is 0, it means that the current prefix group has been processed. Return to step 1 to find a new seed and start processing the next prefix group.

[0043] 2.2) If s_i is not 0, then select the request with the largest s_i, R_smax. If its cost cost_r_smax + used is less than the total system cache capacity, then add it to S, update C and used, and repeat this expansion step.

[0044] Phase Two: Cache Pool Population 3) For all remaining requests Ri that have not been added to S, calculate their net new cost cost_i = |path_ids[i] \ C| relative to the allocated cache set C.

[0045] 4) Sort all remaining requests in ascending order of cost_i (prioritize those with lower costs).

[0046] 5) Iterate through the sorted list. For each request Ri, if used + cost_i ≤ capacity, add it to S and update used = used + cost_i. Otherwise, skip the request.

[0047] 6) When the traversal is complete or the cache capacity is exhausted, the algorithm ends and outputs the optimal scheduling request sequence S for this round.

[0048] (3) Polling scheduling and starvation prevention mechanism The scheduler runs the planning algorithm in "rounds". At the beginning of each round, all requests in the current waiting queue are used as input for planning. Planning continues only after all requests scheduled to the run queue in this round have completed reasoning (generation). <eos>After reaching the maximum length, the scheduler starts a new round of planning and scheduling. This mechanism ensures that each request will be processed within its batch round regardless of the request length or prefix characteristics, effectively avoiding the starvation problem of long requests or requests with unpopular prefixes.

[0049] 3. Specific implementation of the latency-aware batch resource allocation layer This module is another core of the present invention, aiming to jointly optimize TPOT and TTFT. Its workflow is as Figure 3 shown: (1) Estimation of batch time budget 1) For each request Ri in the running queue, calculate its theoretical deadline: t_deadline_i = t_arrival + t_ttft_i + token_num_i * t_tpot_i. Where, t_arrival is the arrival time, t_ttft_i is the TTFT target, token_num_i is the number of generated tokens, and t_tpot_i is the TPOT target.

[0050] 2) Select the request with the most urgent deadline and use its remaining time as the global time budget for this batch: T_budget = min(t_deadline_i - t_current), where t_current is the current system time.

[0051] (2) Decoding and filling stage 3) Obtain all requests in the decoding stage and their current decoding positions.

[0052] 4) According to the pre-established decoding time model, estimate the time cost cost_decode_i required for each decoding request to generate the next token.

[0053] 5) For each decoding request Ri, if cost_decode_i < T_budget, add its next decoding token to this Batch and update the remaining budget T_budget = T_budget - cost_decode_i. If the cost exceeds the budget, skip this request.

[0054] 6) Process all decoding requests in a loop until all requests are processed or the budget is insufficient. After completion, the remaining budget T_budget is used for the pre-filling stage.

[0055] (3) Token selection in the pre-filling stage 7) Set the dedicated pre-filling budget T_prefill = T_budget.

[0056] 8) Retrieve all requests that are in the pre-population stage.

[0057] 9) Based on the pre-established prefill time model, estimate the time cost cost_prefill_i required for each prefill request to process all or part of its remaining tokens.

[0058] 10) For each pre-filled request Ri: 10.1) If cost_prefill_i ≤ T_prefill, then add all remaining prefill tokens to the Batch and update T_prefill = T_prefill - cost_prefill_i.

[0059] 10.2) If cost_prefill_i > T_prefill, then calculate the maximum consecutive token block that can be processed within the T_prefill budget according to the model (i.e., perform block division) and add the token block to the Batch.

[0060] 11) Process pre-filled requests in a loop until all requests have been processed or the budget is exhausted. The final output is a batch consisting of a mixture of decoded tokens and pre-filled token blocks.

[0061] (4) Pre-filling and decoding time modeling Batch processing resource allocation relies on accurate estimations of computation time at different stages within the Transformer architecture. The model is built upon FLOPs (floating-point operations) and the effective hardware computing power Π_eff, and considers the squared cost of the attention mechanism related to the context length. Key parameters (such as Π_eff^prefill, Π_eff^decode, and system constant overhead) are learned and adaptively adjusted online by collecting actual batch execution times during system runtime, ensuring the accuracy of time budget estimation and thus supporting fine-grained latency-aware scheduling.

[0062] The model will then be defined and analyzed in detail: Suppose the target model contains N_layer Transformer layers, the hidden layer dimension is d, the intermediate dimension of the feedforward network (FFN) is d_ff, and the current batch size is b. The effective computational throughput of the hardware during the prefill and decoding stages is denoted as Π_eff^prefill and Π_eff^decode, respectively. For a given token position (whose corresponding context length is denoted as L(position)), the following end-to-end single-token computation time model can be established: Cost_prefill(n,position)=N_layer*[t_attn_prefill(n,position)+t_ffn(n)]+other_overhead_prefill. Cost_decode(position)=N_layer*[t_attn_decode(position)+t_ffn_decode]+other_overhead_decode. First, for a single-layer Transformer, the main computational cost comes from the self-attention sublayer and the feedforward network sublayer. Other operations such as LayerNorm, residual connections, and Softmax have relatively low overhead and can be grouped into the constant overhead term `other_overhead`. Specifically, this includes: (1) Computational modeling of attention sublayer Employing a standard multi-head attention mechanism, the computation of a single layer mainly includes: Q / K / V linear projection, attention score calculation, weighted summation, and output linear projection. Its floating-point computation complexity can be approximated as: FLOPs_attn(n)≈(6*b*n*d^2+2*b*n^2*d+2*b*n^2*d+2*b*n*d^2) ≈8*b*n*d^2+4*b*n^2*d. Where n represents the number of tokens currently being processed.

[0063] 1) Decoding stage Each step generates only one new token (n=1), and attention needs to be calculated with all previously generated tokens (a total of L_decode(position) tokens). Therefore, the FLOPs of a single-layer attention layer in the decoding phase can be simplified as follows: FLOPs_attn_decode(position)≈8*b*d^2+4*b*L_decode(position)*d The corresponding time cost is: t_attn_decode(position)≈[8*b*d^2+4*b*L_decode(position)*d] / Π_eff^decode 2) Pre-filling stage The processing involves a contiguous chunk of input prompts. Assuming we process a chunk of tokens from position s to e (a total of n_chunk = e - s + 1 tokens), the total FLOPs of a single-layer attention on this chunk are the sum of the values ​​calculated at each position: FLOPs_attn_prefill^[s,e]≈∑_{ =s}^{e}(8*b*d^2+4*b* *d) ≈8*b*d^2*n_chunk+4*b*d*[e(e+1)-(s-1)s] / 2 The approximate computation time for the corresponding block-level attention is: t_attn_prefill^[s,e] ≈ FLOPs_attn_prefill^[s,e] / Π_eff^prefill. FFNs typically consist of two linear layers and an intermediate nonlinear activation, with a shape of d -> d_ff -> d. Ignoring the low-order costs of the activation functions, we only consider matrix multiplication FLOPs: (2) Computational modeling of feedforward network sublayers FFNs typically consist of two linear layers and an intermediate nonlinear activation function, with a dimensionality transformation of d -> d_ff -> d. Ignoring the low-order overhead of the activation function, we primarily consider the FLOPs of matrix multiplication: First-level linear transformation: d -> d_ff: FLOPs_ffn_1 ≈ 2 * b * n * d * d_ff. Second-level linear transformation: d_ff -> d: FLOPs_ffn_2 ≈ 2 * b * n * d_ff * d. Therefore, the FLOPs of a single-layer FFN are approximately: FLOPs_ffn(n) ≈ 4 * b * n * d * d_ff. In the pre-filling phase, n is the number of tokens currently being processed, and its time is: t_ffn(n) ≈ [4 * b * n * d * d_ff] / Π_eff^prefill. During the decoding phase, only one new token is generated per step (n = 1), and the time taken is: t_ffn_decode ≈ [4 * b * d * d_ff] / Π_eff^decode. (3) Adaptive learning of model parameters The key parameters in the aforementioned time model—effective computing power Π_eff^prefill and Π_eff^decode, as well as constant overhead terms other_overhead_prefill and other_overhead_decode—cannot be theoretically derived solely from the model structure parameters (N_layer, d, d_ff). They are influenced by multiple factors, including hardware computing power (such as GPU model and Tensor Core utilization), memory bandwidth, kernel implementation efficiency, and batch parallelism.

[0064] To ensure the accuracy of time budget estimation, this system employs an online parameter adaptive learning mechanism. After each batch inference, this mechanism collects the actual execution time and dynamically updates the estimated parameters using an online learning algorithm (such as an approximation method based on stochastic approximation). This ensures that the theoretical model continuously approximates the actual hardware performance under real load. This adaptive process ensures that the time budget upon which latency-aware scheduling relies has high reliability and environmental adaptability, laying a solid technical foundation for guaranteeing service quality goals under high dynamic loads.

[0065] 4. Example of a complete closed-loop scheduling process Combining the above modules, the scheduler of this invention forms a closed loop in its workflow: 1) Request access: After being processed by the front end, the request enters the scheduler's waiting queue.

[0066] 2) Cache planning: The cache planning module starts periodically to generate the optimal execution sequence for requests in the waiting queue.

[0067] 3) Request loading: The run queue loads requests in the optimal sequence and allocates / reuses KV Cache space.

[0068] 4) Batch build: Before each GPU computation, the batch allocation module dynamically builds a hybrid batch based on SLO.

[0069] 5) Execution and Feedback: The execution engine processes the batch, returning the new token and the updated KV Cache status. The completed request is removed and the result is returned.

[0070] 6) Looping and Polling: Requests in the run queue wait for the next batch processing. Once all requests in a round are completed, the system clears the run queue, waiting for new requests in the queue to participate in the next round of cache planning, thus starting a new scheduling cycle.

[0071] In summary, by optimizing request admission and order in the spatial dimension (video memory) through the cache planning layer, and optimizing micro-resource allocation in the time dimension (computation time slice) through the batch processing resource allocation layer, this invention achieves the optimal balance between system throughput and quality of service (TTFT / TPOT) under high concurrency and dynamic load.

[0072] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.< / eos> < / eos> < / eos>

Claims

1. A scheduler for a large-scale model inference framework based on latency and cache awareness, characterized in that, The scheduler includes a tightly cooperating batch resource allocation layer and a prefix tree-based cache planning and scheduling layer; The batch processing resource allocation layer is used to dynamically calculate the batch processing time budget based on the service level target requested in the run queue in each scheduling iteration, and under the constraint of the budget, select decoded tokens and pre-filled tokens from the requests in the run queue to form a mixed batch and send it into the model for calculation, so as to accelerate the completion of pre-filled requests while stabilizing the token generation rate. The prefix tree-based cache planning and scheduling layer is used to periodically select requests from the waiting queue and add them to the running queue. By constructing the requests in the waiting queue into a prefix tree and executing an optimization algorithm, a request scheduling sequence that maximizes the utilization of the key-value cache and the prefix reuse rate between requests is generated to improve the overall throughput of the system.

2. The scheduler according to claim 1, characterized in that, The batch processing resource allocation layer specifically performs the following operations: Dynamic budget calculation: Calculate the tightest time budget for this batch processing based on the arrival time of each request in the run queue, the initial token delay target, the time target per output token, and the current system time; Token selection under budget constraints: Within the tightest time budget, computing resources are allocated to requests in the decoding phase to generate their next token, and the remaining budget is then allocated to requests in the pre-population phase. Pre-fill concurrency control: When the computational cost of a pre-fill request exceeds the remaining budget, the pre-fill request is processed in blocks, and only the tokens that are completed within the budget are added to the current batch.

3. The scheduler according to claim 1, characterized in that, The prefix tree-based cache planning and scheduling layer specifically performs the following operations: Prefix tree construction: For all requests in the waiting queue, construct one or more prefix trees based on their input token sequences, where nodes in the tree represent prefix paths of the token sequences and leaf nodes represent requests; Multi-stage sequence selection: Execute a multi-stage greedy algorithm that includes prefix reuse optimization and cache pool filling stages. Under the constraint of key-value cache capacity, select a set of request sequences that maximizes the reuse of common cache nodes and minimizes the cost of adding new caches. Polling and starvation prevention: The multi-stage greedy algorithm is run in rounds. After each round of scheduling, the running queue is cleared before a new round of planning begins, ensuring that all requests are processed.

4. The scheduler according to claim 3, characterized in that, The multi-stage greedy algorithm includes: Phase 1, Prefix Selection and Expansion: Select the request with the greatest benefit from the common prefix with other requests as the seed, and add requests that share the prefix with the seed to the scheduling sequence to reuse the allocated cache nodes; Phase 2, Cache Pool Filling: For the remaining no-shared or low-shared prefix requests, add them to the scheduling sequence in ascending order of new cache cost until the cache capacity is exhausted.

5. The inference scheduler according to claim 2, characterized in that, The batch processing resource allocation layer relies on the single token computation time model of the pre-filling and decoding stages for budget estimation. The time model is established based on the computation characteristics of the Transformer architecture, and the model parameters are adaptively adjusted by collecting actual execution time online to match specific hardware and runtime environments.

6. The application of the scheduler as described in any one of claims 1 to 5, characterized in that, It is used to optimize request scheduling and resource allocation for different hardware card types, different user request types, and mainstream large language models based on the Transformer architecture.

Images