An iterative level preemptible inference request scheduling method based on delay prediction

Abstract

The application discloses an iterative level pre-emptible inference request scheduling method based on delay prediction and belongs to the technical field of computers. The method comprises two stages of offline performance model calibration and online inference request scheduling. In the offline stage, execution delay data is collected by constructing a synthetic workload, parameter fitting is performed on forward inference delay and key value cache exchange delay, and a calibrated performance model is obtained. In the online stage, a candidate scheduling mapping is generated from a candidate scheduling strategy set according to a system state in each scheduling iteration. The inference execution delay of each candidate scheduling mapping is predicted by using the calibrated performance model, the immediate and waiting delay costs are comprehensively calculated and weighted, the scheduling mapping with the minimum total cost is selected to execute inference, and the system state is updated. The application realizes high-frequency online scheduling with linear complexity through an analyzable performance model and a fixed candidate strategy set, is suitable for resource-limited scenarios, and aims to jointly optimize system responsiveness and generation process smoothness.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to an iterative preemptible inference request scheduling method based on latency prediction. Background Technology

[0002] With the rapid development of Large Language Model (LLM) inference services and the continuous expansion of their application scenarios, inference services are gradually being deployed from centralized data centers to resource-constrained distributed environments (such as the edge). In such environments, the available computing power and storage resources of the system are relatively limited, while user requests exhibit dynamic arrival and high concurrency characteristics, making the inference service system face more complex scheduling problems under resource-constrained conditions.

[0003] Unlike traditional neural network inference tasks, which typically have relatively stable execution latency, LLM inference is usually based on the Transformer architecture and uses an autoregressive generation mechanism to generate output token by token. Its inference process is characterized by iterative execution, uncertain generation length, and high computational overhead. The execution time of an inference request is not only related to the length of the input prompts but also closely related to the semantic complexity of the generated content, thus exhibiting significant uncertainty and volatility. This characteristic makes traditional scheduling methods based on the assumption of fixed execution time difficult to directly apply to LLM inference scenarios.

[0004] To improve the performance of LLM inference services, existing technologies mainly focus on model optimization, execution mechanism optimization, and scheduling strategy design. In terms of model optimization, methods such as low-bit quantization, structured pruning, and knowledge distillation are used to reduce model computational complexity and memory usage, thereby improving inference efficiency. Regarding execution mechanisms, techniques such as key-value cache (KV-cache) reuse, separation of prefill and decode phases, and dynamic batch processing are introduced to reduce redundant computations and improve resource utilization. However, these methods primarily optimize for single inference processes or execution efficiency, and in scenarios with multiple concurrent requests, it remains difficult to effectively guarantee the stability of response latency at the system level.

[0005] In terms of inference request scheduling, existing inference service systems typically employ scheduling strategies based on request arrival order, combined with dynamic batch processing mechanisms to improve throughput performance. In this type of mechanism, requests participate in batch execution at an iterative granularity during the generation process, thereby achieving a certain degree of resource reuse. However, such scheduling strategies often lack fine-grained differentiation of request execution characteristics, easily leading to head-of-line blocking problems in high-concurrency scenarios. This means that long-sequence requests consume significant computing resources, thus delaying the execution of shorter requests and impacting the overall system response performance.

[0006] To alleviate the aforementioned problems, some methods attempt to provide prior information for scheduling decisions by constructing predictive models of inference request generation length or execution time. However, such methods typically rely on large-scale predictive models to ensure prediction accuracy, resulting in high computational overhead and potentially introducing additional system burden in high-frequency scheduling scenarios. Furthermore, different models often require separate training of corresponding predictors, increasing training and deployment costs.

[0007] Furthermore, as inference frameworks supporting iterative execution and preemptive mechanisms develop, systems can interrupt and resume requests during the generation process, thereby improving scheduling flexibility. However, under such mechanisms, request preemption and resumption are usually accompanied by KV-cache swapping in and out operations. If the scheduling strategy does not effectively model the relevant overhead, it may lead to frequent resource swapping, resulting in response latency fluctuations or even performance degradation.

[0008] In summary, existing LLM inference request scheduling methods still have shortcomings in resource-constrained environments with multiple concurrent requests: on the one hand, scheduling strategies based on request arrival order cannot avoid head-of-line blocking problems; on the other hand, existing prediction methods have high computational overhead, making it difficult to meet the needs of high-frequency online scheduling; furthermore, in scenarios supporting preemptive execution, the lack of comprehensive consideration of execution costs and resource exchange overhead can easily affect the stability of system performance. Therefore, there is an urgent need for an inference request scheduling method that can effectively characterize the execution characteristics of inference requests and support efficient scheduling decisions while ensuring controllable computational overhead. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing LLM inference request scheduling methods in environments with multiple concurrent requests and limited resources. It provides an iterative preemptible inference request scheduling method based on latency prediction. Under the premise of ensuring real-time performance and controllable overhead of online scheduling, it jointly optimizes the user-perceived first token latency (TTFT) and time between tokens (TBT) through high-frequency online scheduling decisions, thereby achieving an effective trade-off between system responsiveness and the smoothness of the generation process.

[0010] To address the aforementioned technical problems, the present invention provides the following technical solution: an iterative preemptible inference request scheduling method based on latency prediction, comprising an offline performance model calibration stage and an online inference request scheduling stage, as follows:

[0011] The offline performance model calibration stage includes: collecting inference execution latency data under different configurations based on synthetic workloads, fitting the calibrated performance model, and using the performance model to predict forward inference latency and key-value cache exchange latency.

[0012] The online inference request scheduling phase is executed in iterations, with each scheduling iteration including:

[0013] Generate candidate scheduling maps: Based on the current system state, generate corresponding candidate scheduling maps using at least two preset scheduling strategies;

[0014] Evaluation and selection: The inference iteration execution time corresponding to each candidate scheduling map is predicted using the calibrated performance model, and the comprehensive scheduling cost is calculated based on the execution time. The candidate scheduling map with the minimum comprehensive scheduling cost is selected as the scheduling decision for this round.

[0015] Execution and Update: Based on the scheduling decision, the inference engine performs key-value cache management and forward inference, updates the system state, and continues to the next iteration.

[0016] Preferably, the synthetic workload includes modeling the inference engine resource specifications as resource vectors, enumerating feasible batch specification combinations that satisfy the resource specification constraints, and generating a request set with heterogeneous length for each batch specification.

[0017] The inference execution latency data includes pure pre-fill load latency, pure decoding load latency, mixed pre-fill and decoding load latency, and key-value cache exchange latency. Each configuration is executed multiple times and the median is taken to reduce the impact of system jitter.

[0018] The parameter sets of the forward inference latency model and the key-value cache exchange latency model in the calibrated performance model are fitted independently, and the resulting calibrated performance model does not participate in online updates during system operation.

[0019] Preferably, the forward inference latency model models the GPU internal memory access latency and computation latency separately, and uses a smooth approximation function to take the maximum value of the two to obtain the overall forward inference latency;

[0020] The key-value cache swapping latency model models the swapping out latency and swapping in latency as linear functions, and the total swapping latency is the sum of the swapping out latency and the swapping in latency.

[0021] Preferably, in the forward inference delay model:

[0022] The video memory access latency is obtained by weighting and summing the total number of key-value cache blocks to be written during the pre-filling and decoding stages, and the total number of historical key-value cache blocks to be read during the decoding stage, using read and write overhead coefficients.

[0023] The computational latency is based on the attention computational workload in the pre-filling stage, the attention computational workload in the decoding stage, and the computational workload of the linear layer. The total computational workload is obtained by weighted summation of unit computational costs, and then nonlinearly mapped to the total computational workload through a soft threshold function.

[0024] The overall forward inference latency is obtained by performing a smooth maximum operation on the memory access latency and computation latency; the soft threshold function is used to characterize the computational efficiency changes of the GPU under different load levels, and the smooth maximum operation uses a logarithmic summation exponential function to smoothly approximate the parallel execution behavior of memory access latency and computation latency.

[0025] Preferably, in the key-value cache swapping latency model: the swapping latency is obtained by adding a fixed swapping startup overhead to the product of the total number of key-value cache blocks to be swapped out and the unit swapping bandwidth overhead;

[0026] The swap-in latency is obtained by adding a fixed swap-in startup overhead to the product of the total number of key-value cache blocks to be swapped in and the unit swap-in bandwidth overhead.

[0027] Preferably, the preset scheduling strategy includes a survival priority scheduling strategy and a first-response priority scheduling strategy;

[0028] The survival-first scheduling strategy aims to maintain the continuous progress of existing requests. Under the premise of meeting three resource constraints—the upper limit of the number of concurrent requests of the inference engine, the upper limit of the capacity of the GPU side key value cache block, and the upper limit of the total number of tokens that can be processed in the pre-filling stage of a single iteration—requests are selected to participate in inference according to the queue order.

[0029] For requests that are not selected, perform key-value cache evicting or cleanup based on resource availability, and infer action based on the current location of the request's key-value cache:

[0030] If it resides in GPU memory, add it to the decoding set and keep its resident state unchanged; if it resides in CPU memory, add it to the decoding set and add it to the key-value cache swap-in set; if it does not exist, add it to the pre-filled set.

[0031] The first-response priority scheduling strategy aims to improve the initial response experience of requests. It divides requests in the scheduling queue into two sub-queues based on whether the first token has been generated. Under resource constraints, it first constructs a pre-filled scheduling map for the sub-queues that have not yet generated the first token. Then, under the remaining resource constraints, it calls the survival priority scheduling strategy to schedule the sub-queues that have generated the first token. The scheduling results of the two sub-queues are merged to form the candidate scheduling map for this round.

[0032] Preferably, the calculation of the comprehensive scheduling cost takes into account both the immediate delay cost of participating in inference requests and the waiting delay cost of waiting requests, which are obtained by weighted summation using preset weighting coefficients.

[0033] The instant delay cost corresponds to either the first token delay cost or the inter-token delay cost depending on whether the request has generated the first token.

[0034] For requests that have not yet generated the first token, the difference between the predicted completion time of this round of inference and the arrival time of the request is used as the delay cost for the first token.

[0035] For requests that have generated at least one token, the difference between the prediction completion time of this round of inference and the generation time of the previous token is used as the instantaneous token delay cost.

[0036] The instant average cost is obtained by averaging the instant first token delay cost and the instant inter-token delay cost of all requests and then weighting them.

[0037] The waiting delay cost is estimated based on the predicted execution time of the inference iteration in the idealized next round of execution scenario;

[0038] When constructing an idealized scenario, all requests in the current waiting set are assumed to be included in the next round of inference. Based on the key-value cache management decision of the current round, the required swap-in set for the next round is derived. The pre-filled set and decoding set for the next round are determined according to the status of each waiting request. The calibrated performance model is called to predict the execution time of the next round of inference iteration under this scenario.

[0039] For each waiting request, the predicted execution time of the current and next round of inference iterations is used to estimate the first token delay increment or inter-token delay increment when it gets the inference opportunity in the next round. The average and weighted average waiting cost is then obtained.

[0040] Preferably, the current system state includes the set of currently active requests and the state vector of each request. The state vector includes at least the length of the input prompt token, the cumulative number of generated tokens, the current location of the key-value cache, and the most recently generated token.

[0041] The advantages of this invention compared to the prior art are:

[0042] First, the performance model proposed in this invention is built based on an analytical expression, with a small number of parameters, low computational overhead in the calibration process, and no need for large-scale parameter training, which significantly reduces calibration and deployment costs while ensuring prediction accuracy.

[0043] Second, this invention compresses the original exponential-scale scheduling decision space into a fixed-size candidate strategy set, and uses a calibrated performance model to perform fast cost evaluation and strategy selection in the candidate set. The overall scheduling decision time complexity remains linear, which can meet the real-time requirements of high-frequency online scheduling on computing-limited inference nodes.

[0044] Third, the present invention considers both forward inference latency and KV-cache exchange latency in cost evaluation, which can effectively characterize the resource exchange overhead in iterative preemptible execution scenarios and avoid performance degradation caused by frequent preemption and recovery operations.

[0045] Fourth, the candidate scheduling strategy set designed in this invention covers two representative scheduling preferences: survival priority and first-response priority. It can adaptively select the optimal scheduling mode under different request concurrency and load scenarios, taking into account both system responsiveness and the smoothness of the generation process.

[0046] Fifth, this invention uses preset weighting coefficients to weight the two performance indicators, TTFT and TBT, and supports flexible adjustment and optimization of the emphasis according to the service quality requirements of specific application scenarios, which has strong applicability and scalability. Attached Figure Description

[0047] Figure 1 This is a flowchart of the present invention;

[0048] Figure 2 This is the system architecture diagram of the present invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings.

[0050] Combined with appendix Figure 1 As shown, an iterative preemptible inference request scheduling method based on latency prediction is presented. The method includes an offline performance model calibration stage and an online inference request scheduling stage. The specific steps are as follows:

[0051] Step S101, Construct synthetic workload: Model the inference engine resource specifications as vectors, enumerate feasible batch specification combinations that satisfy resource specification constraints, and generate a request set with heterogeneous length for each batch specification.

[0052] Step S102, collect execution latency data: Drive the inference service to run with the synthetic workload constructed in step S101, collect pure pre-filling, pure decoding, mixed load and KV-cache exchange latency data under different load configurations, repeat the execution multiple times for each configuration and take the median to reduce the impact of system jitter;

[0053] Step S103, calibrate performance model parameters: Based on the latency data collected in step S102, the parameters of the forward inference latency model and the KV-cache swap latency model are fitted respectively to obtain the calibrated performance model; the forward inference latency model models the GPU internal memory access latency and computation latency respectively, and uses the Log-Sum-Exp function to smooth and approximate the maximum value to obtain the overall forward inference latency; the KV-cache swap latency model models the swap-out and swap-in latency respectively with linear functions; the parameter sets of the two models are fitted independently, and the obtained calibrated performance model does not participate in online updates during system operation;

[0054] Step S201, obtain the current system state: at the beginning of each round of scheduling iteration, obtain the current active request set and the state vector of each request. The active requests are arranged in order of arrival time to form a scheduling queue.

[0055] Step S202, Generate candidate scheduling maps: Based on the current system state, corresponding candidate scheduling maps are generated by the Continuation-Priority Scheduling Strategy (CPSS) and the First-Response-Priority Scheduling Strategy (FRPSS), respectively. CPSS aims to maintain the continuous progress of existing requests. Under the premise of satisfying resource constraints, it selects requests to participate in inference according to the queue order. For requests that are not selected, it performs KV-cache swapping or cleanup according to the resource situation. FRPSS is guided by improving the initial response experience of requests. It prioritizes constructing pre-filled scheduling maps for requests that have not yet generated their first token, and then calls CPSS to schedule requests that have generated their first token.

[0056] Step S203, evaluate scheduling cost: For each candidate scheduling map, use the calibrated performance model to predict the forward inference latency and KV-cache exchange latency of this round, and sum them to obtain the predicted execution time of this round of inference iteration; on this basis, calculate the comprehensive scheduling cost, and simultaneously consider the instant TTFT and TBT costs of participating in inference requests and the waiting latency cost of waiting requests, and obtain the total cost of the candidate scheduling map by weighting it with preset weight coefficients;

[0057] Step S204, Select the minimum cost scheduling map: Compare the total cost of each candidate scheduling map, and select the candidate scheduling map with the minimum total cost as the scheduling decision for this round;

[0058] Step S301, perform inference iteration: based on the scheduling decision, drive the inference engine to sequentially perform KV-cache swap-out, swap-in and clear operations, and then perform pre-filling and decoding forward inference on the selected request;

[0059] Step S302, update system status: update the cumulative number of tokens generated for each request, the most recent generation timestamp, and the KV-cache location. After removing completed requests, return to step S201 to continue the next iteration until the inference service ends.

[0060] In specific implementation of the present invention,

[0061] The overall process is divided into two phases: the offline performance model calibration phase and the online inference request scheduling phase. We will first introduce the offline performance model calibration phase:

[0062] Step S101: Construct the synthetic workload. Enumerate feasible batch specification combinations that satisfy the inference engine resource specification constraints, wherein the inference engine resource specifications are modeled as vectors. ,in and These represent the total number of KV-cache memory blocks available to the inference engine on the GPU and CPU, respectively. This indicates the maximum number of tokens that can be stored in a single KV-cache memory block. This indicates the maximum number of KV-cache memory blocks that a single inference request can occupy. This represents the maximum number of requests allowed to participate in the computation simultaneously during a single inference iteration. This represents the maximum number of tokens that can be processed during the pre-filling phase in a single inference iteration. When enumerating feasible batch specifications, it is necessary to cover combinations of different batch sizes and the maximum number of tokens that each request in the batch can occupy. For each batch specification, a request set with heterogeneous lengths is constructed by randomly generating token lengths, and multiple request sets are repeatedly generated for the same batch specification to enrich the sample distribution.

[0063] Step S102: Collect execution latency data. The inference service is driven by various synthetic workloads constructed in step S101. Execution latency data is collected under different load configurations, including pure pre-filling load latency, pure decoding load latency, mixed pre-filling and decoding load latency, and KV-cache exchange latency. This process is repeated multiple times for each workload configuration, and the median of each latency is taken to reduce the impact of system jitter and occasional factors, thereby improving the stability of the calibration data.

[0064] Step S103: Calibrate the performance model parameters. Based on the latency data collected in step S102, perform parameter fitting on the forward inference latency model and the KV-cache exchange latency model, respectively.

[0065] For the forward inference delay model, in the... In the round of iteration, let and These represent the sets of requests for performing pre-padding and decoding operations, respectively. Indicates a request In the The total number of tokens in the round of iterations, of which The length of the input prompt token. This represents the number of output tokens generated so far.

[0066] The GPU's internal memory access latency is modeled as follows. The total number of KV-cache blocks that each request needs to write during the decoding phase is:

[0067]

[0068] The total number of KV-cache blocks to be written during the pre-filling phase is:

[0069]

[0070] The total number of historical KV-cache blocks that need to be read during the decoding phase is:

[0071]

[0072] Based on the above read / write behavior, the GPU's internal memory access latency is modeled as follows:

[0073]

[0074] in This represents the cost of reading the model weights. This represents the unit overhead of writing a single KV-cache block. This represents the unit cost of reading a single KV-cache block.

[0075] In terms of computational overhead, the attention computation workload in the pre-filling stage is:

[0076]

[0077] The computational workload for attention during the decoding phase is:

[0078]

[0079] The computational workload for the linear layer is:

[0080]

[0081] Including attention and linear layer computation, the overall computational workload is:

[0082]

[0083] in , and These represent the unit computation costs for pre-filled attention, decoding attention, and linear layers, respectively. The fixed overhead related to batch size is modeled as follows:

[0084]

[0085] in This refers to the total batch size for this round. To fix startup overhead, This represents the average scheduling overhead coefficient for a single request. A soft threshold function is used to model computational latency to characterize the variation in GPU computational efficiency under different load levels.

[0086]

[0087] in For GPU computing efficiency parameters, This represents the saturation point for GPU computational workload. Since memory access and computation are executed in parallel via a pipeline, the Log-Sum-Exp function is used to smoothly approximate the operation that takes the maximum of the two, resulting in the overall forward inference latency:

[0088]

[0089] in Used to control the smoothness of the approximation function.

[0090] For the KV-cache switching latency model, let and They represent the first Given the set of requests that need to be swapped out and swapped in during each iteration, the total number of KV-cache blocks that need to be swapped out and swapped in are respectively:

[0091]

[0092]

[0093] The swap-out and swap-in delays are modeled as follows:

[0094]

[0095]

[0096] in and This refers to the bandwidth overhead for swapping out and swapping in per unit of KV-cache block. and This is to account for the fixed startup overhead when triggering swap-out and swap-in operations. Since KV-cache swapping operations are performed serially before each round of forward inference, the... The total exchange delay for each round of iterations is:

[0097]

[0098] During parameter calibration, the loss function adopts the L1 norm form, and the prediction deviation is measured by the absolute error between the predicted value and the actual measured value. The parameter sets of the forward inference delay model and the KV-cache exchange delay model are independently optimized and fitted, and iteratively updated using the gradient descent method to obtain a calibrated performance model that can be used for online prediction. This performance model does not participate in online updates during system operation; it serves only as a delay prediction tool in scheduling decisions.

[0099] After completing the offline performance model calibration, the online inference request scheduling phase begins. During this phase, inference requests continue to arrive, and the scheduling system uses iterations as the basic unit of progress. In each scheduling iteration, the following steps are executed sequentially:

[0100] Step S201: Obtain the current system state. In each scheduling iteration... At the beginning, obtain the current active request set. System timestamp and the state vectors of each request ,in For the request Maximum allowed token length (therefore) (corresponding to its maximum output token length). For the request The time of arrival at the system The timestamp of the most recent generation of the token for this request. The current location of its KV-cache ( This indicates that pre-filling has not been performed or the KV-cache has been cleared and therefore does not exist. and These respectively indicate whether the KV-cache is located in GPU video memory or CPU memory. The token most recently generated for this request (initialized to empty when the request first enters the system, and the token in the word segmentation table contains the terminator token). ,when This indicates that the request generation process has been completed. Active requests are arranged in a scheduling queue according to their arrival time. .

[0101] Step S202: Generate candidate scheduling maps. Based on the current system state, candidate scheduling maps are generated sequentially from a predefined set of candidate scheduling strategies. The set of candidate scheduling strategies includes CPSS and FRPSS, and corresponding candidate scheduling maps are generated for each. Each candidate scheduling map determines the set of requests participating in the pre-filling process for this round. The set of requests participating in decoding The set of requests to wait and the corresponding KV-cache swap-in set Exchange set and clear set .

[0102] CPSS's core objective is to maintain the continuous progress of existing requests, based on the active request queue. The system iterates through the requests in sequence, determining the set of requests to participate in inference while satisfying three types of resource constraints: the maximum number of concurrent requests the inference engine can handle, and the maximum number of concurrent requests the inference engine can handle. GPU-side KV-cache block capacity limit And the maximum number of tokens that can be processed in the pre-filling phase of a single iteration. For each request to be inspected Calculate the number of KV-cache blocks it occupies after this round of inference. ,like The number of tokens that this pre-filling request needs to process in this round is: Otherwise If the current resource counter satisfies , and If all three conditions are met, the request will be included in the inference set, and the resource counter will be updated. , and Otherwise, continue scheduling subsequent requests to participate in this round of inference. For requests that pass the feasibility check, their KV-cache locations are used to determine their status. Divide the reasoning actions: If Then it is directly added to the decoding set. And add KV-cache to maintain the collection ;like Then, the KV-cache swap-in set is added simultaneously. With decoding set ;like Then add to the pre-populated set The remaining requests that were not selected to participate in the reasoning process will be added to the waiting set. The system then determines whether to perform swapping or cleanup operations based on the remaining space in the GPU and CPU-side KV-cache: if the requested KV-cache resides on the GPU side and there is insufficient space on the GPU side, it attempts to swap it out to the CPU side; if there is also insufficient space on the CPU side, it adds the KV-cache to the cleanup set. If a request is made to reside on the CPU side but the CPU side capacity is insufficient, it will also be added to the cleanup set. .

[0103] FRPSS prioritizes improving the initial request response experience during scheduling, starting with the scheduling queue. Perform a single traversal, and divide the requests into two sub-queues based on whether the first token has already been generated: if... This indicates that the request has not yet completed the initial token generation; therefore, add it to the sub-queue. Otherwise, add it to the subqueue. Subsequently, under resource constraints, Requests in the process are prioritized for constructing pre-filled scheduling maps. The concurrency constraints, GPU-side KV-cache capacity constraints, and pre-filled token quantity constraints are checked sequentially. Requests that meet the constraints are added to the pre-filled set. It also updates the resource counters, and stops if any constraint is violated; All remaining unselected requests will be added to the waiting set. Complete the task. After processing, the subqueues are processed under the current remaining resource constraints. The CPSS is invoked to make scheduling decisions, and the scheduling results of the two types of sub-queues are finally merged to form a complete candidate scheduling map for this round.

[0104] Step S203: Evaluate the cost based on the performance model. For each candidate scheduling map generated in step S202, predict the inference execution latency using the calibrated performance model obtained in step S103, and calculate the comprehensive scheduling cost.

[0105] Specifically, firstly, based on the composition of each request set in the candidate scheduling map, the forward inference latency for this round is predicted using the calibrated performance model. and KV-cache exchange latency Summing the two values ​​yields the predicted execution time for this round of inference iteration. .

[0106] Then, the request to participate in the reasoning in this round (i.e., from) and Calculate the immediate latency cost for requests that have not yet generated their first token: (Right now (The predicted completion time of this round of reasoning) With the time of request arrival The difference serves as the immediate TTFT cost; for requests that have generated at least one token. (Right now The prediction completion time of this round of reasoning is compared with the generation time of the previous token. The difference is taken as the instantaneous TBT cost. Then, the instantaneous TTFT and TBT costs of all requests are averaged separately, and then weighted according to preset coefficients. The real-time average cost of this round is obtained by weighting the real-time TTFT cost average with the real-time TBT cost average.

[0107] Next, an idealized next-round execution scenario is constructed to calculate the waiting latency cost of the waiting request set: Ignoring GPU resource capacity constraints, the current waiting set is... All requests in the current waiting set are included in the next round of inference. Based on the KV-cache management decision of this round, the required swap-in set for the next round is derived. The pre-filled set and decoding set for the next round are determined according to the status of each waiting request. The calibrated performance model is used to predict the execution time of the next inference iteration under this ideal scenario. For each request in the current waiting set, its TTFT or TBT increment when it gains an inference opportunity in the next round is estimated based on its status using the predicted execution time values ​​of this and the next round of inference iterations. The TTFT and TBT costs of all waiting requests are averaged separately and then weighted. The weighted average cost is obtained by waiting.

[0108] Finally, the immediate average cost is added to the waiting average cost to obtain the total cost of the current candidate scheduling map. This is achieved by adjusting the weighting coefficients. The optimization preference for TTFT and TBT can be flexibly adjusted according to the different emphases on service quality in specific application scenarios.

[0109] Step S204: Select the minimum cost scheduling map. Compare the total costs corresponding to the two candidate scheduling maps, CPSS and FRPSS, and select the candidate scheduling map with the minimum total cost as the scheduling decision for this round. Since the number of candidate scheduling strategies is constant, and the generation complexity and cost evaluation complexity of each candidate strategy are linearly related to the number of current active requests, the overall scheduling decision time complexity remains linear, which can meet the real-time requirements of high-frequency online scheduling.

[0110] After completing the scheduling decisions in steps S201 to S204, the system drives the inference engine according to the selected scheduling map, enters the inference execution phase, actually executes the scheduling results of this round, and updates the system state. Specifically, this includes the following steps:

[0111] Step S301: Perform inference iteration. Based on the scheduling decision determined in step S204, the inference engine first sequentially performs KV-cache swap-out and swap-in operations, i.e., for... The request in the middle swaps its KV-cache from GPU memory to CPU memory. The request in the middle swaps its KV-cache from CPU memory to GPU memory. The request in the middle clears its KV-cache; after completing the KV-cache management operation, it... and The token is sent to the inference engine for forward inference, which involves pre-filling the token and decoding the token. Each request involved in the inference generates an output token.

[0112] Step S302, Update System State. After completing inference in step S301, update the state of each request in the system. This includes the cumulative number of tokens generated for each request. Increment by 1 after participating in pre-filling or decoding, otherwise remain unchanged; timestamp of the most recent token generation. Update to the current system time after inference; otherwise, leave it unchanged; KV-cache resident location. Updated based on scheduling decisions. If the token generated for a request is a terminator... It may have reached the maximum token length limit. If the request is not found, it is removed from the active request set. Then, it is determined whether the system should continue running. If so, the next scheduling iteration begins, and the process returns to step S201; otherwise, the scheduling and inference service ends.

[0113] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0114] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. An iterative preemptible inference request scheduling method based on latency prediction, characterized in that: The process includes an offline performance model calibration phase and an online inference request scheduling phase, with the following steps: The offline performance model calibration stage includes: collecting inference execution latency data under different configurations based on synthetic workloads, fitting the calibrated performance model, and using the performance model to predict forward inference latency and key-value cache exchange latency. The online inference request scheduling phase is executed in iterations, with each scheduling iteration including: Generate candidate scheduling maps: Based on the current system state, generate corresponding candidate scheduling maps using at least two preset scheduling strategies; Evaluation and selection: The inference iteration execution time corresponding to each candidate scheduling map is predicted using the calibrated performance model, and the comprehensive scheduling cost is calculated based on the execution time. The candidate scheduling map with the minimum comprehensive scheduling cost is selected as the scheduling decision for this round. Execution and Update: Based on the scheduling decision, the inference engine performs key-value cache management and forward inference, updates the system state, and continues to the next iteration.

2. The iterative preemptible inference request scheduling method based on latency prediction according to claim 1, characterized in that: The synthetic workload includes modeling inference engine resource specifications as resource vectors, enumerating feasible batch specification combinations that satisfy resource specification constraints, and generating a request set with heterogeneous length for each batch specification. The inference execution latency data includes pure pre-fill load latency, pure decoding load latency, mixed pre-fill and decoding load latency, and key-value cache exchange latency. Each configuration is executed multiple times and the median is taken to reduce the impact of system jitter. The parameter sets of the forward inference latency model and the key-value cache exchange latency model in the calibrated performance model are fitted independently, and the resulting calibrated performance model does not participate in online updates during system operation.

3. The iterative preemptible inference request scheduling method based on latency prediction according to claim 2, characterized in that: The forward inference latency model models the GPU internal memory access latency and computation latency separately, and uses a smooth approximation function to take the maximum value of the two to obtain the overall forward inference latency; The key-value cache swapping latency model models the swapping out latency and swapping in latency as linear functions, and the total swapping latency is the sum of the swapping out latency and the swapping in latency.

4. The iterative preemptible inference request scheduling method based on latency prediction according to claim 3, characterized in that: In the aforementioned forward inference delay model: The video memory access latency is obtained by weighting and summing the total number of key-value cache blocks to be written during the pre-filling and decoding stages, and the total number of historical key-value cache blocks to be read during the decoding stage, using read and write overhead coefficients. The computational latency is based on the attention computational workload in the pre-filling stage, the attention computational workload in the decoding stage, and the computational workload of the linear layer. The total computational workload is obtained by weighted summation of unit computational costs, and then nonlinearly mapped to the total computational workload through a soft threshold function. The overall forward inference latency is obtained by performing a smooth maximum operation on the memory access latency and computation latency; the soft threshold function is used to characterize the computational efficiency changes of the GPU under different load levels, and the smooth maximum operation uses a logarithmic summation exponential function to smoothly approximate the parallel execution behavior of memory access latency and computation latency.

5. The iterative preemptible inference request scheduling method based on latency prediction according to claim 3, characterized in that: In the key-value cache swapping latency model, the swapping latency is obtained by adding a fixed swapping startup overhead to the product of the total number of key-value cache blocks to be swapped out and the unit swapping bandwidth overhead. The swap-in latency is obtained by adding a fixed swap-in startup overhead to the product of the total number of key-value cache blocks to be swapped in and the unit swap-in bandwidth overhead.

6. The iterative preemptible inference request scheduling method based on latency prediction according to claim 1, characterized in that: The preset scheduling strategies include a survival priority scheduling strategy and a first-response priority scheduling strategy. The survival-first scheduling strategy aims to maintain the continuous progress of existing requests. Under the premise of meeting three resource constraints—the upper limit of the number of concurrent requests of the inference engine, the upper limit of the capacity of the GPU side key value cache block, and the upper limit of the total number of tokens that can be processed in the pre-filling stage of a single iteration—requests are selected to participate in inference according to the queue order. For requests that are not selected, perform key-value cache evicting or cleanup based on resource availability, and infer action based on the current location of the request's key-value cache: If it resides in GPU memory, add it to the decoding set and keep its resident state unchanged; if it resides in CPU memory, add it to the decoding set and add it to the key-value cache swap-in set; if it does not exist, add it to the pre-filled set. The first-response priority scheduling strategy aims to improve the initial response experience of requests. It divides requests in the scheduling queue into two sub-queues based on whether the first token has been generated. Under resource constraints, it first constructs a pre-filled scheduling map for the sub-queues that have not yet generated the first token. Then, under the remaining resource constraints, it calls the survival priority scheduling strategy to schedule the sub-queues that have generated the first token. The scheduling results of the two sub-queues are merged to form the candidate scheduling map for this round.

7. The iterative preemptible inference request scheduling method based on latency prediction according to claim 1, characterized in that: The calculation of the comprehensive scheduling cost takes into account both the immediate delay cost of participating in inference requests and the waiting delay cost of waiting requests, which are obtained by weighted summation using preset weighting coefficients. The instant delay cost corresponds to either the first token delay cost or the inter-token delay cost depending on whether the request has generated the first token. For requests that have not yet generated the first token, the difference between the predicted completion time of this round of inference and the arrival time of the request is used as the delay cost for the first token. For requests that have generated at least one token, the difference between the prediction completion time of this round of inference and the generation time of the previous token is used as the instantaneous token delay cost. The instant average cost is obtained by averaging the instant first token delay cost and the instant inter-token delay cost of all requests and then weighting them. The waiting delay cost is estimated based on the predicted execution time of the inference iteration in the idealized next round of execution scenario; When constructing an idealized scenario, all requests in the current waiting set are assumed to be included in the next round of inference. Based on the key-value cache management decision of the current round, the required swap-in set for the next round is derived. The pre-filled set and decoding set for the next round are determined according to the status of each waiting request. The calibrated performance model is called to predict the execution time of the next round of inference iteration under this scenario. For each waiting request, the predicted execution time of the current and next round of inference iterations is used to estimate the first token delay increment or inter-token delay increment when it gets the inference opportunity in the next round. The average and weighted average waiting cost is then obtained.

8. The iterative preemptible inference request scheduling method based on latency prediction according to claim 1, characterized in that: The current system state includes the set of currently active requests and the state vector of each request. The state vector includes at least the length of the input prompt token, the cumulative number of generated tokens, the current location of the key-value cache, and the most recently generated token.

Images