LLM inference task adaptive scheduling method based on scheduling drift index

Abstract

This invention discloses an adaptive scheduling method for LLM inference tasks based on a scheduling drift metric, belonging to the field of task scheduling technology. It includes calculating the effective relaxation time for each request in a candidate request set R, then determining the number of additional tokens generated after each request completes its task, and using the token count to calculate the scheduling value of the request. The method then selects the maximum scheduling value in R and the minimum scheduling value in the current execution batch set to calculate the scheduling drift metric. It determines whether the scheduling drift metric exceeds a preset drift threshold. If so, it selects the request with the highest scheduling value in R and adds it to the cleared execution batch set, returning to step one if R is not empty. Otherwise, it keeps the execution batch set unchanged, selects requests from R based on the remaining memory to update the execution batch set, and returns to step one if R is not empty. This scheduling method avoids frequent scheduling while ensuring the smoothness of the generation process.

Description

Technical Field

[0001] This invention relates to the field of task scheduling technology, and specifically to an adaptive scheduling method for LLM inference tasks based on scheduling drift metrics. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in natural language processing, code generation, information retrieval, and multimodal generation, the resource bottleneck in the inference phase has become increasingly prominent. Model inference is typically divided into a Prefill phase and a Decode phase. The former is a one-time computation process, mainly limited by computing power and bandwidth; the latter is a cyclic generation process, where memory consumption increases linearly with the context length. Especially in multi-user concurrent inference scenarios, the KV cache in the Decode phase continuously accumulates, occupying memory resources, causing new requests to fail to enter the computation queue in a timely manner, resulting in increased response latency and decreased throughput.

[0003] Currently, mainstream inference service frameworks (such as DeepSpeed-Inference and vLLM) typically employ fixed request scheduling strategies, such as First-Come, First-Served (FCFS) or Shortest Job First (SJF). These schemes treat user-generated requests as independent tasks during inference, allocating them sequentially to available GPU resources for execution in a fixed order. Their goal is usually to simplify scheduling logic and ensure the predictability of service responses.

[0004] In multi-user concurrent scenarios, the generation lengths of different tasks vary significantly. If FCFS or SJF scheduling is used, the following problems will occur: FCFS cannot reflect the urgency of tasks, which can easily cause short tasks to wait for a long time; although SJF can shorten the average response time, it can cause long tasks to starve or be frequently interrupted. Therefore, traditional scheduling schemes based on fixed rules cannot simultaneously take into account fairness and memory utilization in large model inference scenarios.

[0005] In recent years, some research has begun to explore the introduction of adaptive batch scheduling mechanisms in large model inference to improve inference efficiency and response performance under conditions of limited GPU memory. Systems such as Apt-Serve dynamically evaluate the GPU memory usage and time gain of each request during the inference execution phase, prioritizing requests with higher "value per unit of GPU memory" for batch execution, and deciding on caching strategies to balance the relationship between time and capacity, thereby improving overall GPU memory utilization and shortening task waiting time.

[0006] However, the Apt-Serve scheme primarily focuses on optimizing memory usage and average response time, failing to effectively consider the temporal smoothness and output continuity of the generation process, making it difficult to guarantee a consistent user experience. Furthermore, this method requires re-evaluating and batch rearranging all requests in each inference iteration, resulting in frequent scheduling, high overhead, and a tendency to cause synchronization blocking and performance fluctuations in a single-GPU environment. In addition, its priority determination is mainly based on memory gain estimation, lacking dynamic awareness of request urgency and generation real-time requirements, making it difficult to achieve a stable and smooth scheduling process in multi-request contention scenarios.

[0007] Meanwhile, existing research has proposed the Fluidity Index (FI) to measure the smoothness and continuity of token output during generation. This index reflects the user's subjective perception of quality during interactive generation. However, in real-world deployment scenarios with limited single-GPU resources, how to simultaneously consider the dynamic balance between memory usage, request timing urgency, and FI gain, and design an adaptively scheduled inference mechanism, remains a key unsolved problem in the field. Summary of the Invention

[0008] To address the aforementioned shortcomings in existing technologies, the adaptive scheduling method for LLM inference tasks based on scheduling drift metrics provided by this invention solves the problems of frequent scheduling and insufficient smoothness in the generation process in existing large model inference tasks.

[0009] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: An adaptive scheduling method for LLM inference tasks based on scheduling drift metrics is provided, comprising the following steps: S1. Iterate through each request in the candidate request set and calculate the corresponding effective relaxation time based on the deadline for each request. S2. Based on the effective relaxation time, determine the number of additional tokens generated for each request after the task is completed, and use the number of tokens to calculate the scheduling value of the request; S3. Select the maximum scheduling value from the candidate request set, and select the minimum scheduling value based on the scheduling values ​​of all requests in the current execution batch set; here, the scheduling value is the request value, and selecting the maximum scheduling value from the candidate request set actually selects the request with the maximum scheduling value and the request with the minimum scheduling value. S4. Calculate the scheduling drift index based on the maximum and minimum scheduling values, and determine whether the scheduling drift index is greater than the preset drift threshold. If yes, proceed to step S5; otherwise, proceed to step S6. S5. Based on the total video memory, a greedy strategy is used to select the requests with the highest scheduling value in the candidate request set and add them to the cleared execution batch set. When the candidate request set is not empty, return to step S1. S6. Keep the execution batch set unchanged, determine the current remaining video memory, select requests from the candidate request set based on the remaining video memory to update the execution batch set, and return to step S1 if the candidate request set is not empty.

[0010] The beneficial effects of this invention are as follows: The core of this invention lies in constructing an adaptive scheduling closed loop that perceives urgency, memory usage, and smoothness by combining relaxation time management, scheduling value quantification, and dynamic batch processing strategies. Specifically, this method first quantifies the time leeway of each request through effective relaxation time (S1), and then predicts its potential to complete ahead of time and contribute additional smoothness (tokens), which is then converted into standardized scheduling value (S2). The innovative scheduling drift metrics (S3-S4) act as "sensors" of the system state, intelligently deciding whether to perform global batch reconstruction (S5) to pursue better overall value or to only perform incremental filling (S6) to maintain scheduling stability by monitoring the severity of the value difference between the candidate request set and the execution batch set. This design enables the system to significantly reduce the computational overhead and performance jitter caused by frequent and indiscriminate scheduling while ensuring smooth generation and real-time response. Ultimately, within limited GPU memory resources, it achieves multi-objective collaborative optimization of system throughput, user experience, and resource utilization.

[0011] Furthermore, the initial time of each request relaxation time for: in, For the request The deadline For the request The estimated execution time; After each generation iteration, the relaxation time update expression is: in, and Let be the relaxation times for the th and th-1th iterations of request i, respectively; For the request The time when the token is generated in the (th-1)th iteration; The preset iteration execution time; The effective relaxation time for the i-th iteration is: in, The maximum relaxation time preset for the system.

[0012] The beneficial effects of the above technical solution are as follows: By introducing an effective relaxation time calculation model, this solution can accurately quantify the urgency of request processing and maintain the temporal continuity of scheduling decisions. This model updates the relaxation time recursively and, combined with a configurable upper limit for relaxation time, ensures both the accuracy of time assessment and avoids excessive impact of long-term requests on current scheduling. The effective relaxation time, as a core parameter, provides a direct basis for subsequently judging the potential for additional request generation and calculating scheduling value, supporting the stable execution of the entire adaptive scheduling strategy.

[0013] Furthermore, step S2 further includes: S21. Calculate the ideal completion time window for the request based on the effective relaxation time: , in, For the i-th request in the candidate request set The deadline; for Effective relaxation time; For the request The ideal completion time window; The parameter is a multiple. The base multiple parameter; For traversal requests The amount of video memory already used; Total video memory; S22. Determine the ideal completion time window Is it greater than or equal to the request? The estimated completion time, and if so, the number of additional tokens generated. If it is k-1, otherwise, the number of additional tokens generated. ; S23, Based on the number of additional tokens generated Calculate the scheduling value of the request: in, For the request The scheduling value; This is the preset iteration execution time.

[0014] The beneficial effects of the above technical solution are as follows: By designing a dynamic ideal completion time window that incorporates the proportion of video memory usage, this solution can accurately assess the additional generation potential of each request under resource constraints, and then quantify it into a directly comparable scheduling value. When calculating the ideal completion time window, a dynamic adjustment factor k based on the current video memory usage rate is introduced, linking the time window estimate to the real-time system load, thus improving the accuracy and adaptability of the assessment. By judging the relative relationship between the time window and the expected completion time, requests that can be completed ahead of time and thus contribute additional smoothness to the system can be intelligently identified. Finally, the number of additional generated tokens is converted into scheduling value per unit time, providing a unified and objective quantitative basis for subsequent batch construction and scheduling drift decisions.

[0015] Furthermore, the expression for calculating the scheduling drift index is as follows: in, For scheduling drift metrics; To maximize scheduling value; This represents the minimum scheduling value.

[0016] The beneficial effects of the above technical solution are as follows: By introducing a normalized scheduling drift metric, this solution can keenly perceive the value difference between candidate requests and executed batches, providing a clear and quantitative decision-making basis for whether to trigger global batch reconfiguration. The expression presents the value difference in a relative proportion, avoiding the scale sensitivity problem caused by absolute values ​​and enhancing the consistency of the system's judgment criteria under different load conditions. This metric effectively balances scheduling stability and resource optimization efficiency, and is a key control parameter for achieving adaptive scheduling.

[0017] Furthermore, the method for determining the execution batch set in step S5 includes: S51. For requests in the candidate request set, sort them in descending order according to their scheduling value; S52. Select requests from the candidate request set and determine... If the condition is met, proceed to step S53; otherwise, output the execution batch set. The currently used video memory, The video memory used for the i-th request; Total video memory; S53. Add the current request to the execution batch set and update... ,make Then return to step S52.

[0018] The beneficial effects of the above technical solution are as follows: This solution, through a greedy selection strategy based on memory constraints, can quickly construct the execution set with the highest total scheduling value when batch reconstruction is triggered, thereby optimizing resource allocation and improving system throughput efficiency. This solution selects requests in descending order of scheduling value and dynamically accumulates memory usage to ensure that as many high-value requests as possible are included within the total memory limit, achieving timely improvement in system performance.

[0019] Furthermore, the method for updating the execution batch set in step S6 includes: S61. Based on the video memory used by the i-th request in the execution batch set. Calculate the video memory used by all requests and the video memory currently in use. ; S62. Determine the currently used video memory. If the value is greater than the total video memory M, then output the execution batch set; otherwise, proceed to step S63. S63. For requests in the candidate request set, sort them in descending order according to their scheduling value; S64. Select requests from the candidate request set and determine... If the condition is met, proceed to step S65; otherwise, output the execution batch set. S65. Add the current request to the execution batch set and update... ,make Then return to step S64.

[0020] The beneficial effects of the above technical solution are as follows: By implementing an incremental update mechanism for batch sets, this solution can fully utilize remaining video memory resources while maintaining scheduling stability. After determining that the system does not require global reconstruction, this method first confirms the current batch video memory usage, and then supplements high-value requests according to value sorting, thereby achieving continuous optimization of resource utilization and processing smoothness with minimal scheduling overhead.

[0021] Furthermore, the batch set generation in steps S5 and S6 must satisfy the objective function of LLM inference task scheduling, and the expression of the objective function is: in, To execute batch sets; Let n be the candidate request set; n is the total number of requests in the candidate request set. For the request The scheduling value; The video memory used for the i-th request; Total video memory; This represents the number of additional tokens generated after the i-th request is completed. For the i-th request in the candidate request set The deadline; The current system time; for Effective relaxation time; This is the preset maximum relaxation time.

[0022] The beneficial effects of the above technical solution are as follows: This solution provides a complete mathematical framework and optimality boundary for scheduling decisions by constructing an optimization model that aims to maximize the overall scheduling value and includes multiple resource and time constraints. The objective function ensures that the scheduling process always systematically pursues the optimal solution for overall smoothness and efficiency under actual constraints such as memory capacity, request time limits, and effective relaxation time, thus theoretically guaranteeing the feasibility of the method and the scheduling quality.

[0023] Furthermore, between steps S3 and S4, the process includes: determining whether the execution batch set B is empty; if so, proceeding to step S5; otherwise, proceeding to step S4. Step S1 also includes determining whether the effective relaxation time of the request is zero. If it is, the scheduling value is set to the maximum scheduling value, i.e., -1, and then proceed to step S2; otherwise, proceed directly to step S2. When generating the execution batch set in step S5, if the remaining video memory of the execution batch set is insufficient to accommodate requests with a scheduling value of -1, then the request with the largest relaxation time in the execution batch set is selected, its video memory is released, and the request with a scheduling value of -1 is stored.

[0024] The beneficial effects of the above technical solution are as follows: On the one hand, if B is empty during the system startup phase or after all batches B have been executed, it will result in the minimum scheduling value. If the scheduling drift index SD is still calculated at this time, it will cause division by zero or logical exception. By first checking whether B is empty and directly entering S5, the system can correctly start the batch reconstruction process in any state, eliminating the risk of system exception under boundary conditions. On the other hand, when generating the execution batch set in step S5, the priority rules and memory clearing mechanism of urgent requests with deadlines in batch generation are clearly defined through unique settings, ensuring that all requests can be executed before the deadline without being discarded.

[0025] Furthermore, step S3 also includes calculating the scheduling value variance within the execution batch set B. : in For the request The scheduling value; This is the average value of all request scheduling in B; This represents the number of requests made in B.

[0026] In step S4, a variance judgment is added to the original SD judgment: if Greater than the preset variance threshold If SD does not exceed SDth, then proceed to step S5 for full reconstruction.

[0027] The beneficial effects of the above technical solution are as follows: The original scheduling drift index SD only judges whether reconstruction is needed by the difference between the maximum scheduling value in the candidate request set R and the minimum scheduling value in B. When the value distribution within B is extremely uneven (some requests have extremely high values, and some have extremely low values), SD may not exceed the threshold, but the batch quality is already poor. By introducing a variance index... It can detect the uniformity of value distribution within a batch and provide an additional refactoring trigger mechanism when SD does not trigger refactoring, further improving the overall scheduling quality of the execution batch set and enabling the system to maintain high-quality batch composition even under complex and variable load scenarios.

[0028] Overall, compared with the prior art, the present invention has the following outstanding advantages: (1) Smoothness and latency co-optimization: This method optimizes smoothness and latency by constructing a scheduling value ( The core of the quantization model is based on the effective relaxation time of the request. Dynamically calculate the ideal completion time window and further infer the number of additional tokens that can be completed ahead of schedule. Within a given time frame, the system prioritizes requests that can be generated ahead of time and have high scheduling value. This ensures real-time response while significantly improving the smoothness of the generation process and the consistency of user experience, effectively reducing overall response latency.

[0029] (2) Memory-aware dynamic scheduling and resource optimization: Under GPU memory constraints, this method uses a greedy strategy and scheduling drift index ( Coordinated control of batch updates and maintenance. In each scheduling decision, the system dynamically selects the optimal set of requests to enter the execution batch based on the current memory usage, balancing memory utilization and inference efficiency, thereby reducing performance jitter and resource waste caused by memory fragmentation or frequent request swapping in and out.

[0030] (3) Adaptive scheduling and overhead control: By introducing a scheduling drift index ( This method achieves adaptive adjustment of scheduling frequency by comparing the data with a preset drift threshold. When changes in system load lead to significant differences in scheduling value, batch reconfiguration is automatically triggered; otherwise, batch stability is maintained, with only incremental updates. This mechanism effectively reduces the computational and synchronization overhead caused by frequent scheduling, improving the overall throughput and stability of the system under dynamic loads.

[0031] (4) End-to-end quality and efficiency balance: This method forms a closed-loop optimization process from "relaxation time perception - scheduling value calculation - drift decision - batch update", which not only solves the problem that traditional fixed scheduling strategies cannot adapt to dynamic request characteristics, but also achieves intelligent balance between smoothness, memory utilization and scheduling overhead through quantitative evaluation and threshold control, ultimately achieving the goal of maximizing the overall system performance under limited resources. Attached Figure Description

[0032] Figure 1 This is a flowchart of an adaptive scheduling method for LLM inference tasks based on scheduling drift metrics. Detailed Implementation

[0033] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0034] refer to Figure 1 , Figure 1 A flowchart of an adaptive scheduling method for LLM inference tasks based on scheduling drift metrics is shown; for example... Figure 1 As shown, the method includes steps S1 to S6.

[0035] In step S1, each request in the candidate request set of the LLM inference task is traversed, and the effective relaxation time corresponding to each request is calculated based on the deadline.

[0036] During implementation, this scheme preferably selects the initial time of each request. relaxation time for: in, For the request The deadline For the request The estimated execution time; After each generation iteration, the relaxation time update expression is: in, and Let be the relaxation times for the th and th-1th iterations of request i, respectively; For the request The time when the token is generated in the (th-1)th iteration; The preset iteration execution time; The effective relaxation time for the i-th iteration is: in, The maximum relaxation time preset for the system can prevent the inflation of scheduling value due to long waiting periods.

[0037] In this solution, based on the GPU memory capacity (8GB) and typical request execution time (10-20ms), a 50% margin is reserved before proceeding. =15ms. This setting ensures that the cumulative relaxation time is bounded, preventing future requests from excessively affecting the current scheduling decision through relaxation time recursion.

[0038] In step S2, based on the effective relaxation time, the number of additional tokens generated after each request completes the task is determined, and the scheduling value of the request is calculated using the number of tokens. In this scheme, the more additional tokens generated, the higher the scheduling value, and the better the smoothness.

[0039] In implementation, step S2 of this solution preferably further includes: S21. Calculate the ideal completion time window for the request based on the effective relaxation time: , in, For the i-th request in the candidate request set The deadline; for Effective relaxation time; For the request The ideal completion time window; The parameter is a multiple. The base multiple parameter; For traversal requests The amount of video memory already used; Total video memory; The value of the base multiplier parameter is related to the business's tolerance for response speed, and depends on the specific computing power of the GPU. The higher the GPU's computing power, the better. The larger the capacity, the higher the throughput; the lower the response tolerance, The smaller the value, the better to ensure real-time performance. In this scheme, the value range is [1.5, 3.0].

[0040] S22. Determine if the ideal completion time window is greater than or equal to the requested time window. The estimated completion time, and if so, the number of additional tokens generated. If it is k-1, otherwise, the number of additional tokens generated. ; ask Estimated completion time = The parameter k is introduced in this scheme to increase tolerance, so that more requests can generate more tokens within the deadline; the preset iteration execution time can be obtained from previous experiments, and is generally set to 25ms.

[0041] S23, Based on the number of additional tokens generated Calculate the scheduling value of the request: in, For the request The scheduling value; For the preset iteration execution time, The higher the target TBT (TimeBetween Tokens), the higher the batch utilization rate and the higher the FI. This solution is the preferred option. Take 25ms.

[0042] It is important to emphasize that the concept of "effective relaxation time" used in this application differs fundamentally in semantics and usage from the traditional relaxation time (Laxity, defined as "deadline - current time - remaining estimated execution time") in the job scheduling field. The effective relaxation time in this application is borrowed from the scheduling slack concept in the FI (Fluency Index) paper (Agrawal et al., "Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems", arXiv:2407.07000, 2024) in the field of LLM inference performance evaluation. The scheduling slack defined in Algorithm 1 of that paper uses a cumulative formula… "The time surplus accumulated by generating tokens in advance is used to measure the potential contribution of requests to FI molecules. It does not include the remaining estimated execution time. Its optimization goal is to maximize the smoothness of generation rather than to evaluate the feasibility of single task deadlines."

[0043] The effective relaxation time in this application is consistent with that in this paper. It is used to quantify the potential contribution rate of candidate requests to FI molecules within the current scheduling cycle (i.e., the scheduling value vᵢ), thereby guiding the dynamic optimization of batches, rather than being directly used as the priority ranking basis for task urgency. The scheduling priority is determined by the scheduling value vᵢ, and the effective relaxation time is only used as an intermediate computational cost of vᵢ.

[0044] In step S3, the maximum scheduling value in the candidate request set is selected, and the minimum scheduling value is selected based on the scheduling values ​​of all requests in the current execution batch set; in this scheme, the current execution batch set belongs to the candidate request set.

[0045] In step S4, the scheduling drift index is calculated based on the maximum scheduling value and the minimum scheduling value, and it is determined whether the scheduling drift index is greater than the preset drift threshold. If so, proceed to step S5; otherwise, proceed to step S6. In implementation, the preferred expression for calculating the scheduling drift index in this scheme is: in, For scheduling drift metrics; To maximize scheduling value; This represents the minimum scheduling value.

[0046] In this scheme, when the scheduling drift metric SD exceeds a preset drift threshold, it indicates that there are requests with high scheduling value in the waiting queue. At this time, the system should trigger a global batch reordering to optimize resource allocation and improve overall smoothness. Conversely, when SD does not reach the preset drift threshold, the system maintains the existing iterative scheduling strategy to ensure the stability and efficiency of the scheduling process. The selection of this threshold is usually based on experience or actual system operating requirements, and the common value range is 0.2 to 0.3.

[0047] In practice, this scheme preferably further includes calculating the scheduling value variance within the execution batch set B in step S3. : in, For the request The scheduling value; This is the average value of all request scheduling in B; The number of requests in B; In step S4, when the scheduling drift index is less than or equal to a preset drift threshold, and before proceeding to step S6, the method further includes determining the variance of the scheduling value. Is it greater than the preset variance threshold? If yes, proceed to step S5; otherwise, proceed directly to step S6.

[0048] In step S5, based on the total video memory, a greedy strategy is used to select requests with higher scheduling value from the candidate request set and add them to the cleared execution batch set. If the candidate request set is not empty, the process returns to step S1. The method for determining the execution batch set in step S5 includes: S51. For requests in the candidate request set, sort them in descending order according to their scheduling value; S52. Select requests from the candidate request set and determine... If the condition is met, proceed to step S53; otherwise, output the execution batch set. The currently used video memory, The video memory used for the i-th request; Total video memory; S53. Add the current request to the execution batch set and update... ,make Then return to step S52.

[0049] In step S5, the "clear" operation refers to removing a request from the execution batch set and clearing its corresponding key-value cache after the request has been executed. This operation is standard practice in LLM inference systems. The execution batch set mentioned in this scheme stores the requests that need to be executed in the current batch. After each scheduling of the execution batch set, the LLM will execute the requests in the set. After a request is executed, it will be removed from the execution batch set and its key-value cache will be cleared, rather than "only adding and not removing". Furthermore, the execution batches consist of selected higher-priority requests, and low-priority requests will not occupy the execution batch set for a long time.

[0050] In practice, this scheme preferably includes, between step S3 and step S4, the following step: determining whether the execution batch set B is empty; if so, proceeding to step S5; otherwise, proceeding to step S4. Step S1 also includes determining whether the effective relaxation time of the request is zero. If it is, the scheduling value is set to the maximum scheduling value, i.e., -1, and then proceed to step S2; otherwise, proceed directly to step S2. When generating the execution batch set in step S5, if the remaining video memory of the execution batch set is insufficient to accommodate requests with a scheduling value of -1, then the request with the largest relaxation time in the execution batch set is selected, its video memory is released, and the request with a scheduling value of -1 is stored.

[0051] In step S6, the execution batch set is kept unchanged, the current remaining video memory is determined, and the execution batch set is updated by selecting requests from the candidate request set based on the remaining video memory. If the candidate request set is not empty, the process returns to step S1.

[0052] In one embodiment of the present invention, the method for updating the execution batch set in step S6 includes: S61. Based on the video memory used by the i-th request in the execution batch set. Calculate the video memory used by all requests and the video memory currently in use. ; S62. Determine the currently used video memory. If the value is greater than the total video memory M, then output the execution batch set; otherwise, proceed to step S63. S63. For requests in the candidate request set, sort them in descending order according to their scheduling value; S64. Select requests from the candidate request set and determine... If the condition is met, proceed to step S65; otherwise, output the execution batch set. S65. Add the current request to the execution batch set and update... ,make Then return to step S64.

[0053] The scheduling method in this scheme is mainly responsible for generating the execution batch set. After obtaining the execution batch set, the CPU executes according to this set. The start and termination of the scheduling depends on whether there are new requests entering, that is, whether the candidate request set is empty. If it is not empty, the scheduling method needs to be executed in a loop. If the candidate request set is empty, the algorithm is terminated until a new request enters and then it is started again.

[0054] The batch set generation in steps S5 and S6 must satisfy the objective function of LLM inference task scheduling. The expression of the objective function is: in, To execute batch sets; Let n be the candidate request set; n is the total number of requests in the candidate request set. For the request The scheduling value; The video memory used for the i-th request; Total video memory; This represents the number of additional tokens generated after the i-th request is completed. For the i-th request in the candidate request set The deadline; The current system time; for Effective relaxation time; This is the preset maximum relaxation time.

[0055] In summary, this solution integrates task time urgency, memory usage, and generation continuity into the scheduling decision, achieving a dynamic balance between memory utilization and generation smoothness during the inference process. Building upon this, the invention further combines an iterative-level padding and batch-level rearrangement hybrid scheduling strategy, enabling the system to adaptively improve generation smoothness and overall user experience while ensuring efficient GPU operation.

Claims

1. A method for adaptive scheduling of LLM inference tasks based on a scheduling drift indicator, the method comprising: determining a drift indicator value for a current scheduling period; and adjusting a scheduling period for a next scheduling period based on the drift indicator value. Including the following steps: S1. Traverse each request in the candidate request set R of the LLM inference task, and calculate the corresponding effective relaxation time based on the deadline of each request. S2. Based on the effective relaxation time, determine the number of additional tokens generated for each request after the task is completed, and calculate the scheduling value of the request based on the number of tokens. S3. Select the maximum scheduling value in the candidate request set R, and select the minimum scheduling value based on the scheduling values ​​of all requests in the current execution batch set B. S4. Calculate the scheduling drift index based on the maximum scheduling value and the minimum scheduling value, and determine whether the scheduling drift index is greater than the preset drift threshold. If it is, proceed to step S5; otherwise, proceed to step S6. S5. Based on the total video memory, a greedy strategy is used to select the requests with the highest scheduling value in the candidate request set and add them to the cleared execution batch set. When the candidate request set is not empty, return to step S1. S6. Keep the execution batch set unchanged, determine the current remaining video memory, select requests from the candidate request set based on the remaining video memory to update the execution batch set, and return to step S1 if the candidate request set is not empty.

2. The adaptive scheduling method for LLM inference tasks based on scheduling drift index according to claim 1, characterized in that, The initial time of each request relaxation time for: in, For the request The deadline For the request The estimated execution time; After each generation iteration, the relaxation time update expression is: in, and Let be the relaxation times for the th and th-1th iterations of request i, respectively; For the request The time when the token is generated in the (th-1)th iteration; The preset iteration execution time; The effective relaxation time for the i-th iteration is: in, The maximum relaxation time preset for the system.

3. The adaptive scheduling method for LLM inference tasks based on scheduling drift index according to claim 1, characterized in that, Step S2 further includes: S21. Calculate the ideal completion time window for the request based on the effective relaxation time. : ， in, The parameter is a multiple. The base multiple parameter; For traversal requests The amount of video memory already used; Total video memory; S22. Determine the ideal completion time window Is it greater than or equal to the request? The estimated completion time, and if so, the number of additional tokens generated. If it is k-1, otherwise, the number of additional tokens generated. ; S23, Based on the number of additional tokens generated Calculate the scheduling value of the request: in, For the request The scheduling value.

4. The adaptive scheduling method for LLM inference tasks based on scheduling drift index according to claim 1, characterized in that, The expression for calculating the scheduling drift metric is: in, For scheduling drift metrics; To maximize scheduling value; This represents the minimum scheduling value.

5. The adaptive scheduling method for LLM inference tasks based on scheduling drift index according to claim 1, characterized in that, The method for determining the execution batch set in step S5 includes: S51. For requests in the candidate request set, sort them in descending order according to their scheduling value; S52. Select requests from the candidate request set and determine... If the condition is met, proceed to step S53; otherwise, output the execution batch set. The currently used video memory, The video memory used for the i-th request; Total video memory; S53. Add the current request to the execution batch set and update... ,make Then return to step S52.

6. The adaptive scheduling method for LLM inference tasks based on scheduling drift index according to claim 1, characterized in that, The method for updating the execution batch set in step S6 includes: S61. Based on the video memory used by the i-th request in the execution batch set. Calculate the video memory used by all requests and the video memory currently in use. ; S62. Determine the currently used video memory. If the value is greater than the total video memory M, then output the execution batch set; otherwise, proceed to step S63. S63. For requests in the candidate request set, sort them in descending order according to their scheduling value; S64. Select requests from the candidate request set and determine... If the condition is met, proceed to step S65; otherwise, output the execution batch set. S65. Add the current request to the execution batch set and update... ,make Then return to step S64.

7. The adaptive scheduling method for LLM inference tasks based on scheduling drift index according to claim 1, characterized in that, The batch set generation in steps S5 and S6 must satisfy the objective function of LLM inference task scheduling. The expression of the objective function is: in, To execute batch sets; Let n be the candidate request set; n is the total number of requests in the candidate request set. For the request The scheduling value; The video memory used for the i-th request; Total video memory; This represents the number of additional tokens generated after the i-th request is completed. For the i-th request in the candidate request set The deadline; The current system time; for Effective relaxation time; This is the preset maximum relaxation time.

8. The adaptive scheduling method for LLM inference tasks based on scheduling drift index according to claim 1, characterized in that, Between steps S3 and S4, the process also includes: determining whether the execution batch set B is empty; if so, proceeding to step S5; otherwise, proceeding to step S4. Step S1 also includes determining whether the effective relaxation time of the request is zero. If it is, the scheduling value is set to the maximum scheduling value, i.e., -1, and then proceed to step S2; otherwise, proceed directly to step S2. When generating the execution batch set in step S5, if the remaining video memory of the execution batch set is insufficient to accommodate requests with a scheduling value of -1, then the request with the largest relaxation time in the execution batch set is selected, its video memory is released, and the request with a scheduling value of -1 is stored.

9. The adaptive scheduling method for LLM inference tasks based on scheduling drift index according to claim 1, characterized in that, Step S3 also includes calculating the scheduling value variance within the execution batch set B. : in, For the request The scheduling value; This is the average value of all request scheduling in B; The number of requests in B; In step S4, when the scheduling drift index is less than or equal to a preset drift threshold, and before proceeding to step S6, the method further includes determining the variance of the scheduling value. Is it greater than the preset variance threshold? If yes, proceed to step S5; otherwise, proceed directly to step S6.

Images