Enhanced large model reasoning framework based on adaptive request scheduling

Abstract

The invention discloses an enhanced large model reasoning framework based on adaptive request scheduling, and the framework comprises a prediction module which is used for predicting the external calling time and output length range of an enhanced large model reasoning request according to the input and calling types of the request before the request enters a queue; the scheduling module is used for selecting a context processing strategy according to the external calling time and the output length range and then predicting the scheduling value of the first stage as a scheduling sequence basis before the external calling is requested to be returned; the scheduling value is updated according to the executed actual uplink and downlink processing strategy and the actual call return length to serve as the scheduling sorting basis of the second request stage; and the dynamic token batch processing module is used for calculating and setting the upper limit of the current batch processing token according to the hardware information and the request pause state of the system, and setting an anti-overload interval constraint at the same time, so that the throughput can be further improved while the SLOs can be met.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of large model inference, and particularly relates to an enhanced large model inference framework based on adaptive request scheduling. BACKGROUND

[0002] In recent years, enhanced large language models (LLMs) have gradually become the core technology paradigm in web services. Unlike traditional LLMs, which rely on fixed pre-trained parameters, lack real-time knowledge, and only support text processing, enhanced LLMs extend the functionality of LLMs by calling external tools (such as web APIs, database queries) and other models during inference. This enables enhanced LLMs to support tasks such as arithmetic calculations, real-time information retrieval, and web interactions, and is gradually becoming a key component in web applications.

[0003] Inference service systems based on enhanced LLMs are also gradually becoming the core infrastructure of the new generation of web, and their inference efficiency and robustness will directly determine the user experience of the web. During the request for enhanced LLM inference, the enhanced LLM identifies the need for real-time information calls and triggers the corresponding tool calls. The system pauses inference during the waiting period for external enhancement modules to respond, and after the response is returned, the system inserts it into the generated sequence and continues the normal generation process.

[0004] In addition, inference systems must efficiently handle a large number of concurrent user requests, which must meet strict service level objective (SLO) requirements (e.g., requiring a time-to-first-token (TTFT) delay of less than a fixed threshold). Therefore, the goal of the inference system is to maximize the number of requests that meet the delay requirement, i.e., to increase the number of completed requests that meet SLOs within a given time, which is often referred to as effective throughput, defined as the number of completed requests that meet SLOs per unit of time, reflecting the system's ability to meet real-time performance requirements.

[0005] The most advanced existing work is also focusing on improving inference performance. vLLM proposes PagedAttention to optimize GPU memory utilization and improve inference efficiency. However, in the enhanced LLM inference process, vLLM directly considers the request as complete and discards the generated context (KV cache) when the external call occurs, and a large amount of recalculation is required after the call returns, resulting in serious waste of GPU resources. InferCept proposes dynamic context management during the call on this basis, dynamically selecting to discard, retain in GPU, or swap to CPU memory based on the call time and context length of the request, thereby significantly alleviating the waste of KV cache and improving the performance of enhanced LLM inference.

[0006] The existing inference framework still faces two major problems in improving the effective throughput of enhanced LLM inference: (1) The first-come-first-serve (FCFS) scheduling strategy leads to excessively high queuing delay, breaking the SLOs. In the existing inference service system, the queuing delay and overall completion time of more requests seriously exceed the SLOs, which will lead to a significant decrease in effective throughput. For example, vLLM and InferCept usually adopt FCFS scheduling, and long requests will block short requests, leading to serious head-of-line (HoL) blocking, thereby causing high delay and throughput reduction. The long context management and return length uncertainty during external calls in enhanced LLM inference will further exacerbate the HoL problem, leading to more serious queuing delay.

[0007] (2) Static token batch processing upper limit. In enhanced LLM, the input, output, and external API call return length of requests differ greatly. The fixed token upper limit cannot adapt to dynamic load and hardware conditions: setting too small will lead to too few processed requests per round and low throughput; setting too large will cause resource contention and excessive recalculation.

[0008] In summary, the existing framework is difficult to further improve the throughput while meeting the SLOs, and therefore a fine-grained scheduling strategy and dynamic token batch processing scheme that combines the characteristics of enhanced LLM are needed. SUMMARY

[0009] In view of the above, the purpose of the present application is to provide an enhanced large model inference framework based on adaptive request scheduling, which introduces an adaptive request scheduling strategy that combines the characteristics of enhanced LLM inference requests and a dynamic batch token mechanism to meet the SLOs while further improving the throughput.

[0010] To achieve the above-mentioned purpose of the application, an enhanced large model inference framework based on adaptive request scheduling provided by an embodiment includes a prediction module, a scheduling module, and a dynamic token batch processing module. The prediction module is configured to predict the external call time and output length range of the enhanced large model inference request based on the input and call type of the request before the request enters the queue. The scheduling module is configured to select a context processing strategy based on the external call time and output length range, and predict the scheduling value of phase one as the scheduling order basis before the request returns from the external call according to the context processing strategy. After completing the actual call of phase one, the scheduling value is updated as the scheduling order basis of phase two according to the actual context processing strategy and the actual return length of the call. The dynamic token batch processing module is configured to calculate and set an upper limit of a current batch token according to hardware information and a suspension request state of the system, and set an anti-overload interval constraint.

[0011] For the scheduling module, after processing the prediction information, different scheduling value processing methods are designed according to the pre-context processing strategy and the request characteristics. Meanwhile, considering the different characteristics before and after the request call and avoiding the amplification of the influence of the prediction error on the system, the scheduling value metric is designed in two stages, and the real feedback information is used for dynamic correction in the second stage to realize stable and efficient priority scheduling. At the same time, a starvation prevention strategy is designed, and the queuing time of the request is considered in the scheduling value.

[0012] For the dynamic token batch processing module, requests with high priority are combined into a batch for processing according to the request ordering. Real-time information in the running process is monitored, and resource adjustment is used to adapt to the runtime load, so as to ensure the stability of the system while maintaining flexible response to the load. Interval constraints are set to prevent transient idle GPU memory fluctuations from causing token budget to expand excessively and memory overload.

[0013] The above three modules together form a complete framework, effectively reducing the queuing delay, alleviating the influence of the prediction error, and maximizing the throughput under diversified enhanced LLM loads.

[0014] Preferably, in the prediction module, a fine-tuned BERT-base-uncased lightweight model is used to predict the output length range and external call time of the enhanced large model inference request. Specifically, the output length is predicted by a classification method, and the output length is discretized into a fixed number of buckets, and only the range to which the output length belongs is predicted. For the external call time, it is regarded as a regression task, and the call time range of each type is collected in the dataset according to the input call type for estimation. The predicted external call time and output length range are used to pre-judge the context processing strategy when the request is called, and provide prior signal support for subsequent scheduling.

[0015] Preferably, in the request call module, the context processing strategy is selected according to the external call time and the output length range, including: Before each enhanced large model inference request enters the waiting queue, the memory waste of each context processing strategy is calculated according to the external call time and the output length range, and the context processing strategy with the smallest memory waste is selected as the selected context processing strategy, wherein the context processing strategy includes a discard strategy, an exchange strategy, and a retention strategy.

[0016] Preferably, in the request call module, the scheduling value of the first stage of the context processing strategy prediction is used as the basis for the scheduling order before the request returns after external call, including: The scheduling value of the selected context processing strategy is calculated based on system overhead and used as the basis for scheduling order before the request returns from an external call. Specifically, this includes the sorting basis in the waiting queue or before the first external call. The entire phase before the call returns includes the pre-filling phase, the generation phase, the memory usage phase during the call, and the swapping phase.

[0017] Preferably, in the scheduling module, updating the scheduling value based on the actual context processing strategy and the actual call return length serves as the scheduling ranking basis for request phase two, including: The system overhead of each stage is calculated using the actual context processing strategy and the actual call return length. Then, the scheduling value corresponding to the actual context processing strategy is calculated based on the system overhead of each stage as the basis for cross-round scheduling decisions. Each stage includes the system overhead of the request context swapping stage, the recalculation stage under the discard strategy, the API call return token processing stage, and the request new round generation stage.

[0018] Preferably, in the scheduling module, after obtaining the scheduling value of phase two, the system overhead of the context processing strategy that may be adopted in the next round is also considered and added to the scheduling value as the final scheduling value of the next round, which also serves as the basis for cross-round scheduling decisions. If the context handling strategy adopted in the next round is the swap strategy, then the system overhead of context swapping needs to be additionally accounted for; If the context handling strategy adopted in the next round is a retention strategy, then the system overhead of memory usage during subsequent API calls must be taken into account.

[0019] Preferably, in the scheduling module, based on the scheduling value or final scheduling value in stage two, an anti-starvation mechanism is introduced. Specifically, a waiting time with a certain weight is added to the corrected scheduling value or final scheduling value. The waiting time is defined as the difference between the current time and the last scheduled time.

[0020] Preferably, in the dynamic token batch processing module, the current batch processing token limit is calculated and set based on hardware information and the system's pause request status, including: Continuously monitor the available video memory capacity and the context video memory occupied by pause requests, and set the GPU memory occupied by pause requests to be preempted by running requests. The sum of the two is taken as the current available capacity, and the current batch token limit should be calculated based on the current available capacity.

[0021] Preferably, the dynamic token batch processing module includes setting an overload protection interval constraint, including: A suitable reference upper limit is determined by running several rounds of static configuration offline. ; When executed online, the maximum number of dynamic tokens is limited to [number]. Within the range, and For smooth control parameters.

[0022] To achieve the above-mentioned objectives, embodiments of the present invention also provide an enhanced large model inference method based on adaptive request scheduling, which employs an enhanced large model inference framework and includes the following steps: When a request arrives, the prediction module is invoked to predict the external call time and output length range of the enhanced large model inference request; The scheduling module selects a context processing strategy based on the external call time and output length range, and uses the context processing strategy to predict the scheduling value of Phase 1 as the basis for the scheduling order of the first round of requests. After the actual call is completed, the scheduling module updates the scheduling value in Phase 2 based on the actual context processing strategy and the actual call return length, and restores the request to different queues according to the actual context processing strategy. In each round of scheduling, the dynamic token batch processing module will calculate and set the current batch processing token limit based on hardware information and the system's pause request status, while setting overload protection interval constraints. Finally, after sorting the requests according to their scheduling value, the requests are selected in turn to fill the batch until a batch is filled and then sent into the model to perform a forward iteration.

[0023] Compared with the prior art, the beneficial effects of the present invention include at least the following: To address the high queuing latency issue caused by the FCFS scheduling strategy in previous solutions, a two-stage adaptive request scheduling strategy combining enhanced LLM inference request characteristics is introduced to optimize request ordering, reduce overall queuing latency, and improve effective throughput. Furthermore, an anti-starvation mechanism is proposed, incorporating request queuing time into the scheduling value considerations to maintain fairness in request scheduling. To address the throughput loss caused by the static batch processing token limit, a new system is implemented that calculates the available GPU memory capacity based on the current system state and request queue information, dynamically setting the batch processing token limit to adapt to load changes and improve throughput. Furthermore, a range constraint is set to maintain stability while achieving high performance. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the structure of the enhanced large model inference framework based on adaptive request scheduling provided in the embodiment. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0027] The technical concept of this invention is as follows: This invention provides an enhanced large model inference framework based on adaptive request scheduling, specifically including: (1) adopting a two-stage adaptive request scheduling strategy for enhanced LLM inference: based on the return length of the request call, the context processing strategy, and the input and output information, design a scheduling value to represent the system overhead and execution time of the request, optimize the request sorting based on the scheduling value, and correct the prediction through real-time monitoring data during runtime, and adaptively adjust the request characteristics and system state. Reduce the overall request queuing delay and improve the effective throughput and inference efficiency. In addition, introduce an anti-starvation mechanism, consider the queuing time of the request, and maintain the fairness of request scheduling; (2) dynamically setting a batch processing token mechanism: based on the hardware information and the paused request status of the system, calculate the upper limit of the batch processing token that should be set at present, adapt to the current system processing capacity and real-time load changes. And set interval constraints to maintain stability while achieving high performance.

[0028] The relevant prior art involved in this invention includes: 1. Basic framework: vLLM framework.

[0029] vLLM, proposed by the University of California, Berkeley, is a high-performance inference framework that has evolved into a mainstream inference project jointly contributed by academia and industry. It aims to significantly improve the throughput and memory efficiency of language model services in real-time scenarios. vLLM initially proposed PagedAttention to effectively manage attention key-value memory, dividing GPU memory into fixed-size blocks. During service requests, it allocates the necessary blocks to the KV cache on demand, replacing the previous method of pre-allocating a whole block of contiguous memory. This greatly reduces GPU memory fragmentation, improves memory efficiency, and thus enhances overall inference performance. With the continuous development of vLLM, optimization techniques such as continuous batch processing, block pre-filling, and offloading the KV cache to the CPU have also been integrated into the framework. InferCept adds support for enhanced LLM inference requests based on the vLLM framework, and this invention is also developed based on the vLLM framework.

[0030] 2. Iterative batch processing Traditional inference scheduling operates at the request level. Requests within the same batch may vary in length and require different numbers of forward iterations. Even if some shorter requests complete first, the batch cannot return a result until all remaining longer requests have finished. Newly arriving requests must also wait for all requests in a batch to complete before joining the new batch, resulting in poor request queuing latency. Orca proposes an iterative scheduling approach to replace the original request-level scheduling. It uses each forward iteration as the scheduling granularity, selecting requests to form a batch before each iteration, generating a token for each iteration, and then proceeding with batch selection for the next iteration. This allows shorter requests that complete first to exit early, and allows newly arriving requests to immediately join the next iteration batch. Orca fully leverages the parallel processing capabilities of GPUs, further improving the overall system throughput.

[0031] 3. Segmented pre-filling Block pre-filling improves GPU utilization and system throughput by breaking down long sequences into smaller blocks and processing them in conjunction with decoding tasks. In traditional continuous batch processing, the pre-filling stage processes the entire input sequence, which is time-consuming and blocks decoding tasks. Block pre-filling reduces computational latency by breaking long sequences into fixed-size blocks (e.g., 512 tokens) and processing one block at a time. Mixed batches include both pre-filling blocks and decoding tasks, making their computational demands complementary and improving overall efficiency. This reduces pipeline bubbles: by mixing small pre-filling tasks with decoding tasks, idle GPU resources are avoided. It also improves decoding latency: decoding tasks are no longer delayed by long blocking times caused by pre-filling tasks, enhancing the streaming generation experience.

[0032] 4. Dynamic Context Processing Strategy In enhanced LLM inference scenarios, when a request is called externally, the request service is considered terminated, the service ends, and the generated context is discarded. After the call returns, the generated token needs to be recalculated, blocking other requests and wasting GPU resources. InferCept addresses this by proposing a dynamic context handling strategy based on the context length at the time of the request call and the expected call time: retain on the GPU, swap to CPU memory, or discard. For example, requests with short call times and short context lengths are better suited to retaining the context on the GPU, while requests with long contexts and very long call times are better suited to discarding the context. InferCept proposes a theoretical calculation formula to calculate the memory waste of each context handling strategy, compares the memory waste of the retain and discard strategies, selects the smaller amount as the request's memory waste, and then sorts the requests based on memory waste. A swapping limit is set; within this limit, request contexts are swapped to CPU memory according to the sorting order. Requests exceeding the swapping limit are handled according to the originally selected retain or discard strategy. This approach, which processes context based on the request's context and call characteristics, adapts well to request requirements, significantly reduces GPU memory waste, and improves overall inference efficiency.

[0033] Based on the aforementioned existing technologies, the embodiments of the present invention provide an enhanced large-model inference framework based on adaptive request scheduling, such as... Figure 1 As shown, it includes a prediction module, a scheduling module, and a dynamic token batch processing module.

[0034] In this embodiment, the prediction module is used to predict the external call time and output length range of the enhanced large model inference request based on the input and call type of the request before the request enters the queue.

[0035] In practice, a fine-tuned BERT-base-uncased model is used for prediction. Efficient scheduling requires information about the output length; however, the actual output length is unknown before decoding begins. Drawing on existing research on LLM inference request length prediction, this lightweight BERT-base-uncased model is fine-tuned to predict the range of output lengths for LLM inference requests and external call times. This model contains only a million parameters, offering fast prediction speeds and negligible computational overhead. Furthermore, its small size ensures it doesn't compete for GPU inference resources. On the other hand, a classification approach is used to predict output lengths, discretizing them into a fixed number of buckets and predicting only the range to which the length belongs. External API call times are treated as a regression task; based on the input call type, the call time range for each type is estimated from the dataset. Specifically, the dataset from InferCept, which mixes six types of enhanced LLM requests, is divided into training and testing sets in a 7:3 ratio, while the ToolBench data from ToolLLM is divided into training and testing sets in a 6:4 ratio. These sets are used to train the model and test its performance. The fine-tuned model achieves 85% accuracy on the ToolBench dataset and 65% accuracy on another dataset across the predicted output length range. For call time prediction, the mean squared errors of the fine-tuned model are 5s and 0.4s, respectively. Currently, the model's prediction accuracy is sufficient to support the implementation of scheduling strategies.

[0036] In enhancing LLM inference services, request scheduling needs to consider the high heterogeneity of request characteristics, including input length, output length, API call time, and return length. Single-stage scheduling typically sorts requests based on predicted length or response time as they enter the system, but this static decision-making can easily amplify prediction errors. If a long request misjudged as short is executed at the head of the queue and a discard strategy is used when the call occurs, a large amount of context needs to be recalculated after the API returns, causing memory consumption and blocking subsequent requests, resulting in the typical head-of-line (HoL) problem. Without runtime correction mechanisms, such errors will accumulate, leading to a sharp increase in queuing latency and a significant reduction in throughput.

[0037] Therefore, an efficient scheduling framework must effectively combine foresight (relying on predicted signals) and adaptability (correcting deviations during operation). Based on this principle, this invention proposes a two-stage scheduling value metric in the scheduling module, performing different evaluations and decisions before and after a request is made, thereby balancing the foresight of predictions with robustness during operation.

[0038] For the scheduling value prediction in Phase 1: When a request arrives, based on the predicted characteristics, output length range, API call time, return length, and context processing strategy, a temporary scheduling value representing the expected system overhead of the request is calculated. By sorting requests according to this scheduling value, the system can prioritize scheduling requests with lower overhead and faster completion, thereby reducing the HoL blocking problem caused by long requests occupying the head of the queue.

[0039] Before each enhanced large model inference request i enters the waiting queue, the memory waste of each context processing strategy is calculated based on the external call time and output length range. Context processing strategies include discarding, swapping, and retaining. The strategy with the least memory waste is then selected as the chosen context processing strategy. : in, , ,as well as Retention strategies Dropping strategy and exchange strategies Memory waste For the time of the external API call to request i, The length of the context when request i is invoked. The memory usage of the KV cache for each token, The processing time for one forward iteration is [time]. This represents the total number of tokens for other requests in the system. To exchange requests between the CPU and GPU i The transmission time required for the context.

[0040] After selecting the context processing strategy, the scheduling value of Phase 1 is predicted based on the context processing strategy. The scheduling order for the first round of requests is determined by several factors, including: firstly, calculating the system overhead for each stage of the selected context processing strategy. In the first round, each stage includes a pre-filling stage, a generation stage, a memory usage stage during the call, and a swapping stage. Dropping and swapping strategies mean that the memory will be released when the call occurs, while retention strategies mean that the memory will be occupied throughout the call. Since it is difficult to predict the next round's situation, the scheduling value of the retention strategy is estimated using the system overhead before the call returns. in, , , as well as These represent the system overhead of the pre-filling phase, the generation phase, the memory usage phase during the call, and the swapping phase corresponding to request i, respectively. This represents the actual input length. This represents the processing time for one forward iteration. The predicted value represents the generated length. Indicates the predicted call interval. and These represent the maximum number of tokens that can be processed in a single forward iteration and the maximum number of tokens that can be swapped out to the CPU in a single iteration, respectively.

[0041] Then, the selected context processing strategy is calculated based on system overhead. scheduling value As the basis for the scheduling order of the first round of requests, each strategy includes pre-filling and decoding phases. In addition, the retention strategy needs to consider the occupancy during the call, and the swapping strategy needs to consider the overhead during the swapping out: + in, , ,as well as These represent the scheduling value of the retention policy, the discard policy, and the swap policy, respectively.

[0042] The scheduling value obtained in Phase 1 It is used as the ordering basis for requests in the waiting queue or before the first external call occurs. However, since prediction errors may accumulate and the system state may change after the call, a second phase is introduced to further optimize the scheduling value through runtime observation and correction.

[0043] Phase two uses runtime adjustments and reordering as the basis for scheduling. While phase one provides prediction-based scheduling value... However, its accuracy is limited by output length, API call latency, and the uncertainty of context processing strategy predictions. These errors can amplify in the request queue, causing priority ordering distortion and increasing overall queuing latency. To address this issue, actual observations are used instead of predicted values ​​during execution to correct the system overhead estimate. For multi-round requests, observations from each round feed back into subsequent predictions, resulting in greater accuracy. The corrected scheduling value is denoted as... This more accurately depicts the actual resource consumption of request i and serves as the basis for reordering after the call.

[0044] After an external call occurs, the request enters a pause queue. The system continuously monitors whether the call returns and records the actual return length and the context handling strategy selected during the pause. First, the monitored information is used to calculate the system overhead for each subsequent stage, including the request context swapping stage, the recalculation stage under the discard strategy, the API call return token processing stage, and the new request generation stage. After the call returns, the request carries a new round of input information into the system; therefore, the new generation length and new call characteristics can be predicted based on this. This indicates the total length of the request that has been processed at the time the call occurred. This indicates the actual length returned by the call.

[0045] in, , , ,as well as These represent the system overhead of the request context swapping phase, the recalculation phase under the discard policy, the API call returned token processing phase, and the new request generation phase, respectively. This indicates the length of the next output after the external call returns; Then, the scheduling value corresponding to the actual context processing strategy is calculated based on the system overhead of each stage, which serves as the basis for cross-round scheduling decisions: in, , ,as well as These represent the updated scheduling value corresponding to the retention, discard, and swap strategies calculated in Phase 2, respectively.

[0046] When an API call returns, the system not only obtains the output of the call but also the updated input context needed for subsequent decoding. Therefore, the scheduling value obtained in Phase Two... Based on this, we further consider the context processing strategies that may be adopted in the next round, thereby obtaining the final scheduling value. If a swapping strategy is used in the next round, the overhead of context swapping needs to be factored in; if a retention strategy is used, the memory usage during subsequent API calls needs to be factored in. Because API types and return lengths typically exhibit stable distribution patterns across similar requests, relatively reliable predictions can be made based on request category characteristics or observations from previous calls. In this way, It can provide a more comprehensive estimate of future resource consumption and serve as a basis for cross-round scheduling decisions.

[0047] in, , ,as well as These represent the final scheduling value corresponding to the retention policy, the discard policy, and the swap policy, respectively.

[0048] In the embodiment, the scheduling value in phase two or the final scheduling value Building upon this, an anti-starvation mechanism is introduced to ensure that long-waiting requests gradually gain higher priority. Specifically, a waiting time with a certain weight is added to the adjusted scheduling value or final scheduling value, and the waiting time is defined as the current time. Compared to the last time it was scheduled The difference: in, This is a weighting factor used to balance fairness and throughput.

[0049] In summary, this invention combines prediction-driven scheduling value estimation (Phase 1), runtime real-time information correction (Phase 2), and an anti-starvation mechanism to form a complete adaptive request scheduling strategy. This design enables the system to achieve a dynamic balance between foresight, adaptability, and fairness, ensuring both high throughput and stable latency performance in enhanced LLM inference services.

[0050] In this embodiment, the dynamic token batch processing module is used to calculate and set the current batch processing token limit based on hardware information and the system's pause request status, while also setting overload protection interval constraints.

[0051] In existing systems, a fixed batch token limit is typically used to protect GPU memory capacity. However, this static approach has significant drawbacks: it results in low throughput when too conservative, and frequent swapping and pauses when too aggressive. To address this, a dynamic token limit adjustment mechanism was designed to adaptively adjust the maximum number of tokens that can be processed in each round based on the runtime resource status.

[0052] Specifically, we continuously monitor two key quantities: (i) free GPU memory capacity, and (ii) context memory occupied by pause requests. The GPU memory used by pause requests can be preempted by running requests, so the sum of these two is considered the current available capacity. Then, based on the available capacity, we calculate the current batch token budget that should be set, as follows: in, This represents the GPU memory used by requests currently in operation at time t. This indicates a request to run. The GPU memory used to indicate the request to pause inference. A request to pause, These represent the video memory occupied by the parameters of the large model itself, the video memory occupied by intermediate variables generated at runtime, and the safety buffer video memory reserved to prevent system crashes (OOM, Out Of Memory). This indicates that the fixed video memory usage cannot be used to store KV cache. This indicates the total available video memory capacity of the GPU. This represents the physically available video memory capacity. This indicates the total amount of video memory that can be allocated to new tasks.

[0053] To prevent the token cap from being excessively inflated during periods of high idle time, which could lead to memory overload, a range constraint is added. A suitable reference upper limit is determined by running several rounds of static configuration offline. When executed online, the maximum number of dynamic tokens is limited. Within the range, and This is to smooth out control parameters. This allows for flexible scaling based on real-time conditions while avoiding over-adjustment due to instantaneous fluctuations.

[0054] In summary, this mechanism of dynamically setting the upper limit of batch tokens can improve both the throughput performance of the batch process and the scheduling efficiency.

[0055] The embodiment also provides an enhanced large model inference method based on adaptive request scheduling, which adopts the above-described enhanced large model inference framework and includes the following steps: When a request arrives, the prediction module is invoked to predict the external call time and output length range of the enhanced large model inference request; The scheduling module selects a context processing strategy based on the external call time and output length range, and uses the context processing strategy to predict the scheduling value of Phase 1 as the basis for the scheduling order of the first round of requests. After the actual call is completed, the scheduling module updates the scheduling value in Phase 2 based on the actual context processing strategy and the actual call return length, and restores the request to different queues according to the actual context processing strategy. In each round of scheduling, the dynamic token batch processing module will calculate and set the current batch processing token limit based on hardware information and the system's pause request status, while setting overload protection interval constraints to prevent fluctuations in extreme situations. Finally, after sorting the requests according to their scheduling value, the requests are selected in turn to fill the batch until a batch is filled and then sent into the model to perform a forward iteration.

[0056] Compared to vLLM and InferCept, this invention achieves significantly superior queuing performance and effective throughput. We tested the scheme using a single NVIDIA RTX 4090 GPU running the GPT-J-6B model, a single NVIDIA H800 GPU running OPT-13B, and two NVIDIA A100 GPUs with 40GB of VRAM each, employing the hybrid enhanced LLM dataset provided by InferCept and the ToolBench dataset provided by ToolLLM. Experimental results show that this scheme consistently delivers the highest effective throughput, exceeding vLLM and InferCept by 4.7-33.1 times and 3.3-13.2 times, respectively. The time-to-first-token (TTFT) latency is up to 96.3% and 95.0% lower than vLLM and InferCept, respectively, and TTFT also reflects request queuing latency. Furthermore, the average token processing latency is 80.3% and 62.7% lower than vLLM and InferCept, respectively. This demonstrates the effectiveness of the proposed two-stage scheduling strategy and dynamic token batching mechanism in reducing overall queuing latency and improving effective throughput.

[0057] Different load fluctuations were used to test the effective throughput of this scheme and the baseline scheme. This invention consistently showed better effective throughput performance. A gamma distribution was used to simulate the request arrival process, and the coefficient of variation was used to control the degree of load fluctuation. When the coefficient of variation was 1.5, the effective throughput of vLLM and InferCept dropped to about 0.1req / s, but this invention still maintained 1.15req / s, showing good stability.

[0058] Ablation tests were conducted on each component of this solution. Firstly, the introduction of dynamic token batching improved the effective throughput. For example, with a request arrival rate of 2.0 req / s, this invention increased the throughput of vLLM from 0.22 req / s and InferCept from 0.28 req / s to 0.35 req / s. This demonstrates that the dynamic batching mechanism can effectively adapt to runtime resource conditions and improve overall system efficiency. Next, the addition of two-stage scheduling further significantly improved the effective throughput and reduced TTFT latency. The effective throughput was increased by 7.13 times and 4.48 times compared to vLLM and InferCept, respectively. TTFT was reduced by 81% and 87.8% compared to vLLM and InferCept, respectively. This is because the FCFS scheduling strategy strictly schedules requests according to arrival order, while the scheduling strategy of this invention can adaptively schedule requests based on request characteristics, thereby reducing overall queuing latency and significantly improving effective throughput. Furthermore, the prediction module of this invention accounts for less than 1% of the overhead, which is negligible.

[0059] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An enhanced large model inference framework based on adaptive request scheduling, characterized in that, The application relates to a request calling method and device for an enhanced large model, and belongs to the technical field of artificial intelligence. The application comprises a prediction module, a scheduling module and a dynamic token batch processing module. The prediction module is used for predicting the external calling time and output length range of an enhanced large model inference request according to the input and calling type of the request before the request enters a queue. The scheduling module is used for selecting a context processing strategy according to the external calling time and output length range, and predicting the scheduling value of stage one as the scheduling order basis before the request occurs external calling return. The dynamic token batch processing module is used for calculating and setting the upper limit of the current batch processing token according to hardware information and the suspension request state of the system, and setting the anti-overload interval constraint.

2. The adaptive request scheduling based enhanced large model inference framework of claim 1, wherein, In the prediction module, a fine-tuned BERT-base-uncased lightweight model is used to predict the output length range and external calling time of the enhanced large model inference request.

3. The adaptive request scheduling based enhanced large model inference framework of claim 1, wherein, In the request calling module, the context processing strategy is selected according to the external calling time and output length range, which comprises the following steps. Before each enhanced large model inference request enters a waiting queue, the memory waste of each context processing strategy is calculated according to the external calling time and output length range, and the context processing strategy with the minimum memory waste is selected as the selected context processing strategy, wherein the context processing strategy comprises a discard strategy, an exchange strategy and a reservation strategy.

4. The adaptive request scheduling based enhanced large model inference framework of claim 1, wherein, In the request calling module, the scheduling value of stage one is predicted as the scheduling order basis before the request occurs external calling return according to the context processing strategy, which comprises the following steps. The scheduling value of the selected context processing strategy is calculated as the scheduling order basis before the request occurs external calling return based on the system overhead, and the sorting basis before the first external calling is included, wherein the whole stage one before the calling return comprises a pre-padding stage, a generation stage, a memory occupation stage during calling and an exchange stage.

5. The adaptive request scheduling based enhanced large model inference framework of claim 1, wherein, In the scheduling module, the scheduling value is updated as the scheduling order basis of stage two of the request according to the actual context processing strategy and actual calling return length, which comprises the following steps. The system overhead of each stage is calculated by using the actual context processing strategy and actual calling return length, and then the scheduling value corresponding to the actual context processing strategy is calculated as the basis for cross-round scheduling decision according to the system overhead of each stage, wherein the stages comprise a request context exchange-in stage, a recalculation stage under the discard strategy, an API calling return token processing stage and a system overhead of a request new round generation stage.

6. The adaptive request scheduling based enhanced large model inference framework according to claim 1 or 5, wherein, In the scheduling module, after obtaining the scheduling value of Phase 2, the system overhead of the context processing strategy that may be adopted in the next round is also considered and added to the scheduling value as the final scheduling value of the next round, which also serves as the basis for cross-round scheduling decisions. If the context handling strategy adopted in the next round is the swap strategy, then the system overhead of context swapping needs to be additionally accounted for; If the context handling strategy adopted in the next round is a retention strategy, then the system overhead of memory usage during subsequent API calls must be taken into account.

7. The adaptive request scheduling based enhanced large model inference framework of claim 6, wherein, In the scheduling module, based on the scheduling value or final scheduling value in stage two, an anti-starvation mechanism is introduced. Specifically, a waiting time with a certain weight is added to the corrected scheduling value or final scheduling value. The waiting time is defined as the difference between the current time and the last scheduled time.

8. The adaptive request scheduling based enhanced large model inference framework of claim 1, wherein, The dynamic token batch processing module calculates and sets the current batch processing token limit based on hardware information and the system's pause request status, including: Continuously monitor the available video memory capacity and the context video memory occupied by pause requests, and set the GPU memory occupied by pause requests to be preempted by running requests. The sum of the two is taken as the current available capacity, and the current batch token limit should be calculated based on the current available capacity.

9. The adaptive request scheduling based enhanced large model inference framework of claim 1, wherein, The dynamic token batch processing module includes setting overload protection interval constraints, including: determining a suitable reference upper limit by running several rounds of static configuration offline ; When executed online, the dynamic token upper limit is limited to wherein and are smoothing control parameters.

10. An enhanced large model inference method based on adaptive request scheduling, characterized in that, Using the enhanced large model inference framework according to any one of claims 1-9, the following steps are performed: When a request arrives, the prediction module is invoked to predict the external call time and output length range of the enhanced large model inference request; The scheduling module selects a context processing strategy based on the external call time and the output length range, and uses the context processing strategy to predict the scheduling value of Phase 1 as the basis for the scheduling order before the first external call returns. After the actual call is completed, the scheduling module updates the scheduling value in Phase 2 based on the actual context processing strategy and the actual call return length, and restores the request to different queues according to the actual context processing strategy. In each round of scheduling, the dynamic token batch processing module will calculate and set the current batch processing token limit based on hardware information and the system's pause request status, while setting overload protection interval constraints. Finally, after sorting the requests according to their scheduling value, the requests are selected in turn to fill the batch until a batch is filled and then sent into the model to perform a forward iteration.

Images