Large model adaptive batch reasoning system and method based on time delay measurement

Abstract

The invention discloses a large-model adaptive batch reasoning system based on time delay measurement, and the system comprises a request receiving and queue management module which is used for monitoring and receiving an LLM reasoning request sent by an external user or an application; the real-time delay sensing and batch monitoring module is used for measuring and analyzing performance indexes directly related to the current batch processing strategy in real time in the LLM reasoning process; the Token-by-Token batch processing and reasoning execution module is used for selecting a corresponding number of requests from the request queue according to the size of the current batch determined by the scheduler, organizing the requests into an effective calculation batch and submitting the effective calculation batch to the LLM reasoning core at the bottom layer to execute one or more decoding steps; and the double-stage batch adjustment decision module is used for obtaining the normalized time delay ratio. The invention also discloses a large-model adaptive batch reasoning method based on time delay measurement. According to the invention, substantive improvement and intelligent management of the overall performance of the LLM reasoning service are realized.

Description

Technical Field

[0001] The present invention belongs to the field of computer science and artificial intelligence technology, and specifically relates to a large-model adaptive batch inference system and method based on delay measurement. Background Art

[0002] With the rapid development of deep learning technology, large language models (LLMs) have become a research hotspot in the field of artificial intelligence and a core driver of industrial application, demonstrating outstanding performance in a wide range of tasks, including natural language understanding, text generation, intelligent question answering, and machine translation. These models, such as the GPT series, BERT series, and LLaMA series, typically contain billions or even trillions of parameters, placing extremely high demands on computing resources and storage bandwidth during both training and inference. In particular, during the inference phase—the stage where models are deployed and applied to provide services—how to effectively control costs and improve resource utilization while ensuring Quality of Service (QoS), such as low latency and high throughput, remains a key challenge facing the industry.

[0003] The core computational process of LLM reasoning, particularly for autoregressive generative tasks such as continuous text generation and conversational interaction, involves encoding the input sequence (prompt) (the prefill phase) and generating the output sequence token by token (word or subword unit) (the decoding phase). At each step in the decoding phase, the model needs to predict the next token based on the currently generated sequence. This process involves numerous matrix operations (such as the Q, K, and V matrix generation and dot product operations in the attention mechanism and linear transformations in the feedforward network (FFN)) as well as loading model parameters from high-bandwidth memory (such as HBM on a GPU) to the compute core. Due to the large number of model parameters, data loading itself consumes considerable memory bandwidth and time.

[0004] The common problems and shortcomings in the current background technology can be summarized as follows: Lack of closed-loop control based on real-time performance feedback: Most batch processing strategies are open-loop, or their feedback mechanisms are not sensitive or precise enough. In particular, for the latency of each token generation, a key metric that directly reflects inference efficiency and user experience, there is a lack of effective real-time measurement and utilization mechanisms to guide the dynamic adjustment of batch size.

[0005] Dynamic drift in the compute-memory bottleneck is difficult to capture: The performance bottleneck of LLM inference can dynamically switch between compute-bound and memory-bandwidth-bound, depending on the current batch size, sequence length combination, model architecture, and hardware characteristics. Static or simple dynamic strategies struggle to adapt to this drift. For example, when the batch size is small, the system may be compute-bound (amortization of memory access costs is low); when the batch size is too large, it may become memory-bandwidth-bound (memory bandwidth saturation).

[0006] Dependence on internal hardware state and unobservability: Some advanced scheduling methods may attempt to leverage underlying hardware metrics such as GPU SM (Streaming Multiprocessor) utilization and memory controller load. However, obtaining these metrics can be delayed and costly, and their interpretation and cross-platform generalization are problematic. Directly measuring user-facing performance metrics (such as latency) may be more straightforward and robust.

[0007] A one-size-fits-all approach conflicts with heterogeneous requests: Fixed batch sizes or simple padding strategies struggle to efficiently process heterogeneous requests with widely varying sequence lengths. Long requests can occupy batch processing resources for extended periods, blocking short requests and causing them to be delayed far beyond their necessary processing time.

[0008] The complexity and limitations of manual tuning: Many systems' batch processing parameters (such as fixed batch size and trigger thresholds for dynamic batching) require extensive experimentation and manual tuning for specific models, hardware, and expected workloads. This tuning process is time-consuming and labor-intensive, and once workload patterns change, the previously optimized parameters may no longer be applicable.

[0009] Therefore, to address the above issues, a large-model adaptive batch inference system and method based on latency measurement is provided. Summary of the Invention

[0010] To address the above-mentioned problems existing in the prior art, the present invention provides a large-model adaptive batch inference system and method based on latency measurement, which can substantially improve the overall performance (efficiency, cost, and user experience) of LLM inference services and achieve intelligent management.

[0011] To achieve the above object, the present invention provides the following technical solutions: One of the present inventions is a large model adaptive batch inference system based on latency measurement, comprising: Request receiving and queue management module, used to monitor and receive LLM inference requests sent by external users or applications; Real-time latency perception and batch monitoring module, used to measure and analyze performance indicators directly related to the current batch processing strategy in real time during LLM inference; The token-by-token batch processing and inference execution module is used to select the corresponding number of requests from the request queue according to the current batch size determined by the scheduler, organize them into a valid computation batch, and submit them to the underlying LLM inference core to execute one or more decoding steps; The two-stage batch adjustment decision module is used to obtain the normalized latency ratio and, combined with the two-stage adjustment strategy, to determine the ideal size of subsequent inference batches; a request interruption, state saving and recovery module, configured to handle the change when the dual-stage batch adjustment decision module decides to significantly reduce the target size of the current batch; Intelligent request scheduling module is used to select requests from the pending request queue to form a group of When a new batch is scheduled, the shortest job first or shortest remaining processing time first scheduling strategy is adopted to further optimize the overall system performance.

[0012] Preferably, in the request receiving and queue management module, each LLM inference request includes but is not limited to: an input text sequence, a maximum expected number of generated tokens, and sampling parameters; One or more pending request queues are maintained internally. Arriving requests are first cached in the queue and wait to be scheduled for execution. The queue adopts a first-in-first-out strategy, or is sorted in combination with subsequent intelligent request scheduling strategies.

[0013] Preferably, in the real-time delay perception and batch monitoring module, performance indicators include but are not limited to the average generation delay of each token; Latency is measured after each inference batch completes one or more decoding steps. A decoding step generates one or more new tokens for all requests in the batch.

[0014] Preferably, the real-time delay perception and batch monitoring module includes: Time-delay data acquisition unit, used to record the current batch The total time it takes to perform one decoding step ; Among them, if the batch shared within requests, and a total of tokens, the average generation delay of each token at the current moment or batch is The calculation formula is: / ; Historical delay recording unit, used to store at least the average token delay of the previous moment or the previous batch , Store latency data within a longer historical window; Normalized delay ratio calculation unit, used to calculate the current delay Delay from the previous moment ratio; The normalized delay ratio is calculated as follows: ; like , indicating that the system performance is stable; if , indicating that the performance has improved, the current batch processing strategy may still have room for optimization, or the system load has been reduced; if , indicating that the generation delay of each token has increased significantly, and the system may be experiencing a bottleneck. is the preset deterioration judgment threshold, slightly greater than 1.

[0015] Preferably, the token-by-token batch processing and inference execution module includes: The dynamic splicing unit of the request sequence is used to logically splice or align the current token sequences of multiple requests selected into the batch in the sequence dimension to form a unified input structure and send it to the LLM model for calculation; The KV cache management unit is used to efficiently access and update the corresponding KV cache for each request in the batch at each step of autoregressive decoding; The dynamic request completion and replacement unit is used to process the completion status of a request in a batch after it has been generated, and allow new requests to be added from the queue to enter the current batch.

[0016] Preferably, in the two-stage batch adjustment decision module, the two-stage adjustment strategy includes two stages: Phase 1, Exploitation Phase: Entry conditions: When the system is in this stage and the normalized delay ratio , and the exploration phase counter The preset number of exploration steps has been reached , indicating that it has run stably for a sufficient number of steps after the last round of exploration; At this stage, the algorithm believes that there is still room for current system resources, and thus adopts a more aggressive strategy to increase the batch size: ,in, is a multiplicative growth factor greater than 1; The second stage, the exploration stage: Entry condition 1: During the utilization phase, a measurement finds the normalized delay ratio ,This strongly indicates that the current batch size has begun to saturate the memory bandwidth or other bottlenecks, resulting in degraded performance; Adjustment behavior 1: In response to deterioration, rapid shrinkage, the system immediately switches to the exploration phase. First, the batch size is drastically reduced: , at the same time, reset the exploration phase counter ,in, is a divisive reduction factor greater than 1; Entry condition 2: After executing the above “rapid contraction”, or the growth condition in the utilization phase is not met but When , the system maintains or enters the exploration and fine-tuning period; Adjustment behavior 2: Cautious detection, linear slight increase, if the most recent measurement , a conservative linear strategy is adopted to increase the batch size: , each execution slightly increases the exploration phase counter ,in, is a small positive integer; Conditions for switching back to the exploitation phase: During the exploration and fine-tuning phase, Successfully achieved , and all steps during Maintain , it indicates that it has been running stably near a new, better batch size. At this time, the two-stage batch adjustment decision module can decide to switch back to the utilization stage and try a larger batch increase.

[0017] Preferably, in the request interruption, state saving and recovery module, Interrupt selection: From the currently running Select from the requests Request to interrupt; State preservation: For each interrupted request, the entire current execution context is fully preserved, including: the generated and confirmed token sequence, the entire KV cache content corresponding to the request, the internal state of the decoder, and the metadata of the original request. The internal state of the decoder includes but is not limited to the state of the random number generator and the stopped condition that has been met. Requeue: The interrupted request and its saved state are put back into the pending request queue; Seamless state recovery: When a previously interrupted request is selected by the scheduler again and added to a new batch, the token-by-token batch processing and inference execution module uses its saved state to seamlessly resume execution from the interruption point.

[0018] A second embodiment of the present invention provides a large model adaptive batch inference method based on latency measurement, comprising: Step S1: Set algorithm parameters and initialize target batch size , pending request queue , the average token delay at the previous moment , Exploration Phase Counter The system operation state is the exploration stage, in which ; Step S2: Receive the newly arrived LLM inference request and add it to the pending request queue. ; Step S3: Use the first-in-first-out strategy or combine it with the subsequent intelligent request scheduling strategy to sort the queues according to the estimated remaining generation length of each request. Sort the requests in Step S4, determine the queue Is it empty? If it is empty, wait for new requests and calculate the batch size based on the current target batch size. , from the queue The head is removed no more than requests, and the total number of tokens or total KV cache requirements do not exceed the hardware limit, forming the current batch to be executed ; Among them, the actual batch size ; Step S5: The current batch to be executed Perform one or more decoding steps to measure and calculate the average token generation delay of the current batch ; Step S6: Make a decision on batch size adjustment based on the dual-stage adjustment strategy status.

[0019] Preferably, in step S1, the algorithm parameters include: deterioration judgment threshold , explore the number of stable steps , using stage growth factors , exploration phase contraction factor and slightly increase the step size in the exploration phase .

[0020] Preferably, in step S6, the state judgment and batch size adjustment decision based on the two-stage adjustment strategy include: like Valid and , then calculate the normalized delay ratio: ; like , the delay deteriorates and enters the contraction and reset phase of the exploration phase. , , that is, execution interruption: the currently executed batch because The sharp decline is too large, Some requests in the queue are interrupted and put back into the queue ; like , latency stabilization or improvement, including: like , then it is in the fine-tuning period of the exploration stage, , ;like , then the exploration and fine-tuning are completed and the maintenance and utilization phase begins. ; like Invalid or , then it is the first iteration or the queue is empty and just recovered, in the initial stage of the exploration phase, try to increase slightly or maintain the current, or , ; exist Update historical delay: ; Check the current batch Check whether there are any requests that have been generated. If so, return the result to the user and remove it from the system. Repeat the above steps until all batches are processed.

[0021] Compared with the prior art, the present invention has the following beneficial effects: The present invention monitors the generation delay of each token in real time and uses this as the core feedback to adjust the batch size. The present invention can more promptly discover and avoid memory bandwidth saturation caused by excessively large batches or insufficient computing resource utilization caused by too small batches; the two-stage adjustment strategy enables the system to operate near the optimal batch size, thereby effectively reducing the end-to-end average delay of a single request from entering the system to completing inference; by dynamically optimizing the batch size, hardware "bubbles" are minimized and the effective utilization of computing resources is improved; the present invention introduces a novel two-stage adaptive batch control logic based on real-time latency measurement, combined with an effective request processing and state management mechanism, which can significantly improve the overall performance of large-scale language model inference services, including reducing latency, increasing throughput, and enhancing system stability and adaptability, while simplifying the complexity of deployment and operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1This is a module diagram of a large model adaptive batch inference system based on delay measurement of the present invention; Figure 2 This is a specific module diagram of the real-time delay perception and batch monitoring module in the present invention; Figure 3 This is a specific module diagram of the token-by-token batch processing and inference execution module in the present invention; Figure 4 This is a flow chart of a large model adaptive batch inference method based on delay measurement of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, a large-model adaptive batch inference system based on latency measurement includes: a request reception and queue management module 1, a real-time latency perception and batch monitoring module 2, a token-by-token batch processing and inference execution module 3, a two-stage batch adjustment decision module 4, a request interruption, state preservation and recovery module 5 and an intelligent request scheduling module 6.

[0025] The request receiving and queue management module 1 is used to monitor and receive LLM inference requests sent by external users or applications.

[0026] In the embodiment, each LLM inference request includes but is not limited to: an input text sequence, a maximum expected number of generated tokens, and sampling parameters; One or more pending request queues are maintained internally. Arriving requests are first cached in the queue and wait to be scheduled for execution. The queue adopts a first-in-first-out strategy, or is sorted in combination with subsequent intelligent request scheduling strategies.

[0027] The real-time latency perception and batch monitoring module 2 is used to measure and analyze the performance indicators directly related to the current batch processing strategy in real time during the LLM reasoning process.

[0028] In the embodiment, the performance indicators include but are not limited to the average generation delay of each token; Latency is measured after each inference batch completes one or more decoding steps. A decoding step generates one or more new tokens for all requests in the batch.

[0029] like Figure 2As shown, the real-time delay perception and batch monitoring module 2 includes: a delay data collection unit 21, a historical delay recording unit 22 and a normalized delay ratio calculation unit 23.

[0030] Delay data collection unit 21, used to record the current batch The total time it takes to perform one decoding step ; Among them, if the batch shared within requests, and a total of tokens, the average generation delay of each token at the current moment or batch is The calculation formula is: / ; The historical delay recording unit 22 is used to store the average token delay of at least the previous moment or the previous batch. , Store latency data within a longer historical window; Normalized delay ratio calculation unit 23, used to calculate the current delay Delay from the previous moment The ratio is the core basis for judging the trend of system performance changes (stable, improving, deteriorating); The normalized delay ratio is calculated as follows: ; like , indicating that the system performance is stable; if , indicating that the performance has improved, the current batch processing strategy may still have room for optimization, or the system load has been reduced; if , indicating that the generation delay of each token has increased significantly. The system may be experiencing a bottleneck, the most common of which is memory bandwidth saturation. is the preset deterioration judgment threshold, slightly greater than 1.

[0031] This module focuses directly on the macro metric (token generation latency) that ultimately impacts user experience and system efficiency, avoiding reliance on complex, difficult-to-access and interpret low-level hardware counters (such as SM utilization and memory controller queue depth). Latency data is relatively easy to obtain and directly reflects the actual impact of batch size changes on performance.

[0032] The token-by-token batch processing and inference execution module 3 is used to select the corresponding number of requests from the request queue according to the current batch size determined by the scheduler, organize them into a valid computing batch, and submit them to the underlying LLM inference core to execute one or more decoding steps.

[0033] like Figure 3 As shown, the Token-by-Token batch processing and inference execution module 3 includes: a dynamic splicing unit 31 of the request sequence, a KV cache management unit 32 and a dynamic request completion and replacement unit 33.

[0034] The request sequence dynamic concatenation unit 31 is used to logically concatenate or align the current token sequences of multiple requests selected into the batch (for the pre-population phase, this is the input prompt; for the decoding phase, this is the generated token sequence plus the next position to be predicted) in the sequence dimension to form a unified input structure, which is then fed into the LLM model for calculation. A KV cache management unit 32 is used to efficiently access and update the corresponding KV cache for each request in the batch at each step of autoregressive decoding; The dynamic request completion and replacement unit 33 is used to process the completion status of a request in a batch after it has been generated (reaching the maximum length or encountering the end symbol EOS), and (if instructed by the two-stage batch adjustment decision module 4) allow new requests to be added from the queue to enter the current batch to continuously maintain high hardware utilization.

[0035] The two-stage batch adjustment decision module 4 is used to obtain the normalized latency ratio and, in combination with the two-stage adjustment strategy, decide the ideal size of the subsequent inference batch.

[0036] In this embodiment, the two-stage adjustment strategy includes two stages: Phase 1, Exploitation Phase: Entry conditions: When the system is in this stage and the normalized delay ratio , and the exploration phase counter The preset number of exploration steps has been reached , indicating that it has run stably for a sufficient number of steps after the last round of exploration; At this stage, the algorithm believes that there is still room for current system resources, and thus adopts a more aggressive strategy to increase the batch size: ,in, is a multiplicative growth factor greater than 1; The second stage, the exploration stage: Entry condition 1: During the utilization phase, a measurement finds the normalized delay ratio ,This strongly indicates that the current batch size has begun to saturate the memory bandwidth or other bottlenecks, resulting in degraded performance; Adjustment behavior 1: In response to deterioration, rapid shrinkage, the system immediately switches to the exploration phase. First, the batch size is drastically reduced: , at the same time, reset the exploration phase counter ,in, is a divisive reduction factor greater than 1; Entry condition 2: After executing the above “rapid contraction”, or the growth condition in the utilization phase is not met but When , the system maintains or enters the exploration and fine-tuning period; Adjustment behavior 2: Cautious detection, linear slight increase, if the most recent measurement , a conservative linear strategy is adopted to increase the batch size: , each execution slightly increases the exploration phase counter ,in, is a small positive integer; Conditions for switching back to the exploitation phase: During the exploration and fine-tuning phase, Successfully achieved , and all steps during Maintain , it indicates that it has been running stably near a new and better batch size. At this time, the two-stage batch adjustment decision module 4 can decide to switch back to the utilization stage and try a larger batch increase.

[0037] The request interruption, state saving and recovery module 5 is used to process the change when the dual-stage batch adjustment decision module 4 decides to significantly reduce the target size of the current batch.

[0038] In this embodiment, the interrupt selection is: Select from the requests Request to interrupt; State preservation: For each interrupted request, the entire current execution context is fully preserved, including: the generated and confirmed token sequence, the entire KV cache content corresponding to the request, the internal state of the decoder, and the metadata of the original request. The internal state of the decoder includes but is not limited to the state of the random number generator and the stopped condition that has been met. Requeue: The interrupted request and its saved state are put back into the pending request queue; Seamless state recovery: When a previously interrupted request is selected by the scheduler again and added to a new batch, the token-by-token batch processing and inference execution module uses its saved state to seamlessly resume execution from the interruption point.

[0039] When batch size reduction is necessary, the state preservation and subsequent recovery mechanisms for interrupted requests are carefully designed to minimize overhead (for example, through zero-copy transfer of the KV cache, with the goal of increasing baseline inference latency by less than 1%). This ensures that the algorithm's flexibility during dynamic adjustments does not come at an excessive performance cost.

[0040] The intelligent request scheduling module 6 is used to select requests from the pending request queue to form a group with a size of When a new batch is scheduled, the shortest job first or shortest remaining processing time first scheduling strategy is adopted to further optimize the overall system performance.

[0041] By dynamically optimizing batch size, hardware "bubbles" (i.e., time spent idle by computing units like GPUs waiting for data or due to batch size) are minimized, effectively improving the utilization of computing resources. Even if batch sizes are temporarily reduced during the exploration phase to mitigate latency degradation, the long-term goal is to find a stable operating point that consistently provides high throughput. When computing resources are not saturated, an aggressive batch growth strategy during the utilization phase helps rapidly increase parallel processing capabilities, thereby increasing the total number of tokens or requests that the kernel system can handle per unit time.

[0042] like Figure 4 As shown in FIG, a large model adaptive batch inference method based on latency measurement includes: Step S1: Set algorithm parameters and initialize target batch size , pending request queue , the average token delay at the previous moment , Exploration Phase Counter The system operation state is the exploration stage, in which .

[0043] In the embodiment, the algorithm parameters include: deterioration judgment threshold , explore the number of stable steps , using stage growth factors , exploration phase contraction factor and slightly increase the step size in the exploration phase .

[0044] Step S2: Receive the newly arrived LLM inference request and add it to the pending request queue. .

[0045] Step S3: Use the first-in-first-out strategy or combine it with the subsequent intelligent request scheduling strategy to sort the queues according to the estimated remaining generation length of each request. Sort the requests in .

[0046] Combined with the first-in-first-out strategy or the subsequent intelligent request scheduling strategy, the present invention can give priority to requests with smaller remaining workloads, thereby significantly reducing the waiting time and completion time of short requests, improving the system's average response speed and user experience fairness. Although in extreme cases, for the sake of overall throughput and the interests of the majority of users, the tail delay of a very small number of particularly long tasks may be slightly increased due to being preempted or waiting for resources in the exploration phase, the algorithm ensures that these long tasks can still be completed correctly in the end through the interrupt state preservation and recovery mechanism, and overall avoids the "head of line blocking" effect of long tasks excessively blocking short tasks.

[0047] Step S4, determine the queue Is it empty? If it is empty, wait for new requests and calculate the batch size based on the current target batch size. , from the queue The head is removed no more than requests, and the total number of tokens or total KV cache requirements do not exceed the hardware limit, forming the current batch to be executed ; Among them, the actual batch size .

[0048] Step S5: The current batch to be executed Perform one or more decoding steps to measure and calculate the average token generation delay of the current batch .

[0049] This invention uses the generation latency of each token, a performance metric directly observable at the application layer, as feedback. This decouples the algorithm logic from the specific hardware, resulting in improved cross-platform versatility and ease of deployment. Requiring only a small number of hyperparameters that are easy to understand and adjust, it reduces the complexity of deployment and maintenance.

[0050] Step S6: Make a decision on batch size adjustment based on the dual-stage adjustment strategy status.

[0051] In an embodiment, the state determination and batch size adjustment decision based on the two-stage adjustment strategy include: like Valid and , then calculate the normalized delay ratio: ; like , the delay deteriorates and enters the contraction and reset phase of the exploration phase. , , that is, execution interruption: the currently executed batch because The sharp decline is too large, Some requests in the queue are interrupted and put back into the queue ; like , latency stabilization or improvement, including: like , then it is in the fine-tuning period of the exploration stage, , ;like , then the exploration and fine-tuning are completed and the maintenance and utilization phase begins. ; like Invalid or , then it is the first iteration or the queue is empty and just recovered, in the initial stage of the exploration phase, try to increase slightly or maintain the current, or , ; exist Update historical delay: ; Check the current batch Check whether there are any requests that have been generated. If so, return the result to the user and remove it from the system. Repeat the above steps until all batches are processed.

[0052] In order to enable those skilled in the art to more clearly understand the technical solution of the present invention, the present invention will be further described in detail below in conjunction with specific embodiments. It should be noted that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0053] Example 1: Construction and operation of a large model adaptive batch inference system based on latency measurement Request reception and queue management module 1: Use a multi-threaded or asynchronous I / O (such as Python's asyncio) web server (such as FastAPI, Flask, gRPC service) as the request entry point to receive inference requests in the form of HTTP or gRPC.

[0054] The request data structure contains: (unique identifier), (Enter token ID list), (expected maximum number of generated tokens), (such as temperature, top_p, etc.), (Request arrival timestamp).

[0055] Internally maintains a Python collections.deque or asyncio.Queue as a queue of pending requests , the new request object is encapsulated and placed in this queue.

[0056] Real-time latency perception and batch monitoring module 2: Latency measurement: After the inference execution module completes each decoding step (that is, all requests in the batch generate a token), record the start time and end time This step takes time .

[0057] Current batch information: Get the current actual execution batch The number of active requests in (Some requests may be completed midway). Assume that at this step, Requests generated tokens (usually ).

[0058] calculate : Current average token delay (if ).

[0059] history Storage: Maintain a variable within the module For storage .

[0060] calculate : (if is valid and greater than 0).

[0061] Parameter settings: For example, =1.08 (an 8% increase in latency is considered a significant degradation).

[0062] Token-by-Token Batch Processing and Inference Execution Module 3: Batch formation: based on the current target batch size And SJF scheduling results, from the queue Remove the request.

[0063] Token sequence processing: For prefill requests: Used directly to construct input.

[0064] For requests in the decoding phase: obtain the last token ID generated in the previous round as input for the current step.

[0065] Use padding technology to align sequences of different lengths within a batch to a common length (usually the length of the longest sequence in the batch) and generate the corresponding attention mask.

[0066] Inference call: Call the forward or generate_step interface of the underlying inference engine, passing in the processed batch input data and KV cache (if it is the decoding stage).

[0067] KV cache management: A mechanism similar to PagedAttention in vLLM is used to manage the KV cache. Each request is assigned a logical KV cache block, which is mapped to contiguous or non-contiguous memory pages in the physical GPU memory.

[0068] After each decoding step, the KV cache of all active requests in the batch is updated. When a request is completed or interrupted, the KV cache block occupied by it is released.

[0069] Two-stage batch adjustment decision module 4: Parameter settings: (Explore Stability 4 Steps); ; ; .

[0070] State variable: current target batch size , exploration phase counter .

[0071] Decision logic: Strictly follow the above step S6, according to , , , To update and .

[0072] Batch Bounds: Ensure Always ≥ 1 and does not exceed a preset system maximum allowable batch size Bmax_system (for example, estimated based on GPU memory).

[0073] Request interrupt, state saving and recovery module 5: Interrupt trigger and selection: When After being significantly reduced, if the currently running batch Size Significantly larger than new (for example, exceeding a certain ratio or absolute difference), an interrupt is triggered. Select The selection strategy can be: prioritize interrupting requests with the least number of generated tokens or the largest remaining estimated workload.

[0074] State preservation: for each interrupted request Create a state object SavedState, containing: req_interrupted.generated_token_ids (list) req_interrupted.kv_cache_pointers (pointing to the identifier or actual data pointer / handle of its KV cache block in the PagedAttention system) req_interrupted.sampling_state (such as random number seed status, etc.) Requeue: Requests that will be interrupted Put it back into the queue along with its SavedState object . A high priority flag can be set for it.

[0075] State recovery: When a request with SavedState is scheduled again: Put it As a historical sequence.

[0076] Restore its KV cache status in the PagedAttention system through kv_cache_pointers to ensure that new calculations can directly use these historical caches.

[0077] Restore sampling_state and add it to the new batch for subsequent token generation.

[0078] Select the intelligent request scheduling module 6 of SJF request scheduling: In the slave queue Before building the batch, All requests in Calculate its remaining length: (If it is the first time scheduling and not pre-populated, then is 0, at this time That is ; if pre-populated, then is the prompt length).

[0079] The queue The request in Sort in ascending order. Then pick the request from the head of the queue.

[0080] Suppose the system has just started up, or is in a stable state and then the load suddenly increases.

[0081] initialization: , , invalid.

[0082] Requests arrive: Multiple requests (with different prompt lengths and ) arrived one after another and entered the queue .

[0083] First batching and execution: SJF scheduling Sort.

[0084] take out requests form batch B1.

[0085] Perform one-step decoding and measure .

[0086] because Ineffective, initial slight increase into the exploration phase: (Assume ). .

[0087] .

[0088] Second batching and implementation: SJF scheduling.

[0089] take out The maximum value is 2 (the actual value may be less than 2 due to insufficient requests in the queue. ) requests constitute batch B2.

[0090] Perform one-step decoding and measure .

[0091] calculate .

[0092] Assumptions ( 1.05) (Because the fixed overhead is diluted by the increase in batch size, the latency may decrease or stabilize in the initial stage): because (1)< (is 4), continue to explore fine-tuning.

[0093] .

[0094] .

[0095] .

[0096] Subsequent iterations (exploration and fine-tuning phase): Repeat the second batching and execution. , It will gradually increase linearly. Also increased.

[0097] Entering the exploitation phase: when achieve (is 4), and the most recent .

[0098] The system determines that it is currently The vicinity is stable and switches to the utilization phase.

[0099] Next adjustment (For example, if the current =5, then ).

[0100] Growth and latency degradation during the utilization phase: The system is larger (For example, 10) is run. If the subsequent measurement continued ,but It will continue to grow exponentially (eg, 10→20→40…).

[0101] Until one time, for example When it increases to 30, the measured Make (For example, 1.12 > 1.08.) This indicates that memory bandwidth is starting to saturate and latency is significantly deteriorating.

[0102] Switch back to the exploration phase (fast contraction): because , the system immediately switches from the exploitation phase back to the exploration phase.

[0103] .

[0104] .

[0105] (If necessary and the current batch is much larger than 15, the interruption mechanism may be triggered to move some requests back to the queue).

[0106] A new round of exploration and fine-tuning: the system is based on a new, reduced (For example, 15) Start running and repeat the second batching and execution logic, perform linear micro-increment and state judgment, and try to find a new stable point near 15 or continue small optimization.

[0107] Through this continuous cycle of "(exploitation phase) bold attempts - (after latency deteriorates) rapid withdrawal - (exploration phase) cautious adjustments - (after stabilization) bold attempts again", this method can dynamically maintain the batch size within an optimal range that is compatible with the current system load and hardware capabilities.

[0108] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A large model adaptive batch inference system based on latency measurement, characterized by: include: Request receiving and queue management module, used to monitor and receive LLM inference requests sent by external users or applications; Real-time latency perception and batch monitoring module, used to measure and analyze performance indicators directly related to the current batch processing strategy in real time during LLM inference; The token-by-token batch processing and inference execution module is used to select the corresponding number of requests from the request queue according to the current batch size determined by the scheduler, organize them into a valid computation batch, and submit them to the underlying LLM inference core to execute one or more decoding steps; The two-stage batch adjustment decision module is used to obtain the normalized latency ratio and, combined with the two-stage adjustment strategy, to determine the ideal size of subsequent inference batches; a request interruption, state saving and recovery module, configured to handle the change when the dual-stage batch adjustment decision module decides to significantly reduce the target size of the current batch; Intelligent request scheduling module is used to select requests from the pending request queue to form a group of When a new batch is scheduled, the shortest job first or shortest remaining processing time first scheduling strategy is adopted to further optimize the overall system performance.

2. The large model adaptive batch inference system based on latency measurement according to claim 1, characterized in that: In the request receiving and queue management module, each LLM reasoning request includes but is not limited to: an input text sequence, a maximum expected number of generated tokens, and sampling parameters; One or more pending request queues are maintained internally. Arriving requests are first cached in the queue and wait to be scheduled for execution. The queue adopts a first-in-first-out strategy, or is sorted in combination with subsequent intelligent request scheduling strategies.

3. The large model adaptive batch inference system based on latency measurement according to claim 1, characterized in that: In the real-time delay perception and batch monitoring module, performance indicators include but are not limited to the average generation delay of each token; Latency is measured after each inference batch completes one or more decoding steps. A decoding step generates one or more new tokens for all requests in the batch.

4. The large model adaptive batch inference system based on latency measurement according to claim 3, characterized in that: The real-time delay perception and batch monitoring module includes: Time-delay data acquisition unit, used to record the current batch The total time it takes to perform one decoding step ; Among them, if the batch shared within requests, and a total of tokens, the average generation delay of each token at the current moment or batch is The calculation formula is: / ； Historical delay recording unit, used to store at least the average token delay of the previous moment or the previous batch , Store latency data within a longer historical window; Normalized delay ratio calculation unit, used to calculate the current delay Delay from the previous moment ratio; The normalized delay ratio is calculated as follows: ； like , indicating that the system performance is stable; if , indicating that the performance has improved, the current batch processing strategy may still have room for optimization, or the system load has been reduced; if , indicating that the generation delay of each token has increased significantly, and the system may be experiencing a bottleneck. is the preset deterioration judgment threshold, slightly greater than 1.

5. The large model adaptive batch inference system based on latency measurement according to claim 4, characterized in that: The token-by-token batch processing and inference execution module includes: The dynamic splicing unit of the request sequence is used to logically splice or align the current token sequences of multiple requests selected into the batch in the sequence dimension to form a unified input structure and send it to the LLM model for calculation; The KV cache management unit is used to efficiently access and update the corresponding KV cache for each request in the batch at each step of autoregressive decoding; The dynamic request completion and replacement unit is used to process the completion status of a request in a batch after it has been generated, and allow new requests to be added from the queue to enter the current batch.

6. The large model adaptive batch inference system based on latency measurement according to claim 4, characterized in that: In the two-stage batch adjustment decision module, the two-stage adjustment strategy includes two stages: Phase 1, Exploitation Phase: Entry conditions: When the system is in this stage and the normalized delay ratio , and the exploration phase counter The preset number of exploration steps has been reached , indicating that it has run stably for a sufficient number of steps after the last round of exploration; At this stage, the algorithm believes that there is still room for current system resources, and thus adopts a more aggressive strategy to increase the batch size: ,in, is a multiplicative growth factor greater than 1; The second stage, the exploration stage: Entry condition 1: During the utilization phase, a measurement finds the normalized delay ratio ,This strongly indicates that the current batch size has begun to saturate the memory bandwidth or other bottlenecks, resulting in degraded performance; Adjustment behavior 1: In response to deterioration, rapid shrinkage, the system immediately switches to the exploration phase. First, the batch size is drastically reduced: , at the same time, reset the exploration phase counter ,in, is a divisive reduction factor greater than 1; Entry condition 2: After executing the above "rapid contraction", or the growth condition in the utilization phase is not met but When , the system maintains or enters the exploration and fine-tuning period; Adjustment behavior 2: Cautious detection, linear slight increase, if the most recent measurement , a conservative linear strategy is adopted to increase the batch size: , each execution slightly increases the exploration phase counter ,in, is a small positive integer; Conditions for switching back to the exploitation phase: During the exploration and fine-tuning phase, Successfully achieved , and all steps during Maintain , it indicates that it has been running stably near a new, better batch size. At this time, the two-stage batch adjustment decision module can decide to switch back to the utilization stage and try a larger batch increase.

7. The large model adaptive batch inference system based on latency measurement according to claim 6, characterized in that: In the request interrupt, state saving and recovery module, Interrupt selection: From the currently running Select from the requests Request to interrupt; State preservation: For each interrupted request, the entire current execution context is fully preserved, including: the generated and confirmed token sequence, the entire KV cache content corresponding to the request, the internal state of the decoder, and the metadata of the original request. The internal state of the decoder includes but is not limited to the state of the random number generator and the stopped condition that has been met. Requeue: The interrupted request and its saved state are put back into the pending request queue; Seamless state recovery: When a previously interrupted request is selected by the scheduler again and added to a new batch, the token-by-token batch processing and inference execution module uses its saved state to seamlessly resume execution from the interruption point.

8. A large model adaptive batch inference method based on latency measurement, characterized by: include: Step S1: Set algorithm parameters and initialize target batch size , pending request queue , the average token delay at the previous moment , Exploration Phase Counter The system operation state is the exploration stage, in which ; Step S2: Receive the newly arrived LLM inference request and add it to the pending request queue. ; Step S3: Use the first-in-first-out strategy or combine it with the subsequent intelligent request scheduling strategy to sort the queues according to the estimated remaining generation length of each request. Sort the requests in Step S4, determine the queue Is it empty? If it is empty, wait for new requests and calculate the batch size based on the current target batch size. , from the queue The head is removed no more than requests, and the total number of tokens or total KV cache requirements do not exceed the hardware limit, forming the current batch to be executed ; Among them, the actual batch size ; Step S5: The current batch to be executed Perform one or more decoding steps to measure and calculate the average token generation delay of the current batch ; Step S6: Make a decision on batch size adjustment based on the dual-stage adjustment strategy status.

9. The large model adaptive batch inference method based on latency measurement according to claim 8, characterized in that: In step S1, the algorithm parameters include: deterioration judgment threshold , explore the number of stable steps , using stage growth factors , exploration phase contraction factor and slightly increase the step size in the exploration phase .

10. The large model adaptive batch inference method based on latency measurement according to claim 8, characterized in that: In step S6, the state judgment and batch size adjustment decision based on the two-stage adjustment strategy include: like Effective and , then calculate the normalized delay ratio: ； like , the delay deteriorates and enters the contraction and reset phase of the exploration phase. , , that is, execution interruption: the currently executed batch because The sharp decline is too large, Some requests in the queue are interrupted and put back into the queue ; like , latency stabilization or improvement, including: like , then it is in the fine-tuning period of the exploration stage, , ;like , then the exploration and fine-tuning are completed and the maintenance and utilization phase begins. ; like Invalid or , then it is the first iteration or the queue is empty and just recovered, in the initial stage of the exploration phase, try to increase slightly or maintain the current, or , ; exist Update historical delay: ; Check the current batch Check whether there are any requests that have been generated. If so, return the result to the user and remove it from the system. Repeat the above steps until all batches are processed.

Images