Large model reasoning method and system based on active load awareness
By employing a proactive load awareness approach, utilizing long-term load prediction and adaptive tuning, and coordinating resource supply and request scheduling, the problem of sudden and heterogeneous loads in large model inference services is solved, achieving a balance between service quality and resource efficiency, and reducing latency spikes and resource waste.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2025-12-05
- Publication Date
- 2026-05-12
AI Technical Summary
Existing large-scale model inference services struggle to balance resource efficiency and service quality when faced with sudden and heterogeneous loads, leading to latency spikes, resource waste, and "front-end congestion."
We adopt a proactive load awareness approach, which uses a closed-loop paradigm of long-term load forecasting, online scheduling, and adaptive tuning to coordinate resource supply and request scheduling. We use a long-term load forecasting model to predict future load, combine response length and urgency to select and schedule instances, and dynamically adjust instance size based on peak resource usage forecasting.
It achieves a steady-state balance between service quality and resource utilization under dynamic and heterogeneous load environments, reduces latency spikes and resource waste, and improves the system's effective throughput and GPU utilization.
Smart Images
Figure CN122021869A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large model inference service technology, and in particular to a large model inference method and system based on proactive load awareness. Background Technology
[0002] In recent years, large language models (LLMs) have become the mainstream in the form of inference services (LMaaS) in cloud computing environments. These services typically need to continuously handle two core tasks in a multi-tenant environment: resource provisioning (i.e., dynamic scaling of instances) and request scheduling (i.e., cross-instance routing and intra-instance batch processing).
[0003] However, the load on large model inference services is highly complex: traffic exhibits high burstiness on short timescales, while showing periodicity on long timescales. Furthermore, user requests show significant heterogeneity in terms of prompt length, expected response length, and Service Level Objectives (SLOs).
[0004] Existing inference systems struggle to balance resource efficiency (cost and utilization) and service quality (latency and SLO) when dealing with such complex loads.
[0005] On the one hand, passive scaling strategies that rely solely on runtime metrics are prone to severe latency spikes and SLO (Solution Time Limit) defaults because large model instances have excessively long cold start times and cannot respond promptly to sudden load peaks. On the other hand, static over-provisioning strategies adopted to cope with peak loads result in a large amount of GPU resources being idle during periods of stable load (troughs), leading to high operating costs and low utilization.
[0006] Furthermore, existing request schedulers often overlook the complex heterogeneity between request length and SLO priority. This can easily lead to "head-of-line blocking," where long-running low-priority requests block short, high-priority requests, thus affecting overall service quality. If the scheduling strategy is too aggressive, attempting to batch requests at high density, it can easily trigger memory overcommitment, causing requests to be evicted or interrupted.
[0007] Therefore, there is an urgent need in this field for a new technical solution to address the technical problem of how to coordinate resource supply and request scheduling under highly bursty and heterogeneous mixed loads, so as to achieve high resource utilization while ensuring service quality (low latency and SLO). Summary of the Invention
[0008] The main objective of this invention is to provide a large model inference method and system based on active load perception to solve the problems raised in related technologies.
[0009] To achieve the above objectives, according to one aspect of the present invention, a large-scale inference method based on proactive load awareness is provided, comprising: predicting the workload for the next period using a long-term load prediction model based on historical load data; determining the basic instance size for the next period based on the predicted workload and a preset average throughput, and performing instance pre-configuration; estimating the response length of the inference request when a inference request is received; selecting a target instance for routing the inference request based on the estimated response length, the current load status of the instance, and the heterogeneity of the response length; performing intra-instance batch processing scheduling within the target instance based on the urgency of the inference request; predicting peak resource consumption during the batch processing process during instance operation; calculating the instance's resource utilization rate based on the peak resource consumption; and dynamically adjusting the instance size based on a comparison of the utilization rate with a preset threshold.
[0010] As a preferred embodiment of the present invention, the long-term load prediction model is trained using an asymmetric loss function, wherein the asymmetric loss function penalizes under-matching (predicted load is lower than actual load) more than it penalizes over-matching (predicted load is higher than actual load).
[0011] As a preferred embodiment of the present invention, the estimation of the response length of the inference request includes: maintaining a conditional probability distribution based on the prompt length and response length of historical requests; and updating the conditional probability distribution using exponential decay and Dirichlet smoothing to estimate the response length of the current inference request.
[0012] As a preferred embodiment of the present invention, the step of selecting a target instance for routing the inference request includes: comprehensively considering the pre-filling stage latency, decoding stage latency, instance congestion factor, and the mismatch between the request hint length and the average hint length of the instance queue, calculating the expected total latency of the inference request on each candidate instance; and selecting the instance with the smallest expected total latency as the target instance.
[0013] As a preferred embodiment of the present invention, the intra-instance scheduling based on the urgency of inference requests includes: calculating a stress score for each inference request, the stress score being used to quantify the urgency of the request from its predicted completion time or service level target deadline; and in batch scheduling, prioritizing the scheduling of requests with higher stress scores, or preempting requests with lower stress scores when necessary.
[0014] As a preferred embodiment of the present invention, the prediction of peak resource usage in the batch processing process includes: determining the earliest completed request in the current batch; and estimating the peak resource usage of the batch processing process based on the currently occupied resources of all requests in the batch, and the additional resource usage required before the earliest completed request is completed.
[0015] According to another aspect of the present invention, a large model inference system based on proactive load awareness includes: a long-term load prediction module for predicting the workload of the next cycle based on historical load data; a cycle planning and provisioning module for determining and pre-configuring the basic instance size for the next cycle according to the predicted workload; an online scheduling module for estimating the response length of an inference request when it is received, and routing the request to a target LLM instance based on the response length, instance status, and heterogeneity, and scheduling it within the target instance based on urgency; and an adaptive tuning module for predicting peak resource consumption during batch processing during instance operation and dynamically adjusting the instance size according to the predicted resource utilization.
[0016] As a preferred embodiment of the present invention, the long-term load prediction module is configured to be trained using an asymmetric loss function to impose a higher penalty on under-match predictions.
[0017] As a preferred embodiment of the present invention, the online scheduling module further includes: a response length sensor for estimating the response length based on historical statistics and Dirichlet smoothing; an instance router for selecting a target instance based on a heterogeneous sensing latency model; and an intra-instance scheduler for performing preemptive scheduling based on pressure scores.
[0018] As a preferred embodiment of the present invention, the adaptive tuning module further includes: a peak working set predictor, used to estimate the peak video memory of a batch by identifying the earliest completed request; and an instance reconfigurator, used to trigger capacity expansion or reclamation actions based on the utilization rate calculated from the peak video memory when it is consistently higher than a high threshold or lower than a low threshold.
[0019] Compared with the prior art, the present invention has the following beneficial effects: Through a closed-loop paradigm of "periodic planning, online scheduling, and adaptive optimization," resource supply and demand scheduling are coordinated.
[0020] During the planning phase, long-term load forecasting models are used for forward-looking instance pre-configuration. This proactively matches foreseeable periodic loads and suppresses tail latency spikes caused by passive scaling up and down and instance cold starts from the source.
[0021] During the scheduling phase, by using lightweight online response length awareness, combined with a two-layer (cross-instance and intra-instance) scheduling strategy that takes into account the heterogeneity of prompt length and the urgency of SLO, the "head-of-line blocking" phenomenon is significantly reduced, and the effective throughput of the system is improved while ensuring SLO.
[0022] During the tuning phase, precise prediction of peak working sets enabled fine-grained adjustments to instance size at runtime. This tuning approach, based on future peak memory load rather than current average load, allows the system to maintain high GPU utilization without triggering memory over-promise, avoiding resource waste caused by over-provisioning.
[0023] In summary, this solution achieves a steady-state balance between service quality (SLO satisfaction rate) and resource efficiency (GPU utilization) under dynamic and heterogeneous load environments. Attached Figure Description
[0024] Figure 1 A schematic diagram of the overall architecture of a large model inference system based on active load awareness is provided for an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the long-term load predictor and the asymmetric loss function used in an embodiment of the present invention; Figure 3 This is a system module block diagram in an embodiment of the present invention.
[0025] Explanation of reference numerals in the attached figures: 100. Long-term load forecasting module; 200. Cyclic planning and provisioning module; 300. Online scheduling module; 310. Response length sensor; 320. Instance router; 330. In-instance scheduler; 400. Adaptive tuning module; 410. Peak working set predictor; 420. Utilization evaluator; 430. Instance reconfigurator; 500. LLM inference instance. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0027] This invention provides a large-model inference system based on proactive load awareness. For example... Figure 1 As shown, this system aims to solve the challenges of resource supply and request scheduling in large-scale model inference services caused by the burstiness, periodicity, and heterogeneity of requests (such as decoding length and SLO targets) of mixed business loads. The system collaboratively optimizes resource efficiency and service quality through a closed-loop paradigm of "periodic planning, online scheduling, and adaptive tuning".
[0028] In a preferred embodiment, such as Figure 3 As shown, the system may include: Long-term load prediction module 100: corresponding Figure 1 The "Long-Term Load Forecaster" in the context of this technology. Its function is to predict workloads for a longer period in the future (e.g., the next week) based on historical arrival traffic.
[0029] Cycle Planning and Supply Module 200: Corresponding Figure 1 The "Instance Expander" module receives the prediction results from the long-term load prediction module 100, combines them with the service throughput capacity, calculates the basic instance size required for the next cycle, and performs a forward-looking "Scaleout / in" action (i.e., adding / reducing instances).
[0030] Online scheduling module 300: Responsible for handling inference requests arriving in real time. This module may further include: Response length sensor 310: Corresponding Figure 1 The "Response Length Sensor" in the example. It lightweightly estimates the response (decoding) length of a request based on historical statistics when the request arrives.
[0031] Example Router 320: Corresponding Figure 1 The "Instance Router" in the context of the request selects a target instance with the shortest expected execution latency based on the estimation results from the Response Length Perceptor 310 and the current state of each LLM instance 500 (such as load and queue heterogeneity).
[0032] In-Instance Scheduler 330 (not shown separately, located inside LLM instance 500): Responsible for batching and scheduling requests within the instance. It employs a preemptive and dynamic batching strategy, prioritizing requests with urgent SLOs to avoid "head-of-line blocking".
[0033] Adaptive tuning module 400: Responsible for fine-grained adjustments to resource configurations during runtime. This module may further include: Peak working set predictor 410: corresponding Figure 1 The "Peak KV Prediction" in this context refers to a function that runs within an LLM instance. It predicts the peak KV cache memory usage during batch processing to prevent memory over-commitment.
[0034] Utilization evaluator 420: Corresponding Figure 1 The "Utilization Assessment" in the context calculates the instance's memory utilization based on the predicted peak memory requirements.
[0035] Instance reconfigurator 430: corresponding Figure 1The "reconfiguration" function in this context determines whether an instance is under high or low load based on utilization assessment results and triggers fine-grained reconfiguration (increase / decrease) decisions. These decisions are then fed back to the lifecycle planning and provisioning module 200 (instance extender) and the instance router 320.
[0036] LLM Inference Example 500: Figure 1 The “LLM instances #1, #2, #3...” are specific computational units that perform large model inference and are usually equipped with GPUs.
[0037] Accordingly, embodiments of the present invention also provide a large model inference optimization method based on proactive load awareness, which can be executed by the aforementioned system. For example... Figure 1 As shown, this method follows a closed-loop process of "periodic planning, online scheduling, and adaptive optimization," and may specifically include the following steps: S1: Periodic Planning Phase – Proactive Load Awareness and Resource Pre-configuration The goal of this phase is to leverage the long-term cyclical patterns of the load to proactively supply resources in order to suppress latency spikes caused by cold starts.
[0038] S1.1: Long-term load forecasting The system first builds a lightweight load predictor, such as... Figure 2 The stacked backtracking-prediction model is shown.
[0039] The model analyzes the historical arrival sequence of tokens. Modeling is performed. The model includes... Each stack contains 1 stack. One residual projection block. The calculation process for each block can be represented as follows: ; ; ; in, For the input of the block, For activation functions (such as ReLU). For learnable weights, and These are backcast and forwardcast projectors, respectively. This is the interpretable part of the backtracking. This is the prediction part.
[0040] Workload forecast for the next cycle By summing the prediction outputs of all blocks, we obtain: ; To ensure that predictions better align with service objectives (i.e., "overfitting rather than underfitting"), a service-oriented asymmetric loss function is used during model training. (like Figure 2 (As shown in the top left corner) ; in, It is the prediction error (predicted value - actual value). This indicates over-matching (prediction is too high). This indicates underperformance (the forecast is too low). This is the system-defined load tolerance (acceptable short-term overload limit). It is set via... The model was guided to address severe mismatch ( They will be subject to hefty penalties.
[0041] S1.2: Instance Scale Direct Push and Pre-configuration Based on the predicted load of the next window And services based on historical data. average throughput The system directly estimates the number of instances required. : ; System calculation and current instance count The difference And trigger the expansion / contraction action: if Then generate A new instance; if Then isolate An isolated instance will not accept new requests after completing an in-transit request.
[0042] S2: Online Service Phase – Joint Scheduling of Length-Aware and Heterogeneous Instances The goal of this phase is to efficiently schedule heterogeneous real-time requests (with different prompt / response lengths and different SLOs) at a fine-grained time scale.
[0043] S2.1: Response Length Awareness When a new request (The length of the prompt is...) Upon arrival, the system first estimates its response length. System maintenance recently. The history window of the requests .
[0044] First, the conditional frequency is estimated using exponential decay and a Gaussian kernel counter. : ; in, It is the time decay factor. Control the weight diffusion of similar prompts. It is an indicator function.
[0045] Then, the conditional distribution is obtained using Dirichlet smoothing. : ; in, It is a smoothing factor that guarantees a non-zero probability for rare lengths. It is a window The set of response lengths after deduplication. The system samples from this distribution to obtain the estimated response lengths. .
[0046] S2.2: Obtain the estimated length of the shortest delay route across instances Then, instance router 320 made a request. Choose the best LLM instance The routing strategy takes into account the latency of both the pre-padding and decoding stages, and introduces the heterogeneity of the cue length as a penalty.
[0047] First, define With examples Length mismatch of requests in the queue : ; in, It is an example The average prompt length for requests in the current queue.
[0048] ask In the example The estimated execution time for the two phases is as follows: ; ; in, It is an empirical cost coefficient, which quantifies the sensitivity of the token to the pre-filling and decoding stages, respectively.
[0049] Routing decisions aim to minimize weighted execution latency, where the weights... As a congestion factor: ; in It is an example Current GPU utilization.
[0050] S2.3: Intra-instance iterative scheduling The request is routed to the instance. Then, the in-instance scheduler 330 is responsible for adding it to the batch queue. To reduce "head-of-line blocking" (long tasks blocking short tasks), the scheduler introduces a preemption mechanism and uses a "pressure score" to quantify the urgency of requests: ; in, It predicts the total response length. This is the number of decoded tokens. It is the SLO deadline. This is the execution time. The larger the value, the closer the request is to completion or the SLO deadline, and the higher the urgency.
[0051] When a new request arrives, if the current batch If the quota is not full, include it directly; otherwise, include it within the approved quota. Compare the smallest requests; if the new request... If the value is higher, preemption occurs. When a request is completed, the BATCHGEN algorithm is triggered to dynamically adjust the batch limit for the next batch. This is to maximize throughput while satisfying SLO (Standard Operating Level).
[0052] S3: Adaptive Tuning Phase – Peak Working Set Prediction and Instance Reconfiguration The goal of this phase is to make rapid, fine-grained adjustments to the instance size during runtime, based on actual decoding dynamics and memory usage.
[0053] S3.1: Peak Working Set (KV) Prediction During continuous decoding, the memory usage of the KV cache changes dynamically, typically reaching a local peak when a certain request in the batch is completed. To avoid high-overhead full-process tracking, the system only performs evaluation at the "peak candidate iteration" point.
[0054] In a batch First, find the earliest completed request. : ; in, It is a request The predicted total length, This is the decoded length.
[0055] System estimation in Upon completion, the batch Peak memory required for: ; in, It is the memory footprint (e.g., number of bytes) per token. This indicates the length, the first item. The first item is the currently used KV video memory, and the second item is all the KV video memory in the batch. The request is in Before completion (still need) Additional space required for the step.
[0056] S3.2: Utilization Assessment and Instance Reconfiguration Based on predicted peak memory Utilization evaluator 420 calculation example Memory utilization: ; in, It is an example The total capacity of video memory.
[0057] Instance reconfigurator 430 based on utilization Does it remain above the high threshold? (e.g., 0.95) or below the low threshold (e.g., 0.30) to trigger scaling up or down. The decision logic can be expressed as: ; in, At the moment of decision The number of instances is adjusted (+1 for expansion, -1 for reclamation). In the future The empirical frequency of an event occurring within a decoding iteration (or prediction window). It is the trigger sensitivity threshold (e.g., 0.10).
[0058] The first line indicates: if there is a high load ( The frequency of occurrence exceeds Then, it involves forward-looking capacity expansion.
[0059] The second line indicates: if the load is low ( The frequency of occurrence exceeds If so, then an instance will be recycled.
[0060] Through the coordinated work of these three stages, the system and method proposed in this invention transform observable load patterns (long-term periodicity, short-term length correlation) into executable supply and scheduling decisions, forming a closed loop of "planning-scheduling-optimization". This can suppress cold start latency through proactive supply and continuously optimize GPU utilization and effective throughput without triggering overload through heterogeneous sensing scheduling and adaptive correction, ensuring SLO satisfaction rate.
[0061] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A large-model inference method based on proactive load awareness, characterized in that, include: Based on historical load data, the workload for the next cycle is predicted using a long-term load forecasting model. Based on the predicted workload and the preset average throughput, the base instance size for the next cycle is determined, and instance pre-configuration is performed. When an inference request is received, the response length of the inference request is estimated; based on the estimated response length, the current load status of the instance, and the heterogeneity of the prompt length, a target instance is selected for routing the inference request; Within the target instance, in-instance batch processing scheduling is performed based on the urgency of the inference request; During instance operation, the peak resource usage during batch processing is predicted; the instance's resource utilization is calculated based on the peak resource usage; and the instance size is dynamically adjusted based on a comparison of the utilization with a preset threshold.
2. The method according to claim 1, characterized in that, The long-term load prediction model is trained using an asymmetric loss function, which penalizes underfitting more than overfitting.
3. The method according to claim 1, characterized in that, The estimation of the response length of the inference request includes: maintaining a conditional probability distribution based on the prompt length and response length of historical requests; and updating the conditional probability distribution using exponential decay and Dirichlet smoothing to estimate the response length of the current inference request.
4. The method according to claim 1, characterized in that, The step of selecting a target instance for routing the inference request includes: comprehensively considering the pre-filling stage latency, decoding stage latency, instance congestion factor, and the mismatch between the request hint length and the average hint length of the instance queue, calculating the expected total latency of the inference request on each candidate instance; and selecting the instance with the minimum expected total latency as the target instance.
5. The method according to claim 1, characterized in that, The in-instance scheduling based on the urgency of inference requests includes: calculating a stress score for each inference request, the stress score being used to quantify the urgency of the request from its predicted completion time or service level target deadline; and in batch scheduling, prioritizing requests with higher stress scores, or preempting requests with lower stress scores when necessary.
6. The method according to claim 1, characterized in that, The prediction of peak resource usage during the batch processing includes: identifying the earliest completed request in the current batch; and estimating the peak resource usage of the batch processing based on the currently occupied resources of all requests in the batch, and the additional resource usage required before the earliest completed request is completed.
7. A large-model inference system based on active load awareness, characterized in that, include: Long-term load forecasting module (100) is used to forecast the workload for the next cycle based on historical load data; The cycle planning and supply module (200) is used to determine and pre-configure the base instance size for the next cycle based on the predicted workload; An online scheduling module (300) is used to estimate the response length of an inference request when it is received, and to route the request to a target LLM instance (500) based on the response length, instance status and heterogeneity, and to schedule the request within the target instance based on urgency; and an adaptive tuning module (400) is used to predict the peak resource usage during the batch processing during instance operation, and to dynamically adjust the instance size according to the predicted resource utilization.
8. The system according to claim 7, characterized in that, The long-term load prediction module (100) is configured to be trained using an asymmetric loss function to impose a higher penalty on undermatch predictions.
9. The system according to claim 7, characterized in that, The online scheduling module (300) further includes: A response length perceptron (310) is used to estimate the response length based on historical statistics and Dirichlet smoothing; Instance router (320) is used to select target instances based on a heterogeneous awareness latency model; And an in-instance scheduler (330) for performing preemptive scheduling based on pressure scores.
10. The system according to claim 7, characterized in that, The adaptive tuning module (400) further includes: A peak working set predictor (410) is used to estimate the peak memory of a batch by identifying the earliest completed request; And an instance reconfigurator (430) for triggering capacity expansion or reclamation actions based on the utilization rate calculated based on the peak memory, when it is consistently above a high threshold or below a low threshold.