An energy-saving method for large model inference based on fine-grained DVFS
By using iterative GPU frequency control and latency predictors, the problems of energy waste and service quality degradation in large language model inference services are solved, achieving highly energy-efficient GPU frequency regulation that is suitable for various models and load scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2026-04-23
- Publication Date
- 2026-05-29
Smart Images
Figure CN122114185A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to an energy-saving method for large model inference based on fine-grained DVFS. Background Technology
[0002] With the continuous development of artificial intelligence technology, Large Language Models (LLMs) have been widely used in scenarios such as intelligent question answering, text generation, code assistance, dialogue interaction, knowledge retrieval and enhanced generation, mathematical reasoning, and multimodal processing. Compared with traditional machine learning models, large language models are typically characterized by large parameter scale, long inference chains, intensive computation and storage access, and significant fluctuations in online service load. Therefore, in actual deployment, they are highly dependent on computing resources, especially graphics processing units (GPUs).
[0003] Currently, large language model inference services generally use GPUs as the primary computing unit to handle tasks such as model forward propagation, attention calculation, key-value cache read / write, and token generation. To ensure inference throughput and response speed, existing systems typically set GPUs to high operating frequencies, and in some scenarios, even maintain them at near-maximum frequencies for extended periods to maximize computational performance. However, in real-world online service scenarios, the arrival rate of inference requests is usually not constant, system load fluctuates over time, and GPUs are not always at full capacity. Furthermore, service quality requirements from the business side are not solely based on throughput but are often constrained by Service-Level Objectives (SLOs), such as Time-To-First-Token (TTFT), Time-Between-Tokens (TBT), and end-to-end latency.
[0004] In the aforementioned application scenarios, if the GPU runs at a high frequency for an extended period, significant power redundancy often occurs even when the system has performance margins, leading to energy waste. Conversely, simply reducing the GPU frequency to lower energy consumption may increase inference latency, impacting service quality and potentially causing SLO (Service Level Agreement) defaults. Therefore, how to reasonably adjust the GPU frequency while ensuring inference service latency constraints to reduce energy consumption during large language model inference and improve energy efficiency is a crucial problem that needs to be addressed in current technologies.
[0005] Existing large language model inference processes typically include a Prefill phase and a Decode phase. The Prefill phase primarily performs a one-time parallel computation on the input prompts, usually exhibiting strong computational intensive characteristics. The Decode phase, on the other hand, generates output step-by-step according to the tokens, requiring the reading of model parameters and historical key-value caches at each step, generally resembling a memory-bandwidth-constrained scenario. Because the two phases differ significantly in computational patterns, memory access patterns, and frequency sensitivity, adopting a uniform frequency strategy often fails to balance performance and energy consumption. Specifically, when the GPU frequency is set too low, the latency of the Prefill phase can increase significantly, thus affecting the speed of generating the first token; when the GPU frequency is set too high, the performance gain from frequency increases in the Decode phase is limited due to the primary constraint of memory bandwidth, while power consumption continues to increase, leading to decreased energy efficiency.
[0006] Furthermore, with the continuous evolution of large language model architectures and inference modes, existing inference workloads exhibit greater heterogeneity and dynamism. For example, the introduction of the Mixture-of-Experts (MoE) model means that different tokens may activate different expert modules during inference, resulting in more dispersed GPU memory access; inference-enhanced generation modes such as Chain-of-Thought will lengthen the output sequence, further increasing the proportion of the Decode stage; at the same time, the improvement rate of GPU computing power is generally faster than the improvement rate of GPU memory bandwidth, leading to more and more inference tasks exhibiting memory wall dominance. In such scenarios, the impact of GPU frequency changes on inference latency and power consumption is more complex, making it difficult to adapt existing fixed-frequency or coarse-grained frequency tuning strategies.
[0007] To address the aforementioned issues, some research has attempted to apply Dynamic Voltage and Frequency Scaling (DVFS) to GPU power management to optimize inference services. However, existing solutions typically suffer from the following shortcomings: First, frequency control granularity is coarse, often set only at the request, batch, or scheduling levels, making it difficult to reflect real-time load changes during inference iterations. Second, some solutions rely on output length prediction or future request load prediction, but in actual online services, request length, output length, and user behavior have high uncertainty, easily causing prediction bias and affecting the accuracy of frequency decisions. Third, some solutions fail to fully consider the differences in frequency sensitivity between the Prefill and Decode phases, often adopting a uniform frequency strategy, making it difficult to simultaneously optimize energy efficiency and latency. Fourth, some solutions introduce significant prediction and frequency switching overhead during operation; if executed synchronously on the critical inference path, this may have additional impacts on system throughput and response time, even negating energy-saving benefits.
[0008] In large language model inference systems that employ continuous batch processing, dynamic mixed batching, and online scheduling for parallel execution, a single iteration often contains inference tasks of different request types, context lengths, and stages, with frequent load changes between iterations. In such cases, using static frequency or request-level frequency control not only fails to capture energy-saving opportunities during low-load iterations but may also lead to service quality degradation due to improper frequency reduction during high-load or latency-sensitive iterations. Therefore, current technology still lacks a GPU energy-saving control scheme that combines iteration-level runtime status, service-level latency constraints, and dynamic frequency adjustment capabilities.
[0009] In summary, although existing technologies have recognized the role of GPU frequency regulation in reducing inference energy consumption, there are still problems in application scenarios for online inference services for large language models, such as coarse frequency control granularity, insufficient load adaptability, strong dependence on uncertainty, insufficient differentiation of stage features, and high operating overhead.
[0010] Therefore, those skilled in the art are dedicated to developing an energy-saving method for large-model inference based on fine-grained DVFS. This provides a technical solution that enables fine-grained adjustment of GPU frequency and reduces system power consumption during the large language model inference process while ensuring service quality. Summary of the Invention
[0011] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is to build a high-efficiency LLM service framework, which utilizes fine-grained GPU DVFS and iterative frequency optimization to reduce energy consumption as much as possible while strictly meeting the latency SLO.
[0012] To achieve the above objectives, this invention provides a large model inference energy-saving method based on fine-grained DVFS, including an iterative GPU frequency control mechanism. Before each inference iteration begins, the GPU frequency used for the iteration is dynamically selected based on the current iteration's running status, batch composition, and service-level latency budget.
[0013] Furthermore, taking service iterations as a unit and combining operational characteristics in mixed batches, including the number of Prefill Tokens, the number of Decode Tokens, the length of the Prefill context, and the length of the Decode context, execution latency at different GPU frequencies is predicted, and the lowest feasible frequency is selected while meeting latency constraints. The operational characteristics mentioned above... Furthermore, it includes an iterative frequency-delay predictor that uses the running characteristics of the current iteration and candidate frequencies to predict the iteration execution delay. The predictor is constructed using a machine learning model, and after offline sampling and training, it infers the delay values at different frequencies online.
[0014] Furthermore, the iterative features are discretized into buckets, and the frequency results are directly reused for similar states that have already been decided.
[0015] Furthermore, the scheduling and frequency decisions for the next iteration are executed in parallel with the GPU execution pipeline of the current iteration, and the frequency settings are decoupled from the main inference thread.
[0016] Existing GPU DVFS power-saving solutions typically set frequencies at the request, scheduling, or batch levels, making it difficult to adjust frequencies in real-time for the iterative process in large language model inference. Since continuous batch processing, dynamic mixed batching, and alternating prefill / decode execution are common in LLM online services, the load variation in a single iteration is significant. Using coarse-grained or static frequency control methods can easily lead to increased latency in the first token during the prefill phase, wasted energy during the decode phase, and difficulty in balancing energy saving and latency goals when batches change. This invention proposes an iterative-level GPU frequency control mechanism that dynamically selects the GPU frequency used for each inference iteration before it begins, based on the current iteration's running state, batch composition, and service-level latency budget, instead of using a uniform frequency at the request or task level. This invention decentralizes inference service control granularity to the "iteration" level. Specifically, it uses the current service iteration as a unit, combining operational characteristics such as the number of Prefill Tokens, Decode Tokens, Prefill context length, and Decode context length in the current mixed batch to predict the execution latency of that iteration at different GPU frequencies. Then, while meeting latency constraints, it selects the lowest feasible frequency, thereby achieving on-demand power supply and on-demand operation. Since the computational and memory access loads of different iterations in LLM inference are inconsistent, iteration-level control can more accurately reflect actual load changes, avoiding redundant overhead caused by a uniform frequency strategy.
[0017] Existing DVFS optimization schemes typically rely on output length prediction, request length prediction, or historical statistical models to infer the load. However, in practical LLM online services, the number of output tokens has significant uncertainty, especially in dialogue generation, inferential answering, and MoE model scenarios, where prediction errors are large and can easily affect the accuracy of frequency decisions. Furthermore, the online inference overhead of complex models is high, making them unsuitable for real-time execution in each iteration. This invention proposes an iterative frequency-latency predictor that no longer relies on output length prediction but directly utilizes the runtime characteristics and candidate frequencies of the current iteration to predict the iteration execution latency at that frequency. The predictor uses lightweight features available at runtime as input, including but not limited to: the number of Prefill tokens in the current iteration; the number of Decode tokens in the current iteration; the Prefill context length; and the Decode context length. These features characterize the computational structure and memory access pressure of the current batch, thus reflecting the sensitivity of GPU frequency changes to latency. The predictor can be built using machine learning models, such as gradient boosting tree models, XGBoost models, or other lightweight prediction models. After offline sampling training, it can quickly infer latency values at different candidate frequencies in the online phase.
[0018] While iterative control offers high precision, synchronously executing prediction, decision-making, and frequency switching in each iteration introduces significant CPU overhead. In particular, the GPU frequency switching interface itself has millisecond-level latency, which, if located on the critical path, can cause GPU waiting, thus negating energy savings. This invention introduces two levels of low-overhead runtime optimization: 1. Bucketed frequency decision cache: Discretely bucketing iterative features, directly reusing frequency results for similar states that have already been decided; 2. Asynchronous pipelining and decoupled frequency switching: Parallelizing the scheduling and frequency decision-making of the next iteration with the GPU execution of the current iteration, and decoupling the frequency setting action from the main inference thread. For continuous batch processing scenarios, adjacent iterations often exhibit strong temporal locality and feature similarity. By bucketing iterative features, similar states are mapped to the same cache key, and upon a hit, the previous frequency decision is directly reused, reducing the number of predictor calls. Simultaneously, by introducing an independent control process or asynchronous queue, frequency switching is decoupled from the main inference path, preventing scheduling, prediction, and DVFS setting from sequentially blocking GPU forward execution.
[0019] Compared with the prior art, the present invention has the following obvious substantive features and significant advantages: 1. This invention significantly improves GPU energy efficiency, reducing energy waste caused by idling and excessively high frequency operation; better adapts to continuous batch processing and dynamic mixed batch scenarios; improves overall Tokens / Joule performance without compromising latency SLO; balances service quality and energy saving goals, making it suitable for online inference system deployment.
[0020] 2. This invention avoids strong dependence on output length, improving system robustness; it can adapt to dynamic, bursty, and heterogeneous request loads; it achieves lower online prediction overhead, making it suitable for iterative real-time control; compared to complex neural prediction models, it balances accuracy and speed, reducing operational burden.
[0021] 3. This invention significantly reduces controller online overhead; increases GPU effective busy time; reduces GPU idle waiting and forward execution blocking; makes iterative DVFS engineering feasible; and improves throughput while maintaining energy efficiency.
[0022] 4. The energy-saving framework for large-model inference based on fine-grained DVFS proposed in this invention has strong practical application value and industrialization feasibility.
[0023] 4.1. Can be directly deployed on existing LLM inference systems
[0024] This invention does not require a reconstruction of the underlying inference framework. Instead, it integrates with existing inference service engines as a lightweight control layer, working collaboratively with the scheduler and GPU DVFS interface. This solution can be integrated into common large model service frameworks such as vLLM, TensorRT-LLM, and TGI, exhibiting strong compatibility and portability.
[0025] 4.2. Applicable to various models and various load scenarios.
[0026] Because this invention uses iterative runtime features for decision-making, it does not rely on a specific model structure or fixed output length, and therefore is applicable to: Dense models; MoE models; dialogue workloads; code generation workloads; complex inference workloads; and mixed high-concurrency and low-concurrency scenarios. Especially in inference tasks with a high proportion of decoding and limited memory, this solution demonstrates significant energy-saving advantages.
[0027] 4.3. The energy-saving effect is clear and easy to implement in engineering projects.
[0028] Experimental results show that, while maintaining a relatively constant SLO (Solution Time Limit), this solution can achieve a maximum energy consumption reduction of 25.4%, an average energy saving of approximately 21.5%, and an average Tokens / J efficiency improvement of 1.28 times. For data centers, cloud inference platforms, and enterprise private deployment scenarios, this means: reduced GPU power costs; reduced rack cooling pressure; improved computing power utilization; and greater value in green computing and carbon emission optimization.
[0029] 4.4. Possesses good stability and scalability.
[0030] This invention achieves stable online control capabilities through the collaborative operation of three modules: a predictor, a controller, and a low-overhead runtime. It still has room for expansion for future larger-scale models, more concurrent requests, and more complex hybrid batch processing scenarios. Further expansion is possible to: multi-GPU collaborative frequency tuning; GPU and memory frequency collaborative control; cross-node frequency coordination in distributed inference scenarios; and joint optimization with instance elastic scaling.
[0031] 4.5. It is conducive to the construction of green data centers and computing infrastructure.
[0032] Data centers are becoming increasingly sensitive to energy efficiency and carbon footprint. This invention can be used as an energy-optimizing component in LLM inference servers, helping enterprises reduce operating costs without sacrificing user experience. It aligns with the development trends of green computing, low-carbon AI, and energy conservation and emission reduction in computing infrastructure, and has significant social benefits and promising prospects for industrial application.
[0033] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description
[0034] Figure 1 This is a system architecture diagram of a preferred embodiment of the present invention; Figure 2 This is a frequency selection algorithm of a preferred embodiment of the present invention; Figure 3 This is a preferred embodiment of the ternary asynchronous pipeline mechanism of the present invention; Figure 4 This is a preferred embodiment of the decoupled multi-process DVFS of the present invention; Figure 5 This is an experimental result of energy consumption and energy efficiency of Qwen3-8B, a preferred embodiment of the present invention; Figure 6 This is an experimental result showing the SLO achievement rate of Qwen3-8B, a preferred embodiment of the present invention. Detailed Implementation
[0035] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.
[0036] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.
[0037] This document describes the design and implementation of EcoInfer, a large-model inference energy-saving framework based on fine-grained DVFS (Dynamic Voltage and Frequency Scaling). The overall goal is to build a highly energy-efficient LLM service framework that utilizes fine-grained GPU DVFS and iterative frequency optimization to minimize energy consumption while strictly meeting latency SLO (Standard Loss Limiting).
[0038] EcoInfer does not replace the underlying service engine; instead, it is integrated into the inference engine as a lightweight control layer, working in conjunction with the scheduler and the GPU DVFS interface. Its core design principle is simple: frequency decisions should be made at the service iteration level, because workload characteristics and latency sensitivities can differ significantly between iterations in continuous batch processing and mixed batch processing. To make this iterative-level control feasible, EcoInfer combines three tightly coupled components: a latency predictor for estimating iteration latency at candidate frequencies, a controller for selecting the target frequency that satisfies SLO and is energy-efficient, and a low-overhead runtime path that minimizes online decision-making and execution overhead. Figure 1 The overall system architecture was demonstrated.
[0039] 1. Overall Overview
[0040] EcoInfer sits between the LLM service runtime and the GPU DVFS interface. While the service engine remains responsible for request acceptance, continuous batch processing, and iterative execution, EcoInfer enhances this process through iterative-level frequency control. At the start of each iteration, the scheduler exposes the current runtime state to EcoInfer, including batch composition, request status, queue status, and the iterative-level latency budget derived from the current SLO constraints. Based on this information, EcoInfer determines the target GPU frequency for the current iteration and passes this decision to the runtime layer for execution via the DVFS interface. This placement allows EcoInfer to utilize the service state visible to the scheduler without intrusive modifications to the execution semantics of the underlying inference engine. The resulting control loop is scheduler-assisted and iteratively driven. For each iteration, EcoInfer first extracts a compact workload description from the current batch state. It then evaluates the behavior of the iteration at different candidate GPU frequencies, selects a target frequency that satisfies the SLO, and applies the decision before or simultaneously with execution on the corresponding GPU.
[0041] EcoInfer breaks down this control loop into three cooperating design components. First, an iterative frequency-latency predictor maps the runtime state of mixed batches and candidate GPU frequencies to latency estimates. This module addresses the issue that modern service iterations often involve both prefill and decode work, making simple phase-based modeling insufficient. Second, an SLO-aware GPU frequency controller combines these predicted latency estimates with the iterative budget provided by the scheduler to select the lowest feasible frequency from a sweet-spot-guided range of candidate frequencies. This module implements the core decision logic, translating load awareness into more energy-efficient and SLO-compliant frequency selection. Third, a low-overhead runtime layer makes this control loop feasible in latency-sensitive service systems by reducing the overhead of repeated predictions and overlapping the remaining control costs with GPU execution.
[0042] This decomposition reflects EcoInfer's three key design requirements. The system must provide accurate and lightweight online latency estimates, these estimates must be translated into safe and energy-efficient frequency decisions under heterogeneous latency constraints, and excessive CPU-side overhead must be avoided to prevent offsetting the benefits of fine-grained DVFS.
[0043] 2. Iterative Frequency-Delay Predictor
[0044] To achieve fine-grained DVFS control, EcoInfer must estimate the latency of a service iteration at different GPU frequencies before actually applying them. This capability is crucial for the controller to evaluate candidate frequencies and select the lowest frequency that still meets the latency target. Direct online trial and error is not feasible because changing frequencies on the critical path and measuring the resulting latency incurs unacceptable overhead. Therefore, EcoInfer introduces an iteration-level frequency-latency predictor, a lightweight, machine learning-driven runtime module for predicting the execution latency of the current iteration at a given GPU frequency. Designing such a predictor is not straightforward because modern LLM service engines no longer execute iterations strictly in a phase-separated manner. In continuous batching and mixed batching, prefill and decode requests may be packaged into the same iteration and executed together through attention and other forward operators. Therefore, an iteration cannot be accurately characterized by a simple prefill / decode label. Its latency actually depends on the fine-grained composition of the workload in the batch, including how many prefill and decode tokens are processed together and how long their respective contexts are. This characteristic makes coarse-grained, stage-based predictions insufficient to support runtime DVFS decisions. EcoInfer addresses this challenge by modeling latency at the iteration granularity. The predictor takes the current iteration feature and the target GPU frequency as input and outputs the prediction execution latency at that frequency. Feature design follows a system-oriented principle: the predictor should capture the minimum workload description that determines the shape of the operator tensor, as these shapes directly determine the computational and memory traffic during forward propagation, thus determining the sensitivity of iterations to GPU frequency scaling.
[0045] Based on this principle, four iterative features are used: `num_prefill_tokens`, `num_decode_tokens`, `prefill_context_length`, and `decode_context_length`. These features collectively describe the token composition and context state of the mixed batch. The number of prefill and decode tokens determines the workload contributed by various requests in the current iteration, while the corresponding context length determines the effective sequence dimension of attention and related operators. Together, they characterize the computational intensity, memory access behavior, and sensitivity to frequency changes during iteration, without requiring explicit binary stage classification.
[0046] The predictor must simultaneously meet two competing requirements. On the one hand, the prediction error must be low enough, because inaccurate delay estimation may cause the controller to choose an overly aggressive frequency, thus violating the SLO; or choose an excessively high frequency, missing energy-saving opportunities.
[0047] On the other hand, prediction overhead must be as small as possible because the predictor is invoked online in the service path, and its own cost will directly offset the benefits of fine-grained DVFS control.
[0048] Therefore, EcoInfer opted for a lightweight machine learning model rather than a hand-built parsed model or a heavier neural network predictor. Parsed models struggle to achieve accuracy in mixed-batch execution scenarios because kernel behavior depends on the interplay between token composition, context length, and device frequency. Conversely, more complex learning models, while more expressive, incur excessive runtime overhead when used online. Several candidate models were explored, including Ridge, Random Forest, and MLP, before XGBoost was ultimately chosen to train the IFLP. The XGBoost-based IFLP achieved the optimal balance between prediction accuracy and inference overhead in the scenario.
[0049] To train IFLP, offline profiling data was first collected by performing representative iterations at different GPU frequencies and recording their actual latency. These profiling samples were then used to pre-train the model before deployment. At runtime, EcoInfer performs only lightweight inference: given the current iteration features and candidate frequencies, IFLP returns the estimated latency of that iteration at the specified frequency. In experiments, this design achieved a mean absolute percentage error (MAPE) of only 1.13%, while keeping the predictor inference latency below 4ms.
[0050] Overall, the iterative frequency-delay predictor forms the basis of the EcoInfer control closed loop. By transforming the runtime state of a hybrid batch into an accurate and low-overhead delay estimate, it enables the controller to infer the feasibility of different frequency choices without expensive online probing, thus achieving a practical SLO-aware DVFS at the iterative granularity.
[0051] 3. SLO-aware GPU frequency controller
[0052] The next challenge is how to translate the predicted latency into specific GPU frequency decisions for each service iteration. EcoInfer addresses this by using an SLO-aware GPU frequency controller, aiming to minimize energy consumption per iteration while maintaining latency constraints. In general, the controller receives features from the scheduler for the current iteration, queries the predictor across a discrete set of candidate frequencies, and selects the lowest frequency that still satisfies the iteration-level SLO budget for prediction. Figure 2 This decision-making process is illustrated. Specifically, offline profiling reveals a lower bound inspired by the "sweet spot" frequency; below this bound, further frequency reduction often leads to a significant increase in latency and even an increase in total power consumption. Therefore, the search space is restricted to the interval [Freq(floor), Freq(max)], discretized in steps of 100 MHz. Within this range, lower frequencies generally mean lower power consumption without entering the significantly inefficient region on the left side of the U-shaped curve. In other words, EcoInfer selects the smallest feasible frequency from all candidate frequencies that satisfy the current iteration budget. Since the candidate set has been restricted to the range inspired by the sweet spot, this smallest feasible frequency is also the optimal energy-saving choice under the SLO constraint.
[0053] The controller also introduces a simple yet crucial load-aware protection mechanism. When the scheduler indicates that the waiting queue is not empty, the system is already under high load. In this case, priority should be given to ensuring throughput and latency margins, rather than pursuing energy saving. EcoInfer completely skips the prediction and selection process and directly sets the target frequency to Freq(max). This strategy prevents the controller from wasting optimization overhead in scenarios with significant overload, because in such scenarios, aggressive energy saving not only has limited benefits but may also exacerbate queuing delays.
[0054] Under normal load, the controller evaluates the candidate set in a predictor-guided manner. For the current iteration's feature vector, it is paired with all candidate frequencies to form a batch of predictor inputs. The predictor then performs batch inference and returns a latency estimate for each candidate frequency. Batch prediction reduces decision overhead compared to evaluating candidate frequencies one by one and aligns naturally with the predictor's inference process. Once all predicted latency is obtained, the controller applies the selection rule from the above equation to determine the target frequency. If no candidate frequency meets the latency budget, EcoInfer will also fall back to Freq(max) to maintain quality of service.
[0055] Finally, after determining the target frequency, the controller passes it to the runtime layer and executes it through the GPU DVFS interface.
[0056] Overall, the SLO-aware GPU frequency controller is the core of EcoInfer's decision-making. It connects the predictor's latency estimation with the runtime DVFS mechanism through a simple yet effective strategy: prioritizing performance under high loads, and selecting the lowest feasible frequency from a sweet spot-inspired candidate range in other cases. This strategy enables EcoInfer to translate iterative workload information into energy-efficient and SLO-compliant frequency decisions in real-time scenarios.
[0057] 4. Low-overhead runtime optimization for iterative control
[0058] While the predictors and controllers described in the previous two sections have achieved accurate and SLO-aware frequency selection, they are not sufficient on their own to make iterative DVFS practical. A direct synchronous implementation would still introduce a non-negligible CPU-side control overhead on the critical path of each service iteration. To address this challenge, EcoInfer designed two complementary runtime optimizations: first, a bucketed frequency decision cache to reduce the frequency of frequency prediction calls; and second, an asynchronous pipelining and decoupled frequency execution mechanism to hide the remaining missed path overhead during GPU execution.
[0059] (1) Bucket-based frequency decision caching
[0060] The first source of overhead is the repeated calls to the predictor during the frequency selection process. EcoInfer mitigates this cost by leveraging a key characteristic of continuous batch processing: iterative-level temporal locality. During stable decoding phases, adjacent iterations often exhibit highly similar workload characteristics. For example, when the set of running requests remains constant, the number of decode requests typically increases only slightly in context length between iterations. The workload only changes significantly when new requests enter the prefill phase or when existing requests have been generated. Therefore, many consecutive iterations have similar frequency requirements, allowing for the direct reuse of recent decisions.
[0061] Based on this observation, EcoInfer introduces a bucketed frequency decision cache. The cache key is constructed by bucketing four iterative features: num_prefill_tokens, num_decode_tokens, prefill_context_length, and decode_context_length, with a fixed bucketing granularity. The cache value is the target frequency selected by the controller for the corresponding bucket state. This cache is managed using an LRU strategy to ensure that recently occurring iterative patterns are retained and reused. When a cache hit occurs, EcoInfer directly reuses the target frequency in the cache, skipping the predictor-based selection process, thereby eliminating most of the decision overhead along the hit path.
[0062] A key design trade-off here lies in the choice of bucket granularity. If the buckets are too large, iterations with significantly different workload characteristics may map to the same cache entry, reducing decision accuracy and potentially harming SLO compliance or energy efficiency. If the buckets are too small, cache reuse opportunities decrease, resulting in a low hit rate that fails to deliver any real speedup. In our deployment, the bucket size was set to 128, retaining only the 100 most recent cache entries. This configuration strikes a good balance between reusability and accuracy, achieving an average cache hit rate of approximately 73% in practice.
[0063] (2) Asynchronous pipelining and decoupled DVFS execution
[0064] While the decision cache eliminates most predictor calls, cache misses still incur significant CPU overhead. Furthermore, even after frequency selection is complete, directly applying the target frequency via pyNVML can block the inference process and further lengthen the critical path. To address these remaining overheads, EcoInfer employs asynchronous pipelining and a decoupled DVFS execution mechanism.
[0065] The core idea is to overlap the control preparation work for the next iteration while the GPU is executing the current iteration. That is, while the GPU is performing the forward propagation of the current iteration, the CPU simultaneously completes scheduling and performs predictor-based frequency selection for the next iteration when needed. The selected target frequency is then placed in a target frequency queue. After the current GPU forward propagation is complete, the runtime can directly retrieve the prepared decision from the queue without waiting for the scheduling and control logic to execute serially. This pipelined design removes most of the scheduling and decision latency from the GPU's critical path, thereby minimizing GPU idle time and improving GPU utilization, as shown in Figure 3.
[0066] EcoInfer further decouples the inference process from DVFS execution by placing frequency execution in a separate CPU process. Specifically, as... Figure 4 As shown, the inference runtime and the frequency controller operate independently of each other. The DVFS process retrieves the target frequency from the queue and asynchronously calls pyNVML. This separation prevents the blocking behavior of the frequency setting API from interfering with the main inference process. Therefore, in typical cases, scheduling, frequency decision-making, and DVFS execution are no longer executed serially with GPU computation.
[0067] These two optimizations together make iterative control feasible. Decision caching reduces the frequency of predictor searches, while asynchronous pipelining and decoupled execution hide the remaining overhead of missed paths behind continuous GPU execution. Combined, they transform iterative DVFS from a theoretically effective but costly mechanism into a low-overhead runtime control loop that can be deployed in low-latency LLM service systems.
[0068] 5. Effectiveness Evaluation: EcoInfer was evaluated on a single-node LLM service test platform. All experiments were conducted on a high-performance inference server equipped with two Intel Xeon Platinum 8369B CPUs and eight NVIDIA A100 GPUs, each with 80 GB HBM. The GPUs were interconnected via NVLink. ShareGPT traces were used to simulate real-world online service traffic.
[0069] Figure 5 The total energy consumption and energy efficiency (TPJ) of EcoInfer and the baseline vLLM on the Qwen3-8B model are shown at different request rates. Experimental results show that the present invention achieves energy savings of up to 23.1%, with a corresponding peak TPJ improvement of 30%. Meanwhile, Figure 6 This demonstrates that EcoInfer significantly reduces energy consumption while maintaining almost complete delay guarantees.
[0070] This result demonstrates that EcoInfer successfully translates workload-related performance margins into energy savings without significantly compromising latency guarantees.
[0071] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A large-model inference energy-saving method based on fine-grained DVFS, characterized in that, This includes an iterative GPU frequency control mechanism that dynamically selects the GPU frequency to be used in each iteration before the start of each inference iteration, based on the current iteration's running status, batch composition, and service-level latency budget.
2. The energy-saving method for large model inference based on fine-grained DVFS as described in claim 1, characterized in that, Based on service iterations and combined with the operational characteristics of mixed batches, the execution latency at different GPU frequencies is predicted, and the lowest feasible frequency is selected under the premise of meeting latency constraints.
3. The energy-saving method for large model inference based on fine-grained DVFS as described in claim 2, characterized in that, The operational characteristics include the number of Prefill Tokens, the number of Decode Tokens, the length of the Prefill context, and the length of the Decode context.
4. The energy-saving method for large model inference based on fine-grained DVFS as described in claim 1, characterized in that, It includes an iterative frequency-delay predictor that uses the running characteristics of the current iteration and candidate frequencies to predict the iteration execution delay.
5. The energy-saving method for large model inference based on fine-grained DVFS as described in claim 4, characterized in that, The predictor is built using a machine learning model.
6. The energy-saving method for large model inference based on fine-grained DVFS as described in claim 5, characterized in that, The predictor employs a lightweight prediction model.
7. The energy-saving method for large model inference based on fine-grained DVFS as described in claim 5, characterized in that, The predictor includes a gradient boosting tree model and an XGBoost model.
8. The energy-saving method for large model inference based on fine-grained DVFS as described in claim 5, characterized in that, The predictor, after offline sampling and training, infers the time delay values at different frequencies online.
9. The energy-saving method for large model inference based on fine-grained DVFS as described in claim 1, characterized in that, Discretely divide the iterative features into buckets, and directly reuse the frequency results for similar states that have already been decided.
10. The energy-saving method for large model inference based on fine-grained DVFS as described in claim 1, characterized in that, The scheduling and frequency decisions for the next iteration are performed in parallel with the GPU execution of the current iteration, and the frequency settings are decoupled from the main inference thread.