Prospective reasoning method for dynamically predicting output length based on hardware load state

Abstract

A speculative reasoning method for dynamically predicting output length based on a hardware load state comprises the following steps: for a fixed batch size, obtaining a saturation prediction length capable of realizing throughput saturation of each round of reasoning; dynamically setting a predicted output length of each request according to a hardware resource load state; the hardware resource load state comprises a saturation prediction length, a batch size and a real-time video memory residual capacity; and constructing the prediction tree based on the token acceptance rate of each layer in the historical statistical prediction tree, and taking the prediction output length as the total prediction token number of the prediction tree. The invention provides a mechanism for dynamically adjusting the predicted output length on the basis of hardware load states such as GPU-accept computing power, batch size and video memory occupation, the computing power limit of a large model reasoning system is sampled and evaluated, and the optimal predicted output length of each request is dynamically determined in combination with real-time batch size and video memory occupation, so that the optimal prediction output length of each request is obtained. The GPU utilization rate can be maximized under different reasoning loads, the throughput and resource occupation are balanced, and therefore the overall reasoning performance of the system is remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, specifically to a speculative inference method based on dynamic prediction of output length under hardware load conditions. It is applicable to efficient inference systems for large-scale language models (LLMs), and in particular, improves the utilization of computing resources and throughput by dynamically controlling the predicted output length during the decoding stage. Background Technology

[0002] In recent years, with the widespread application of large-scale language models (such as GPT and LLaMA) in tasks such as dialogue generation, code completion, and text understanding, the complexity of their inference computation has increased dramatically. How to improve inference efficiency has become a key issue in current research and engineering optimization. Currently, large-scale model inference methods generally adopt a phased processing of "prefill + decode". The decoding stage is usually characterized by serialization and computational fragmentation, resulting in low GPU utilization and limiting system throughput.

[0003] To address the computational inefficiency of the decoding stage in large-scale model inference, researchers proposed the Speculative Decoding method. This method introduces an independent draft model or auxiliary prediction module to pre-generate multiple candidate token sequences, which are then batch-verified in parallel by the main model. This significantly improves the parallelism and computational efficiency of the decoding stage while maintaining the quality of the generated tokens. In practical deployments, the number of tokens predicted in speculative inference (i.e., the prediction output length) has a crucial impact on inference performance: a prediction output length that is too short limits parallel computing capabilities and reduces throughput; while a prediction output length that is too long leads to a large amount of invalid computation, increases the verification burden on the main model, and may even cause GPU-accelerator resource bottlenecks. Especially in high-volume request scenarios, a fixed prediction output length can easily lead to an excessive number of tokens being predicted, significantly increasing the GPU-accelerator load and memory overhead, affecting system stability; while in low-volume scenarios, the same prediction output length may result in insufficient GPU-accelerator utilization and wasted computational resources. Existing methods often set the prediction output length based on expert experience and generally lack the ability to adjust for inference states (such as batch size, GPU-accelerator load, etc.), making it difficult to cope with the throughput and latency trade-offs in dynamic operating environments.

[0004] To further improve the verification efficiency of predicted tokens, speculative inference methods (such as SpecInfer and CaPE) introduce prediction strategies based on draft trees. These methods no longer use fixed-length linear prediction sequences but instead construct prediction trees across multiple candidate paths, combined with a prediction token pruning mechanism to reduce the verification overhead of invalid tokens. Methods like EAGLE2 further propose a dynamic prediction tree structure, using the confidence of candidate tokens and set thresholds for dynamic pruning to improve overall acceptance. Although these speculative decoding methods control the prediction output length through dynamic pruning, they still lack a mechanism for dynamic adjustment based on system hardware conditions such as GPU accelerator utilization. This makes it difficult to effectively balance the prediction output length and hardware resource utilization under different inference loads. Furthermore, the confidence of each node in the prediction tree is obtained by multiplying the probabilities of nodes along the path; the longer the path, the lower the cumulative confidence, and lower-level nodes are more likely to be pruned prematurely, limiting the overall acceptable maximum prediction output length. Therefore, how to dynamically control the predicted output length while considering the system resource status, and at the same time improve the effective token reception length, has become one of the key challenges for further optimization of speculative inference. Summary of the Invention

[0005] To address the problem that existing speculative inference methods generally use a fixed predicted output length, which makes it difficult to adapt to different hardware resource states and batch processing scales, this invention proposes a speculative inference method based on dynamic prediction of output length according to hardware load state.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A speculative inference method for dynamically predicting output length based on hardware load status includes the following steps:

[0008] Step 1: For a fixed batch size, obtain the saturation prediction length that can achieve saturation of inference throughput in each round;

[0009] Step 2: Dynamically set the predicted output length for each request based on the hardware resource load status; the hardware resource load status includes the saturation predicted length, the batch size, and the remaining real-time video memory capacity;

[0010] Step 3: Construct a prediction tree based on the token acceptance rate of each layer in the historical statistical prediction tree, and use the predicted output length as the total number of predicted tokens in the prediction tree.

[0011] Preferably, in step 1, for the batch size B, the prediction length L of a single request is gradually increased, thereby increasing the total number of prediction tokens P. The inference throughput for each round is calculated as follows:

[0012]

[0013] Wherein, Latency(P) represents the delay, indicating the time required to complete one inference.

[0014] Preferably, when the increase in the inference throughput per round is less than a preset threshold, the total number of prediction tokens P is the saturation prediction length.

[0015] Preferably, in step 2, the KV cache space overhead required for the predicted output length under the batch size B is calculated based on the memory estimation function Mem(B,T); simultaneously, the real-time remaining memory capacity M is obtained through a software interface. free The system determines whether the resource constraints under the current hardware conditions are met based on this; the predicted output length is calculated as follows:

[0016]

[0017] Among them, P max The saturation prediction length is given.

[0018] Preferably, step 3 includes:

[0019] Define the key parameters of the prediction tree; the key parameters include:

[0020] The maximum depth is D, which represents the longest consecutive number of tokens generated for each predicted path;

[0021] The number of nodes in layer l is N. l , representing the number of candidate tokens in this layer;

[0022] The total number of predicted tokens in the prediction tree satisfies the following formula:

[0023]

[0024] Preferably, step 3 further includes:

[0025] Set minimum acceptance rate threshold r min Starting from the root node of the historical statistical prediction tree, the process unfolds layer by layer downwards. When the average acceptance rate of a certain layer is lower than the minimum acceptance rate threshold r... min When the time is reached, stop expanding and set the current layer number to the maximum depth D of the prediction tree;

[0026] The predicted number of tokens for each layer is set using an incremental node allocation strategy based on acceptance rate; the number of nodes in layer l is N. l The setting principles are as follows:

[0027]

[0028] Where r l It is the average acceptance rate of the l-th layer tokens in the historical statistical prediction tree, where δ is the regularization term; normalization yields N. l :

[0029]

[0030] Preferably, the method further includes step 4: estimating the confidence level based on the joint probability of the node's position in the prediction tree and its path, and prioritizing the expansion of nodes with high confidence levels.

[0031] Preferably, in step 4, for the candidate node n in the l-th layer of the prediction tree, its path is:

[0032]

[0033] The token sequence was generated sequentially through d(n) predictions.

[0034] Define the confidence level C(n) of node n as the joint probability of all tokens on the path of that node:

[0035]

[0036] in, This represents the probability of generating the aforementioned token sequence.

[0037] As a preferred approach, each candidate node is sorted according to its confidence level C(n), and the N node with the highest confidence level is selected. l The nodes constitute the actual node set of the l-th layer:

[0038]

[0039] Top-N l This means sorting the candidate set according to their confidence scores and retaining the top N. l One result; the set of nodes in each layer based on the previous layer. Expand the dataset, evaluate the confidence level of the newly generated nodes, and then select the subset with the highest confidence level. The entire prediction tree is constructed layer by layer until the maximum depth D is reached.

[0040] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0041] Unlike traditional speculative inference schemes that rely on experience to set a fixed prediction output length, this invention proposes a mechanism that dynamically adjusts the prediction output length based on hardware load conditions such as GPU accelerator computing power, batch size, and memory usage. This method samples and evaluates the computing power limit of the large model inference system and combines this with real-time batch size and memory usage to dynamically determine the optimal prediction output length for each request. This mechanism can maximize GPU utilization under different inference loads and balance throughput and resource consumption, thereby significantly improving the overall inference performance of the system.

[0042] To address the problem that traditional methods, which prune prediction trees solely based on confidence levels, often result in the removal of long-path nodes, thus limiting the maximum acceptable prediction length, this invention proposes a dynamic prediction tree construction method based on the token acceptance rate at each layer. This method dynamically sets the maximum depth of the prediction tree based on the token acceptance rate at each layer. Simultaneously, it employs an incremental node allocation strategy, retaining a small number of high-quality nodes in high-acceptance-rate layers near the root node to reduce redundant computation, while appropriately expanding the prediction token nodes in deeper layers where the acceptance rate rapidly declines, thereby increasing the candidate space and compensating for the loss of effective tokens due to low acceptance rates. This method not only effectively controls the size of the prediction tree but also retains more token candidates on long paths, improving the acceptance rate of long-path tokens and significantly increasing the acceptable prediction length for speculative inference. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the system framework of Embodiment 1 of the present invention. Detailed Implementation

[0044] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0045] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0046] Example 1:

[0047] like Figure 1This invention presents a speculative inference method based on dynamic prediction of output length under hardware load conditions. First, it comprehensively considers the GPU-accelerator's computing power, memory usage, and current batch size to dynamically adjust the predicted output length for each round of speculative decoding, improving resource utilization and system throughput under different load conditions. Second, it dynamically determines the depth and width of the prediction tree by statistically analyzing the token acceptance rate at each layer of the prediction tree—the average probability that all candidate tokens are accepted by the main model at a given layer. Finally, during the gradual generation of candidate tokens in the draft model, this method combines the joint probability of the paths to estimate the confidence of candidate tokens, filtering them to construct a compact prediction tree with higher verification efficiency. Through this method, the invention can dynamically control the predicted output length under diverse load scenarios, optimize the token verification path, increase the maximum acceptable length of predicted tokens, and enhance the overall performance and efficiency of large-model inference systems.

[0048] The specific steps include the following detailed steps:

[0049] Step 1: For a fixed batch size, obtain the saturation prediction length P that can achieve saturation of inference throughput in each round. max ;

[0050] To maximize the utilization of GPU-accelerator computing resources while ensuring stable throughput during the speculative inference phase, this method first performs computational sampling and performance analysis on the inference task under the current hardware environment, and evaluates the maximum number of tokens that can be supported per round of inference under the current hardware conditions, thereby providing a reasonable length upper limit for subsequent stages.

[0051] Because the KV Cache is frequently accessed and updated each time a new token is generated during the decoding phase, bandwidth bottlenecks and insufficient computational load occur. Therefore, setting an appropriate upper limit for the saturation prediction length requires obtaining the maximum number of tokens that the GPU can handle per iteration under current hardware conditions through computing power benchmark tests. The specific method is as follows: for a fixed batch size B, i.e., the number of requests processed in each inference, gradually increase the prediction length L of a single request, thereby increasing the total number of predicted tokens P, and record the throughput (tokens / s) metric for each round of inference.

[0052]

[0053] Wherein, Latency(P) represents the delay, indicating the time required to complete one inference.

[0054] As the predicted output length P increases, the throughput will initially rise rapidly, then level off until the increase is no longer significant. When the relative increase in throughput is less than a preset threshold ε (5% in this embodiment), the predicted length that makes the throughput gain tend to saturate is determined, which is the saturated predicted length P that can achieve throughput saturation under the current hardware. max :

[0055]

[0056] The saturation prediction length P max This typically corresponds to the critical point where the GPU transitions from a memory-bound state to a compute-bound state. As a basis for setting the upper limit of the prediction output length, it helps to ensure that speculative inference fully releases the hardware performance bottleneck without significantly increasing the verification phase time.

[0057] Step 2: Dynamically set the predicted output length T for each request based on the hardware resource load status; the hardware resource load status includes the saturation prediction length P. max Batch size B and remaining real-time video memory M free ;

[0058] Before each round of speculative decoding begins, this invention uses the saturation prediction length P, obtained in step 1, to make the GPU-accelerator reach its computing power limit. max Based on the current batch size B and the real-time remaining video memory capacity M free By dynamically adjusting the prediction output length for each request based on hardware resource load status, the processing capacity of each round of inference is maximized and resources are fully utilized.

[0059] Specifically, the system first calculates the KV cache space overhead required for processing batches of size B with a predicted output length of T per request, based on the memory estimation function Mem(B,T). Simultaneously, it obtains the M... free This represents the additional available video memory capacity beyond the allocated video memory under the current inference task. The system uses this to determine whether the resource constraints under the current hardware conditions are met. The final prediction output length T for each request is determined by the following formula:

[0060]

[0061] This method achieves high throughput by dynamically setting the predicted token length for each request under the dual constraints of computing power and memory resources, ensuring that resources are not overloaded. This improves the utilization of hardware resources in the speculative inference verification stage.

[0062] Step 3: Construct a prediction tree based on the token acceptance rate of each layer in the historical statistical prediction tree, and use the prediction output length T as the total number of predicted tokens in the prediction tree;

[0063] After obtaining the prediction output length T for each round, this step constructs a prediction tree structure using the token acceptance rate of each layer in the prediction tree based on historical statistics, thereby increasing the maximum acceptable length of the predicted token. Here, the token acceptance rate refers to the probability that a token generated by the draft model is validated and accepted by the main model. For each layer of this paper, the token acceptance rate refers to the average probability that a token is accepted by the main model at each step when the draft model generates a sequence along a single path, without using a prediction tree.

[0064] The prediction tree structure uses the current context state as the root node and expands the candidate tokens layer by layer to form multiple candidate paths, thereby improving the parallelism of the inference and verification phases. The core design goal of this step is to maximize the number of valid tokens ultimately accepted by the main model while keeping the prediction output length fixed.

[0065] The key parameters of the prediction tree are defined as follows:

[0066] ● The maximum depth of the tree is D, which represents the longest consecutive number of tokens generated for each predicted path;

[0067] ●The number of nodes in layer l is N l , representing the number of candidate tokens in this layer;

[0068] ● The total number of predicted tokens for the entire tree is T, which is the predicted output length dynamically adjusted in step 2, satisfying the following formula:

[0069]

[0070] The dynamic construction process of the prediction tree based on the token acceptance rate at each layer is as follows:

[0071] Maximum Depth Setting: To improve the overall acceptance rate of the predicted path and reduce redundant calculations, this step sets a minimum acceptable acceptance rate threshold r. min And using historical statistical data, the average acceptance rate r of the lth stratum l Starting from the root node of the historical statistical prediction tree, the tree expands layer by layer downwards. When the average acceptance rate of a certain layer is lower than r... min When the time is reached, stop expanding and set the current level to the maximum depth D of the prediction tree.

[0072] Node allocation strategy for each layer: To more effectively utilize the token acceptance rate characteristic, the predicted number of tokens for each layer adopts an incremental node allocation strategy based on the acceptance rate. Near the root node, the token acceptance rate is higher, so the number of candidate tokens can be reduced to decrease redundant computation; however, in deeper layers closer to the leaf nodes, the acceptance rate decreases layer by layer, requiring the addition of more candidate tokens to compensate for the lower acceptance probability. Therefore, the number of nodes N in layer l is... l The setting principles are as follows:

[0073]

[0074] Where r l It is the average acceptance rate of the l-th layer token in the statistical historical process, and δ is the regularization term (e.g., set to 10). -3 To avoid division by zero, the actual number of nodes N per layer can be obtained through normalization. l :

[0075]

[0076] Ultimately, under limited resources, we achieved joint control over the prediction tree depth and the number of tokens at each layer, constructing a dynamic prediction tree structure with strong coverage and a longer number of accepted tokens.

[0077] Step 4: Estimate the confidence level based on the node's position in the prediction tree and the joint probability on its path, and prioritize expanding nodes with high confidence levels.

[0078] To further improve the acceptance rate and resource utilization efficiency of predicted tokens, this step adopts a confidence-based dynamic prediction tree expansion method. This step estimates the confidence level based on the current node's position in the prediction tree and the joint probability along its path, thereby prioritizing the expansion of nodes that are more likely to be accepted by the main model, increasing the overall predicted token acceptance length, and avoiding resource waste on low-probability paths.

[0079] Specifically, for a candidate node n at level l in the prediction tree, its path is:

[0080]

[0081] This node starts from the context root node and generates the token sequence through d(n) predictions. This represents the probability that the draft model generates this token. To measure the confidence of this path, the confidence function C(n) of node n is defined as the joint probability of all tokens on the path of this node:

[0082]

[0083] This confidence score takes into account the depth of the node and the context formed by its ancestor tokens, and measures the probability that the path as a whole will be accepted by the main model.

[0084] During the expansion of the tree structure, the prediction tree starts from the root node and expands hierarchically. For the l-th level, it starts from the set of selected nodes in the previous level. In this process, each node generates several candidate tokens based on its context, forming a candidate node set. This step sorts the candidate nodes based on their confidence level C(n) and selects the N with the highest confidence level. l The nodes constitute the actual node set of the l-th layer: Top-N l This means sorting the candidate set according to their confidence scores and retaining the top N. l One result.

[0085] The set of nodes in each layer based on the previous layer Expand the dataset, evaluate the confidence level of the newly generated nodes, and then select the subset with the highest confidence level. The entire prediction tree is constructed layer by layer until the preset maximum tree depth D is reached.

[0086] Finally, through the aforementioned confidence-based node selection and layer-by-layer expansion process, a complete prediction tree structure is constructed. This step preserves prediction tokens with high acceptance rates, further improving the maximum acceptance length of prediction tokens and the overall throughput during the inference phase.

[0087] Comparative example:

[0088] The following is a specific application of the method of the present invention: an experiment and analysis of a speculative inference method based on dynamic prediction of output length under hardware load conditions:

[0089] The hardware environment used in the application is based on a single H100 GPU and the Ubuntu 22.04 operating system. A complete CUDA software environment was built to support AI model inference experiments. The specific environment settings are shown in Table 1 below.

[0090] Table 1:

[0091]

[0092] In the experiment, the main model used was the llama-2-7b-chat model from the LLaMA series, and the draft model used was EAGLE-llama2-chat-7B. The test dataset used was the standard inference evaluation set GSM8K, and batch processing scenarios with different input and output lengths were set up. The comparison model was the EAGLE1(base) model with batch processing capabilities, and the evaluation metric was inference throughput (Tokens / s).

[0093] Different test examples:

[0094] BatchSize InputLength OutputLength base This invention promote 8 256 512 61.75 76 123.08％ 8 512 512 66.08 77.25 116.90％ 8 1024 1024 35.94 38.87 108.15％ 16 256 128 412.49 446.62 108.27％ 32 256 128 683.8 844.67 123.53％ 32 256 512 191.97 236.74 123.32％ 32 512 512 194.46 226.32 116.38％ 32 1024 1024 107.71 107.71 110.60％

[0095] Under the same batch size, the dynamic prediction output length control mechanism proposed in this invention significantly improves inference throughput performance in medium to large-scale input / output length scenarios, with a maximum improvement of 123%. This method maximizes the utilization of GPU-accelerator resources in the main model verification stage by dynamically adjusting the prediction output length, thereby effectively improving the system throughput.

Claims

1. A speculative reasoning method based on dynamically predicting output length according to hardware load status, characterized in that, The method comprises the following steps: Step 1, for a fixed batch size, obtain a saturated prediction length capable of achieving saturated inference throughput per round; Step 2, dynamically set the prediction output length of each request according to the hardware resource load state; the hardware resource load state comprises the saturated prediction length, the batch size and the real-time memory remaining capacity; Step 3, based on the historical statistics, predict the token acceptance rate of each layer in the prediction tree, and construct the prediction tree, wherein the prediction output length is used as the total prediction token number of the prediction tree.

2. The speculative reasoning method based on dynamically predicting output length according to hardware load status of claim 1, wherein, In the step 1, for the batch size B, the prediction length L of a single request is gradually increased, thereby increasing the total prediction token number P, and the inference throughput per round is calculated as follows: Wherein, Latency(P) is the delay, which represents the time required to complete one inference.

3. The speculative reasoning method based on dynamically predicting output length according to hardware load status of claim 2, wherein, When the increase amplitude of the inference throughput per round is less than a preset threshold, the total prediction token number P is the saturated prediction length.

4. The speculative reasoning method based on dynamically predicting output length according to hardware load status of claim 3, wherein, In step 2, based on the memory estimation function Mem(B, T), the KV cache space overhead required for the predicted output length under the batch size B is calculated; at the same time, the real-time memory remaining capacity M is obtained through a software interface free Accordingly, the system judges whether the resource constraint under the current hardware condition is met; the predicted output length is calculated as follows: where P max is the saturation prediction length.

5. The speculative reasoning method based on dynamically predicting output length according to hardware load status of claim 1, wherein, The step 3 comprises: Defining the key parameters of the prediction tree; the key parameters comprise: The maximum depth is D, which represents the longest token number generated continuously by each prediction path; The number of nodes in the first layer is N l , which represents the number of candidate tokens in the layer; The total prediction token number of the prediction tree satisfies the following formula:

6. The speculative reasoning method based on dynamically predicting output length according to hardware load status of claim 5, wherein, The step 3 further comprises: Setting a minimum acceptance rate threshold r min Starting from the root node of the history statistics prediction tree, expanding layer by layer downwards, when the average acceptance rate of a certain layer is lower than the minimum acceptance rate threshold r min , stopping expansion, and setting the current layer number as the maximum depth D of the prediction tree; The prediction token number setting of each layer adopts an incremental node allocation strategy based on acceptance rate; the node number N of the lth layer l The setting principle is as follows: where r l is the lth layer token average acceptance rate of the historical statistics prediction tree, and δ is a regularization term; and N l is normalized to obtain N 7. The speculative reasoning method based on dynamically predicting output length according to hardware load status of claim 1, wherein, Further comprising step 4, according to the position of the node in the prediction tree and the joint probability estimation confidence of the path thereof, the node with high confidence is preferentially expanded.

8. The speculative reasoning method based on dynamically predicting output length according to hardware load status of claim 7, wherein, In the step 4, for a candidate node n of the lth layer in the prediction tree, the path thereof is: The token sequence is generated in turn by d(n) times of prediction Defining the confidence C(n) of the node n as the joint probability of all tokens on the path of the node: wherein, denotes the probability of generating the said token sequence.

9. The speculative reasoning method based on dynamically predicting output length according to hardware load status of claim 8, wherein, Sort the candidate nodes according to their confidence level C(n), and select the N with the highest confidence level. l The nodes constitute the actual set of nodes in the I-th layer: where Top-N l denotes that from the candidate set, rank by confidence score, keep top-N l results; in each layer, based on the node set of the last layer expand, and evaluate the confidence of the newly generated nodes, and select the highest confidence subset from them build the whole prediction tree layer by layer, until the maximum depth D is reached.

Images