Model reasoning acceleration method and device, computer equipment and readable storage medium

Abstract

The invention relates to a model reasoning acceleration method and device, computer equipment and a readable storage medium. The method comprises the steps of inputting obtained initial text data into a target reasoning model for reasoning, scheduling and executing draft token generation, context token retrieval and probability calculation prediction distribution of an initial text in parallel through an asynchronous execution engine, and sequentially obtaining a draft token set, a retrieval token set and probability distribution of tokens; integrating the draft token set and the retrieval token set to generate a target draft tree; and based on a tree attention mechanism, performing recursive verification on the target grassy tree according to the probability distribution, determining an effective token sequence from the target grassy tree, updating the initial text data according to effective tokens, and executing the step of inputting the initial text data into the target reasoning model for reasoning until a preset ending condition is met. By adopting the method, the hardware resource utilization rate and the model reasoning speed can be improved on the basis of not sacrificing accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model inference acceleration, in particular to a model inference acceleration method, device, computer equipment and readable storage medium. BACKGROUND

[0002] Large language models (LLMs) have shown outstanding performance in various benchmark tests, further highlighting their practical value. Such models are mainly based on the Transformer architecture and use an autoregressive (AR) decoding method, which can effectively capture complex dependencies and generate coherent sequences. However, due to the large size of the model, large language models also face significant computational overhead and latency bottlenecks during inference. For example, the GLM-10B model runs at a speed of only 101 tokens per second on a single Nvidia A100 GPU. This poses a major challenge to the widespread deployment and application of large language models.

[0003] Therefore, there is a need for a model inference acceleration method that improves hardware resource utilization and model inference speed without sacrificing accuracy. SUMMARY

[0004] Therefore, there is a need for a model inference acceleration method that improves hardware resource utilization and model inference speed without sacrificing accuracy.

[0005] In a first aspect, the present application provides a model inference acceleration method, comprising:

[0006] obtaining initial text data;

[0007] inputting the initial text data into a target inference model for inference, scheduling parallel execution of draft token generation, retrieval context token and probability calculation prediction distribution of the initial text by an asynchronous execution engine, and sequentially obtaining a draft token set, a retrieval token set and a probability distribution of the token;

[0008] integrating the draft token set and the retrieval token set to generate a target draft tree; the nodes in the target draft tree represent integrated tokens determined based on the integration of the draft token set and the retrieval token set;

[0009] based on a tree attention mechanism, recursively verifying the target draft tree according to the probability distribution, determining an effective token sequence from the target draft tree, and updating the initial text data according to the effective token to execute the step of inputting the initial text data into the target inference model for inference until a preset end condition is met.

[0010] In one of the embodiments, the retrieving context tokens comprises:

[0011] determining a knowledge-enhanced data storage D;

[0012] obtaining a current context associated with the initial text data, extracting k token suffixes from the current context;

[0013] retrieving context tokens in the data storage D according to the k token suffixes, obtaining an initial retrieval token set of the initial text data;

[0014] for each initial retrieval token in the initial retrieval token set, performing secondary sorting according to a character length and a frequency of being retrieved of each initial retrieval token, determining candidate retrieval tokens in which the character length and the frequency of being retrieved are both in the top N from the sorted initial retrieval token set;

[0015] determining a retrieval token set according to the top N candidate retrieval tokens.

[0016] In one of the embodiments, the determining a knowledge-enhanced data storage D comprises:

[0017] obtaining an original data set associated with the initial text data;

[0018] transforming the original data set into a vector through an embedding layer in a large language model, and obtaining respective hierarchical features corresponding to each layer by respectively passing the vector through a shallow layer, an intermediate layer and a deep layer of the large language model;

[0019] splicing each hierarchical feature to obtain a spliced feature;

[0020] decoding the spliced feature by using a token decoding layer of the large language model to obtain a knowledge representation, enhancing a preset data storage D according to the knowledge representation, and obtaining a knowledge-enhanced data storage D.

[0021] In one of the embodiments, the integrating the draft token set and the retrieval token set to generate a target drafting tree comprises:

[0022] constructing a model drafting tree according to the draft token set, and constructing a retrieval tree according to the retrieval token set;

[0023] fusing the model drafting tree and the retrieval tree to obtain a fused drafting tree;

[0024] identifying at least one shared prefix of the fused drafting tree based on longest prefix matching;

[0025] For each shared prefix of the fusion draft tree, all nodes with the shared prefix in the fusion draft tree are merged into a single path to obtain a target draft tree.

[0026] In one of the embodiments, the target draft tree is recursively verified according to the probability distribution to determine an effective token sequence from the target draft tree, including:

[0027] All nodes in the target draft tree are arranged in a hierarchical traversal order to construct an attention mask matrix;

[0028] The structural features of each node in the target draft tree are encoded according to the attention mask matrix to obtain corresponding encoded features;

[0029] The probability of each node is determined according to the encoded features of each node through single forward propagation;

[0030] An effective token sequence is determined from the target draft tree according to the probability distribution and the probability of each node.

[0031] In one of the embodiments, the effective token sequence is determined from the target draft tree according to the probability distribution and the probability of each node, including:

[0032] The probability of each node is verified in sequence according to the probability distribution. If the token acceptance ratio of a target node is greater than or equal to a preset acceptance threshold, the target node and all child nodes corresponding to the target node are retained, and the child nodes are verified to determine an effective token sequence from the target draft tree. The target node is any one of the nodes; and / or,

[0033] If the token acceptance ratio of the target node is less than the preset acceptance threshold, the same level to which the target node belongs in the target draft tree is determined, all descendant nodes of the target node are removed, and unverified sibling nodes in the same level are verified.

[0034] If the token acceptance ratio of all sibling nodes in the same level of the target node is less than the preset acceptance threshold, the last accepted node is resampled.

[0035] In a second aspect, the application further provides a model inference acceleration device, including:

[0036] A data acquisition module acquires initial text data;

[0037] The parallel execution module is configured to input the initial text data into a target reasoning model for reasoning, schedule parallel execution of draft token generation, retrieval context token and probability calculation prediction distribution of the initial text by an asynchronous execution engine, and sequentially obtain a draft token set, a retrieval token set and a probability distribution of tokens;

[0038] The draft tree construction module is configured to integrate the draft token set and the retrieval token set to generate a target draft tree. A node in the target draft tree is configured to represent an integrated token determined based on integration of the draft token set and the retrieval token set.

[0039] The verification module is configured to perform recursive verification on the target draft tree based on a tree attention mechanism according to the probability distribution, determine an effective token sequence from the target draft tree, and update the initial text data according to the effective token to perform the step of inputting the initial text data into the target reasoning model for reasoning until a preset ending condition is met.

[0040] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0041] obtaining initial text data;

[0042] inputting the initial text data into a target reasoning model for reasoning, scheduling parallel execution of draft token generation, retrieval context token and probability calculation prediction distribution of the initial text by an asynchronous execution engine, and sequentially obtaining a draft token set, a retrieval token set and a probability distribution of tokens;

[0043] integrating the draft token set and the retrieval token set to generate a target draft tree. A node in the target draft tree is configured to represent an integrated token determined based on integration of the draft token set and the retrieval token set.

[0044] performing recursive verification on the target draft tree based on a tree attention mechanism according to the probability distribution, determining an effective token sequence from the target draft tree, and updating the initial text data according to the effective token to perform the step of inputting the initial text data into the target reasoning model for reasoning until a preset ending condition is met.

[0045] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the following steps:

[0046] obtaining initial text data;

[0047] inputting the initial text data into a target reasoning model for reasoning, scheduling parallel execution of draft token generation, retrieval context token and probability calculation prediction distribution of the initial text by an asynchronous execution engine, sequentially obtaining a draft token set, a retrieval token set and a probability distribution of tokens;

[0048] integrating the draft token set and the retrieval token set to generate a target drafting tree; a node in the target drafting tree is used to represent an integrated token determined based on integration of the draft token set and the retrieval token set;

[0049] based on a tree attention mechanism, recursively verifying the target drafting tree according to the probability distribution, determining an effective token sequence from the target drafting tree, and updating the initial text data according to the effective token to execute the step of inputting the initial text data into a target reasoning model for reasoning until a preset ending condition is met.

[0050] In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the following steps:

[0051] obtaining initial text data;

[0052] inputting the initial text data into a target reasoning model for reasoning, scheduling parallel execution of draft token generation, retrieval context token and probability calculation prediction distribution of the initial text by an asynchronous execution engine, sequentially obtaining a draft token set, a retrieval token set and a probability distribution of tokens;

[0053] integrating the draft token set and the retrieval token set to generate a target drafting tree; a node in the target drafting tree is used to represent an integrated token determined based on integration of the draft token set and the retrieval token set;

[0054] based on a tree attention mechanism, recursively verifying the target drafting tree according to the probability distribution, determining an effective token sequence from the target drafting tree, and updating the initial text data according to the effective token to execute the step of inputting the initial text data into a target reasoning model for reasoning until a preset ending condition is met.

[0055] The aforementioned model inference acceleration methods, devices, computer equipment, computer-readable storage media, and computer program products, during the model inference acceleration process, utilize an asynchronous execution engine to schedule and execute the initial text draft token generation, context token retrieval, and probability calculation prediction distribution in parallel. The draft token set and the retrieved token set are integrated to generate a target draft tree. Based on a tree attention mechanism, the target draft tree is recursively verified according to the probability distribution to determine the valid token sequence. The initial text data is updated based on the valid tokens, and the inference is repeated until a preset termination condition is met. By parallelizing the three subtasks of draft token generation, context retrieval, and probability distribution calculation to improve the concurrent utilization of hardware computing units, and by using a tree attention mechanism to recursively verify the entire tree to confirm multiple valid tokens at once while maintaining the semantic consistency of the original model's probability distribution, the technical effect of effectively alleviating the computational overhead and latency bottleneck in the large language model inference process without sacrificing output accuracy can be achieved, resulting in a dual improvement in inference speed and hardware resource utilization. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a diagram illustrating the application environment of a model inference acceleration method in one embodiment.

[0058] Figure 2 This is a flowchart illustrating the pre-retrieval and pre-verification process in one embodiment;

[0059] Figure 3 This is a schematic diagram illustrating the multi-engine parallel design of Talon's core architecture in one embodiment.

[0060] Figure 4 This is a flowchart illustrating a method for retrieving a context token in one embodiment;

[0061] Figure 5 This is a schematic diagram illustrating the construction of data storage D in one embodiment;

[0062] Figure 6 This is a schematic diagram of a hybrid drafting strategy process in one embodiment;

[0063] Figure 7 This is a schematic diagram of a recursive verification strategy based on a tree attention mechanism in one embodiment;

[0064] Figure 8 This is a schematic diagram of the overall process of a model inference acceleration method in one embodiment;

[0065] Figure 9 This is a schematic diagram of the pre-retrieval and post-verification process in one embodiment;

[0066] Figure 10 This is a schematic diagram of a model inference acceleration framework in one embodiment;

[0067] Figure 11 This is a structural block diagram of a model inference acceleration device in one embodiment;

[0068] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0070] With the development of large-scale language models, they also face significant computational overhead and latency bottlenecks during inference. To alleviate these problems, speculative decoding is proposed to accelerate LLM inference without sacrificing accuracy. This strategy uses a draft model to efficiently generate k tokens as predictions for future decoding steps in the LLM. The LLM then verifies these tokens, and successfully verified tokens are adopted as the decoding results, thus ensuring the quality of the generated content. By concentrating computational resources on verifying pre-generated tokens, speculative decoding significantly reduces the memory operations required to access LLM parameters, ultimately improving overall inference efficiency.

[0071] However, while speculative decoding in this approach shows significant acceleration potential through its "draft-verification" framework, the mutual waiting problem caused by the sequential dependency between the draft generation and verification stages leads to low hardware resource utilization. Furthermore, retrieval-based draft generation methods perform poorly in general domains, and model-based draft generation methods are less efficient in knowledge-intensive scenarios, resulting in poor acceleration of model inference.

[0072] Therefore, there is a need for a model inference acceleration method that can improve hardware resource utilization and model inference speed without sacrificing accuracy.

[0073] In one embodiment, such as Figure 1As shown, a method for accelerating model inference is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0074] Step 102: Obtain initial text data.

[0075] The initial text data, also known as the input prefix x, can serve as the starting text for the model's inference process, guiding the generation of subsequent token sequences. The initial text data can be obtained, but is not limited to, reading the initial text string from user input, system cache, or external interfaces.

[0076] Step 104: Input the initial text data into the target inference model for inference. The asynchronous execution engine schedules and executes the initial text draft token generation, retrieves context tokens, and calculates the probability prediction distribution in parallel, thereby obtaining the draft token set, the retrieved token set, and the probability distribution of the tokens in sequence.

[0077] Understandably, during the target model inference acceleration process, although the draft model Md generates γ speculative tokens (i.e., draft tokens) through the γ forward pass sequence, the target model Mp remains idle despite having access to the current prefix x. Simultaneously, while the target model Mp verifies γ candidate tokens (i.e., retrieves tokens), the draft model Md remains idle, waiting for verification to complete. These factors lead to underutilized computing resources, reduced hardware resource utilization, and an impact on model inference speed.

[0078] The target inference model can be a large-scale language model based on the Transformer architecture. This model can be used to perform language understanding and generation tasks, outputting the probability distribution of tokens and the generation results (e.g., image recognition, data prediction). The asynchronous execution engine can be a runtime control component that schedules multiple computational subtasks to execute in a non-blocking, parallel manner. It can be used to decouple high-latency subtasks and improve the concurrent utilization of hardware computing units such as GPUs. For example, the asynchronous execution engine can dynamically allocate computing resources to execute independent subtasks based on event-driven or task queue mechanisms.

[0079] The draft tokens in the draft token set are generated by the draft model based on the initial text of the input; for example, the draft model Md is generated by... Forward pass order generation Speculative tokens (i.e., draft tokens). Retrieval tokens in the retrieval token set can be retrieved from external or internal memory via a context-based retrieval mechanism using the retrieval drafter Mr. The probability distribution of the tokens can be the predicted probability vector of the target inference model for the next token. For example, in a text generation task, the target model calculates the probability of each possible subsequent token based on the currently input text (input prefix), and then selects the token with the highest probability as the next content to be generated, thus progressively generating complete text. The probability distribution of the tokens can be determined by multiple speculative tokens and retrieval tokens. Candidate tokens are validated, and the final valid sequence is output. It should be noted that the model can consider only tokens with high probabilities based on the probability distribution P, while ignoring tokens with extremely low probabilities, thereby reducing unnecessary computation and improving search efficiency. Based on the probability distribution P, multiple... The verification method for candidate tokens can be, but is not limited to, a recursive verification strategy, or the use of multiple different models. Candidate tokens can be evaluated or verified based on probability thresholds, etc. Probability threshold-based verification can be performed on each... A probability threshold is set for candidate tokens. If the probability of a candidate token in the probability distribution P is higher than the threshold, the token is considered valid; otherwise, it is considered invalid.

[0080] For example, inputting initial text data into the target inference model for inference can be achieved by encoding the text into an embedding vector, feeding it into the target inference model for forward computation, and using an asynchronous execution engine for concurrent scheduling. Each task independently computes one of three subtasks: draft token generation, context token retrieval, and probability calculation to predict the distribution. Each task's processing result is obtained independently. This approach, driven by concurrent execution, eliminates serial dependencies, solves the idle problem of the target model during draft token generation, and addresses the idle state of the draft model during the verification phase, thereby improving hardware resource utilization.

[0081] In one exemplary embodiment, such as Figure 2 The diagram shows the pre-retrieval and pre-validation flowchart. By obtaining the current prefix x of the input, the asynchronous execution engine schedules concurrently, independently calculating three subtasks: initial text draft token generation, retrieval context token, and probability calculation and prediction distribution. Specifically, the target model Mp pre-calculates the probability distribution based on the current prefix of the input, obtaining the probability distribution P. The draft model generates γ speculative tokens (also called draft tokens) based on the current prefix of the input through a γ forward pass sequence. The retrieval drafter Mr retrieves the retrieval tokens based on the current prefix of the input through suffix matching. Multiple γ candidate tokens are obtained by integrating the draft tokens and retrieval tokens, and the probability distribution P is used to evaluate these multiple γ candidate tokens. Candidate tokens are validated and decisions are made, resulting in a valid sequence. Based on this valid sequence, a target drafting tree can be constructed. A tree attention mechanism is used to recursively validate the target drafting tree, determining the valid token sequence. The initial text data is then updated based on the valid tokens, and the initial text data is input into the target inference model for inference, continuing until a preset termination condition is met. In an exemplary embodiment, such as... Figure 3 As shown, a schematic diagram of Talon's core architecture, employing a multi-engine parallel design, is provided. For each round of model inference, the Pre-Retrieve & Pre-Verify process involves parallel execution of draft token generation, context token retrieval, and probability calculation to predict the distribution. Here, Xd represents the draft token generated by the Draft model, Xr represents the retrieval token generated by the retrieval model, y is the prediction token LLMs token generated by the base model (i.e., the target model), p represents the probability distribution predicted by the target model, √ indicates that the token has been received, × indicates that the token has not been received, and the crossed-out circle represents that the subsequent process will not proceed.

[0082] Step 106: Integrate the draft token set and the retrieval token set to generate the target draft tree; the nodes in the target draft tree are used to represent the integrated tokens determined based on the integration of the draft token set and the retrieval token set.

[0083] The target drafting tree can be a tree-like multi-path candidate structure constructed by fusing draft tokens and retrieval tokens. The target drafting tree can represent multiple possible generation branches and supports the simultaneous verification of multiple tokens. The integration method can be, but is not limited to, merging the prefix-shared paths of draft and retrieval tokens to form a branched tree data structure; it can also be achieved through existing methods, which will not be elaborated upon here. The integration token can be the node content in the target drafting tree representing the fused draft and retrieval information.

[0084] For example, integrating the draft token set and the retrieval token set to generate the target draft tree can be achieved by comparing the commonalities of the prefixes of the two sets, merging the differences in branches, and constructing a tree structure. For instance, this can be achieved by merging shared prefixes and retaining divergent paths as child nodes based on a Trie tree structure, or by using dynamic programming to align the sequences of the two sets and construct a minimum common supersequence tree, thereby forming a compact multi-candidate path representation and reducing redundant computation.

[0085] Step 108: Based on the tree attention mechanism, recursively verify the target drafting tree according to the probability distribution, determine the valid token sequence from the target drafting tree, update the initial text data according to the valid tokens, and perform the step of inputting the initial text data into the target inference model for inference until the preset termination condition is met.

[0086] The tree attention mechanism can be a computational method suitable for tree-like structures, supporting recursive verification of the entire tree. It can be used to batch-confirm valid tokens while maintaining semantic consistency of the original model. The recursive verification method can be a recursive verification strategy based on the tree attention mechanism, which enables a single forward computation of the label probability for all nodes in the draft tree. Specifically, it can traverse the tree nodes in hierarchical order, using the attention mask matrix M to simultaneously encode two key structural features: a positional embedding vector representing the hierarchical depth and explicit parent-child dependencies.

[0087] A valid token sequence can be a subsequence of tokens that has been verified by the tree attention mechanism to conform to the distribution of the target model. A valid token sequence can be used as a reliable output to be appended to the initial text data to advance the next round of inference. A preset termination condition can be a set of rules that control the termination of inference. The preset termination condition can be used to determine when the inference process stops, avoiding infinite generation.

[0088] For example, based on the tree attention mechanism, recursively verifying the target drafting tree according to the probability distribution can start from the root node, recursively calculate the joint probability of tokens on each path, and compare it with a threshold. Recursively calculating the joint probability of tokens on each path and comparing it with the threshold can be done by traversing the tree nodes in hierarchical order, using the attention mask matrix M to simultaneously encode two key structural features, confirming multiple valid tokens, and then outputting a sequence of valid tokens. Further, updating the initial text data based on valid tokens and performing the step of inputting the initial text data into the target inference model for inference until a preset termination condition is met can be achieved by appending valid tokens to the initial text to form new input, repeating the inference loop until the termination condition is triggered. The probability distribution can be used to filter the tokens corresponding to each node in the target drafting tree to initially select tokens that meet the requirements, then calculate the probability of the target node in the target drafting tree, calculate the proportion of tokens accepted for each target node xi (Ptarget / Qdraft), and compare it with a preset acceptance threshold. A comparison is made. A node is rejected when the ratio does not meet the threshold requirement.

[0089] Taking the real-time response of an online dialogue system as an example, the initial output text is: "Please help me write an application letter." By calling the target inference model, the asynchronous execution engine starts simultaneously: 1) The draft module quickly generates variations starting with "Dear..." or "Hello!..."; 2) The retrieval module extracts relevant template fragments from the user's historical dialogues; 3) The main model calculates the probability distribution of the next token. The results of these three are merged into a target draft tree. After verification by the tree attention mechanism, "Dear XXXX:" is confirmed as a valid sequence. This sequence is appended back to the input, and the next round of inference begins until a complete letter is generated and the EOS token termination condition is triggered. The entire process confirms multiple tokens in a single iteration, significantly reducing GPU idle waiting time.

[0090] In the aforementioned model inference acceleration method, during the model inference acceleration process, the asynchronous execution engine schedules the parallel execution of draft token generation, context token retrieval, and probability calculation to predict the distribution of the initial text. The draft token set and the retrieved token set are integrated to generate the target draft tree. The asynchronous execution paradigm decouples draft generation and verification, effectively eliminating the synchronization bottleneck. Based on the tree attention mechanism, the target draft tree is recursively verified according to the probability distribution to determine the valid token sequence. The initial text data is updated according to the valid tokens, and the inference is repeated until the preset termination condition is met. By parallelizing the three subtasks of draft token generation, context retrieval, and probability distribution calculation to improve the concurrent utilization of hardware computing units, and by using the tree attention mechanism to recursively verify the entire tree to confirm multiple valid tokens at once while maintaining the semantic consistency of the original model's probability distribution, the computational overhead and latency bottleneck in the large language model inference process can be effectively alleviated without sacrificing output accuracy, achieving the technical effect of improving both inference speed and hardware resource utilization.

[0091] In one exemplary embodiment, such as Figure 4 As shown, a method for retrieving context tokens is provided, specifically including the following:

[0092] Step 402: Determine the knowledge-enhanced data storage D.

[0093] Specifically, the knowledge-enhanced data storage D can be a structured, indexed database storing external knowledge. This can provide highly relevant and low-redundancy external knowledge sources for retrieval context tokens, such as knowledge information in the medical or financial fields. Alternatively, the knowledge-enhanced data storage D can be a selected or constructed structured knowledge base that supports token-level indexing and efficient retrieval. This can be achieved by pre-injecting specific domain knowledge and establishing a corresponding index structure, thereby providing external knowledge support for context retrieval.

[0094] Step 404: Obtain the current context associated with the initial text data, and extract k token suffixes from the current context.

[0095] Among them, the k token suffixes can be a fixed number of consecutive token sequences truncated from the end of the current context, which can be used as retrieval keys to represent the key semantic features of the current generation state and to match relevant entries in the knowledge base.

[0096] Step 406: Retrieve context tokens from data storage D based on the k token suffixes to obtain the initial retrieval token set for the initial text data.

[0097] For example, using k token suffixes as query conditions, a similarity or exact match retrieval is performed in the knowledge-enhanced data storage D. Context tokens are retrieved in data storage D based on the k token suffixes to obtain the initial retrieval token set for the initial text data.

[0098] Step 408: For each initial retrieval token in the initial retrieval token set, perform a secondary sorting based on the character length and detection frequency of each initial retrieval token, and select the candidate retrieval tokens from the sorted initial retrieval token set whose character length and detection frequency are both in the top N.

[0099] The character length can be the number of characters contained in a single search token, and the search frequency can be the cumulative number of times the token is hit during the search process or the statistical frequency.

[0100] For example, for each initial retrieval token in the initial retrieval token set, a secondary sort is performed based on the character length and detection frequency of each initial retrieval token. This can be achieved by calculating the character length and detection frequency of each token and sorting them jointly along these two dimensions. The subset of tokens in the sorted set that simultaneously satisfy both the character length and detection frequency ranking in the top N positions can be used as representatives of high-quality, highly reusable semantic units, thereby constructing the final retrieval token set.

[0101] Step 410: Determine the set of retrieval tokens based on the top N candidate retrieval tokens.

[0102] For example, a dynamic suffix matching retrieval algorithm is used to achieve real-time knowledge acquisition and determine the retrieval token set. For instance, given the current context s=(x1,…,xt), the system extracts ℓ-token suffixes for precise matching, constructing a frequency-weighted candidate set S={(tj,fj)}. This process not only boasts fast retrieval speed but, more importantly, naturally filters out the most relevant candidate tokens through frequency statistics.

[0103] In the above embodiments, by determining the knowledge-enhanced data storage D, the current context associated with the initial text data is obtained, and k token suffixes are extracted from it. Based on the k token suffixes, an initial retrieval token set is obtained by retrieving data storage D. Then, each initial retrieval token is sorted a second time according to its character length and the frequency of being detected. Candidate retrieval tokens with both character length and frequency of being detected are selected from the top N. Finally, the retrieval token set is determined accordingly. By introducing a structured external knowledge base as the retrieval source, using lightweight suffixes as query keys, and combining a two-dimensional sorting and intersection filtering mechanism, the accuracy of the retrieval token set is improved and the cross-domain token acceptance rate is increased.

[0104] To dynamically integrate model-based tag generation and retrieval-based tag matching to maximize acceptance rate, this strategy fully leverages the predictive power of the Md drafting model and combines it with context-sensitive tags in the knowledge-enhanced data repository D to form an efficient integration scheme. Therefore, an adaptive hybrid drafting strategy is proposed. In an exemplary embodiment, a method for storing knowledge-enhanced data D is provided, specifically including:

[0105] Obtain the original dataset associated with the initial text data; transform the original dataset into vectors through the embedding layer in the large language model; pass the vectors through the shallow, intermediate, and deep layers of the large language model to obtain the corresponding layered features for each layer; concatenate the layered features to obtain the concatenated features; decode the concatenated features using the token decoding layer of the large language model to obtain the knowledge representation; enhance the preset data storage D based on the knowledge representation to obtain the knowledge-enhanced data storage D.

[0106] Understandably, Talon's hybrid drafting strategy leverages the dual advantages of deep fusion model prediction and retrieval matching to construct an adaptive intelligent drafting system. The core architecture of this strategy is built upon knowledge-enhanced data storage (D). The construction of this adaptive intelligent drafting system employs a method based on surface-level distribution filling using a tokenizer, distribution alignment of multi-layer outputs from the target model, layer-specific token injection, and hierarchical feature fusion. Surface-level distribution filling based on a tokenizer: A tokenizer is a tool that segments text into individual words or tokens. Surface-level distribution filling refers to analyzing the surface features of the text (such as word frequency, part-of-speech, etc.) based on tokenization and filling in the distribution information of these features according to certain rules or models. Distribution alignment of multi-layer outputs from the target model: The target model typically has multiple hidden layers, and the output of each hidden layer contains feature information at different levels. Distribution alignment refers to adjusting the distribution of these different layer outputs to a similar state. Layer-specific token injection: Injecting specific token information into different layers of the model. Hierarchical feature fusion: Effectively integrating features extracted from different layers of the model. Features at different layers have different levels of abstraction. Lower-level features typically contain more detailed information, while higher-level features are more abstract and semantic. In other words, hierarchical feature fusion generates enriched tokens that capture both syntactic accuracy and semantic richness by connecting the hidden states of different Transformer layers and applying nonlinear transformations. This multi-level fusion ensures that the retrieval token maintains both the accuracy of the underlying syntactic structure and the coherence of the higher-level semantics, laying the foundation for high-quality drafting. For example, in image recognition tasks, hierarchical feature fusion can combine lower-level edge and texture features with higher-level object category features, improving the accuracy of image recognition.

[0107] The original dataset can be an external text collection that is related to the initial text data in terms of domain or semantics. It can be used as a source for constructing the knowledge-enhanced data storage D, providing the knowledge content to be encoded.

[0108] Each layer of features can be an intermediate activation representation output from the shallow, intermediate, and deep layers of a large language model, respectively. These can carry semantic information at different levels of abstraction for subsequent fusion. The shallow layers of a large language model can be the first few layers near the input in a Transformer architecture, used to capture local syntactic structure and surface formal features. For example, the shallow layers of a large language model can include, but are not limited to, the first and second layers of the Transformer block, low-level self-attention modules, and surface feedforward networks.

[0109] The intermediate layers of a large language model can be several layers within the Transformer architecture, used to encode semantic compositions and contextual dependencies at a moderate level of abstraction. For example, the intermediate layers of a large language model can include, but are not limited to, Transformer blocks at layers 3-6, cross-sentence semantic integration layers, and topic transition layers.

[0110] The deeper layers of a large language model can be the later layers near the output in a Transformer architecture, which can be used to extract highly abstract global semantics and task-related representations. For example, the deeper layers of a large language model can include, but are not limited to, Transformer blocks at layers 7-12, high-level inference layers, and semantic cohesion layers.

[0111] For example, an initial unenhanced structured knowledge base, i.e., a preset data storage D, is obtained. The original dataset is transformed into vectors through the embedding layer in the large language model. The vectors are then passed through the shallow, intermediate, and deep layers of the large language model to obtain the corresponding layered features for each layer. The layered features are concatenated to obtain concatenated features. The concatenated features are decoded using the token decoding layer of the large language model to obtain knowledge representations. The preset data storage D is then enhanced based on the knowledge representations to obtain knowledge-enhanced data storage D.

[0112] In one exemplary embodiment, such as Figure 5 The diagram shows the construction of data storage D. Green represents the Computed layer, white represents the Skipped layer, arrows represent jumping directly from the last green layer to the last layer, and yellow represents the Combined layer to be connected. Figure 5 The diagram illustrates how the original dataset is transformed into vectors through the embedding layer in the large language model. These vectors are then processed through shallow, intermediate, and deep layers, as well as multiple layers of shallow, intermediate, and deep concatenation. Finally, the output is obtained through the token decoding layer to enrich the knowledge retrieval base. Furthermore, the detection results are sorted a second time based on their length and the frequency of the retrieved tokens. The top N tokens in terms of frequency and length are selected as candidate retrieval results.

[0113] In the above embodiments, the concatenated features are decoded by the token decoding layer of the large language model to obtain a knowledge representation. The preset data storage D is then enhanced based on the knowledge representation to obtain a knowledge-enhanced data storage D. By utilizing the shallow, intermediate, and deep layers of the large language model to extract semantic features of different abstract granularities, and concatenating them, the model's own token decoding layer reconstructs them into a knowledge representation. This enhances the semantics of the preset data storage D, embedding a deep semantic association highly aligned with the target inference model within the knowledge-enhanced data storage D. This significantly improves the semantic matching accuracy and relevance in subsequent retrieval stages, makes the retrieval token more accurate and has higher information density, and maximizes the acceptance rate.

[0114] Based on the above, a knowledge-enhanced data storage D is constructed, and a retrieval token set is determined using a dynamic suffix matching retrieval algorithm based on data storage D. The following step involves integrating the obtained retrieval token set and draft token set.

[0115] In one exemplary embodiment, the draft token set and the retrieval token set are integrated to generate a target drafting tree, including:

[0116] A model drafting tree is constructed based on the set of draft tokens, and a retrieval tree is constructed based on the set of retrieval tokens. The model drafting tree and the retrieval tree are merged to obtain a fused drafting tree. Based on the longest prefix matching, at least one shared prefix of the fused drafting tree is identified. For each shared prefix of the fused drafting tree, all nodes with shared prefixes in the fused drafting tree are merged into a single path to obtain the target drafting tree.

[0117] The model drafting tree can be a tree structure constructed from a set of draft tokens, used to represent candidate branches of the draft generation path. The model drafting tree can be obtained by inserting the draft tokens into the tree structure sequentially, merging shared prefixes into common nodes, or by constructing it sequentially using a Trie tree insertion algorithm; no specific limitation is made here. For example, a tree structure organized according to certain semantic and syntactic rules in the draft token set yields the model drafting tree. The root node in the model drafting tree can be the current input prefix, and child nodes other than the root node can represent draft tokens or semantic information associated with draft tokens.

[0118] A retrieval tree can be a tree-like structure constructed from a set of retrieval tokens, used to represent candidate sequence paths retrieved from context or knowledge sources. The retrieval tree can, but is limited to, being obtained by organizing the set of retrieval tokens into a tree structure semantically or temporally, preserving the diversity of the original retrieval results. The root node of the retrieval tree can be the current input prefix, and child nodes other than the root node can represent the corresponding retrieval tokens or keywords. The model draft tree and the retrieval tree are merged, which can be done by starting from the root node and merging all paths of the two trees, preserving their respective branch structures.

[0119] Furthermore, frequency pruning can be applied to the constructed search tree to simplify its structure and improve the efficiency of subsequent processing. Frequency pruning counts the frequency of each search term (token) in the document collection. Search terms with extremely low frequencies have little impact on search results. Therefore, these low-frequency search terms can be removed from the index, reducing its size, increasing retrieval speed, and consequently improving model inference speed.

[0120] For example, a model draft tree is constructed based on a set of draft tokens, and a retrieval tree is constructed based on a set of retrieval tokens. The model draft tree and the retrieval tree are then merged to obtain a merged draft tree. For this merged draft tree, the longest prefix matching (LPM) technique is used to identify shared prefixes. All nodes in the merged draft tree with shared prefixes are merged into a single path to obtain the target draft tree. For example, if a user inputs "implement quicksort using Python", the system generates a set of draft tokens such as ['defquicksort', 'defqsort', 'functionsort'], and a set of retrieval tokens such as ['defquick_sort', 'defpartition']. The model draft tree and the retrieval tree are constructed separately and then merged into a merged draft tree, where "def" is a shared prefix for multiple paths. The shared prefix is ​​identified using longest prefix matching, and all branches starting with "def" are merged into a single path to form the target draft tree.

[0121] In an exemplary embodiment, based on the aforementioned data storage D, context token retrieval and dual-tree fusion are performed, such as... Figure 6 As shown, a flowchart of a hybrid drafting strategy is provided, which specifically includes: constructing a knowledge base (i.e., constructing data storage D), which can be processed through the shallow, intermediate and deep layers of a large language model and the output of LLM-Head to obtain the hierarchical features of each layer, and then the hierarchical features of each layer are fused by multi-layer hidden states, i.e., spliced, to obtain spliced ​​features and thus obtain the knowledge-enhanced data storage D.

[0122] Based on the defined knowledge-enhanced data storage D, a retrieval operation is performed on the initial text data using a retrieval drafter. Specifically, a candidate set S (i.e., a set of retrieval tokens) is constructed through suffix matching retrieval. An initial retrieval tree Tr is then constructed based on the retrieval token set, and frequency pruning is applied to obtain the final retrieval tree. This initial retrieval tree is then fused with the model drafting tree Traft, determined by the draft token set, to obtain the target drafting tree Tcombine. The specific implementation can be achieved as described above and will not be elaborated upon here. It is understandable that this approach maximizes acceptance rate by dynamically integrating model-based tag generation with retrieval-based tag matching. This strategy fully leverages the predictive power of the Md drafting model and combines it with context-relevant tags in the knowledge-enhanced data storage D to form an efficient integration scheme.

[0123] In the above embodiments, by employing the longest prefix matching (LPM) technique to identify shared prefixes and merging matching segments into a single path, redundant computation can be effectively eliminated, significantly improving processing efficiency. The entire fusion mechanism not only solves the problem of secondary growth in forward propagation time in large language models but also ensures semantic consistency of the generated sequences through structured optimization. While preserving the semantic integrity of multi-source candidates, it significantly reduces the number of nodes and memory accesses required during recursive verification of the tree attention mechanism, thereby improving hardware resource utilization.

[0124] In one exemplary embodiment, a recursive verification strategy based on a tree attention mechanism is provided, such as... Figure 7 The diagram shows a flowchart of a recursive verification strategy based on a tree attention mechanism, which includes:

[0125] Based on the tree attention mechanism, the target drafting tree is recursively verified according to the probability distribution to determine the valid token sequence from the target drafting tree. This includes: arranging all nodes in the target drafting tree in hierarchical traversal order to construct an attention mask matrix; encoding the structural features of each node in the target drafting tree according to the attention mask matrix to obtain their corresponding encoded features; determining the probability of each node based on the encoded features of each node through a single forward propagation; and determining the valid token sequence from the target drafting tree by applying the SpecInfer criterion, based on the probability distribution and the probability of each node.

[0126] In this process, nodes in the target drafting tree represent an integration token, and their attribute information can include the token's position within the generation path. The hierarchical traversal order can be a sequence of node visits from root to leaf and from left to right within the same level, providing a deterministic node arrangement for constructing the attention mask matrix and ensuring that structural dependencies are correctly modeled. The attention mask matrix can be a binary mask tensor used to constrain the legal attention range in tree attention computation. It can be used to mask illegal cross-branch attention connections, preserve contextual dependencies within the path, and guarantee semantic consistency. For example, the attention mask matrix can set positions where attention flow is allowed to be 1 and others to 0, based on the hierarchical traversal order and node parent-child / sibling relationships.

[0127] Structural features include vectorized representations of a node's position in the target drafting tree and its topological relationships with other nodes. These topological relationships can be explicit parent-child dependencies. In other words, the attention mask matrix M simultaneously encodes two key structural features: positional embeddings representing hierarchical depth and explicit parent-child dependencies. A single forward propagation verification can perform a complete model forward computation in parallel on the encoded features of all nodes in the target drafting tree. This allows for the output of the probabilities of all nodes in the entire tree at once, avoiding repetitive computation per token.

[0128] The valid token sequence is determined by comparing the calculated acceptance rate of each token with a preset acceptance threshold. The acceptance rate can be determined by the ratio of the predictions from the large model to the small model. The large model prediction can be the probability calculated by the target model, while the small model prediction can be the prediction output from the draft model.

[0129] Furthermore, based on the probability distribution and the probability of each node, a valid token sequence is determined from the target drafting tree, including at least one of the following:

[0130] The probability of each node is verified sequentially according to the probability distribution. If the proportion of tokens received by the target node is greater than or equal to the preset acceptance threshold, the target node and all child nodes (i.e., subtrees) corresponding to the target node are retained. Verification processing (i.e., continuing depth-first processing) is performed on all child nodes to determine the valid token sequence from the target drafting tree. The target node is any node among the nodes.

[0131] The probability of each node includes Ptarget calculated using the target model and its probability Qdraft calculated using the smaller model. The token acceptance rate can be the ratio of the token corresponding to the target node to its probability Qdraft calculated using the smaller model and its Ptarget calculated using the target model. A preset acceptance threshold is also included. This can be a pre-defined confidence threshold used to determine whether a token is acceptable. If the token acceptance rate is greater than or equal to the preset acceptance threshold, the target node is accepted; otherwise, it is rejected. This significantly improves inference speed by reducing the number of calculations in the target model. The probability distribution obtained from the pre-calculated probability distribution of the target model is used to initially screen tokens in the target drafting tree, selecting tokens that meet the model's expectations, and then calculating the token acceptance rate for the target node.

[0132] If the percentage of tokens received by the target node is less than the preset acceptance threshold, then the target node is determined to be at the same level in the target drafting tree, all descendant nodes of the target node are removed, and unverified sibling nodes at the same level are verified; if the percentage of tokens received by all sibling nodes at the same level of the target node is less than the preset acceptance threshold, then sampling is performed from the last accepted node.

[0133] In this context, "same level" can be the set of all nodes in the target drafting tree that have the same depth as the target node. Descendant nodes can be all child nodes of the target node and all lower-level nodes formed by recursively expanding their child nodes. Sibling nodes can be other nodes that share the same parent node as the target node and are at the same level.

[0134] Understandably, the system first arranges all nodes in the target drafting tree in hierarchical traversal order, constructing a special attention mask matrix M. This matrix simultaneously encodes two key structural features: positional embeddings representing hierarchical depth and explicit parent-child dependencies. A recursive depth-first algorithm is used to validate each traversed target node. Specifically, for each target node xi, the system calculates the percentage of accepted fusion tokens corresponding to the target node, where τ is a preset acceptance threshold. When a target node is rejected, the system automatically removes all descendant subtrees of that node and moves on to the next unvalidated sibling node at the same level for further processing. Another core advantage of this validation strategy lies in its robust error recovery capability and quality control mechanism: when all sibling nodes at the same level are rejected, the system does not get stuck but automatically starts a resampling procedure from the last accepted ancestor node.

[0135] For example, if the percentage of tokens received by the target node is greater than or equal to the preset acceptance threshold, the target node and all child nodes corresponding to the target node are retained. When the verification is successful, the target node and its subtree are marked as valid and included in the subsequent processing scope. Verification processing is performed on all child nodes. This can be achieved by taking the retained child nodes as new target nodes, recursively executing the verification logic, and then implementing deep verification by continuing the valid path.

[0136] If the token acceptance rate of a target node is less than a preset acceptance threshold, the target node is rejected, and all descendant nodes of the target node are removed. This can be done by pruning all branch nodes below the target node from the target drafting tree. The same level to which the target node belongs in the target drafting tree is determined, and unverified sibling nodes in the same level are verified. This can be done by continuing to evaluate the remaining unprocessed sibling nodes in the current level. If the token acceptance rate of all sibling nodes in the same level of the target node is less than the preset acceptance threshold, sampling is performed again from the last accepted node. This can be done by falling back to the most recent valid node when the entire layer of verification fails, triggering a new round of draft generation and tree construction.

[0137] In the recursive verification strategy based on the tree attention mechanism described above, by constructing an attention mask matrix M, the target model can complete the probability calculation of all nodes in the entire drafting tree through a single forward propagation, completely changing the inefficient mode of traditional node-by-node verification. When the target node is rejected, the system automatically removes all descendant subtrees of that node and turns to the next unverified sibling node at the same level for further processing. This not only significantly reduces computational overhead, but more importantly, it maintains the semantic coherence of the verification process, ensuring that only token sequences that conform to the distribution characteristics of the target model will be ultimately accepted. Even in the face of consecutive verification failures, the system can still maintain the generation progress, avoiding the sequence interruption problem common in traditional methods. By fully utilizing the original distribution characteristics of the target model Mp through the resampling process, new tokens that conform to the contextual semantics are generated, thereby maintaining the lossless quality of generation.

[0138] In an exemplary embodiment, before integrating the draft token set and the retrieval token set to generate the target draft tree, the draft token set and the retrieval token set can be initially screened according to the probability distribution of the tokens to select tokens that meet the model quality requirements. The target draft tree is then constructed based on the tokens that meet the model quality requirements. The target draft tree is recursively verified based on the tree attention mechanism to determine the valid token sequence from the target draft tree. The initial text data is updated according to the valid tokens, and the step of inputting the initial text data into the target inference model for inference is executed until the preset termination condition is met. The specific implementation method can be implemented in the manner defined above, and will not be elaborated here.

[0139] In one exemplary embodiment, such as Figure 8 As shown, a schematic diagram of the overall process of a model inference acceleration method is provided, which specifically includes:

[0140] The initial text data, i.e., the input prefix, is acquired. An asynchronous execution engine schedules and executes in parallel the generation of draft tokens, retrieval of context tokens, and probability calculation of the predicted distribution. Specifically, the model drafter Md generates corresponding speculative tokens (i.e., draft tokens) to obtain a draft token set, and the retrieval drafter Mr retrieves context tokens to obtain a retrieval token set. The draft token set and the retrieval token set are integrated to generate a target drafting tree. A validation decision unit validates the target drafter based on the probability distribution obtained from the probability calculation of the predicted distribution, and judges the validation result. If the validation passes (acceptance), a valid token sequence is output, and the input prefix is ​​updated according to this valid token sequence (updated input prefix × + new token), and then the next round of decoding is executed. If the validation fails, resampling is required to update the input prefix × + new token, and then the next round of decoding is executed until a preset end condition is met, ending the model inference.

[0141] Understandably, based on this, idle resources can be utilized during the verification phase, such as... Figure 9 The diagram shows the pre-retrieval and post-verification process. Md generates subsequent tokens for the next round of drafting, Mr retrieves context tokens to prepare for the next round of chaining, and the verification probability is calculated through the target model Mp for parallel verification processing. This eliminates the idle state of the draft model during the verification phase. The above asynchronous speculative decoding architecture decouples the draft generation and verification processes in time, eliminates the mutual waiting problem, achieves full utilization of the pipeline, improves resource utilization, and accelerates the model inference speed.

[0142] It should be noted that the embodiments described above are applicable to large model inference based on the Transformer framework, such as Vicuna-13B, LLaMA-Instruct-3.1-8B, and DeepSeek-R1-LLaMA-8B. First, select a pre-trained large language model that requires speculative decoding. Second, select the corresponding draft model architecture. Train the model using the ShareGPT and ULTRACHAT-200K datasets, employing the AdamW optimizer. After training the draft model, it can be used to accelerate inference during the inference process.

[0143] In this embodiment, by decoupling the draft generation and verification processes in an asynchronous execution architecture, the synchronous bottleneck is eliminated, i.e., the underutilized computing resources are parallelized to maximize the hardware utilization in the speculative decoding process. Furthermore, an adaptive hybrid strategy is adopted to dynamically fuse model prediction and retrieval methods, thereby maximizing the label acceptance rate in different domains. On this basis, not only is the generation quality maintained, but it also exhibits a 4.04x–6.52x speedup performance in benchmark tests and model families, far exceeding the existing Eagle2 and Eagle3, thus improving the model inference speed.

[0144] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0145] Based on the same inventive concept, this application also provides a model inference acceleration framework for implementing the model inference acceleration method described above. The solution provided by this framework is similar to the implementation scheme described in the above method; therefore, the specific limitations of one or more model inference acceleration framework embodiments provided below can be found in the limitations of the model inference acceleration method described above, and will not be repeated here.

[0146] In one exemplary embodiment, such as Figure 10 As shown, a model inference acceleration framework is provided, including: an input module, an asynchronous execution engine, a model drafter, a retrieval drafter, and a validation decision maker, wherein:

[0147] The input module is used to input initial text data; when the initial text data is input into the target inference model, the asynchronous execution engine 1004 performs the initial text draft token generation, retrieves the context token, and calculates the probability prediction distribution by parallel scheduling of the parallel execution model drafter 1006 and the retrieval drafter, thereby obtaining the draft token set, the retrieval token set, and the probability distribution of the tokens in sequence.

[0148] By integrating the draft token set and the retrieval token set, a target draft tree is generated. Nodes in the target draft tree represent the integrated tokens determined based on the integration of the draft token set and the retrieval token set. The verification decision-maker is called through the asynchronous execution engine. Based on the tree attention mechanism, the target draft tree is recursively verified according to the probability distribution. The valid token sequence is determined from the target draft tree. The initial text data is updated according to the valid tokens, and the steps of inputting the initial text data into the target inference model for inference are executed until the preset termination condition is met.

[0149] The aforementioned model inference acceleration framework, based on the Talon architecture, employs two complementary strategies: a pre-retrieval and pre-verification strategy and a pre-retrieval and post-verification strategy. These strategies maximize hardware utilization during speculative decoding by parallelizing underutilized computing resources. Additionally, an adaptive hybrid drafting strategy is used to maximize acceptance rate by dynamically integrating model-based tag generation with retrieval-based tag matching.

[0150] Based on the same inventive concept, this application also provides a model inference acceleration device for implementing the model inference acceleration method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more model inference acceleration device embodiments provided below can be found in the limitations of the model inference acceleration method described above, and will not be repeated here.

[0151] In one exemplary embodiment, such as Figure 11 As shown, a model inference acceleration device is provided, including: a data acquisition module 1102, a parallel execution module 1104, a draft tree construction module 1106, and a verification module 1108, wherein:

[0152] Data acquisition module 1102 acquires initial text data;

[0153] The parallel execution module 1104 is used to input the initial text data into the target inference model for inference. The asynchronous execution engine schedules the parallel execution of the initial text's draft token generation, context token retrieval, and probability calculation prediction distribution, thereby obtaining the draft token set, the retrieval token set, and the token probability distribution in sequence.

[0154] The drafting tree construction module 1106 is used to integrate the draft token set and the retrieval token set to generate the target drafting tree; the nodes in the target drafting tree are used to represent the integrated token determined based on the integration of the draft token set and the retrieval token set.

[0155] The verification module 1108 is used to recursively verify the target drafting tree based on the tree attention mechanism and the probability distribution, determine the valid token sequence from the target drafting tree, update the initial text data according to the valid tokens, and perform the step of inputting the initial text data into the target inference model for inference until the preset termination condition is met.

[0156] The aforementioned model inference acceleration device, during the model inference acceleration process, schedules the parallel execution of draft token generation, context token retrieval, and probability calculation and distribution prediction of the initial text through an asynchronous execution engine. It integrates the draft token set and the retrieved token set to generate a target draft tree. Based on the tree attention mechanism, it recursively verifies the target draft tree according to the probability distribution and determines the valid token sequence. It updates the initial text data according to the valid tokens and repeats the inference until the preset termination condition is met. By parallelizing the three subtasks of draft token generation, context retrieval, and probability distribution calculation to improve the concurrent utilization of the hardware computing unit, and by using the tree attention mechanism to recursively verify the entire tree to confirm multiple valid tokens at once while maintaining the semantic consistency of the original model's probability distribution, it can effectively alleviate the computational overhead and latency bottleneck in the inference process of large language models without sacrificing output accuracy, and achieve the technical effect of improving both inference speed and hardware resource utilization.

[0157] In one exemplary embodiment, the model reasoning acceleration device further includes a retrieval module for determining knowledge-enhanced data storage D;

[0158] Get the current context associated with the initial text data, and extract k token suffixes from the current context;

[0159] Based on the k token suffixes, retrieve the context tokens in data storage D to obtain the initial retrieval token set for the initial text data;

[0160] For each initial retrieval token in the initial retrieval token set, a secondary sort is performed based on the character length and detection frequency of each initial retrieval token, and candidate retrieval tokens with both the character length and detection frequency in the top N are selected from the sorted initial retrieval token set.

[0161] The set of retrieval tokens is determined based on the top N candidate retrieval tokens.

[0162] In one exemplary embodiment, the model inference acceleration device further includes a database building module for acquiring the original dataset associated with the initial text data;

[0163] The original dataset is transformed into vectors through the embedding layer in the large language model. The vectors are then passed through the shallow, intermediate, and deep layers of the large language model to obtain the corresponding hierarchical features for each layer.

[0164] The features of each layer are concatenated to obtain the concatenated features;

[0165] The concatenated features are decoded using the token decoding layer of the large language model to obtain a knowledge representation. The pre-set data storage D is then enhanced based on the knowledge representation to obtain the knowledge-enhanced data storage D.

[0166] In an exemplary embodiment, the drafting tree construction module 1106 is used to construct a model drafting tree based on a set of draft tokens, and to construct a retrieval tree based on a set of retrieval tokens;

[0167] The model draft tree and the retrieval tree are merged to obtain a merged draft tree;

[0168] Based on longest prefix matching, identify at least one shared prefix of the merged drafting tree;

[0169] For each shared prefix in the merged drafting tree, all nodes with shared prefixes in the merged drafting tree are merged into a single path to obtain the target drafting tree.

[0170] In an exemplary embodiment, the verification module 1108 is used to arrange all nodes in the target drafting tree in hierarchical traversal order and construct an attention mask matrix;

[0171] The structural features of each node in the target drafting tree are encoded based on the attention mask matrix to obtain their respective encoded features.

[0172] The probability of each node is determined by a single forward propagation based on the coding characteristics of each node.

[0173] Based on the probability distribution and the probability of each node, the valid token sequence is determined from the target drafting tree.

[0174] In an exemplary embodiment, the verification module 1108 is used to verify the probability of each node sequentially according to the probability distribution. If the proportion of tokens received by the target node is greater than or equal to a preset acceptance threshold, the target node and all child nodes corresponding to the target node are retained, and verification processing is performed on all child nodes to determine a valid token sequence from the target drafting tree; the target node is any one of the nodes; and / or,

[0175] If the percentage of tokens received by the target node is less than the preset acceptance threshold, then the target node is determined to be at the same level in the target drafting tree, all descendant nodes of the target node are removed, and unverified sibling nodes at the same level are verified.

[0176] If the percentage of tokens received by all sibling nodes at the same level of the target node is less than the preset acceptance threshold, then resampling will begin from the last accepted node.

[0177] Each module in the aforementioned model inference acceleration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0178] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a model inference acceleration method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0179] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0182] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0184] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0186] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for accelerating model inference, characterized in that, The method includes: Get the initial text data; The initial text data is input into the target inference model for inference. The asynchronous execution engine schedules and executes the initial text's draft token generation, context token retrieval, and probability calculation and prediction distribution in parallel, thereby obtaining the draft token set, the retrieval token set, and the token probability distribution in sequence. The draft token set and the retrieval token set are integrated to generate a target drafting tree; the nodes in the target drafting tree are used to represent the integrated tokens determined based on the integration of the draft token set and the retrieval token set. Based on the tree attention mechanism, the target drafting tree is recursively verified according to the probability distribution, a valid token sequence is determined from the target drafting tree, and the initial text data is updated according to the valid tokens. The step of inputting the initial text data into the target inference model for inference is then performed until a preset termination condition is met.

2. The method according to claim 1, characterized in that, The retrieval context token includes: Determine the data storage D for knowledge enhancement; Obtain the current context associated with the initial text data, and extract k token suffixes from the current context; Based on the k token suffixes, retrieve context tokens from the data storage D to obtain the initial retrieval token set for the initial text data; For each initial retrieval token in the initial retrieval token set, a secondary sort is performed based on the character length and detection frequency of each initial retrieval token, and candidate retrieval tokens whose character length and detection frequency are both in the top N from the sorted initial retrieval token set are determined. The retrieval token set is determined based on the top N candidate retrieval tokens.

3. The method according to claim 2, characterized in that, The determined knowledge-enhanced data storage D includes: Obtain the original dataset associated with the initial text data; The original dataset is transformed into vectors through the embedding layer in the large language model. The vectors are then passed through the shallow, intermediate, and deep layers of the large language model to obtain the corresponding hierarchical features for each layer. The layered features are concatenated to obtain the concatenated features; The concatenated features are decoded using the token decoding layer of the large language model to obtain a knowledge representation. The preset data storage D is then enhanced based on the knowledge representation to obtain a knowledge-enhanced data storage D.

4. The method according to claim 1, characterized in that, The process of integrating the draft token set and the retrieval token set to generate the target drafting tree includes: A model drafting tree is constructed based on the set of draft tokens, and a retrieval tree is constructed based on the set of retrieval tokens; The model drafting tree and the retrieval tree are fused to obtain a fused drafting tree; Based on longest prefix matching, at least one shared prefix of the fused drafting tree is identified; For each shared prefix of the fused drafting tree, all nodes in the fused drafting tree with the shared prefix are merged into a single path to obtain the target drafting tree.

5. The method according to claim 1, characterized in that, The tree attention mechanism, which recursively verifies the target drafting tree according to the probability distribution and determines a valid token sequence from the target drafting tree, includes: Arrange all nodes in the target drafting tree in hierarchical traversal order to construct an attention mask matrix; The structural features of each node in the target drafting tree are encoded according to the attention mask matrix to obtain their respective encoded features. The probability of each node is determined by a single forward propagation based on the encoding features of each node. Based on the probability distribution and the probability of each node, a valid token sequence is determined from the target drafting tree.

6. The method according to claim 5, characterized in that, The step of determining the valid token sequence from the target drafting tree based on the probability distribution and the probability of each node includes: According to the probability distribution, the probability of each node is verified sequentially. If the proportion of tokens received by the target node is greater than or equal to a preset acceptance threshold, the target node and all its child nodes are retained. Verification processing is performed on all child nodes to determine a valid token sequence from the target drafting tree. The target node is any one of the nodes; and / or, If the percentage of tokens received by the target node is less than the preset acceptance threshold, then the same level to which the target node belongs in the target drafting tree is determined, all descendant nodes of the target node are removed, and unverified sibling nodes in the same level are verified. If the token acceptance rate of all sibling nodes at the same level as the target node is less than the preset acceptance threshold, then resampling is performed from the last accepted node.

7. A model inference acceleration framework, characterized in that, The model inference acceleration framework includes: The input module is used to input initial text data; An asynchronous execution engine is used to schedule the model drafter to generate draft tokens for the initial text when the initial text data is input into the target inference model for inference. The model drafter retrieves context tokens from the initial text data, and the target model pre-execution probability calculation predicts the distribution, thereby obtaining the draft token set, the retrieval token set, and the token probability distribution in sequence. By integrating the draft token set and the retrieval token set, a target draft tree is generated. Nodes in the target draft tree represent the integrated tokens determined based on the integration of the draft token set and the retrieval token set. The asynchronous execution engine calls the verification decision-maker based on the tree attention mechanism to recursively verify the target draft tree according to the probability distribution, determine the valid token sequence from the target draft tree, update the initial text data according to the valid tokens, and execute the steps of inputting the initial text data until the preset termination condition is met.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images