Speculative decoding method, device and system for text sequence generation and storage medium

Abstract

The application discloses a speculative decoding method, device and system for text sequence generation and a storage medium, and belongs to the technical field of large language model reasoning. The method comprises the following steps: sorting the association scores between the current draft query and each block proxy key-value pair from large to small, using top-k relevant blocks corresponding to the top-k rankings to form a draft attention context, and ensuring that the content transmitted to the draft text generation model is accurate; inputting the draft attention context into the draft text generation model to obtain candidate tokens to establish a draft tree; grouping and fusing the leaf nodes in the draft tree for verification, using the first rejection depth feedback in the verification process for the next round of draft tree construction, finding the longest verified path, adding the current historical token sequence, and deleting the text sequence requested by the user as the text answer requested by the user. The application can alleviate the execution overhead problem caused by long sequence generation, and finally improve the efficiency of text sequence generation.

Description

Technical Field

[0001] This application belongs to the field of large language model inference technology, and more specifically, relates to a speculative decoding method, apparatus, system and storage medium for text sequence generation. Background Technology

[0002] In the practical application of large language models, core tasks such as automatic code generation, complex scientific reasoning, multi-turn intelligent agent interaction, and the creation of long texts (tens of thousands of words) generally require the model to continuously generate thousands or even tens of thousands of word sequences under long context constraints. The reasoning logic of these long sequence generation tasks differs fundamentally from that of short text scenarios: as the generation steps progress, the inference system needs to continuously store, update, and access an ever-expanding key-value cache, and each decoding step requires attention calculations covering the entire historical context. This makes the requirements for hardware memory access bandwidth, cache scheduling strategies, and parallel computing resource allocation for long sequence generation far higher than those for ordinary short text scenarios, and it has become a performance bottleneck that urgently needs to be overcome in the current deployment of large models.

[0003] As a current mainstream autoregressive decoding acceleration technology, speculative decoding's core idea is to use a lightweight draft generation model with far fewer parameters than the main model to quickly pre-generate a draft sequence containing multiple candidate words. This draft sequence is then passed to the more accurate main target model for parallel verification of the entire candidate sequence in one go. When multiple consecutive candidate words pass the main model's verification, the system can directly output multiple valid words in a single forward inference process. This significantly reduces the number of decoding rounds that the main model originally needed to execute sequentially word by word, fundamentally reducing the serial overhead of autoregressive decoding and achieving a significant improvement in inference speed.

[0004] Most mainstream speculative decoding solutions in the industry currently employ a unified speculative strategy with fixed length, fixed width, or fixed tree structure, with optimization efforts almost entirely focused on improving the overall average acceptance rate of candidate words. While this design achieves the expected acceleration in scenarios with short sequence generation and minimal changes in context, it exposes several shortcomings in long sequence generation scenarios: First, the unified speculative strategy struggles to adapt to the differentiated benefit characteristics of different decoding positions. The validation costs of the main model at different decoding positions are not consistent. As historical context continues to accumulate, the computational complexity of attention and the access overhead of the KV cache in the later stages of the generation process are significantly higher than in the early stages. The actual acceleration benefits brought by the same number of successful speculative words vary greatly at different positions, and traditional strategies cannot allocate resources specifically, making it difficult to fully unleash the potential of hardware acceleration. Second, there is a context redundancy access problem in the draft generation stage. Under long context conditions, the effective context length that the draft model can bear is limited. Directly loading the complete historical KV cache introduces a large amount of unnecessary memory access overhead, while simple truncation easily loses key long-distance information, lowering the quality of candidate word generation. Third, dynamically adjusting the candidate structure can lead to irregular validation loads. If the candidate tree shape is dynamically adjusted to improve the acceptance rate, non-uniform leaf nodes, irregular prefix sharing relationships, and differentiated workloads will appear in the verification stage. Using the traditional fixed-shape verification method is very likely to cause problems such as redundant calculations and uneven thread utilization.

[0005] The aforementioned problems combine to cause existing speculative decoding solutions to perform far less efficiently than theoretically expected in long sequence generation scenarios. There is an urgent need in the industry for a speculative decoding solution adapted to long sequence generation scenarios of large language models, capable of systematically alleviating the execution overhead caused by the difficulty of adapting a uniform speculative strategy to positional gains, redundant access to draft context, and irregular validation loads in long sequence generation, while maintaining the consistency of the target text generation model's output. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the prior art, this application provides a speculative decoding method, apparatus, system and storage medium for text sequence generation, which is used to solve the technical problem of low actual execution efficiency of speculative decoding in the existing long sequence generation process.

[0007] To achieve the above objectives, according to one aspect of this application, a speculative decoding method for text sequence generation is provided, comprising: S1: Obtain the current historical token sequence corresponding to the user request, the historical key-value cache, and the current draft query generated by the draft text generation model at the current decoding position; the historical key-value cache includes the key vector and value vector corresponding to each token in the current historical token sequence; S2: Divide the current historical token sequence into multiple token blocks, and generate a block proxy key-value pair for each token block to evaluate the relevance of the key vector and value vector; S3: Sort the association scores between the current draft query and each block proxy key-value pair from largest to smallest, and select the token blocks corresponding to the top k ranked block proxy key-value pairs as top-k related blocks; combine the top-k related blocks with the most recent token that has not yet formed a complete token block to form a draft attention context; S4: Input the draft attention context into the draft text generation model to make it output multiple candidate tokens; construct a draft tree using the multiple candidate tokens under the constraints of preset node budget, preset maximum depth, preset maximum branch width and preset maximum branch layer; S5: Use the target text generation model to perform fusion verification on the leaf node groups in the draft tree. During the verification process, reject the deep feedback for the first time for the next round of draft tree construction, and add the longest verified path to the current historical token sequence; determine whether the user request is completed. If it is, delete the text sequence of the user request from the current historical token sequence as the text answer to the user request; otherwise, return to S1.

[0008] Further, step S2 includes: dividing the current historical token sequence into multiple non-overlapping token blocks of fixed length; and for each token block, calculating the salience weight corresponding to each token within that token block. Based on the significance weights, the key vectors and value vectors in the token block are weighted and aggregated to obtain the weighted center representation of the key vectors within the block. Weighted center representation of the sum vector ; in, , Let be the key vector and value vector corresponding to the j-th token in the key vector of the i-th token block. Let L be the value vector of the i-th token block, L be the number of tokens in each block, i be the token block index, and j be the position index of the token within the block. For the summation index, For non-zero positive numbers, Norm representation; The weighted center representation of the key vector corresponding to each token block is concatenated with the boundary keys corresponding to the head token and the tail token. The weighted center representation of the value vector corresponding to each token block is concatenated with the boundary values ​​corresponding to the head token and the tail token. Then, the block proxy key-value pair is formed by linear projection. The block proxy key-value pair includes a block proxy key vector and a block proxy value vector.

[0009] Furthermore, the block proxy key vector is represented as: The block proxy value vector is represented as: ;in, Let be the weight of the key vector within the block corresponding to the i-th token block. Let be the weight of the value vector within the block corresponding to the i-th token block. This represents the weighted center representation of the key vectors within the block. Let be the weighted center representation of the value vector within the i-th token block. and These are the boundary keys of the first and last tokens in the i-th token block, respectively; and These are the boundary values ​​of the first and last tokens in the i-th token block, respectively.

[0010] Furthermore, the association score is expressed as: Where i is the index of the token block, and t is the current decoding time step. For the current draft search, Let i be the block proxy key vector corresponding to the i-th token block. Let be the block proxy value vector corresponding to the i-th token block, where the superscript T indicates transpose.

[0011] Furthermore, S3 also includes: constructing a contiguous block storage area for storing the top-k related blocks; the contiguous block storage area is used to transform the logically discrete k top-k related blocks into physically contiguous access areas; supporting incremental loading and slot reclamation through a mapping table from block identifiers to area offsets; and maintaining soft residency of highly reused KV cache blocks in the cache hierarchy through an access policy window.

[0012] Furthermore, the construction of the draft tree in S4 includes: maintaining verification arrival counts and first rejection counts for each depth position of the draft tree, and calculating the empirical rejection risk based on the smoothing coefficient; selecting high-risk depths as branch layers, adding candidate branches at the branch layers, and maintaining chain expansion at other depths to reduce unnecessary candidate expansion under a fixed node budget.

[0013] Further, S5 includes: Leaf nodes are grouped according to their nearest branch ancestor in the draft tree to obtain data blocks corresponding to each group of leaf nodes; leaf nodes on candidate paths with the same nearest branch ancestor are placed in the same leaf node group; the target text generation model is used to fuse and verify each data block to obtain verified paths; Add the longest verified path to the current historical token sequence; determine whether the user request has been completed. If so, delete the text sequence of the user request from the current historical token sequence as the text response to the user request; otherwise, return to S1.

[0014] According to another aspect of this application, a speculative decoding apparatus for text sequence generation is provided, comprising: The acquisition module is used to acquire the current historical token sequence corresponding to the user request, the historical key-value cache, and the current draft query corresponding to the draft text generation model; the historical key-value cache includes the key vector and value vector corresponding to the current historical token sequence; The generation module is used to divide the current historical token sequence into multiple token blocks and generate a block proxy key-value pair for each token block for evaluating the relevance of the key vector and value vector; The composition module is used to sort the association scores between the current draft query and each block proxy key-value pair from largest to smallest, select the token blocks corresponding to the top k ranked block proxy key-value pairs as top-k related blocks; and combine the top-k related blocks with the most recent token that has not yet formed a complete token block to form a draft attention context. A construction module is used to input the draft attention context into the draft text generation model so that it outputs multiple candidate tokens; and to construct a draft tree using the multiple candidate tokens under the constraints of a preset node budget, a preset maximum depth, a preset maximum branch width, and a preset maximum branch layer. The verification module is used to perform fusion verification on the leaf node groups in the draft tree using the target text generation model. During the verification process, the first rejection of the deep feedback is used for the next round of draft tree construction. The longest verified path is added to the current historical token sequence. It determines whether the user request has been completed. If it has, the text sequence of the user request is deleted from the current historical token sequence as the text answer to the user request. Otherwise, it returns to the acquisition module.

[0015] According to another aspect of this application, a speculative decoding system for text sequence generation is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the speculative decoding method for text sequence generation.

[0016] According to another aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the speculative decoding method for generating the text sequence.

[0017] Overall, compared with the prior art, the above-described technical solutions conceived in this application can achieve the following beneficial effects: (1) This invention provides a speculative decoding method for text sequence generation. The method sorts the association scores between the current draft query and each block proxy key-value pair from largest to smallest, selecting the top k ranked block proxy key-value pairs corresponding to the token blocks as top-k related blocks. The top-k related blocks, together with the most recent tokens that have not yet formed complete token blocks, constitute a draft attention context. This reduces computational complexity while ensuring the accuracy of the content transmitted to the draft text generation model. The draft attention context is input into the draft text generation model to output multiple candidate tokens and build a draft tree. The leaf nodes in the draft tree are grouped and fused for verification. During the verification process, the first rejection of deep feedback is used for the next round of draft tree construction. The longest verified path is found and added to the current historical token sequence until the user request is completed. Through the above technical solution, this invention can alleviate the problems of inconsistent positional benefits, redundant access to draft contexts, and execution overhead caused by irregular verification loads in long sequence generation, while maintaining the consistency of the target text generation model's output, ultimately improving the efficiency of text sequence generation.

[0018] (2) In this scheme, the weighted center representation of the key vector corresponding to each token block is concatenated with the boundary keys corresponding to the first and last tokens, and the weighted center representation of the value vector corresponding to each token block is concatenated with the boundary values ​​corresponding to the first and last tokens. Then, the block proxy key-value pairs are formed through linear projection. The block proxy key-value pairs include a block proxy key vector and a block proxy value vector. This design can suppress weakly correlated noise and highlight tokens that contribute more to subsequent generation through significant weighting. The boundary key values ​​can supplement the context transition information that may be lost in the weighted aggregation, thereby improving the semantic coverage and stability of the block representation while reducing the overhead of block-level correlation estimation.

[0019] (3) The block proxy key vector described in this scheme is represented as: The block proxy value vector is represented as: This design allows for block-level sorting without scanning the entire historical key-value cache token by token, and does not replace the token-level key-value pairs within the selected token block, thus balancing the efficiency of relevance filtering with the accuracy of draft attention representation.

[0020] (4) The correlation score described in this scheme is expressed as: This design utilizes the inner product of the current draft query and the block surrogate key vector to reflect the degree of structural alignment, and the inner product of the current draft query and the block surrogate value vector to reflect the degree of semantic compatibility, then performs scaling normalization using the square of the query modulus. This allows for dynamic selection of relevant token blocks based on the current decoding position, reducing interference from irrelevant historical information in draft generation and improving the quality of candidate tokens.

[0021] (5) This scheme constructs a contiguous block storage area for storing the top-k related blocks; supports incremental loading and slot reclamation through a mapping table from block identifier to region offset; and maintains soft residency of highly reused KV cache blocks in the cache hierarchy through an access policy window. This design transforms logically discrete top-k related blocks into physically contiguous access areas through the contiguous block storage area, and utilizes the mapping table from block identifier to region offset to incrementally load new blocks and reclaim invalid slots; the access policy window increases the retention probability of highly reused KV cache blocks in the cache hierarchy. Therefore, it reduces redundant relocation and non-contiguous memory access, improving the memory merging efficiency during the draft attention stage.

[0022] (6) This scheme maintains verification arrival counts and first rejection counts at each depth of the draft tree, and calculates the empirical rejection risk based on a smoothing coefficient; high-risk depths are selected as branch layers, and candidate branches are added at these layers, while maintaining chain expansion at other depths. This design estimates the rejection risk at each depth based on the verification arrival counts and first rejection counts, adding candidate branches only at high-risk depths and maintaining chain expansion at other depths. This allows candidate resources to be concentrated at locations more likely to be rejected under a fixed node budget, increasing the probability that at least one candidate path passes through high-risk depths, while reducing low-return branches and ineffective nodes.

[0023] (7) In this scheme, leaf nodes are grouped according to their nearest branch ancestor in the draft tree to obtain data blocks corresponding to each group of leaf nodes; leaf nodes on candidate paths with the same nearest branch ancestor are placed in the same leaf node group; and the data blocks are fused and verified to obtain verified paths. This design groups the leaf nodes according to their nearest branch ancestor and packages each group into a fixed-capacity data block; during fusion verification, the shared prefix of the candidate paths in the same group is reused based on the ancestor path mask and the paging key-value cache. This can transform the irregular draft tree into a more regular parallel working unit, reducing the repeated loading of shared prefixes, uneven thread load, and kernel scheduling overhead. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the speculative decoding method for text sequence generation provided in Embodiment 1 of the present invention.

[0025] Figure 2 This is a schematic diagram illustrating the principle of the speculative decoding method for text sequence generation provided in Embodiment 1 of the present invention.

[0026] Figure 3 This is a schematic diagram of the utility-guided drafting and differentiated speculative execution engine structure provided in Embodiment 1 of the present invention.

[0027] Figure 4 This is a schematic diagram of the Chunk contiguous region and APW-guided KV cache soft residency mechanism provided in Embodiment 1 of the present invention.

[0028] Figure 5 This is a schematic diagram of the adaptive speculative tree, topology regularization working unit, and prefix block reuse mechanism provided in Embodiment 1 of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Furthermore, the technical features involved in the various embodiments described below can be combined with each other as long as they do not conflict with each other.

[0030] Example 1 This embodiment 1 provides a speculative decoding method for text sequence generation, such as... Figure 1 and Figure 2 As shown, it includes: S1-S5.

[0031] S1: Obtain the current historical token sequence, historical key-value cache, and current draft query corresponding to the draft text generation model corresponding to the user request; the historical key-value cache includes the key vector and value vector corresponding to the current historical token sequence.

[0032] S2: Divide the current historical token sequence into multiple token blocks, and generate a block proxy key-value pair for each token block to evaluate the relevance of the key vector and value vector.

[0033] S3: Sort the association scores between the current draft query and each block proxy key-value pair from largest to smallest, and select the token blocks corresponding to the top k ranked block proxy key-value pairs as top-k related blocks; combine the top-k related blocks with the most recent token that has not yet formed a complete token block to form a draft attention context.

[0034] S4: Input the draft attention context into the draft text generation model to output multiple candidate tokens. Using these candidate tokens, construct a draft tree under constraints of a preset node budget, a preset maximum depth, a preset maximum branch width, and a preset maximum branch depth. The branch width at each depth of the draft tree is determined based on the first rejection depth of the target text generation model in a previous validation round.

[0035] S5: Use the target text generation model to perform fusion verification on the leaf node groups in the draft tree. During the verification process, reject the deep feedback for the first time for the next round of draft tree construction, and add the longest verified path to the current historical token sequence; determine whether the user request is completed. If it is, delete the text sequence of the user request from the current historical token sequence as the text answer to the user request; otherwise, return to S1.

[0036] In S1, the historical token sequence, historical key-value (KV) cache, and current draft query corresponding to the user request are obtained. The historical token sequence corresponding to the user request is a token sequence that has been confirmed and submitted by the target text generation model; the historical KV cache includes key vectors and value vectors corresponding to processed historical tokens; the current draft query includes the query vector or its equivalent representation used by the draft text generation model when generating the next candidate token at the current decoding position.

[0037] It should be noted that during the first execution of S1, the current historical token sequence corresponding to the user request refers to the user request itself, denoted as A0. Then, S2-S5 are executed, resulting in the longest verified path, denoted as A1, which is added to A0, thus creating the new current historical token sequence {A0+A1}. During the second execution of S1, the current historical token sequence is {A0+A1}. Then, S2-S5 are executed, resulting in the longest verified path, denoted as A2, which is added to {A0+A1}, thus creating the new current historical token sequence {A0+A1+A2}. Similarly, during the nth round of execution S1, the current historical token sequence is {A0+A1+...+An-1}. Then, executions S2-S5 are performed to obtain the longest verified path, denoted as An, which is added to {A0+A1+...+An-1}, resulting in the new current historical token sequence {A0+A1+...+An-1+An}. If it is determined at this point that the user request has been completed, A0 is removed from {A0+A1+...+An-1+An} to obtain the sequence {A1+...+An-1+An}, which is considered the text response to the user request.

[0038] Regarding the historical KV cache, it consists of the key and value vectors corresponding to the current historical token sequence, continuing the notation from the example above. In the first round of executing S1, it is the key and value vector of the user request A0; in the second round of executing S1, it is the key and value vector of the sequence {A0+A1}; and so on, it is the key and value vector of the sequence {A0+A1+...+An-1}.

[0039] In S2, the historical token sequence X is divided into fixed-length, non-overlapping token blocks. Let the historical token sequence be... If each token block contains L tokens, then multiple token blocks can be obtained. Where M is approximately T / L. L can be determined based on the cache block size, attention calculation granularity, and hardware relocation granularity. For example, in one implementation, L can be set to 64; in other implementations, L can be set to 32, 128, or other values ​​adapted to the execution platform.

[0040] Optional implementation methods, such as Figure 3 As shown, step S2 includes: dividing the current historical token sequence into multiple non-overlapping token blocks of fixed length; and for each token block, calculating the salience weight corresponding to each token within that token block. Based on the significance weights, the key vectors and value vectors in the token block are weighted and aggregated to obtain the weighted center representation of the key vectors within the block. Weighted center representation of the sum vector ;in, , Let be the key vector and value vector corresponding to the j-th token in the key vector of the i-th token block. Let L be the value vector of the i-th token block, L be the number of tokens in each block, i be the token block index, and j be the position index of the token within the block. For the summation index, For non-zero positive numbers, The block proxy key-value pair is formed by using norm representation; concatenating the weighted center representation of the key vector corresponding to each token block with the boundary keys corresponding to the first and last tokens, and concatenating the weighted center representation of the value vector corresponding to each token block with the boundary values ​​corresponding to the first and last tokens. The block proxy key-value pair includes a block proxy key vector and a block proxy value vector.

[0041] As an optional implementation, the block proxy key vector is represented as: The block proxy value vector is represented as: ;in, Let be the weight of the key vector within the block corresponding to the i-th token block. Let be the weight of the value vector within the block corresponding to the i-th token block. This represents the weighted center representation of the key vectors within the block. Let be the weighted center representation of the value vector within the i-th token block. and These are the boundary keys of the first and last tokens in the i-th token block, respectively; and These are the boundary values ​​of the first and last tokens in the i-th token block, respectively.

[0042] Specifically, for each token block Based on the token-level key vector and value vector within the block, construct block proxy key-value pairs. Specifically, for a block... The j-th token in the dataset has its key vector and value vector represented as follows: and Based on and The norm is used to calculate the significance weight. This assigns higher weights to tokens with strong activations in both the key and value vectors in the block representation. Subsequently, saliency-weighted aggregation is performed on the key and value vectors respectively to obtain weighted central representations of the key and value vectors within the block. .

[0043] To avoid losing context transition information at block boundaries through simple aggregation, the boundary key-value pairs corresponding to the first and last tokens of the token block are further extracted. Then, the weighted center representation is concatenated with the first and last boundary key-value pairs, and a linear projection is used to form the block surrogate key vector and block surrogate value vector corresponding to the token block. The linear projection matrix can be derived from the projection parameters of the draft text generation model or a fixed projection matrix predetermined during the inference deployment phase. The block surrogate key-value pairs are primarily used for relevance ranking and do not replace the token-level key-value pairs in the selected token block.

[0044] In S3, relevant contexts are selected based on the association between the current draft query and each block surrogate key-value pair. Specifically, the current draft query qt is used to perform association calculations with both the block surrogate key vector and the block surrogate value vector, and the calculation results are scaled and normalized to obtain the block-query association score. As one implementation method, It can be determined by the inner product of the query and the block surrogate key vector, the inner product of the query and the block surrogate value vector, and the query norm. Sort multiple token blocks and select the top k (top-k) relevant blocks from the top of the sort.

[0045] To balance long-range relevance with short-range local continuity, the nearest tokens to the current decoding position that have not yet formed a complete token block are added to the top-k relevant blocks to obtain a draft attention context. Thus, when generating candidate tokens, the draft text generation model can access relevant historical token blocks while also preserving the continuous context near the current position.

[0046] As an optional implementation, S3 further includes: constructing a contiguous block storage area for storing the top-k related blocks; the contiguous block storage area is used to transform the logically discrete k top-k related blocks into physically contiguous access areas; supporting incremental loading and slot reclamation through a mapping table from block identifiers to area offsets; and maintaining soft residency of highly reused KV cache blocks in the cache hierarchy through an access policy window.

[0047] Specifically, such as Figure 4Reuse-aware residency management is performed on the top-k relevant blocks. Since the top-k relevant blocks selected by S3 may logically be distributed across different locations in the historical context, direct access to the original KV cache may result in non-contiguous memory accesses. Therefore, a contiguous block storage region is constructed to store the KV cache blocks corresponding to the current relevant block in a packaged manner. A mapping table is maintained between block identifiers and offsets in the contiguous block storage region, enabling draft attention computation to locate the physically contiguous region from the logical block identifier through the mapping table.

[0048] Between adjacent decoding steps, the top-k relevant block sets typically overlap to some extent. The relevant block sets of the current decoding step are compared with those of the previous decoding step. Only newly added relevant blocks are loaded into the contiguous block storage area, and slots occupied by no longer selected relevant blocks are reclaimed. For relevant blocks that continue to be selected, their positions in the contiguous block storage area are maintained to reduce redundant relocation.

[0049] An access policy window is set for the contiguous block storage area to maintain soft residency of highly reused key-value cache blocks in the cache hierarchy. Soft residency refers to increasing the retention probability of the top k (top-k) relevant blocks in the cache through cache residency hints or equivalent cache control mechanisms without altering the program semantics. The access policy window can be configured based on the size of the contiguous block storage area and the device cache capacity, and the window can be released or reset after the drafting stage to avoid unnecessary cache usage for subsequent verification calculations.

[0050] In S4, the draft attention context is input into the draft text generation model to obtain multiple candidate tokens. A draft tree is then constructed using these candidate tokens under the constraints of the previous preset node budget, maximum depth, maximum branch width, and maximum branch depth. The maximum branch width is determined as follows: a depth-related rejection risk distribution is generated based on the first rejection depth of the target text generation model in previous validation rounds, and the maximum branch width of the draft tree is determined based on this rejection risk distribution. Then, candidate tokens are generated based on the draft attention context, and the draft tree is constructed. The draft tree uses the end position of the historical token sequence corresponding to the user request as its root node. Each node in the draft tree represents a candidate token appended along the path from the root node to that node. The construction of the draft tree is subject to the preset node budget B, maximum depth D, and maximum branch depth. and maximum branch level The constraints are as follows: node budget B limits the number of candidates in a round of speculation, maximum depth D limits the longest candidate path length, and maximum branch width... Used to limit the candidate width of a single branch layer, maximum number of branch layers. Used to limit the depth at which width expansion is performed in the draft tree.

[0051] In S4, the draft tree structure is adjusted based on historical validation feedback from the target text generation model. Specifically, after each round of validation, the depth at which the target text generation model first rejects a candidate token in that round is recorded. For each depth d of the draft tree, a validation arrival count is maintained. and first rejection count Verify arrival count This represents the number of times the verification process reaches depth d, with the first rejection count being the largest. This represents the number of times the first rejection occurs at depth d. The empirical rejection risk is calculated using the smoothing coefficient α (alpha). ,For example Experience rejects risk level This indicates the relative risk of a candidate path being interrupted at a corresponding depth. No more than [number missing] paths are selected from each depth location. Each high-risk depth is designated as a branch layer, and the branch width of each branch layer is determined based on the level of risk. At the branch layer, multiple candidate tokens are generated for the leading node to increase the likelihood that at least one candidate path passes through that depth. At the non-branch layer, chain expansion is maintained to reduce the consumption of invalid nodes.

[0052] During the draft tree expansion process, leading nodes can be selected based on the output distribution of the draft text generation model and the cumulative draft probability. For each selected leading node, candidate tokens are chosen from the output distribution of the draft text generation model for expansion, and the new node is recorded as a data structure containing the token value, parent node index, node depth, and cumulative score. The expansion process ends when the node budget B is reached, the maximum depth D is reached, or there are no more expandable leading nodes.

[0053] As an optional implementation, S5 includes: grouping leaf nodes according to their nearest branch ancestor in the draft tree to obtain data blocks corresponding to each group of leaf nodes; wherein leaf nodes on candidate paths with the same nearest branch ancestor are placed in the same leaf node group; using the target text generation model to perform fusion verification on each data block to obtain verified paths; adding the longest verified path to the current historical token sequence; determining whether the user request has been completed, and if so, deleting the text sequence of the user request from the current historical token sequence as the text answer to the user request; otherwise, returning to S1.

[0054] Furthermore, such as Figure 5 As shown, topology-aware fusion verification is performed. Different leaf nodes in the adaptive draft tree may share prefixes of different lengths. Leaf node queries are grouped according to the nearest branch ancestor of each leaf node, so that candidate paths with the same nearest branch ancestor are placed in the same group; then each group of leaf nodes is packaged into a fixed-capacity data block to form regular working units on parallel hardware.

[0055] The fusion verification kernel maintains a logically continuous attention view based on a paginated key-value cache. For shared prefix key-value cache blocks within the same data block, the kernel parses the required key-value blocks based on the ancestor path mask and pagination index, loads the corresponding key-value blocks of the shared prefix once, and reuses them within the data block. For the non-shared parts of different candidate paths, the kernel accesses the corresponding key-value blocks according to each candidate node path. In this way, while maintaining the semantic integrity of the target text generation model verification, it can reduce the repeated loading of shared prefixes and reduce the scheduling overhead caused by irregular verification.

[0056] After the target text generation model outputs the validation unnormalized probability scores (logits) for each inferred node, the longest accepted candidate path is determined based on the target text generation model distribution and the draft candidates: The token corresponding to this path is then submitted to the current historical token sequence. If a rejection occurs at a certain depth, this initial rejection depth is used for the next round of draft tree construction. Thus, the draft tree structure can be iteratively adjusted based on the actual validation results of the target text generation model.

[0057] Example 2 This embodiment provides a speculative decoding device for text sequence generation, including: an acquisition module, a generation module, a composition module, a construction module, and a verification module.

[0058] The acquisition module is used to acquire the current historical token sequence corresponding to the user request, the historical key-value cache, and the current draft query corresponding to the draft text generation model; the historical key-value cache includes the key vector and value vector corresponding to the current historical token sequence; The generation module is used to divide the current historical token sequence into multiple token blocks and generate a block proxy key-value pair for each token block for evaluating the relevance of the key vector and value vector; The composition module is used to sort the association scores between the current draft query and each block proxy key-value pair from largest to smallest, select the token blocks corresponding to the top k ranked block proxy key-value pairs as top-k related blocks; and combine the top-k related blocks with the most recent token that has not yet formed a complete token block to form a draft attention context. A construction module is used to input the draft attention context into the draft text generation model so that it outputs multiple candidate tokens; and to construct a draft tree using the multiple candidate tokens under the constraints of a preset node budget, a preset maximum depth, a preset maximum branch width, and a preset maximum branch layer. The verification module is used to perform fusion verification on the leaf node groups in the draft tree using the target text generation model. During the verification process, the first rejection of the deep feedback is used for the next round of draft tree construction. The longest verified path is added to the current historical token sequence. It determines whether the user request has been completed. If it has, the text sequence of the user request is deleted from the current historical token sequence as the text answer to the user request. Otherwise, it returns to the acquisition module.

[0059] Example 3 This embodiment provides a speculative decoding system for text sequence generation, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the speculative decoding method for text sequence generation.

[0060] This system can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0061] Example 4 This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the speculative decoding method for generating the text sequence.

[0062] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0063] Example 5 This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of this application.

[0064] The technical features of the embodiments described above can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "as in another example" in this application are intended to illustrate the application and are not intended to limit the application.

[0065] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A speculative decoding method for text sequence generation, characterized in that, include: S1: Obtain the current historical token sequence, historical key-value cache, and current draft query corresponding to the draft text generation model corresponding to the user request; the historical key-value cache includes the key vector and value vector corresponding to the current historical token sequence; S2: Divide the current historical token sequence into multiple token blocks, and generate a block proxy key-value pair for each token block to evaluate the relevance of the key vector and value vector; S3: Sort the association scores between the current draft query and each block proxy key-value pair from largest to smallest, and select the token blocks corresponding to the top k ranked block proxy key-value pairs as top-k related blocks; combine the top-k related blocks with the most recent token that has not yet formed a complete token block to form a draft attention context; S4: Input the draft attention context into the draft text generation model so that it outputs multiple candidate tokens; A draft tree is constructed using multiple candidate tokens under the constraints of a preset node budget, a preset maximum depth, a preset maximum branch width, and a preset maximum branch level. S5: Use the target text generation model to perform fusion verification on the leaf node groups in the draft tree. During the verification process, reject the deep feedback for the first time for the next round of draft tree construction, and add the longest verified path to the current historical token sequence; determine whether the user request is completed. If it is, delete the text sequence of the user request from the current historical token sequence as the text answer to the user request; otherwise, return to S1.

2. The speculative decoding method for text sequence generation as described in claim 1, characterized in that, S2 includes: The current historical token sequence is divided into multiple non-overlapping token blocks of fixed length; for each token block, the salience weights corresponding to each token within that block are calculated. Based on the significance weights, the key vectors and value vectors in the corresponding token blocks are weighted and aggregated to obtain the weighted center representation of the key vectors within the block. Weighted center representation of the sum vector ; in, , Let be the key vector and value vector corresponding to the j-th token in the key vector of the i-th token block. Let L be the value vector of the i-th token block, L be the number of tokens in each block, i be the index of the token block, and j be the position index of the token in the i-th token block. For the summation index, For non-zero positive numbers, Norm representation; The weighted center representation of the key vector corresponding to each token block is concatenated with the boundary keys corresponding to the head token and the tail token. The weighted center representation of the value vector corresponding to each token block is concatenated with the boundary values ​​corresponding to the head token and the tail token. Then, the block proxy key-value pair is formed by linear projection. The block proxy key-value pair includes a block proxy key vector and a block proxy value vector.

3. The speculative decoding method for text sequence generation as described in claim 2, characterized in that, The block proxy key vector is represented as follows: ; The block proxy value vector is represented as follows: ; in, Let be the weight of the key vector within the block corresponding to the i-th token block. Let be the weight of the value vector within the block corresponding to the i-th token block. This represents the weighted center representation of the key vectors within the block. Let be the weighted center representation of the value vector within the i-th token block. and These are the boundary keys of the first and last tokens in the i-th token block, respectively; and These are the boundary values ​​of the first and last tokens in the i-th token block, respectively.

4. The speculative decoding method for text sequence generation as described in claim 1, characterized in that, The association score is represented as follows: ； Among them, subscript The index for the token block. This is the current decoding time step. For the current draft search, Let i be the block proxy key vector corresponding to the i-th token block. Let be the block proxy value vector corresponding to the i-th token block, where the superscript T indicates transpose.

5. The speculative decoding method for text sequence generation as described in claim 1, characterized in that, S3 further includes: constructing a contiguous block storage area for storing the top-k related blocks; the contiguous block storage area is used to transform the logically discrete k top-k related blocks into physically contiguous access areas; supporting incremental loading and slot reclamation through a mapping table from block identifier to area offset; and maintaining soft residency of highly reused KV cache blocks in the cache hierarchy through an access policy window.

6. The speculative decoding method for text sequence generation as described in claim 1, characterized in that, The draft tree constructed in S4 includes: maintaining verification arrival counts and first rejection counts at each depth position of the draft tree, and calculating the empirical rejection risk based on the smoothing coefficient; selecting high-risk depths as branch layers, adding candidate branches at the branch layers, and maintaining chain expansion at other depths to reduce unnecessary candidate expansion under a fixed node budget.

7. The speculative decoding method for text sequence generation as described in claim 1, characterized in that, S5 includes: Leaf nodes are grouped according to their nearest branch ancestor in the draft tree to obtain data blocks corresponding to each group of leaf nodes; leaf nodes on candidate paths with the same nearest branch ancestor are placed in the same leaf node group; the target text generation model is used to fuse and verify each data block to obtain verified paths; Add the longest verified path to the current historical token sequence; determine whether the user request has been completed. If so, delete the text sequence of the user request from the current historical token sequence as the text response to the user request; otherwise, return to S1.

8. A speculative decoding apparatus for text sequence generation, characterized in that, include: The acquisition module is used to acquire the current historical token sequence corresponding to the user request, the historical key-value cache, and the current draft query corresponding to the draft text generation model; the historical key-value cache includes the key vector and value vector corresponding to the current historical token sequence; The generation module is used to divide the current historical token sequence into multiple token blocks and generate a block proxy key-value pair for each token block for evaluating the relevance of the key vector and value vector; The composition module is used to sort the association scores between the current draft query and each block proxy key-value pair from largest to smallest, select the token blocks corresponding to the top k ranked block proxy key-value pairs as top-k related blocks; and combine the top-k related blocks with the most recent token that has not yet formed a complete token block to form a draft attention context. The building module is used to input the draft attention context into the draft text generation model so that it outputs multiple candidate tokens; A draft tree is constructed using multiple candidate tokens under the constraints of a preset node budget, a preset maximum depth, a preset maximum branch width, and a preset maximum branch level. The verification module is used to perform fusion verification on the leaf node groups in the draft tree using the target text generation model. During the verification process, the first rejection of the deep feedback is used for the next round of draft tree construction. The longest verified path is added to the current historical token sequence. It determines whether the user request has been completed. If it has, the text sequence of the user request is deleted from the current historical token sequence as the text answer to the user request. Otherwise, it returns to the acquisition module.

9. A speculative decoding system for text sequence generation, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the speculative decoding method for generating text sequences according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the speculative decoding method for generating text sequences according to any one of claims 1 to 7.

Images