Hybrid candidate source fused multi-sequence speculative sampling method and system

Abstract

The application relates to the technical field of artificial intelligence, and discloses a multi-sequence speculation sampling method of Hybrid candidate source fusion. The method respectively constructs a history matching candidate set, a secondary feedback candidate set and a distribution exploration candidate set based on a current generated prefix, aligns candidate sequences to a unified root token, removes and combines to form a multi-candidate structure, allocates a differentiated maximum extension length according to branch rank or search confidence weight difference, performs capacity control under the maximum draft budget constraint, submits a target model for verification, and writes the real output of the target model at the verification failure position into a secondary feedback pool for subsequent round search reuse. The method is helpful to improve the average accepted token number of single verification of the target model and reduce the reasoning delay.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multi-candidate speculative sampling acceleration technique in autoregressive generative model inference systems. Background Technology

[0002] With the increasing prevalence of large language models in scenarios such as dialogue interaction, code generation, document writing, and edge smart terminals, the inference efficiency of autoregressive generative models has become a key factor affecting service response latency and computing resource utilization. In cloud-based inference service scenarios, high-concurrency requests require the inference system to maximize the number of effective output tokens generated by each forward call to the target model within limited GPU memory and computing budget, in order to reduce single-token inference latency and improve throughput. In scenarios with high context repetition, such as code generation, frequently occurring templated fragments and repetitive expression patterns in the code to be generated, if they can be reused when they reappear in similar contexts, can help reduce the number of forward calls to the target model. In edge deployment scenarios, limited GPU memory and RAM conditions impose more stringent constraints on the size of candidate data structures and index maintenance overhead.

[0003] Speculative sampling, which constructs a sequence of candidate drafts before submitting them to the target model to validate multiple tokens at once, is an important approach to improving the efficiency of autoregressive inference. However, some existing implementations rely on a single candidate source to construct the draft sequence, resulting in limited candidate information dimensions and unstable candidate hit rates when faced with novel contexts. Furthermore, some existing implementations typically use the target model output at validation failure locations only for correction in the current step, rather than continuously organizing it into searchable candidate sources for subsequent rounds, leading to underutilization of validation feedback information. In addition, some existing implementations lack differentiated budget allocation mechanisms for candidate branches with different confidence levels and feedback pool lifecycle management capabilities, potentially causing low-confidence candidates to consume excessive node budgets and insufficient utilization of target model validation resources. Therefore, there is an urgent need for an inference acceleration solution that can integrate multi-dimensional candidate sources, enable continuous reuse of validation feedback, and balance branch budget control and memory capacity management. Summary of the Invention

[0004] The purpose of this application is to provide a multi-sequence speculative sampling method and system for hybrid candidate source fusion to solve the problems mentioned in the background art.

[0005] This application provides a multi-sequence speculative sampling method for hybrid candidate source fusion, applied to an inference system of an autoregressive generative model, wherein the inference system includes a target model, and the method includes: Get the token sequence corresponding to the currently generated prefix; Based on the matching results of the token sequence in the historical token sequence, a historical matching candidate set is constructed; Based on the token sequence, pattern matching is performed in the secondary feedback pool to construct a secondary feedback candidate set; wherein, the secondary feedback pool is used to organize the corrected token sequence or subsequent token sequence determined by the target model at the historical verification failure position according to the pattern; Based on the token distribution output from the previous verification phase, construct a candidate set for distribution exploration; At least a portion of the candidate sequences in the historical matching candidate set, the secondary feedback candidate set, and the distributed exploration candidate set are aligned to a unified root token, and the aligned candidate sequences are deduplicated and merged to form a multi-candidate structure. Different maximum extension lengths are assigned to different candidate branches in the multi-candidate structure based on the candidate branch's rank, candidate source, or retrieval confidence weight. Capacity control is performed on the multi-candidate structure under the maximum draft budget constraint to truncate or stop adding candidate branches that exceed the budget; The capacity-controlled multi-candidate structure is submitted to the target model for verification, and the current round output token is determined based on the continuous acceptance length of each candidate branch. The corrected token sequence or subsequent token sequence determined by the target model at the location where the verification failed in this round is written into the secondary feedback pool for retrieval when constructing the secondary feedback candidate set in subsequent rounds.

[0006] In a preferred embodiment, if the current round of verification results in full acceptance and there are no verification failure positions, the writing process to the secondary feedback pool is not triggered; the secondary feedback pool is divided into a standard area and a short sequence area according to the writing sequence length; writing the corrected token sequence or subsequent token sequence determined by the target model at the verification failure position in this round into the secondary feedback pool includes: Obtain the corrected token sequence or subsequent token sequence continuously output by the target model at the verification failure position; if the corrected token sequence or subsequent token sequence encounters an end token, truncate it to the position before the end token; use the number of valid tokens after truncation as the actual sequence length; if the actual sequence length is zero, discard the entry; Before writing, a pre-screening process is performed: the secondary feedback pool is queried to see if there is already a candidate sequence that is completely consistent with the sequence to be written. If it exists, the sequence is not written repeatedly and the hit count and the most recent hit time step of the existing entries are updated. The total number of entries to be written in the current round is counted. If the total number of entries to be written exceeds the product of the preset single-round writing ratio limit and the capacity limit of the secondary feedback pool, only the first few entries with the longest actual sequence length are retained for writing. After pre-screening, if the actual sequence length is not less than the preset minimum write sequence length threshold, the corresponding entry is written to the standard area; if the actual sequence length is greater than zero and less than the minimum write sequence length threshold, the corresponding entry is written to the short sequence area. Multiple different modified token sequences or subsequent token sequences are allowed to be stored under the same pattern key, and the maximum number of records stored under the same pattern key is a preset value. When writing, the number of records under the same pattern key reaches... If the record with the lowest hit count under that key is discarded, a new record will be written. When the total number of entries in the secondary feedback pool exceeds the capacity limit, capacity eviction is performed. The order of capacity eviction is as follows: first, eviction of the entry with the lowest eviction priority score from the short sequence area is performed until the total number of entries does not exceed the capacity limit or the short sequence area is cleared; if the capacity limit is still exceeded, then eviction of the entry with the lowest eviction priority score from the standard area is performed; wherein, the eviction priority score... The calculation method is as follows: in, To add 1 to the hit count for this entry, we prevent newly written entries from being immediately discarded because a zero-hit entry would have a score of zero. This is the current value of the global round counter; Add 1 to the denominator for the most recent hit time step of this entry to reflect time decay and prevent division by zero; The actual sequence length stored for this entry; The minimum write sequence length threshold, and The ratio is used to give longer sequences a higher retention priority.

[0007] In a preferred embodiment, the secondary feedback pool establishes a multi-level index layer with at least two different pattern lengths; when writing the corrected token sequence or subsequent token sequence determined by the target model at the location of the current verification failure into the secondary feedback pool, it further includes: Extract the context token fragment before the verification failure position, and extract token fragments of corresponding lengths according to the pattern length of each level in the multi-level index layer as the pattern key of each level. Then, write the same entry to be written into the corresponding level index layer according to the pattern key of each level. The step of performing pattern matching in the secondary feedback pool based on the token sequence to construct a secondary feedback candidate set includes: starting from the index layer corresponding to the longest pattern length, extracting the last token fragment of the token sequence as the current query pattern for retrieval; if the current index layer does not hit, then using the truncation suffix normalization method—removing the frontmost token of the current query pattern while keeping the last token unchanged—to obtain the next level query pattern, and stepping back to the index layer corresponding to the shorter pattern length for retrieval; stopping the backtracking after the first hit in any index layer, and adding the hit corrected token sequence or subsequent token sequences as candidate sequences to the secondary feedback candidate set; When the length of the token sequence is less than the shortest pattern length in the multi-level index layer, the retrieval in the secondary feedback pool is skipped, and the secondary feedback candidate set is empty. When the secondary feedback pool includes a standard area and a short sequence area, retrieval is prioritized in the index corresponding to the standard area within the same index level. If the standard area is not found, retrieval is then performed in the index corresponding to the short sequence area. A corresponding retrieval confidence weight is assigned based on the pattern length of the hit level, calculated as follows: in, In the first retrieval confidence weight when a level-one index is hit; For the first The length of the pattern corresponding to the level index; The longest pattern length among the multi-level index layers; the longer the matched pattern, the better. The closer the value is to 1.0, the higher the relevance of the candidate sequence to the current context. The capacity control of the multi-candidate structure under the maximum draft budget constraint includes: when truncation priority sorting needs to be performed among candidate sequences from different candidate sources, the candidate sequences in the secondary feedback candidate set are arranged in descending order according to the retrieval confidence weight to determine the retention priority; if multiple candidate sequences have the same retrieval confidence weight, they are arranged in descending order according to the hit count as the secondary sorting basis; when a retrieval hit occurs, the hit count and the most recent hit time step of the hit entry are updated; if no hit occurs at all levels of the index layer, the secondary feedback candidate set is an empty set, and the system reverts to a dual-source fusion mode that relies only on the historical matching candidate set and the distributed exploration candidate set, without updating the hit count of any entry.

[0008] In a preferred embodiment, constructing a candidate set for distribution exploration based on the token distribution output from the previous verification phase includes: Retrieve the next token logits generated in the previous verification phase, and select the top-k candidate tokens from them. }, among which the highest probability As the unified root token; for each Construct an extended prefix and query its subsequent matching sequence in the matching pool; if a match is found, form a sequence with the prefix... The initial candidate branch; if not found, the single-token branch is retained. As exploration candidates; thus obtaining the distribution exploration candidate set; Aligning at least some candidate sequences to a unified root token includes: if the first token of a candidate sequence in the historical matching candidate set or the secondary feedback candidate set is consistent with the unified root token, then removing its first token and using it as a subsequent candidate under the unified root token; if the first token of the candidate sequence is inconsistent with the unified root token, then retaining the candidate sequence as an independent candidate branch and adding it to the multi-candidate structure if capacity control allows. The step of assigning different maximum extension lengths to different candidate branches based on their rank includes: for the first candidate branch from the distributed exploration candidate set... There are 10 candidate branches, based on a preset branch budget function. Allocate the maximum extension length; wherein This represents the rank order determined based on the top-k candidate tokens; candidate branches with higher rank are assigned a longer maximum extension length, and candidate branches with lower rank are assigned a shorter maximum extension length; the branch budget function... It is any one of the following: a lookup function, a linear decay function, an exponential decay function, or a dynamic function related to the logits score; The capacity control of the multi-candidate structure includes: dynamically calculating the overall size of the candidates during the process of adding candidate sequences to the multi-candidate structure; if the newly added candidate sequence will cause the overall size of the candidates to exceed the maximum draft budget, then only the longest prefix portion that can be accommodated in the candidate sequence is retained, or the addition of subsequent candidate sequences is stopped.

[0009] This application also provides a two-stage feedback pool management method for speculative sampling in an autoregressive generative model inference system, the inference system including a target model, the method comprising: After the target model performs verification on the candidate draft sequence, the corrected token sequence or subsequent token sequence determined by the target model at the verification failure position is obtained; Extract the context token fragment before the verification failure position, and construct a pattern key according to the preset pattern length; The pattern key is associated with the corrected token sequence or subsequent token sequence and written into the secondary feedback pool; In subsequent rounds, a query pattern is constructed based on the terminal token fragment of the currently generated prefix. Candidate sequences matching the query pattern are retrieved from the secondary feedback pool and used as secondary feedback candidates to participate in the candidate draft construction. The hit count and the most recent hit time step are updated for the retrieved entries. When the total number of entries in the secondary feedback pool exceeds the preset capacity limit, elimination is performed based on the entry's hit information and sequence length.

[0010] In a preferred embodiment, before writing to the secondary feedback pool, a write pre-screening is performed: if the length of the sequence to be written is zero, it is discarded; a duplicate detection is performed on the sequence to be written, and if a candidate sequence that is completely identical to the sequence to be written already exists in the secondary feedback pool, it is not written repeatedly, and the hit count and the most recent hit time step of the existing entries are updated; the total number of entries to be written in the current round is counted, and if it exceeds the product of the preset single-round writing ratio limit and the capacity limit, only the first few entries with the longest actual sequence length are retained for writing; The secondary feedback pool includes a standard area and a short sequence area. During writing, the tokens are written to the corresponding storage areas according to the comparison results between the length of the modified token sequence or subsequent token sequence and the preset minimum write sequence length threshold. During eviction, the item with the lowest eviction priority score is first evictioned from the short sequence area. If the short sequence area is still overcapacity after being cleared, it is then evictioned from the standard area. The secondary feedback pool establishes a multi-level index layer with at least two different pattern lengths. During retrieval, the query starts from the index layer corresponding to the longest pattern length. If no match is found, the query is backtracked step by step to the index layer corresponding to the shorter pattern length using the truncation suffix normalization method. The backtracking stops after a match is found at a certain level. The retrieval confidence weight is assigned to the candidate sequence that has been hit based on the ratio of the pattern length of the hit level to the pattern length of the longest level.

[0011] This application also provides a multi-sequence speculative sampling system for hybrid candidate source fusion, applied to an inference system for an autoregressive generative model, wherein the inference system includes a target model, and the system includes: The historical candidate construction module is used to construct a set of historical matching candidates based on the matching results of the token sequence corresponding to the currently generated prefix in the historical token sequence; The feedback candidate construction module is used to perform pattern matching in the secondary feedback pool based on the token sequence to construct a secondary feedback candidate set, wherein the secondary feedback pool is used to organize the corrected token sequence or subsequent token sequence determined by the target model at the historical verification failure position according to the pattern. The candidate exploration module is used to construct a set of candidate distributions based on the token distribution output from the previous verification phase. The candidate fusion module is used to align at least a portion of the candidate sequences in the historical matching candidate set, the secondary feedback candidate set, and the distributed exploration candidate set to a unified root token, and to perform deduplication and merging on the aligned candidate sequences to form a multi-candidate structure. The budget allocation module is used to allocate different maximum extension lengths to different candidate branches in the multi-candidate structure based on the candidate branch's rank, candidate source, or retrieval confidence weight. A capacity control module is used to perform capacity control on the multi-candidate structure under the maximum draft budget constraint, so as to truncate or stop adding candidate branches that exceed the budget; The verification module is used to submit the capacity-controlled multi-candidate structure to the target model for verification, and determine the current round output token based on the continuous acceptance length of each candidate branch. The feedback writing module is used to write the corrected token sequence or subsequent token sequence determined by the target model at the position of the verification failure in this round into the secondary feedback pool, so that the feedback candidate construction module can retrieve it in subsequent rounds.

[0012] This application also provides an electronic device, including a processor and a memory; the memory is used to store computer programs or instructions; the processor is used to execute the computer programs or instructions to implement the above-described method.

[0013] This application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the above-described method.

[0014] This application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the above-described method.

[0015] The technical solution proposed in this application uses the token sequence corresponding to the currently generated prefix as a unified input basis, and coordinates with each other at multiple levels, including candidate source organization, feedback pool reuse, branch budget allocation, and memory capacity control, to achieve a systematic improvement in resource utilization during the target model verification stage of the autoregressive generative model inference system. The following section elaborates on the technical effects produced by each technical means and their synergistic cooperation.

[0016] At the end-to-end overall method level, this method, in each generation iteration, is based on the current generation prefix. Construct candidate sets of historical matches respectively Secondary feedback candidate set and distribution exploration candidate set Three candidate sequences from different sources were used to align the three types of candidate sequences to a unified root token. After deduplication and merging are performed to form a multi-candidate structure, different maximum extension lengths are assigned to different candidate branches based on their rank, source, or retrieval confidence weight. Capacity control is then performed under the maximum draft budget constraint before submitting to the target model for one-time validation. The corrected token sequence or subsequent token sequence determined by the target model at the validation failure location is written to the secondary feedback pool. Historical matching candidates can utilize continuation segments already appearing in historical token sequences, secondary feedback candidates can reuse the true subsequent token sequences determined by the target model at historical validation failure locations, and distribution exploration candidates supplement the candidate starting points under the current output distribution based on the token distribution output in the previous validation phase. All three participate in candidate construction in the same round of inference, increasing the number and diversity of effective candidate paths under the same maximum draft budget, which helps improve the average number of accepted tokens per validation by the target model. The deduplication and merging operations after aligning to a unified root token eliminate the duplicate occupation of node budget by redundant candidate sequences from different candidate sources, allowing the limited draft budget to be concentrated on truly differentiated candidate paths, increasing the proportion of effective candidates in the candidate structure. The multi-candidate structure can be organized in the form of candidate trees, tries, arrayed trees, or node tables, merging shared prefixes in the data structure. This facilitates mapping to the input data such as token sequences, position IDs, attention masks, and branch IDs required for batch verification by the target model. Furthermore, this method does not rely on additional candidate draft models; instead, it directly constructs multi-branch candidates through hybrid candidate source fusion, eliminating the need for additional GPU memory and computational overhead during deployment. The "verification → writing → retrieval → re-verification" feedback loop formed by steps 300 and 900 transforms the true output given by the target model at the verification failure location from one-time error correction information into a reusable candidate source, improving the hit probability and verification pass rate of subsequent candidates in scenarios where similar prefixes recur.

[0017] At the write management level of the secondary feedback pool, the write process is not triggered when the system determines that the current round of verification results are all accepted and there are no verification failure positions, thus avoiding the waste of storage resources by invalid write operations. The pre-screening process performed before writing includes truncating when the corrected token sequence encounters a terminator token (preventing invalid tokens from entering the feedback pool), discarding zero-length entries (preventing entries with no substantial content from occupying capacity), duplicate detection and hit count updates for existing completely identical candidate sequences (eliminating redundant entries and increasing the diversity of effective candidates under limited capacity), and handling situations where the total number of entries to be written in a single round exceeds a preset proportion limit. With capacity limit The product process retains only the entry with the longest actual sequence length—this collectively filters out low-contribution and redundant entries, and prevents new entries from flooding in and overwriting historically high-value entries during a round of abnormally large-scale validation failures, thus maintaining the temporal stability of the secondary feedback pool. The secondary feedback pool is divided into standard areas based on the length of the written sequence. and short sequence regions The hierarchical storage structure, coupled with a strategy that prioritizes shorter sequence areas over standard areas during eviction, extends the lifespan of entries with longer sequence lengths and higher reuse value in the standard area, even under limited pool capacity. Longer sequences can accept more tokens per verification in subsequent checks, thus increasing their reuse value and improving the effective candidate density. Multiple different candidate sequences can be stored under the same pattern key, with a preset maximum number of records. This is the upper limit; when the number of records under this key reaches... The record with the lowest hit count is evicted before being written, increasing candidate diversity under the same pattern while controlling retrieval noise. For capacity eviction, a eviction priority score is used. The composite evaluation mechanism integrates hit frequency ( Adding 1 prevents newly written entries from being immediately evicted due to zero hits, and time decay ( The entry that is further back from the nearest hit scores lower (add 1 to prevent division by zero) and sequence length ( The three factors (giving higher retention priority to longer sequences) help differentiate the actual reuse value of different entries under the same capacity limit compared to a pure LRU eviction strategy—for example, distinguishing between "long sequences that occasionally hit but are accepted every time" and "short sequences that frequently hit but are only accepted with a single token each time." A batch cleanup mode triggered when the amount of eviction in a single instance is too large can quickly restore the pool capacity to a manageable state.

[0018] At the level of multi-level pattern length indexing and step-by-step backtracking retrieval, the secondary feedback pool is based on at least two different pattern lengths. A multi-level index is established. During writing, the same entry is written to the corresponding index level using the pattern key at each level. During retrieval, the entry is retrieved from the index with the longest pattern length. Starting from the corresponding index layer, within the same index layer, standard regions are prioritized over short sequence regions for retrieval. If no match is found in the current index layer, a truncation and suffix normalization method is used—removing the leading token of the query pattern while keeping the trailing token unchanged—and gradually backtracking to the index layer corresponding to the shorter pattern. Backtracking stops after the first hit at a certain level. This mechanism, while maintaining priority for exact matches, allows the system to still obtain candidates through shorter patterns when exact long patterns are not found, reducing the probability of an empty retrieval set and improving the retrieval recall rate of the secondary feedback pool in low-frequency pattern scenarios. A retrieval confidence weight is assigned based on the pattern length of the hit level. Driven truncation sorting during capacity control phase -- Higher-ranking candidates are preferred for retention. Lower-ranking candidates are prioritized for truncation when capacity is limited—adapting truncation decisions to the matching accuracy between the candidate sequence and the current context. Upon a successful retrieval, the hit count and the most recent hit time step are updated for the matched entry, ensuring that the elimination priority score reflects the actual retrieval usage of the entry. When all levels fail to match, the secondary feedback candidate set is empty, the system reverts to a dual-source fusion mode of historical matching candidate sets and distributed exploration candidate sets, and no entry hit count is updated. This approach ensures the system can still function normally during cold starts or in low-frequency pattern scenarios. Retrieving a prefix shorter than the pattern length of the shortest index level is skipped directly, avoiding invalid retrieval operations.

[0019] At the level of candidate construction and branch budget allocation, the system selects the top-k candidate tokens from the logits distribution of the previous verification phase. The highest probability As a unified root token, for each Construct an extended prefix and query subsequent matching sequences in the first-level matching pool. If a match is found, a sequence is formed. If the initial candidate branch is not found, a single-token branch is retained as an exploration candidate, thus forming a distributed exploration candidate set. This step utilizes information from the current output distribution of the target model to explore the continuation possibilities under different starting tokens, compensating for the limitation of insufficient recall of historical matching candidates when faced with novel contexts. For sequences in historical matching candidates and secondary feedback candidates where the first token matches the root token, the first token is removed before being appended; sequences where they do not match are retained as candidate candidates. This achieves an ordered organization of three types of heterogeneous candidate sources within a unified structure. (Branch budget function) Different maximum extension lengths are assigned to different candidate branches based on their rank. Higher-ranked candidate branches receive longer validation depths to fully utilize their continuous acceptance potential, while lower-ranked candidate branches are limited to shorter exploration ranges to avoid wasting candidate nodes. This function can employ a lookup table function, a linear decay function, an exponential decay function, or a dynamic function related to logits scores, or it can be correlated with candidate source confidence or retrieval hit level. Capacity control helps prevent candidate structures submitted to the target model from becoming too large, causing computational overhead and memory usage during the validation phase to exceed hardware capacity. This is achieved by dynamically calculating the overall size of the candidates and retaining only the longest accommodating prefix portion or stopping subsequent candidate additions when new candidates would exceed the maximum draft budget. Branch budget control and capacity control work together to prioritize candidate paths more likely to be accepted by the target model within a fixed budget.

[0020] At the independent management level of the secondary feedback pool, this management method covers the entire lifecycle from obtaining the corrected token sequence at the verification failure position, writing it according to the pattern key association, retrieving and updating the hit information according to the current prefix in subsequent rounds, to performing elimination based on the hit information and sequence length when the capacity is exceeded, forming a closed-loop management of "write-retrieve-update-elimination". This management method enables the secondary feedback pool to independently provide reusable feedback on the actual output of the target model for candidate draft construction, even in inference systems that do not adopt a complete three-source fusion scheme. When the above management method further introduces write pre-screening, hierarchical storage of standard area and short sequence area, step-by-step back retrieval of multi-level index layer, and retrieval confidence weight based on the pattern length ratio, the storage efficiency, retrieval recall and capacity utilization of the secondary feedback pool are all improved in a targeted manner at their respective levels. The synergistic effect enables the secondary feedback pool to maintain a high effective candidate density and retrieval efficiency under limited memory conditions.

[0021] At the system level, the historical candidate construction module, feedback candidate construction module, and exploration candidate construction module generate three types of candidate sets and output them to the candidate fusion module. The candidate fusion module outputs multiple candidate structures, which are then processed sequentially by the budget allocation module and the capacity control module before being submitted to the verification module. The verification results from the verification module are output to the feedback writing module to update the secondary feedback pool. These modules work collaboratively through data flow, forming a complete processing chain from candidate construction to verification output and then to feedback writing. At the electronic device level, the processor executes computer programs or instructions stored in memory to implement the above method. The index structure and entries of the secondary feedback pool reside in memory, while the forward inference operation of the target model is executed on the processor (GPU, NPU, inference accelerator card, or general-purpose processor). This allows the method to be deployed on different types of electronic devices such as servers, workstations, edge computing devices, or edge inference devices. The technical solution of computer-readable storage media and computer program products enables the above method to be deployed and executed in software form on multiple hardware platforms.

[0022] The aforementioned processes collectively contribute to the internal execution efficiency of the inference system: candidate source fusion enhances the complementarity of candidate sources; the secondary feedback pool preserves the true outputs from failed target model verification locations; root token alignment and deduplication reduce candidate structure redundancy; branch budget and maximum draft budget control the scale of verification inputs; and target model verification and failure write-back form a closed-loop feedback loop. These techniques do not implement a single sampling rule in isolation, but rather form an organically unified technical solution around candidate construction, candidate reuse, candidate organization, and capacity constraints in the target model verification phase. Together, they serve the systematic performance improvement goals of increasing the average number of tokens accepted per verification of the target model, reducing the number of forward calls, and lowering inference latency.

[0023] The specification of this application contains numerous technical features distributed across various technical solutions. Listing all possible combinations of these technical features (i.e., technical solutions) would make the specification excessively lengthy. To avoid this problem, the various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which should be considered as described in this specification), unless such a combination of technical features is technically infeasible. For example, one example discloses feature A+B+C, and another example discloses feature A+B+D+E. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; they cannot be used simultaneously. Feature E can technically be combined with feature C. Therefore, the solution A+B+C+D should not be considered as described because it is technically infeasible, while the solution A+B+C+E should be considered as described. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall process of a multi-sequence speculative sampling method for Hybrid candidate source fusion according to an embodiment of this application.

[0025] Figure 2 This is a schematic diagram illustrating the writing, retrieval, step-by-step rollback, and capacity eviction process of a secondary feedback pool according to an embodiment of this application.

[0026] Figure 3 This is a schematic diagram illustrating the multi-candidate structure, branch budget allocation, capacity control, and target model batch verification input mapping according to embodiments of this application.

[0027] Figure 4 This is a structural block diagram of a multi-sequence speculative sampling system and electronic device for Hybrid candidate source fusion according to an embodiment of this application. Detailed Implementation

[0028] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0029] Explanation of some concepts: A token is a discrete symbol unit generated by a word segmenter and processed by an autoregressive generative model.

[0030] The pattern refers to the matching key consisting of one or more consecutive tokens at the end of the current generated prefix.

[0031] The root token is the starting token shared by multiple candidate branches in the candidate structure.

[0032] Target model verification refers to the process of submitting candidate draft sequences or multiple candidate structures to the target model, and having the target model perform acceptance or rejection judgments on each position of the candidate tokens and determine the continuous acceptance length.

[0033] A candidate draft sequence is a sequence of tokens to be verified constructed by the inference system before the target model is submitted for verification.

[0034] The historical matching candidate set refers to the set of candidate sequences constructed based on the matching results of the token sequence corresponding to the current generated prefix in the historical token sequence.

[0035] The secondary feedback pool (secondary pool) refers to the searchable candidate source of corrected token sequences or subsequent token sequences determined by the target model at the historical verification failure positions according to the pattern.

[0036] The secondary feedback candidate set refers to the set of candidate sequences obtained by matching the token sequence corresponding to the current generated prefix with the pattern in the secondary feedback pool.

[0037] The distributed exploration candidate set refers to the set of candidate sequences constructed based on the token distribution output from the previous verification phase.

[0038] A multi-candidate structure refers to a candidate organization structure formed by aligning, deduplicating, and merging candidate sequences from at least one candidate source, and used to submit the target model for one-time validation.

[0039] Branch budget refers to the maximum number of extension tokens allocated to a candidate branch.

[0040] The maximum draft budget (maximum candidate budget) refers to the maximum candidate size constraint allowed for a multi-candidate structure before submitting it for target model verification. It is used to match hardware computing power and GPU memory capacity. The maximum draft budget is a positive integer, and its measurement object can be the total number of candidate nodes, the total number of candidate tokens in the multi-candidate structure, or the equivalent candidate size converted from the available GPU memory budget of the inference device. In the same inference task, the system uses the same measurement method to count the overall candidate size.

[0041] The consecutive acceptance length refers to the number of tokens that a candidate sequence passes the target model's verification consecutively, starting from its initial position.

[0042] The standard area refers to the logical storage partition within the secondary feedback pool used to store entries whose sequence length is not less than the preset minimum write sequence length threshold.

[0043] The short sequence area refers to the logical storage partition within the secondary feedback pool used to store entries whose sequence length is greater than zero and less than the preset minimum write sequence length threshold.

[0044] The retrieval confidence weight is a weight value calculated based on the ratio of the length of the pattern at the hit level to the length of the longest level pattern in the secondary feedback pool. It is used to characterize the matching accuracy between the candidate sequence and the current context.

[0045] The elimination priority score is a composite evaluation value calculated by combining the hit count of an item, the most recent hit time step, and the sequence length. It is used to determine the retention or elimination priority of each item when the capacity of the secondary feedback pool exceeds the limit.

[0046] Truncation suffix normalization refers to the operation of removing the leading token of the current query pattern while keeping the trailing token unchanged during the step-by-step back-down retrieval process of a multi-level index in order to construct the next level query pattern.

[0047] In the embodiments of this application, k in top-k is a positive integer; the maximum number of stored records under the same pattern key. It is a positive integer; the upper limit of the secondary feedback pool capacity. A positive integer; minimum write sequence length threshold An integer not less than 1; maximum single-round write ratio. satisfy The set of multi-level pattern lengths satisfies ,in The longest pattern length. This is the shortest pattern length.

[0048] The following is a brief summary of some of the innovative aspects of this application: In summary, the technical solution of this application does not simply stack historical matching, logits exploration, or cache reuse in parallel. Instead, it addresses the system performance bottleneck of the continuous acceptance length of a single validation of the target model in an autoregressive generative model inference system by establishing a mutually constraining and interdependent collaborative mechanism among candidate source fusion, feedback pool organization, branch budget, and capacity control. Specifically, the historical matching candidate set... (Step 200) Secondary Feedback Candidate Set (Step 300) and the distribution of candidate sets (Step 400) The functional roles of the three types of candidate sources are heterogeneous and cannot be substituted for each other. Provides the ability to reuse long historical segments. Provides the true subsequent token sequence determined by the target model at the verification failure location. Feedback reuse capability Provides the ability to explore the current output distribution—and all three must be accessed via a unified root token. The alignment, deduplication, and merging operations (step 500) are necessary to form a multi-candidate structure that can be verified by the target model in one go. Without this alignment and deduplication step, repeated sequences from different candidate sources will redundantly occupy node budgets, causing the branch budget function to... The capacity control in steps 600 and 700 fails to make reasonable resource allocation and truncation decisions between redundant and differential paths. More importantly, the secondary feedback pool is not a typical cache structure; it uses the pattern near the verification failure location as the key and... A specific organization method for values, and multi-level indexing. The step-by-step backtracking retrieval mechanism with truncation and suffix normalization (steps 310-350), and the retrieval confidence weight. The assignment rules and capacity control phase are as follows The descending order of the sorting logic for determining truncation and retention priorities forms a continuous technical chain, encompassing write timing, index construction, retrieval strategies, and capacity control decisions. This chain transforms the target model's actual output at the verification failure location from a one-time error correction message into a candidate source that can be continuously retrieved according to the pattern and participate in capacity competition with a priority commensurate with its matching accuracy. The write operation in step 900 and the retrieval operation in step 300 thus constitute a feedback loop of "verification → write → retrieval → re-verification." Simultaneously, the hierarchical storage structure (standard area) With short sequence region ) and elimination priority score --Comprehensive Hit Count Time decay factor With sequence length factor -- Together with the short-sequence region priority elimination strategy, we ensure limited capacity. Longer sequence entries gain longer lifespans. This elimination criterion is directly related to the number of consecutively accepted tokens and the actual reuse value of the entries during the target model's verify phase, rather than an arbitrary cache replacement strategy. The aforementioned three-source fusion, root-aligned deduplication, feedback closed-loop writing and retrieval, rank branch budget, and capacity control are interdependent and cooperate with each other to form a complete processing chain. This collaborative approach is not readily apparent to those skilled in the art when facing individual problems such as limited information dimensions from a single candidate source or the lack of accumulated feedback from verification failures.

[0049] Furthermore, through long-term and in-depth research on speculative sampling techniques in autoregressive generative model inference systems, the inventors of this application have discovered that existing solutions have several interrelated technical bottlenecks in the organization and resource utilization of candidate sources for target model verification. The root cause of these bottlenecks is not the inadequacy of a single link, but the lack of systematic coordination among the four links of candidate construction, verification feedback handling, branch budget allocation, and candidate structure capacity management.

[0050] Through analysis of numerous real-world inference task execution trajectories, the inventors discovered that relying solely on a single candidate source based on historical N-gram matching leads to decreased candidate recall when faced with novel contexts or low-frequency patterns, resulting in fluctuations in the length of consecutive acceptances. Furthermore, the true subsequent token sequences provided by the target model at validation failure locations contain the most accurate continuation information for the current context; however, some existing solutions simply use this information for correction in the current step before discarding it. The inventors recognized that organizing these true outputs into searchable candidate sources according to contextual patterns and re-participating them in candidate construction when similar prefixes appear later can form a continuously accumulating feedback loop. However, the effective operation of this loop depends on the coordinated support of write filtering, hierarchical storage, multi-level index retrieval, and capacity eviction.

[0051] The inventors also observed that in multi-candidate source fusion scenarios, candidate branches from different sources and with varying confidence levels significantly differ in their contribution to validation. If all branches are allocated the same extension depth indiscriminately, low-confidence branches will crowd out the resource space of high-confidence candidates. Furthermore, without alignment and deduplication of multi-source candidates, identical sequences will repeatedly occupy node budgets. Based on this, the inventors recognized that differentiated branch budgets must be implemented according to rank, candidate source, and retrieval confidence weights, and capacity control must be performed under global draft budget constraints to ensure that candidate structures prioritize candidate paths more likely to be accepted by the target model within a fixed budget. Based on the above in-depth research, the inventors proposed a Hybrid candidate source fusion multi-sequence speculative sampling scheme with a two-level feedback pool as its core. The implementation process of this application is described in detail below through specific embodiments.

[0052] To make the objectives, technical solutions, and beneficial effects of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this application. Unless otherwise specified, the embodiments and features described in this application can be combined with each other. Those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope thereof.

[0053] In the following embodiments, the method described in this application is applied to an inference system for an autoregressive generative model. This inference system includes a target model, which is a large language model to be accelerated for inference. The target model performs forward inference operations on a processor (e.g., GPU, NPU, inference accelerator card, or general-purpose processor). The core objective of the inference system is to increase the average number of tokens accepted per verification by the target model, thereby reducing the number of forward calls to the target model, lowering single-token inference latency, and improving overall inference throughput, without relying on additional candidate draft models or with only extremely low draft costs. This is achieved by constructing multiple candidate draft sequences and submitting them to the target model for one-time verification.

[0054] The following defines the key terms used in this application. "Token" refers to a discrete symbolic unit generated by the tokenizer and processed by the autoregressive generative model. "Pattern" refers to a matching key consisting of one or more consecutive tokens at the end of the current prefix. "Root token" refers to the starting token shared by multiple candidate branches in the candidate structure. "Branch budget" refers to the maximum number of extended tokens allocated to a candidate branch. "Continuous acceptance length" refers to the number of tokens that a candidate sequence continuously passes the target model verification from its starting position. "Secondary feedback pool" refers to a searchable candidate source of corrected token sequences or subsequent token sequences determined by the target model at historical verification failure positions according to the pattern. "Draft sequence" refers to a sequence of tokens to be verified constructed by the system before submission for target model verification. "Standard area" and "short sequence area" refer to two logical storage partitions within the secondary feedback pool, divided according to the length of the written sequence—the standard area stores entries with a sequence length not less than a preset minimum written sequence length threshold, and the short sequence area stores entries with a sequence length less than this threshold but greater than zero.

[0055] Example 1: A Multi-Sequence Speculative Sampling Method for Fusion of Three Hybrid Candidate Sources This embodiment describes the main flow of the core method of this application, namely the end-to-end speculative sampling process. This process is executed cyclically in each generation iteration of the inference system, see [link to documentation]. Figures 1 to 4 The specific steps are as follows: Step 100: Obtain the token sequence corresponding to the current generated prefix. At the beginning of each generation iteration, the inference system reads all the output tokens that the target model has already determined, concatenates them with the original input tokens, and forms the current generated prefix. ,in This is the total length of the current prefix. This prefix is ​​stored in the inference system's memory and serves as the base input for all subsequent candidate construction steps.

[0056] Step 200: Based on the matching results of the currently generated prefix in the historical token sequence, construct a historical matching candidate set. Specifically, the system retrieves multiple historical matches corresponding to the current pattern from the primary matching pool. The primary matching pool stores token sequences generated during the historical generation process or externally provided reference text. The system extracts the current prefix. The last few tokens are used as a suffix pattern. The historical positions that match this pattern are queried in the primary matching pool. The subsequent token sequences after each matching position are extracted to form a historical matching candidate set. The data source for the primary matching pool can be the generated context of the current inference request, pre-indexed reference text, or generated sequences from other requests in the same batch, depending on the configuration of the inference system. In multi-level matching mode, if the exact pattern is not matched, relevant patterns can be retrieved further through normalized fallback. The purpose of this step is to leverage continuation patterns that have appeared in historical text to provide long-segment reuse capabilities for candidate construction.

[0057] Step 300: Based on the currently generated prefix, perform pattern matching in the secondary feedback pool to construct a secondary feedback candidate set. The system uses the current prefix... The system uses the last token fragment as the query pattern and retrieves candidate sequences matching that pattern from the secondary feedback pool. The entries stored in the secondary feedback pool originate from the failure positions of the target model during historical validation—when a candidate draft sequence is rejected by the target model during validation, the actual subsequent token sequence given by the target model at that rejection position is extracted, organized according to the pattern, and written into the secondary feedback pool. The key difference between the secondary feedback pool and a regular cache is that its stored entries are not arbitrary historical token sequences, but rather inference feedback candidate sources with the pattern near the validation failure position as the key and the actual subsequent token sequence determined by the target model at that position as the value. Because these sequences come directly from the actual output of the target model, their consistency with the target model's distribution is high, thus increasing the probability of being accepted again in subsequent similar contexts. After a successful retrieval, the system uses the matched corrected token sequence or subsequent token sequence as candidate sequences to form a secondary feedback candidate set. The core value of this step lies in transforming the target model output, originally intended only for error correction in the current step, into a reusable candidate source for subsequent rounds, forming a feedback loop of "verification → writing → retrieval → re-verification". The detailed writing, retrieval, and management mechanisms of the secondary feedback pool will be explained in Examples 2 to 4.

[0058] Step 400: Based on the token distribution output from the previous verification phase, construct a candidate distribution exploration set. The system reads the next token logits distribution generated in the previous verification phase and selects the top-ranked tokens from it. One candidate token { The highest probability among them. This serves as the unified root token for the current round. For each candidate token... System construction extended prefix It then queries its subsequent matching sequence in the primary matching pool. If a match is found, a sequence is formed. The initial candidate branch; if not found, the single-token branch is retained. [ ] as exploration candidates. This yields a distributed exploration candidate set. The purpose of this step is to utilize information about the current output distribution of the target model to explore the continuation possibilities under different starting tokens, thus compensating for the limitation of insufficient recall of historical matching candidates when faced with novel contexts.

[0059] In the first iteration of inference, if the token distribution output from the previous verification phase does not exist, the system can call the target model based on the current generated prefix to obtain the next token logits, and use these next token logits as input to construct the distribution exploration candidate set; alternatively, the system can set the distribution exploration candidate set to empty and construct the candidate structure only based on the historical matching candidate set and the secondary feedback candidate set. In subsequent iterations, the system constructs the distribution exploration candidate set based on the token distribution output from the previous verification phase.

[0060] The three types of candidate sources each play a different functional role: historical matching candidates provide the ability to reuse long segments, secondary feedback candidates provide the ability to reuse feedback from the actual output of the target model, and distribution exploration candidates provide the ability to explore the current output distribution. Incorporating all three into the candidate construction process improves the coverage of multiple candidates compared to a single candidate source. It should be noted that the execution order of steps 200 to 400 is not strictly limited; they can be executed sequentially, or partially or entirely in parallel, as long as all three candidate sets are ready before proceeding to step 500.

[0061] Step 500: Align at least a portion of the candidate sequences from the historical matching candidate set, the secondary feedback candidate set, and the distributed exploration candidate set to a unified root token, and perform deduplication and merging on the aligned candidate sequences to form a multi-candidate structure. Specifically, for and For each candidate sequence, the system checks whether its first token matches the unified root token. If they match, the first token of the candidate sequence is removed, and the remaining part is attached to the multi-candidate structure as a subsequent candidate under the root token; if they do not match, the candidate sequence is retained as a candidate alternative. After alignment, the system performs deduplication on all candidate sequences—if two candidate sequences from different sources are completely identical after alignment, only one is retained to avoid duplicate occupation of candidate node budget. The deduplicated candidate sequences are merged and organized into a unified multi-candidate structure. This multi-candidate structure can be organized in memory using data structures such as candidate trees, tries, arrayed trees, or node tables. The above process can be represented as: in This indicates the operation of aligning candidate sequences to a unified root token. This indicates a deduplication operation. This indicates the merging of sets. This forms the final set of fused candidates. The technical significance of root alignment and deduplication is that candidate sequences from three different sources are organized uniformly in the same structure, avoiding the waste of candidate budget by duplicate nodes among multi-source candidates, concentrating validation resources on truly different candidate paths, and enabling the target model to validate multiple candidate branches simultaneously in a single forward computation.

[0062] Step 600: Assign different maximum extension lengths to different candidate branches in the multi-candidate structure based on the candidate branch's rank, candidate source, or retrieval confidence weight. Specifically, for the first candidate branch from the distributed exploration candidate set... There are 10 candidate branches, and the system uses a preset branch budget function. The maximum extension length is allocated accordingly. Candidate branches with higher rank (i.e., higher probability of logits) are assigned longer maximum extension lengths, thus having greater potential for continuous acceptance during verification; candidate branches with lower rank (i.e., lower probability of logits) are assigned shorter maximum extension lengths, retaining only limited exploration depth. The technical effect of this differentiated allocation is that, under a fixed total candidate node budget (draft budget), the system allocates more resources to candidate directions with higher probability of acceptance, while not completely abandoning exploration opportunities in low-probability directions, thereby achieving a better balance between candidate coverage and resource utilization. Branch budget function It can be implemented using any of the following: a lookup function, a linear decay function, an exponential decay function, or a dynamic function related to the logits score. It can also be associated with candidate source confidence or retrieval hit level. The specific form of the branch budget function will be further explained in Example 5.

[0063] Step 700: Perform capacity control on the multi-candidate structure under the maximum draft budget constraint to truncate or stop adding candidate branches that exceed the budget. During the process of gradually adding candidate sequences to the multi-candidate structure, the system dynamically calculates the overall candidate size (e.g., the total number of candidate nodes). If a newly added candidate sequence would cause the overall candidate size to exceed the preset maximum draft budget, the system will only retain the longest prefix portion that can be accommodated in that candidate sequence, or directly stop adding subsequent candidate sequences. This capacity control mechanism helps prevent the computational overhead and memory usage during the verification phase from exceeding the hardware's capacity due to the candidate structure being too large.

[0064] Furthermore, when truncation priority ranking needs to be performed among candidate sequences from different candidate sources, candidate sequences from the secondary feedback candidate set can be sorted in descending order of retrieval confidence weight to determine retention priority. Candidates with higher retrieval confidence weights are retained first, while candidates with lower retrieval confidence weights are truncated first when capacity is limited. If multiple candidate sequences have the same retrieval confidence weight, they are sorted in descending order of hit count as a secondary ranking criterion. This mechanism ensures that exact matching candidates are retained first when capacity is limited, and candidates with different matching levels receive retention priorities commensurate with their matching accuracy. At the same time, candidate branches exceeding their allocated maximum extension length are also truncated first. Thus, branch budget control and capacity control work together to improve the efficiency of target model validation. The specific calculation method of retrieval confidence weights will be explained in Example 3.

[0065] Step 800: Submit the capacity-controlled multi-candidate structure to the target model for verification, and determine the output token for the current round based on the consecutive acceptance length of each candidate branch. The target model performs parallel verification on all candidate branches in the multi-candidate structure. For each candidate branch, the target model starts from the beginning of the branch and checks each candidate token to see if it matches the output distribution of the target model at the corresponding position. The number of consecutively verified tokens is the consecutive acceptance length of that branch. The system selects the path with the longest consecutive acceptance length from all candidate branches, determines the accepted tokens on that path as the output result of the current round, and updates the current generation prefix.

[0066] Step 900: Write the corrected token sequence or subsequent token sequence determined by the target model at the position where the verification failed in this round into the secondary feedback pool for retrieval when constructing the secondary feedback candidate set in subsequent rounds. For a candidate branch rejected at a certain position during the verification process, the system extracts the actual output token sequence given by the target model at the rejection position and its subsequent positions, and extracts the context token fragment before the failure position to construct a patternkey according to preset rules. This patternkey is then associated with the corresponding actual output token sequence and written into the secondary feedback pool. Further, if the current round's verification result is all accepted (i.e., there are no verification failure positions), the writing process to the secondary feedback pool is not triggered. If the corrected token sequence or subsequent token sequence encounters an end-of-strike (EOS) token, it is truncated to the position before the end-of-strike token before writing is performed.

[0067] Steps 100 to 900 constitute a complete generation iteration loop. During inference, this loop is executed repeatedly until the target model outputs the end-of-line token or reaches the preset maximum generation length. Steps 300 and 900 together form a feedback loop of "verification → writing → retrieval → re-verification"—the actual output given by the target model at the verification failure point is no longer a one-time error correction information, but is accumulated as a sustainable and reusable candidate source, participating again in the construction of candidate drafts when similar prefixes appear later. As the rounds progress, the number of feedback candidates accumulated in the secondary feedback pool gradually increases, and in long-term operation, it automatically enriches the correction sequence of the region with the largest deviation between the target model distribution and the candidate draft distribution.

[0068] Example 2: Writing and Feedback Reuse Process of the Secondary Feedback Pool This embodiment elaborates in detail on the specific writing and retrieval mechanism of the secondary feedback pool involved in steps 300 and 900 in Embodiment 1.

[0069] The writing process to the secondary feedback pool includes the following sub-steps.

[0070] Step 910: Initialize the hierarchical storage structure of the secondary feedback pool. Before inference begins, the system creates two storage areas in memory for the secondary feedback pool. Standard area Used to store write sequence lengths that are not less than a preset minimum write sequence length threshold. Entries; short sequence region Used to store write sequence lengths greater than zero but less than zero. The entries. Both storage areas use a hash dictionary structure, where the key is a tuple of patterntoken and the value is a list of candidate entries. Each candidate entry contains the following fields: the actual subsequent sequence. Write time step Hit count (Initial value is 0), most recent hit time step (Initially equal to) ) and sequence length The system simultaneously initializes the global entry counter. and global round counter The system also sets a capacity limit for the secondary feedback pool. Its value is determined by the available memory of the inference service and the length of the target model context.

[0071] The technical significance of this hierarchical storage structure is that, during subsequent capacity eviction, entries in the short-sequence area are prioritized for eviction, thereby allowing long-sequence entries with higher reuse value in the standard area to have a longer lifespan, and improving the effective candidate density of the secondary feedback pool under limited capacity.

[0072] Step 920: Verify the actual output sequence at the failed position. In step 800 of Example 1, when a candidate draft sequence is at position... When verification fails, the system extracts the target model at the location. The actual output token sequence given in the following positions. ,in This represents the number of valid tokens continuously output by the target model at this position. If... If the system encounters a terminator token (EOS), it truncates the string to the position before the terminator token. Simultaneously, the system extracts the context token fragment preceding the failed position and constructs the corresponding pattern key according to the preset pattern length. If the prefix length of the failed position is insufficient to form a pattern key at any level, no entry is generated for that failed position.

[0073] Step 930: Write Pre-screening. Before performing the actual write operation, the system performs the following checks on each entry to be written: Step 931 -- Length Filtering: Record . like If the value is zero (meaning the target model did not produce valid output at the verification failure point), the entry is discarded directly, and no subsequent write steps are performed. This filter prevents entries with no substantial content from occupying the secondary feedback pool capacity.

[0074] Step 932 – Duplicate Detection: The system checks the secondary feedback pool to see if a duplicate entry already exists and is yet to be written. A completely identical candidate sequence. If it already exists, it will not be written again; only the hit count of the existing entry will be updated. ) and the most recent hit time step ( This detection eliminates redundant entries, increasing the diversity of effective candidates under a fixed capacity.

[0075] Step 933 -- Hierarchical Marking: For non-duplicate entries that have passed the pre-screening, if Not less than If it is, then it is marked as "standard write" and "target write". ; like Greater than zero and less than If it is, then it is marked as "short sequence write", and the target write is... .

[0076] Step 934 -- Single Round Write Limit: The system counts the total number of entries to be written in the current round. . like Exceeding the preset single-round write ratio limit With the upper limit of the secondary feedback pool capacity The product of (i.e.) If ), then only retain The longest front Only a few entries are written, and the rest are discarded. The technical purpose of limiting the amount of data written in a single round is to prevent a large influx of new entries from causing all historically high-value entries to be discarded when an abnormally large number of verification failures occur in a certain round, thereby maintaining the temporal stability of the secondary feedback pool.

[0077] Step 940: Tiered storage write. For entries marked "standard write", the system inserts them into the standard area. For entries marked "short sequence write", the system inserts them into the short sequence region. Each written candidate entry contains , (set as) ), (Initially 0) (Initially) )as well as Furthermore, multiple different modified token sequences or subsequent token sequences are allowed to be stored under the same pattern key to increase candidate diversity under the same pattern. The maximum number of records stored under the same pattern key is a preset value. When writing, the number of records under the same patternkey has reached [a certain threshold]. If the key is not specified, the record with the lowest hit count under that key will be discarded before a new record is written. After each write operation, the global entry counter will be updated. The corresponding increase is made. If memory allocation fails during writing, the system skips the write operation but does not interrupt the inference process.

[0078] When the secondary feedback pool establishes a multi-level index layer with at least two different pattern lengths, the writing in step 940 further includes: extracting the context token fragment before the verification failure position, truncating token fragments of corresponding lengths according to the pattern length of each level in the multi-level index layer as the pattern key for each level, and writing the same entry to be written to the corresponding level index layer with the pattern key for each level. This multi-level writing mechanism allows the same entry to be matched by query patterns of different lengths in subsequent searches. The specific structure and retrieval process of the multi-level index layer will be described in Example 3.

[0079] The retrieval process of the secondary feedback pool has been outlined in step 300 of Embodiment 1. Its core logic is as follows: Construct a query pattern using the end token fragment of the currently generated prefix, perform matching in the secondary feedback pool, and add the matched candidate sequences to the secondary feedback candidate set. When a retrieval is successful, update the hit count and the most recent hit time step of the matched entry. If all retrievals fail, the secondary feedback candidate set is empty, and the system reverts to a dual-source fusion mode that relies solely on the historical matching candidate set and the distributed exploration candidate set, without updating the hit count of any entry. The details of the multi-level indexing and step-by-step rollback of the retrieval will be elaborated in Embodiment 3.

[0080] It should be noted that in the initial stage of inference, the secondary feedback pool is empty (cold start state), and all searches return an empty set. The system relies on the historical matching candidate set and the distribution exploration candidate set to operate in a dual-source fusion mode. As the verify process continues, the secondary feedback pool gradually accumulates entries, and in long-term operation, it automatically enriches the correction sequence of the region with the largest deviation between the target model distribution and the candidate draft distribution, and the feedback reuse effect gradually becomes apparent.

[0081] Furthermore, in one embodiment, when a candidate branch fails verification at position k, the system at least obtains the corrected token determined by the target model at position k. If it is necessary to obtain the subsequent token sequence after the corrected token, the system first adds the corrected token to the current generation prefix to form a corrected generation prefix; then, based on the corrected generation prefix, the system continues to autoregressively output a token sequence from the target model that does not exceed a preset length L_{real_max}, and uses this token sequence as the subsequent token sequence. If the token sequence encounters a terminator token, it is truncated to the position before the terminator token. Thus, the subsequent token sequence written to the secondary feedback pool is determined based on the corrected context, rather than directly using the subsequent output after the rejected candidate path.

[0082] Example 3: Multi-level pattern length index and step-by-step backtracking retrieval process This embodiment elaborates on the multi-level index structure and hierarchical retrieval mechanism of the secondary feedback pool.

[0083] The secondary feedback pool establishes a multi-level index layer with at least two different pattern lengths. Let the set of multi-level pattern lengths be { },in , This refers to the number of index levels. The system assigns a level to each index level. A separate index layer is established, implemented using a hash dictionary structure. When writing entries, the pattern key for the same entry is truncated from the context token fragment preceding the verification failure position, according to the length of each level, and written to the corresponding level of the index layer. For example, for { The three-level index configuration of {4, 3, 2} failed at the following location. The keys at each location are tuples consisting of the last four tokens, the last three tokens, and the last two tokens, respectively.

[0084] When the inference system does not have enough available memory to create all index levels, the system will reduce the number of index levels. The system reduces the number of levels to the maximum accommodating level, starting with the level with the shortest pattern length. For example, in the original three-level configuration {4, 3, 2}, if memory is only sufficient to create two levels of indexes, it will be reduced to {4, 3}. This adaptive degradation mechanism is particularly useful in memory-constrained edge deployment scenarios.

[0085] During the search, the system executes a hierarchical search process that backtracks step by step. The specific steps are as follows: Step 310: The system starts from the longest pattern length The corresponding index layer begins the search. Extract the currently generated prefix. end Each token is used as the query pattern. If the current prefix length Less than But not less than a certain level The length is then directly derived from that level. Start the search. If... Less than the shortest level If the prefix is ​​too short, it is insufficient to form a key at any level, so the secondary feedback pool retrieval is skipped, and the secondary feedback candidate set becomes empty.

[0086] Step 320: In the standard area Retrieve in the corresponding level of the index layer If a hit occurs, return all corresponding values. Record and label the hit level. Simultaneously, for each hit record, the hit count and the most recent hit time step are updated: , .

[0087] Step 330: If no match is found in the standard area Then in the short sequence region Search within the corresponding level That is, within the same index level, the standard area is retrieved before the short sequence area.

[0088] Step 340: If neither the standard region nor the short sequence region is hit, the system uses truncation and suffix normalization to remove the missing sequences. The first token is retained, while the last token remains unchanged -- this yields the next level query pattern. Reverted to the sub-ministerial level For the corresponding index layer, repeat steps 320 to 330. The key to the truncation suffix normalization method is that each backtracking only removes the front token of the key, while the end token remains unchanged, thereby maintaining the consistency between the retrieved key and the current prefix end and avoiding semantic shifts caused by random truncation.

[0089] Step 350: Rollback level by level until a match is found or all levels are missed. The system follows the "stop on first hit" principle—once a match is found at a certain level, it does not rollback to a shorter level. This principle ensures that the pattern matching accuracy of the returned results is the highest level currently achievable. If all levels are missed, the secondary feedback candidate set is empty, and the system rolls back to a dual-source fusion mode that relies solely on the historical matching candidate set and the distributed exploration candidate set, without updating the hit count of any entries.

[0090] For in the first The system assigns a retrieval confidence weight to results that are hit at the index level. The calculation method is as follows: in, For the first The length of the pattern corresponding to the level index. This is the longest pattern length among the multi-level index layers. The longer the matched pattern, the better. The closer the value is to 1.0, the higher the relevance of the candidate sequence to the current context. For example, in the {4, 3, 2} configuration, when a 4-gram hit occurs... When 3-gram hits When 2-gram hits This weight is used in the capacity control phase of step 700 to determine the cutoff and retention priority of the secondary feedback candidates. Candidates with higher values ​​are retained first. Candidates with lower values ​​are preferentially truncated when capacity is limited. If multiple candidate sequences exist... If the values ​​are the same, count the hits. Descending order is used as the basis for secondary sorting.

[0091] The technical significance of multi-level hierarchical retrieval and step-by-step fallback is that, while maintaining the priority of exact matching, even if the exact long key is not hit, the system can still obtain candidates through the shorter key, reduce the probability of empty search set, and improve the retrieval recall rate of the secondary feedback pool in low-frequency pattern scenarios.

[0092] Example 4: Capacity Elimination and Batch Cleanup of the Secondary Feedback Pool This embodiment details the capacity elimination process when the total number of entries in the secondary feedback pool exceeds the capacity limit.

[0093] Step 950: Determine if eviction is triggered. After each write operation in Step 940, the system checks... Is it greater than If so, then capacity eviction will be triggered.

[0094] Step 960: Calculate the elimination priority score for each item. and Calculate the elimination priority score for all entries. The calculation method is as follows: in, For the first Each item is assigned a priority score for elimination; the higher the score, the greater the item's value for retention. The hit count for this entry reflects the cumulative number of times this entry has been retrieved in the past. (In the molecule) The increment operation is used to prevent newly written entries with zero hits from being immediately eliminated with a score of zero. If no increment is made, newly written entries that have not yet been retrieved will receive zero points and may be deleted in the same round of elimination, preventing the secondary feedback pool from accumulating new entries. This is the current value of the global round counter. The most recent hit time step for this entry. This reflects the time decay effect—the further back in time an entry has been hit, the lower its score; adding 1 is used to prevent this effect. Exactly equal to When division by zero occurs. The actual sequence length stored for this entry. This is the preset minimum write sequence length threshold. This factor serves to give longer, true subsequent sequences a higher retention priority—longer sequences accept more tokens per validation and have higher reuse value.

[0095] In summary, the above formula evaluates the actual reuse value of an entry for subsequent validation from three dimensions: hit frequency, time decay, and sequence length. Compared to pure LRU eviction, it can distinguish between "long sequences that occasionally hit but are always accepted" and "short sequences that frequently hit but are only accepted with a single token each time." For example, a long sequence that occasionally hits but is always accepted ( Moderate (larger), and a short sequence that frequently hits but is only accepted once per hit ( higher Compared to a very small score, a composite score can reasonably classify the former as an item with higher retention value.

[0096] Step 970: Perform tiered elimination. The elimination order is: first from the short sequence region. Elimination The lowest entry, up to No more than or Cleared; if capacity is still exceeded, retrieve from the standard area. Elimination The lowest entry, up to No more than The reason for prioritizing the elimination of short sequence entries is that the number of tokens that can be accepted in a single verification of short sequence entries is limited, and their reuse value is usually lower than that of long sequence entries in the standard area. Prioritizing their elimination can free up capacity space for long sequence entries in the standard area.

[0097] Step 980: Batch Cleanup Mode. When the number of entries to be deleted in a single cleanup exceeds... The preset ratio (e.g., exceeding) This indicates that the write speed far exceeds the normal replacement speed, triggering the system's batch cleanup mode: clearing all data at once. All entries, and delete them. Entries with a priority score lower than the global median are eliminated. The purpose of the batch cleanup mode is to quickly restore the secondary feedback pool to a manageable capacity.

[0098] As an alternative, the system can also employ a pure LRU eviction strategy—eviction of the entry with the earliest most recent hit time step when the total number of entries exceeds the capacity limit. This alternative has lower implementation complexity and is suitable for scenarios where generation task context switching is infrequent and pool capacity is ample (e.g., long text continuous generation scenarios). However, it cannot distinguish the reuse value differences of entries of different lengths when the pool capacity is small. Under this alternative, the hierarchical storage structure, write pre-screening, and multi-level hierarchical retrieval remain unchanged.

[0099] Example 5: Rank-based branch budget allocation and candidate structure capacity control process This embodiment elaborates on the branch budget allocation and capacity control mechanism in steps 600 and 700 of Embodiment 1.

[0100] In step 400, the system selects the top-k candidate tokens from the logits distribution of the previous verification phase. }. this The candidate tokens are sorted by probability from high to low, forming a rank sort, and the rank value is... Starting from 0 and increasing sequentially. Highest probability. ( It serves as the unified root token and is also the starting token with the highest confidence level.

[0101] The core logic of branch budget allocation is: For the first branch from the distributed exploration candidate set... There are 10 candidate branches, and the system uses a preset branch budget function. Allocate the maximum extension length for this branch. Rank higher ( Candidate branches with smaller values ​​and higher probabilities are assigned a longer maximum extension length, and those with lower rank are assigned a lower maximum extension length. Candidate branches with higher probabilities (and lower probabilities) are assigned shorter maximum extension lengths. The technical basis for this differentiated budget allocation is that candidate tokens with higher probabilities are more likely to match the true output of the target model, and therefore their subsequent extension sequences are more likely to be accepted by the target model. Assigning them a longer extension depth can increase the upper limit of the continuous acceptance length of that branch. Conversely, candidate tokens with lower probabilities are less likely to be accepted, and assigning them too long an extension depth would result in wasted candidate nodes.

[0102] Branch budget function Various implementation methods can be used, including but not limited to: table lookup functions (pre-setting a fixed maximum extension length for each rank position), ,in The functions can be configured using various methods, including array maximum index, linear decay (maximum extension length decreases linearly with rank), exponential decay (maximum extension length decreases exponentially with rank), or dynamic functions related to logits scores (dynamically calculating the extension length based on the actual probability value of each candidate token). Different function forms are suitable for different inference scenarios: in scenarios with high content redundancy, lookup table functions are advantageous due to their simple configuration; in scenarios requiring dynamic adjustment of exploration depth based on output distribution, dynamic functions related to logits scores provide finer control. Branch budget functions can also be associated with candidate source confidence or retrieval hit level, allowing candidate branches from different sources to obtain extension lengths appropriate to their confidence levels.

[0103] Regarding candidate structure capacity control, the system maintains a dynamic counter to monitor the overall candidate size in real time as candidate sequences from three types of candidate sources are gradually added to the multi-candidate structure. This size can be represented by the total number of candidate nodes, the total number of candidate tokens, or equivalent metrics. If a newly added candidate sequence causes the overall candidate size to exceed the preset maximum draft budget, the system retains only the longest accommodating prefix portion of that candidate sequence (i.e., truncates it to a position where the total size just does not exceed the budget), or directly stops adding subsequent candidate sequences. Truncation is performed when the total candidate structure size exceeds the maximum draft budget, and the selection order of the truncation objects follows the branch budget function. The maximum allocated extension length is used as the basis—candidate branches exceeding their allocated maximum extension length are preferentially truncated. For candidates from the secondary feedback candidate set, when it is necessary to perform truncation priority ranking among different candidate sources, confidence weights are retrieved. Candidates with higher values ​​are retained first. Candidates with lower values ​​are truncated.

[0104] Example 6: Organization of Candidate Structures In the above embodiments, the multi-candidate structure processed by steps 500 to 700 needs to be stored in the memory of the inference system with an appropriate data structure to support subsequent submission to the target model for batch verification. Optional organizational forms include: a candidate tree structure, where the root token is the root node, and each candidate branch is attached as a subtree. When multiple candidate sequences share the same prefix token, these shared tokens are stored only once in the tree, and different candidate sequences branch at the first diverging token. This common prefix merging method reduces the number of redundant candidate nodes; a prefix tree (trie) structure, which reduces node redundancy through prefix merging, with each node corresponding to a token value; and an array-based tree or node table form, where each candidate token is assigned a node number, and branch relationships are described through parent pointers or children lists. All candidate nodes are arranged in a linear array, suitable for mapping to the tensor structure required for batch verification of the target model. Regardless of the data structure used, its core function is to organize multiple candidate branches under a unified root token, support deduplication and merging operations, and be able to be converted into the batch verification input format required by the target model when submitting for target model verification.

[0105] Furthermore, when candidate structures need to be submitted to the target model for verification, the system maps the candidate structures to the input for batch verification of the target model. Specifically, the system arranges the token values ​​of all nodes in the candidate tree into a one-dimensional token sequence, and generates a corresponding position ID sequence (reflecting the absolute position of each token in its respective candidate branch) and an attention mask matrix (reflecting which preceding tokens each token can notice). In addition, the system labels each candidate node with a branch ID or an equivalent branch identifier, so that the continuous acceptance length of each branch can be quickly located based on the branch identifier after verification.

[0106] Example 7: System Example This embodiment describes a multi-sequence speculative sampling system for hybrid candidate source fusion. This system is applied to the inference system of an autoregressive generative model, wherein the inference system includes a target model.

[0107] The system includes the following functional modules: A historical candidate construction module, used to construct a historical matching candidate set based on the matching results of the token sequence corresponding to the currently generated prefix in the historical token sequence, providing the ability to reuse long historical segments for candidate construction. This module searches for historical positions in the primary matching pool that match the end pattern of the current prefix and extracts the corresponding continuation token sequence. A feedback candidate construction module, used to perform pattern matching in the secondary feedback pool based on the token sequence, constructing a secondary feedback candidate set, so that the true subsequent token sequence given by the target model at the historical verification failure position can participate in the candidate construction of subsequent rounds. This module may include a multi-level index retrieval submodule, performing candidate retrieval according to the hierarchical retrieval mechanism described in Embodiment 3. An exploration candidate construction module, used to construct a distribution exploration candidate set based on the token distribution output in the previous verification stage, providing the ability to explore the current output distribution for candidate construction. A candidate fusion module, used to align at least some candidate sequences in the historical matching candidate set, the secondary feedback candidate set, and the distribution exploration candidate set to a unified root token, and to perform deduplication and merging on the aligned candidate sequences to form a multi-candidate structure. The budget allocation module allocates different maximum extension lengths to different candidate branches in a multi-candidate structure based on their rank, candidate source, or retrieval confidence weight, achieving differentiated branch budget control. The capacity control module performs capacity control on the multi-candidate structure under the maximum draft budget constraint, truncating or stopping the addition of candidate branches exceeding the budget. The verification module submits the capacity-controlled multi-candidate structure to the target model for verification and determines the current round's output token based on the continuous acceptance length of each candidate branch. The feedback writing module writes the corrected token sequence or subsequent token sequence determined by the target model at the verification failure location in this round into a secondary feedback pool for retrieval by the feedback candidate construction module in subsequent rounds. This module may include a pre-screening submodule and a capacity elimination submodule, executing the pre-screening process in Example 2 and the capacity elimination process in Example 4, respectively.

[0108] The modules described above work collaboratively via data flow: the historical candidate construction module, the feedback candidate construction module, and the exploration candidate construction module each generate three types of candidate sets and output them to the candidate fusion module; the candidate fusion module outputs multiple candidate structures to the budget allocation module and the capacity control module; the capacity control module outputs the final candidate structure to the verification module; and the verification results from the verification module are output to the feedback writing module to update the secondary feedback pool. These modules can be implemented by the processor executing computer programs or instructions stored in memory, or they can be implemented partially or entirely through dedicated hardware logic.

[0109] Example 8: Electronic Devices and Storage Media This embodiment describes an electronic device. The electronic device includes a processor and a memory. The memory stores computer programs or instructions and intermediate data during the inference process (including data structures such as the index structure, candidate structure, and primary matching pool of the secondary feedback pool). The processor executes the computer program or instructions to implement any of the methods described in Embodiments 1 to 6, or to implement the secondary feedback pool management method described in Embodiments 2 to 4. The processor can be at least one of a GPU, NPU, inference accelerator card, or general-purpose processor. The memory can include volatile memory (such as DRAM) and non-volatile memory (such as SSD or HDD). The electronic device can be a server, workstation, edge computing device, or edge inference device. In edge or edge device deployment scenarios, the system can adopt a lightweight two-level index structure (retaining only the long key level and the short key level) and a limited-capacity secondary feedback pool to adapt to limited GPU and memory conditions.

[0110] This application also provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the methods described in any of the above embodiments. The computer-readable storage medium includes, but is not limited to, random access memory, read-only memory, flash memory, optical disk, or magnetic disk.

[0111] This application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the methods described in any of the above embodiments.

[0112] Example 9: Alternatives and Variations The technical solution of this application allows for a variety of alternative implementation methods. Under the premise of keeping the core technical idea of ​​this application unchanged, several typical variations are listed below.

[0113] Variant A: Pure LRU eviction strategy. The eviction decision rule in Example 4 is simplified from a formula-based composite score to a pure LRU strategy—when eviction is triggered, the [missing information] is removed. The earliest entry. The hierarchical storage structure, write pre-filtering, and multi-level hierarchical retrieval remain unchanged. This variant is suitable for scenarios where the generation task context switching is infrequent and the pool capacity is sufficient, and the implementation complexity is low. However, when the pool capacity is small, it cannot distinguish the differentiated reuse value of long sequences and short sequences.

[0114] Variant B: Two-level index structure. This increases the number of levels in a multi-level index. The value is fixed at 2, retaining only the long key and short key levels, and eliminating the intermediate levels. The hierarchical storage structure, write pre-filtering, and capacity eviction mechanism remain unchanged. This variant is suitable for lightweight deployment scenarios where the inference service has limited memory and needs to reduce index structure overhead. While it reduces index maintenance overhead, the fallback granularity becomes coarser, and the transition from precision to recall is less smooth.

[0115] Variant C: Fixed window pattern. Uses a fixed length for all entries. The pattern is used as the key, and no multi-level index is built. The secondary feedback pool is organized in the manner of "pattern → actual subsequent sequence", and the hierarchical storage structure, write pre-screening and capacity eviction remain unchanged. This variant is suitable for scenarios with high context repetition and high pattern hit rate (such as templated code snippets in code generation). It has low key construction overhead and fast retrieval speed, but there is no fallback path when dealing with low-frequency patterns.

[0116] Variant D: Candidate Source Expansion. In addition to historical N-gram matching candidates, secondary feedback pool candidates, and logits distribution exploration candidates, hybrid candidate sources can also access external memory retrieval sequences or other statistical candidate sources. As long as the core framework of multi-source candidate fusion and branch budget control is not changed, they are all alternative solutions in this application.

[0117] Variant E: Branch-budget function substitution. Branch-budget function In addition to table lookup functions, linear decay functions, exponential decay functions, or dynamic functions related to logits scores can also be used, and they can also be associated with candidate source confidence or retrieval hit level.

[0118] Variant F: TTL Elimination. In addition to composite score elimination and pure LRU elimination, pure LFU elimination (sorted only by hit frequency) or TTL (Time to Live) based expiration elimination can also be used—each record is given a fixed number of survival steps and is automatically deleted after timeout. In short text dialogue scenarios, TTL elimination provides more concise control over the timeliness of entries.

[0119] Streamlined calculation examples The following is a flowchart example to illustrate the complete process of the secondary feedback pool in actual inference, from writing to retrieval and then to participation in capacity control.

[0120] Assuming the current global round Current generated prefix prefix length Not less than the pattern length of the longest index level. Multi-level indexes are configured as {} = {4, 3, 2}, minimum write sequence length threshold .

[0121] In the current round of verify, a candidate is in position Verification failed. The system proceeded to step 920 to extract the target model at the location. The actual output sequence is given at [location]. Sequence length The system performs pre-screening in step 930: In step 931... Not less than The process involves several steps: length filtering; step 932 checks the secondary feedback pool for duplicate entries, passing duplicate detection; step 933 marks it as "standard write"; and step 934 ensures the total number of entries to be written in the current round does not exceed the single-round write ratio limit. The system then executes step 940 to extract the failure locations. Keys at each level: It is a tuple consisting of the last four tokens. It is a tuple consisting of the last three tokens. This is a tuple consisting of the last two tokens. The system indexes the standard region at three levels respectively... , , Insert a new record.

[0122] In subsequent rounds At that time, the new prefix The last four tokens are exactly the same as those previously stored. Matching. The system extracts the query pattern in step 310. In step 320, the standard area level is retrieved. The index layer, hit retrieval confidence weight The system returned... As a secondary feedback candidate, update the entry. and The candidate carries Proceed to step 500 for Hybrid integration and step 700 for capacity control.

[0123] Suppose that in another round, the last four tokens of the new prefix do not match. However, the last 3 tokens match In step 340, the system reverts to the level using truncation and postfix normalization. Hit , The candidate carries The Hybrid fusion process proceeds to step 500, and in the capacity control phase of step 700, its retention priority is lower than... Candidates.

[0124] Assuming continuous operation Exceed The system triggers elimination in step 950. In step 970, elimination is first performed on the short sequence region. The elimination priority score for each item is calculated. Eliminate the item with the lowest score until... Restore to within the capacity limit. If the capacity is still exceeded after clearing, try again from [the previous page]. Elimination The lowest entry.

[0125] The embodiments of this application will be further explained and illustrated by example below.

[0126] The specific method of mapping candidate structures to target model batch validation inputs For example, the system arranges the token values ​​of all nodes in the candidate tree into a one-dimensional sequence according to depth-first traversal order, and assigns a globally unique node number to each node. The attention mask is constructed according to the following rules: for nodes... Its attention mask allows the node to focus on the path from the root node to the node. All ancestor nodes and nodes on the path It cannot focus on non-ancestor nodes. The position id is assigned according to the depth of each node on its path—the root node's position id is equal to the length of the current prefix. Its direct child nodes are And so on. The branch ID assigns an integer identifier to each complete path from the root node to a leaf node, and all nodes on the same path share this branch ID. Thus, the target model can distinguish the causal relationships between different branches in a single forward computation using the attention mask, maintain the correctness of positional encoding using the position ID, and distinguish the continuous acceptance length of each branch in the validation results using the branch ID.

[0127] The number of valid tokens continuously output by the target model at the validation failure point Method of determination For example, during the validation phase, the target model computes the output logits in parallel for all nodes in the candidate structure. When the position... After a candidate token at a certain location is rejected by the target model, the system directly reads the target model's token at that location. The argmax token output at the location is used as Since validation is performed in a single forward pass, the target model is at position. Subsequent logits outputs have been generated synchronously, and the system reads the positions sequentially. The argmax token at the location is used as Until the end of the candidate branch path is reached or an EOS token is encountered. Equal to the position of the end of the path minus The value of .

[0128] Specific implementation of the first-level matching pool For example, the primary matching pool is implemented using a hash table structure, with a fixed-length n-gram suffix as the key and a list of positions of that suffix in the historical token sequence as the value. The data source for the historical token sequence includes all tokens confirmed by the target model in the current session and tokens in the user input prompt. After each round of verification and confirmation of the output token, the system appends the newly confirmed token to the primary matching pool.

[0129] Step 800 verifies the specific criteria for acceptance judgment. For example, in greedy decoding mode, the system uses argmax consistency judgment—if the candidate token matches the argmax token of the target model's output logits at that position, it is judged as "accepted"; otherwise, it is judged as "rejected". In sampling decoding mode, the system uses rejection sampling judgment based on probability ratio—according to the standard verification criteria of speculative sampling, the ratio of the target model distribution to the draft distribution is used as the acceptance probability.

[0130] Memory layout and key-value cache interaction in GPU or NPU deployment scenarios For example, in a GPU deployment scenario, the token ID array, attention mask matrix, and position ID array in the candidate structure are assembled on the host side and then asynchronously transmitted to the GPU memory. The target model performs a forward computation on the GPU with the total number of candidate nodes as the sequence dimension, outputting the logits distribution at each candidate position. The target model's KV cache is expanded according to the position of the candidate nodes during the verification phase. After verification, only the KV cache entries corresponding to the accepted paths are retained, and the remaining entries are reclaimed. The index structure and entries of the secondary feedback pool reside in the host memory and do not occupy GPU memory. After the verification results are compared on the GPU side, they are transmitted back to the host side, where the continuous acceptance length of each branch is determined and the generated prefix is ​​updated. The above data transmission and forward computation process can be pipelined with the candidate construction steps of the next round to further reduce end-to-end latency.

[0131] Lightweight deployment configuration on edge devices For example, in a scenario deployed on the client side, the number of levels in a multi-level index. The maximum capacity of the secondary feedback pool is set to 2, and the pattern length set is configured as {3, 2}. Set to a low value (e.g., several hundred) to accommodate the limited memory space of the end device. The limit can be appropriately increased to further reduce the amount of space occupied by short sequence entries. The candidate structure is organized using an array-based node table to reduce pointer jump overhead. Branch budget function. A lookup table function is used, with the table length equal to the top-k values, to avoid runtime computation. The maximum draft budget for the overall candidate pool is dynamically determined based on the available video memory on the device. With this configuration, the memory footprint of the secondary feedback pool and its index structure can be kept relatively small, while still retaining the core capabilities of feedback reuse and hierarchical retrieval.

[0132] Compare experimental data or benchmark results For example, a comparison is made under the conditions of the same target model, the same set of input prefixes, and the same maximum candidate node budget. Using only a single historical N-gram candidate source as the baseline, the average number of accepted tokens is improved and the number of forward calls to the target model is reduced after adopting a three-source hybrid candidate source fusion. Using no secondary feedback pool as the baseline, after introducing secondary feedback pool feedback reuse, in generation scenarios where similar prefixes appear repeatedly (e.g., templated code snippets in code generation, repetitive expression patterns in long text generation), the hit rate of the secondary feedback pool candidates gradually increases with the number of execution rounds, the validation pass rate of subsequent candidates is higher than the baseline scheme, and the average consecutive acceptance length is correspondingly increased; in scenarios where similar prefixes appear less frequently, the hit rate of the secondary feedback pool is lower, and the performance difference between the two schemes is not significant. Using no rank branch budget as the baseline, after introducing a branch budget allocation that decreases according to rank, under the same maximum draft budget, the number of candidate nodes decreases while the average number of accepted tokens does not decrease. Using single-level fixed pattern retrieval as a baseline, the empty set rate was reduced after employing multi-level pattern backtracking retrieval. The specific magnitude of the above comparison results is affected by the characteristics of the target model and the type of generation task.

[0133] The above embodiments have the following technical effects By simultaneously utilizing candidate sequences from three different sources—historical N-gram matching candidates, secondary feedback pool candidates, and exploration candidates based on output distribution—the candidate construction process covers three information dimensions: historical fragment reuse, real feedback from the target model, and distribution exploration. Compared to relying on only a single candidate source, it can increase the number and diversity of effective candidate paths within the same maximum draft budget, which helps to improve the average number of accepted tokens per validation of the target model.

[0134] Because the secondary feedback pool reorganizes the actual outputs of the target model at the historical verification failure positions into candidate sources retrieved by pattern, forming a feedback loop of "verification → writing → retrieval → re-verification", the target model outputs that were originally discarded after the current step of error correction are continuously reused. In the generation scenario where similar prefixes appear repeatedly, the candidate hit probability and verification pass rate in subsequent rounds are improved.

[0135] By introducing a branch budget allocation that decreases according to rank, high-confidence candidates gain a longer validation depth, while low-confidence candidates are limited to a shorter range. This reduces the number of nodes occupied by low-confidence candidates under a fixed total draft budget, which helps to curb candidate size expansion and improve resource utilization during the validation phase.

[0136] Since deduplication and merging are performed after alignment to a unified root token, duplicate sequences from different candidate sources no longer occupy candidate node budgets, allowing the limited draft budget to be allocated to truly differentiated candidate paths, which helps to increase the proportion of effective candidates in the candidate structure.

[0137] Because the secondary feedback pool is divided into a standard region and a short sequence region based on the length of the write sequence, and the short sequence region is prioritized over the standard region during eviction, the lifespan of longer sequence entries in the standard region is extended under limited pool capacity, thus increasing the effective candidate density. Pre-screening, including length filtering, duplicate detection, and single-round write volume limits, filters out low-contribution and redundant entries, increasing the diversity of effective candidates under a fixed capacity and maintaining the temporal stability of the secondary feedback pool.

[0138] By using a composite score that combines hit frequency, time decay, and sequence length for elimination sorting, compared to pure LRU elimination, the system can more accurately retain entries with higher actual reuse value under the same capacity limit. This helps the secondary feedback pool to automatically enrich corrected sequences that contribute significantly to subsequent verification during long-term operation.

[0139] Due to the hierarchical retrieval and step-by-step backtracking mechanism of the multi-level pattern length index, even if the exact long pattern is not matched, the system can still obtain candidates through the shorter pattern, which reduces the probability of empty search set and improves the retrieval recall rate of the secondary feedback pool in low-frequency pattern scenarios.

[0140] Since the retrieval confidence weight drives the truncation ranking during the capacity control phase, exact matching candidates are preferentially retained when capacity is limited. Candidates of different matching levels receive a retention priority commensurate with their matching accuracy, so that the truncation decision during the capacity control phase is adapted to the degree of relevance between the candidate sequence and the current context.

[0141] Since this approach does not rely on additional candidate draft models, but instead constructs multi-branch candidates directly through Hybrid candidate source fusion, it does not require additional loading of small model memory and computational overhead during deployment, which helps to reduce the deployment cost of the inference system.

[0142] It should be noted that the actual extent of the above-mentioned technical effects is affected by factors such as the characteristics of the target model, the type of generated task, the repetition of the context, the configuration of the secondary feedback pool capacity, and the selection of parameters, and may vary in different application scenarios.

[0143] It should be noted that in the application documents of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In the application documents of this application, if it refers to performing an action according to an element, it means performing the action at least according to that element, including two cases: performing the action only according to that element, and performing the action according to that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.

[0144] All documents mentioned in this application are considered to be incorporated in their entirety into the disclosure of this application so that they can serve as a basis for modifications if necessary. Furthermore, it should be understood that after reading the foregoing disclosure of this application, those skilled in the art can make various alterations or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. A multi-sequence speculative sampling method using hybrid candidate source fusion, applied to an inference system of an autoregressive generative model, wherein the inference system includes a target model, characterized in that, The method includes: Get the token sequence corresponding to the currently generated prefix; Based on the matching results of the token sequence in the historical token sequence, a historical matching candidate set is constructed; Based on the token sequence, pattern matching is performed in the secondary feedback pool to construct a secondary feedback candidate set; wherein, the secondary feedback pool is used to organize the corrected token sequence or subsequent token sequence determined by the target model at the historical verification failure position according to the pattern; Based on the token distribution output from the previous verification phase, construct a candidate set for distribution exploration; At least a portion of the candidate sequences in the historical matching candidate set, the secondary feedback candidate set, and the distributed exploration candidate set are aligned to a unified root token, and the aligned candidate sequences are deduplicated and merged to form a multi-candidate structure. Different maximum extension lengths are assigned to different candidate branches in the multi-candidate structure based on the candidate branch's rank, candidate source, or retrieval confidence weight. Capacity control is performed on the multi-candidate structure under the maximum draft budget constraint to truncate or stop adding candidate branches that exceed the budget; The capacity-controlled multi-candidate structure is submitted to the target model for verification, and the current round output token is determined based on the continuous acceptance length of each candidate branch. The corrected token sequence or subsequent token sequence determined by the target model at the location where the verification failed in this round is written into the secondary feedback pool for retrieval when constructing the secondary feedback candidate set in subsequent rounds.

2. The method according to claim 1, characterized in that, If the current round of verification results in all acceptances and there are no verification failures, then the writing process to the secondary feedback pool will not be triggered. The secondary feedback pool is divided into a standard area and a short sequence area according to the length of the written sequence; The step of writing the corrected token sequence or subsequent token sequence determined by the target model at the location of the current verification failure into the secondary feedback pool includes: Obtain the corrected token sequence or subsequent token sequence continuously output by the target model at the verification failure position; if the corrected token sequence or subsequent token sequence encounters an end token, truncate it to the position before the end token; use the number of valid tokens after truncation as the actual sequence length; if the actual sequence length is zero, discard the entry; Before writing, a pre-screening process is performed: the secondary feedback pool is queried to see if there is already a candidate sequence that is completely consistent with the sequence to be written. If it exists, it is not written repeatedly and the hit count and the most recent hit time step of the existing entries are updated. The total number of entries to be written in the current round is counted. If the total number of entries to be written exceeds the product of the preset single-round writing ratio limit and the capacity limit of the secondary feedback pool, only the first few entries with the longest actual sequence length are retained for writing. After pre-screening, if the actual sequence length is not less than the preset minimum write sequence length threshold, the corresponding entry is written to the standard area; if the actual sequence length is greater than zero and less than the minimum write sequence length threshold, the corresponding entry is written to the short sequence area. Multiple different modified token sequences or subsequent token sequences are allowed to be stored under the same pattern key, and the maximum number of records stored under the same pattern key is a preset value. When writing, the number of records under the same pattern key reaches... If the record with the lowest hit count under that key is discarded, a new record will be written. When the total number of entries in the secondary feedback pool exceeds the capacity limit, capacity eviction is performed. The order of capacity eviction is as follows: first, eviction of the entry with the lowest eviction priority score from the short sequence area is performed until the total number of entries does not exceed the capacity limit or the short sequence area is cleared; if the capacity limit is still exceeded, then eviction of the entry with the lowest eviction priority score from the standard area is performed; wherein, the eviction priority score... The calculation method is as follows: in, To add 1 to the hit count for this entry, we prevent newly written entries from being immediately discarded because a zero-hit entry would have a score of zero. This is the current value of the global round counter; Add 1 to the denominator for the most recent hit time step of this entry to reflect time decay and prevent division by zero; The actual sequence length stored for this entry; The minimum write sequence length threshold, and The ratio is used to give longer sequences a higher retention priority.

3. The method according to claim 1, characterized in that, The secondary feedback pool establishes a multi-level index layer with at least two different pattern lengths; when writing the corrected token sequence or subsequent token sequence determined by the target model at the location of the current verification failure into the secondary feedback pool, it further includes: Extract the context token fragment before the verification failure position, and extract token fragments of corresponding lengths according to the pattern length of each level in the multi-level index layer as the pattern key of each level. Then, write the same entry to be written into the corresponding level index layer according to the pattern key of each level. The step of performing pattern matching in the secondary feedback pool based on the token sequence to construct a secondary feedback candidate set includes: starting from the index layer corresponding to the longest pattern length, extracting the last token fragment of the token sequence as the current query pattern for retrieval; if the current index layer does not hit, then using truncation and suffix normalization—removing the frontmost token of the current query pattern while keeping the last token unchanged—to obtain the next level query pattern, and backtracking step by step to the index layer corresponding to the shorter pattern length for retrieval; stopping the backtracking after the first hit in any index layer, and adding the hit corrected token sequence or subsequent token sequences as candidate sequences to the secondary feedback candidate set; When the length of the token sequence is less than the shortest pattern length in the multi-level index layer, the retrieval in the secondary feedback pool is skipped, and the secondary feedback candidate set is empty. When the secondary feedback pool includes a standard area and a short sequence area, retrieval is prioritized in the index corresponding to the standard area within the same index level. If the standard area is not found, retrieval is then performed in the index corresponding to the short sequence area. A corresponding retrieval confidence weight is assigned based on the pattern length of the hit level, calculated as follows: in, In the first retrieval confidence weight when a level-one index hits; For the first The length of the pattern corresponding to the level index; The longest pattern length among the multi-level index layers; the longer the matched pattern, the better. The closer the value is to 1.0, the higher the relevance of the candidate sequence to the current context. The capacity control of the multi-candidate structure under the maximum draft budget constraint includes: when truncation priority sorting needs to be performed among candidate sequences from different candidate sources, the candidate sequences in the secondary feedback candidate set are arranged in descending order according to the retrieval confidence weight to determine the retention priority; if multiple candidate sequences have the same retrieval confidence weight, they are arranged in descending order according to the hit count as the secondary sorting basis; when a retrieval hit occurs, the hit count and the most recent hit time step of the hit entry are updated; if no hit occurs at all levels of the index layer, the secondary feedback candidate set is an empty set, and the system reverts to a dual-source fusion mode that relies only on the historical matching candidate set and the distributed exploration candidate set, without updating the hit count of any entry.

4. The method according to claim 1, characterized in that, The process of constructing a candidate set for distribution exploration based on the token distribution output from the previous verification phase includes: Retrieve the next token logits generated in the previous verification phase, and select the top-k candidate tokens from them. }, among which the highest probability As the unified root token; for each Construct an extended prefix and query its subsequent matching sequence in the matching pool; if a match is found, form a sequence with the prefix... The initial candidate branch; if not found, the single-token branch is retained. As exploration candidates; thus obtaining the distribution exploration candidate set; Aligning at least some candidate sequences to a unified root token includes: if the first token of a candidate sequence in the historical matching candidate set or the secondary feedback candidate set is consistent with the unified root token, then removing its first token and using it as a subsequent candidate under the unified root token; if the first token of the candidate sequence is inconsistent with the unified root token, then retaining the candidate sequence as an independent candidate branch and adding it to the multi-candidate structure if capacity control allows. The step of assigning different maximum extension lengths to different candidate branches based on their rank includes: for the first candidate branch from the distributed exploration candidate set... There are 10 candidate branches, based on a preset branch budget function. Allocate the maximum extension length; wherein This represents the rank order determined based on the top-k candidate tokens; candidate branches with higher rank are assigned a longer maximum extension length, and candidate branches with lower rank are assigned a shorter maximum extension length; the branch budget function... It is any one of the following: a lookup function, a linear decay function, an exponential decay function, or a dynamic function related to the logits score; The capacity control of the multi-candidate structure includes: dynamically calculating the overall size of the candidates during the process of adding candidate sequences to the multi-candidate structure; if the newly added candidate sequence will cause the overall size of the candidates to exceed the maximum draft budget, then only the longest prefix portion that can be accommodated in the candidate sequence is retained, or the addition of subsequent candidate sequences is stopped.

5. A two-stage feedback pool management method for speculative sampling in an autoregressive generative model inference system, wherein the inference system includes a target model, characterized in that, The method includes: After the target model performs verification on the candidate draft sequence, the corrected token sequence or subsequent token sequence determined by the target model at the verification failure position is obtained; Extract the context token fragment before the verification failure position, and construct a pattern key according to the preset pattern length; The pattern key is associated with the corrected token sequence or subsequent token sequence and written into the secondary feedback pool; In subsequent rounds, a query pattern is constructed based on the terminal token fragment of the currently generated prefix. Candidate sequences matching the query pattern are retrieved from the secondary feedback pool and used as secondary feedback candidates to participate in the candidate draft construction. The hit count and the most recent hit time step are updated for the retrieved entries. When the total number of entries in the secondary feedback pool exceeds the preset capacity limit, elimination is performed based on the entry's hit information and sequence length.

6. The method according to claim 5, characterized in that: Before writing to the secondary feedback pool, a write pre-screening is performed: if the length of the sequence to be written is zero, it is discarded; a duplicate detection is performed on the sequence to be written; if a candidate sequence that is completely identical to the sequence to be written already exists in the secondary feedback pool, it is not written repeatedly, and the hit count and the most recent hit time step of the existing entries are updated; the total number of entries to be written in the current round is counted; if it exceeds the product of the preset single-round writing ratio limit and the capacity limit, only the first few entries with the longest actual sequence length are retained for writing. The secondary feedback pool includes a standard area and a short sequence area. During writing, the tokens are written to the corresponding storage areas according to the comparison results between the length of the modified token sequence or subsequent token sequence and the preset minimum write sequence length threshold. During eviction, the item with the lowest eviction priority score is first evictioned from the short sequence area. If the short sequence area is still overcapacity after being cleared, it is then evictioned from the standard area. The secondary feedback pool establishes a multi-level index layer with at least two different pattern lengths. During retrieval, the query starts from the index layer corresponding to the longest pattern length. If no match is found, the query is backtracked step by step to the index layer corresponding to the shorter pattern length using the truncation suffix normalization method. The backtracking stops after a match is found at a certain level. The retrieval confidence weight is assigned to the candidate sequence that has been hit based on the ratio of the pattern length of the hit level to the pattern length of the longest level.

7. A hybrid candidate source fusion multi-sequence speculative sampling system, applied to the inference system of an autoregressive generative model, wherein the inference system includes a target model, characterized in that, The system includes: The historical candidate construction module is used to construct a set of historical matching candidates based on the matching results of the token sequence corresponding to the currently generated prefix in the historical token sequence; The feedback candidate construction module is used to perform pattern matching in the secondary feedback pool based on the token sequence to construct a secondary feedback candidate set, wherein the secondary feedback pool is used to organize the corrected token sequence or subsequent token sequence determined by the target model at the historical verification failure position according to the pattern. The candidate exploration module is used to construct a set of candidate distributions based on the token distribution output from the previous verification phase. The candidate fusion module is used to align at least a portion of the candidate sequences in the historical matching candidate set, the secondary feedback candidate set, and the distributed exploration candidate set to a unified root token, and to perform deduplication and merging on the aligned candidate sequences to form a multi-candidate structure. The budget allocation module is used to allocate different maximum extension lengths to different candidate branches in the multi-candidate structure based on the candidate branch's rank, candidate source, or retrieval confidence weight. A capacity control module is used to perform capacity control on the multi-candidate structure under the maximum draft budget constraint, so as to truncate or stop adding candidate branches that exceed the budget; The verification module is used to submit the capacity-controlled multi-candidate structure to the target model for verification, and determine the current round output token based on the continuous acceptance length of each candidate branch. The feedback writing module is used to write the corrected token sequence or subsequent token sequence determined by the target model at the position of the verification failure in this round into the secondary feedback pool, so that the feedback candidate construction module can retrieve it in subsequent rounds.

8. An electronic device, characterized in that, It includes a processor and a memory; the memory is used to store computer programs or instructions; the processor is used to execute the computer programs or instructions to implement the method as described in any one of claims 1 to 4, or to implement the method as described in any one of claims 5 to 6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1 to 4, or the method as described in any one of claims 5 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1 to 4, or the method as described in any one of claims 5 to 6.

Images