Npu-oriented large language model speculation reasoning acceleration method and system

Abstract

The application relates to the technical field of artificial intelligence model inference acceleration, and discloses an NPU-oriented large language model speculation inference acceleration method, system, device and storage medium. The method divides a KV Cache into a committed area, a candidate differential area and a rollback index area, and constructs a verification window descriptor containing a base address of the committed area and a base address of the candidate differential area for a candidate token; a draft execution graph writes candidate K / V data into the candidate differential area without updating a committed area write pointer; a verify execution graph merges differential K / V with accepted prefixes to the committed area after verification according to the descriptor; and rejected suffixes are released through a rollback index without performing a recovery copy on the committed area; and the draft execution graph and the verify execution graph are executed in a pipeline through a descriptor queue. The method helps to reduce KV rollback costs and execution graph synchronization overhead.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence model inference acceleration technology, and in particular to verification window orchestration and key-value cache rollback management technology in speculative inference of large language models for neural network processors (NPUs). Background Technology

[0002] Large language models based on the Transformer architecture have been widely used in the field of natural language processing. At the deployment level, an increasing number of scenarios require large language models to be deployed on devices equipped with Neural Processing Units (NPUs) to meet requirements regarding inference latency, power consumption, or data privacy. For example, conversational intelligent assistants running on mobile terminals need to perform autoregressive decoding on the NPU to achieve low-latency token-by-token text generation; code completion or meeting minutes generation services deployed on edge inference servers need to continuously process high-concurrency long contextual inference requests on the NPU; and in enterprise knowledge base question answering and private inference appliance scenarios, model inference needs to complete the stepwise decoding of long sequences under the limited on-chip SRAM and external storage bandwidth of the NPU.

[0003] Speculative inference, as an important means to improve the throughput of autoregressive decoding, uses a lightweight draft model to pre-generate multiple candidate tokens, which are then verified by the target model all at once. When a candidate token is accepted, multiple tokens are advanced simultaneously, thus reducing the number of calls to the target model. However, existing technologies have several limitations when deploying speculative inference on the NPU. If the temporary K / V data generated by candidate tokens is directly written to the main K / V cache, a large-scale recovery copy of the written area is required after rejection. This recovery overhead increases with the rejection length, the number of model layers, and the K / V bit width. Frequent synchronization between the draft execution graph and the verify execution graph makes it difficult for NPU computing units and DMA channels to form a continuous pipelined execution. The verification window length and verification batch size are usually fixed and difficult to dynamically adjust with changes in request type, context length, acceptance rate, and NPU load status. Furthermore, the recovery path after token rejection involves additional overhead such as main K / V cache relocation, position index correction, and attention mask reconstruction. Therefore, there is an urgent need for a technical solution that can collaboratively design the submission and release of candidate key-value data, the orchestration of verification windows, and the pipeline scheduling of execution graphs on the NPU. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for accelerating speculative reasoning of large language models for NPU, so as to solve the problems mentioned in the background art.

[0005] This application provides a method for accelerating speculative inference of large language models for NPUs, which is executed by the NPU runtime scheduler on an NPU configured with on-chip SRAM, DMA channels, and an execution graph mechanism. The method includes: Construct the draft execution graph for generating candidate tokens and the verify execution graph for verifying the candidate tokens using the target model; The KV Cache is divided into a committed area, a candidate difference area, and a rollback index area. The committed area stores the K / V data of accepted tokens, the candidate difference area stores the differential K / V data generated by candidate tokens, and the rollback index area records the mapping relationship between candidate tokens and candidate difference blocks. The draft execution graph generates candidate tokens based on the K / V data of the committed area, writes the differential K / V data corresponding to the candidate tokens into the candidate differential block of the candidate differential partition, does not update the write pointer of the committed area, and generates the corresponding rollback index entry in the rollback index area. A verification window descriptor is constructed for the candidate token and written to the descriptor queue. The verification window descriptor includes at least the candidate token address, the position offset, the submitted zone base address, the candidate difference zone base address, and the rollback index address. The verify execution graph reads the verification window descriptor from the descriptor queue, reads the candidate token, the K / V data of the submitted area and the candidate difference block according to the verification window descriptor, performs target model verification on the candidate token, and outputs the accept prefix length and rejection position; Based on the accepted prefix length, the differential K / V data of the candidate differential blocks corresponding to the accepted prefix are merged into the committed area, and the write pointer and committed sequence length of the committed area are updated; Based on the rejection position, the rollback index entries corresponding to the rejected candidate token and its subsequent candidate tokens are marked as released to release the corresponding candidate differential blocks, and no recovery copy is performed on the committed area; The draft execution graph and the verify execution graph are executed in a pipelined manner during the NPU runtime through the descriptor queue.

[0006] In a preferred embodiment, the verification window descriptor further includes: window number, length of the committed sequence before the window starts, number of candidate tokens, number of candidate branches, verification strategy number, verification precision mode, on-chip cache budget, window cutoff condition, window generation number, commit snapshot, candidate differential block status bitmap, commit completion event identifier, release completion event identifier, window commit abort flag, and merging method; The window generation number is a monotonically increasing number used to distinguish different verification windows in pipeline overlap; the candidate differential block state bitmap is used to indicate the state of the rollback index item corresponding to each candidate token in the current verification window; the merging method records the merging method of the accepted prefix, which is taken from one of DMA scatter, on-chip copy, page table remapping, and pointer swapping. The submitted snapshot Record number The submitted area status before the start of each verification window is represented as a triplet: in, This is the write pointer to the committed area mentioned before the start of this verification window. The length of the submitted sequence, in units of tokens, before the start of this verification window. This is the KV page table version before the start of this verification window.

[0007] In a preferred embodiment, the rollback index entry is bound to the window generation number. The rollback index entry records at least the window generation number of the verification window to which it belongs, the token sequence number within the window, the layer number, the KV head group number, the candidate differential block address, the target location, and the status. The target location is the location where the differential K / V data of the corresponding candidate token should be written to the submitted area after the candidate token is accepted. When the verify execution graph performs target model verification on the candidate token, it checks whether the window generation number in the corresponding rollback index item is consistent with the window generation number in the verification window descriptor when reading the candidate difference block. If they are inconsistent, the corresponding candidate token is considered rejected. When the inconsistency involves all candidate tokens in the current verification window, the window submission abort flag is set, the current verification window is rolled back to the state of the submission snapshot record, and all candidate difference blocks of the current verification window are released.

[0008] In a preferred embodiment, the differential K / V data in the candidate differential blocks corresponding to the accepted prefix are merged into the committed area, employing a two-stage submission process consisting of a pre-submission check phase and a formal submission phase. During the pre-commit check phase, the rollback index entries corresponding to the accept prefix are checked one by one in token order, and the following conditions are met: the status is ready; the window generation number matches the window generation number in the verification window descriptor; the target position matches the length of the committed sequence according to the commit snapshot. The target location of the committed area is consistent with the token sequence number within the window; the KV page table version in the committed snapshot. The submission actions of other verification windows have not changed the status; the candidate differential blocks of each layer and each KV head group of the corresponding candidate token are all in the committable state; when all conditions are met, the corresponding rollback index item is set from the ready state to the pre-commit state, the corresponding bit is set in the candidate differential block state bitmap, and the corresponding candidate differential block is locked so that it is not reused by the free list before the release is completed; when any rollback index item does not meet the conditions, the rollback index item that has been set to the pre-commit state is restored to the ready state, the window submission abort flag is set, and the current verification window as a whole reverts to the state of the submission snapshot record; During the formal submission phase, the candidate differential blocks corresponding to the rollback index items in the pre-submission state are merged into the target position of the submitted area according to the merging method, based on the token order, and the corresponding rollback index items are set to the submitted state. After the commit completion event corresponding to the commit completion event identifier is returned, the write pointer of the committed area, the length of the committed sequence, and the KV page table version are atomically advanced: in, For the first The length of the accept prefix for each verification window. , , These are the write pointer to the committed area after the push, the length of the committed sequence, and the KV page table version. A version advance function to ensure that the KV page table version is monotonically incremented; Before the submission completion event returns, when the draft execution graph generates candidate tokens for the next verification window, it only reads the length of the submitted sequence. Previously committed key-value data is not read from candidate differential blocks that are in the pre-commit state.

[0009] In a preferred embodiment, the state of the rollback index entry switches between idle, writing, ready, pre-commit, committed, and released; When releasing the rejected candidate token and the candidate differential block corresponding to the subsequent candidate token, if the state of the corresponding rollback index item is not in the pre-commit state, it is directly set to the released state, and the corresponding candidate differential block is added to the free list of the candidate differential partition; if the state of the corresponding rollback index item is in the pre-commit state, it is released after the commit completion event returns; after the release is completed, the release completion event flag is set. The release and acceptance prefix submission are executed in parallel, releasing write pointers that do not cross the submitted area before the current verification window begins. Furthermore, the release process does not read, write, or rebuild the committed area and its page tables.

[0010] In a preferred embodiment, the merging method is adaptively determined by the NPU runtime based on the page alignment condition between the candidate differential block corresponding to the accept prefix and the target location of the submitted region, the queue congestion of the DMA channel, and the on-chip resource status: When the page alignment condition is met and the target position is not occupied by other verification windows, page table remapping or pointer swapping is used. Otherwise, DMA scatter is used when the number of fragments in the candidate differential block does not exceed the capacity of the DMA scatter descriptor queue and the DMA queue congestion does not exceed the congestion boundary. If none of the above conditions are met, on-chip copying will be used.

[0011] In a preferred embodiment, the descriptor queue employs a double-buffered approach: one set of verification window descriptors is consumed by the verify execution graph, and the other set is continued to be written by the draft execution graph; the verify execution graph consumes the first set of descriptors... When the verification window descriptor of the first verification window is prepared, the draft execution graph is ready for the first verification window. The NPU runtime synchronizes the candidate difference partition readability, submission completion, and release completion status through event flags; when the candidate difference partition space is insufficient or the accept prefix is ​​not determined, the draft execution graph enters a waiting state without triggering host-side reconstruction of the execution graph; Based on at least one of the following factors in recent verification windows: acceptance rate, logit margin, context length, on-chip cache usage, and DMA queue status, dynamically adjust at least one of the following factors in subsequent verification windows: number of candidate tokens, number of candidate branches, verification batch size, verification precision strategy, and candidate differential partition capacity. No. The required candidate difference partition capacity for each verification window satisfy: in, For the first The number of candidate tokens for each verification window. The number of layers in the target model. For the number of KV heads, For the dimension of a single head, The bit width of the differential K / V data is expressed in bits, and a coefficient of 2 indicates that it includes both K and V types of cache. When the current free capacity of the candidate difference partition Less than When the number of candidate tokens for the next verification window is converged, the following formula is used: : in, This is the floor function; When the acceptance rate of multiple consecutive verification windows is lower than the threshold, the number of idle blocks in the candidate differential partition is lower than the threshold, the output confidence is lower than the threshold, or the power consumption or temperature of the NPU exceeds the threshold, the conservative decoding strategy is switched to. The conservative decoding strategy includes at least one of shortening the number of candidate tokens, reducing the number of candidate branches, improving verification accuracy, and switching to single-step decoding of the target model.

[0012] This application provides a large language model speculative inference acceleration system for NPUs, running on an NPU configured with on-chip SRAM, DMA channels, and an execution graph mechanism. The system includes: The model execution graph construction module is used to construct the draft execution graph for generating candidate tokens and the verify execution graph for verifying the candidate tokens using the target model. The KV rollback cache management module is used to divide the KV Cache into a committed area, a candidate difference area, and a rollback index area, and to manage the candidate difference blocks and corresponding rollback index entries written by the draft execution graph into the candidate difference area. The NPU window scheduling module is used to determine the number of candidate tokens, the number of candidate branches, the verification batch, the verification precision strategy, and the candidate difference partition capacity of the verification window, for use by the draft execution graph to generate candidate tokens; The verification window descriptor generation module is used to construct verification window descriptors for candidate tokens and write them into the descriptor queue; The fusion verification operator module is used to read candidate tokens, position offsets, K / V data of the submitted area and candidate difference blocks from the verification window descriptors in the descriptor queue by the verify execution graph, perform target model verification on the candidate tokens, and output the accept prefix length and rejection position; The submission and release module is used to merge the differential K / V data in the candidate differential block corresponding to the acceptance prefix into the submitted area according to the acceptance prefix length, and release the rejected candidate token and the candidate differential block corresponding to its subsequent candidate token according to the rejection position, without performing a recovery copy on the submitted area; The system is used to perform the above methods.

[0013] This application provides an electronic device including an NPU, a memory, and a processor, wherein the memory stores program instructions that, when executed, cause the electronic device to perform the above-described method.

[0014] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0015] This application utilizes a combination of techniques, including verification window descriptor 131-driven NPU window-level scheduling, KV Cache 140 three-partition management, candidate KV differential submission and release, and dual execution graph descriptor queues 130 pipelined, to reduce the overhead of target verification, KV Cache rollback, candidate token synchronization, and execution graph switching when performing speculative inference on a large language model on the NPU, while maintaining the consistency of the target model verification results. The technical effects and inherent relationships of the above techniques are explained from multiple technical dimensions below.

[0016] At the physical layout level of the KV Cache, this application divides the KV Cache 140 into three physical regions: the committed area 141, the candidate difference area 142, and the rollback index area 143 (e.g., ...). Figure 2 As shown in Figure 121, the draft execution writes the differential K / V data corresponding to the candidate token into the candidate differential block of the candidate differential partition 142, without updating the write pointer of the committed area 141, and generates the corresponding rollback index entry in the rollback index area 143. This three-partition isolation write mechanism physically separates the K / V data of the candidate token from the historical K / V data of the committed area 141. When a candidate token is rejected, only the rollback index entry needs to be marked as released to release the corresponding candidate differential block, without performing a recovery copy on the committed area 141. Compared with the scheme that directly writes candidate KV to the main KV Cache, in the recovery scenario of rejected suffixes, the amount of K / V recovery migration in the committed area is zero, and the release operation mainly involves the state update of the rollback index entry and the maintenance of the free list, avoiding the need to update the state of the rollback index entry and the maintenance of the free list as the number of rejected candidate tokens and the number of model layers increase. and KV bit width Linear growth.

[0017] At the NPU runtime scheduling level, this application constructs a verification window descriptor 131 for candidate tokens (e.g., Figure 2(As shown) and written to descriptor queue 130. The verification window descriptor 131 includes at least the candidate token address, position offset, base address of committed area 141, base address of candidate differential block 142, and rollback index area 143 address. The NPU runtime 120 schedules draft generation, verifying, KV commit, and release operations in one go based on this descriptor, escalating the host-side synchronization per token to window-level scheduling. When the verification window descriptor 131 further includes extended fields such as window generation number, candidate differential block status bitmap, commit completion event identifier commit_fence_id, release completion event identifier release_fence_id, window commit abort flag, commit snapshot, and merge method, the NPU runtime 120 can obtain more granular scheduling information: commit snapshot... The record of the stable state of the committed area 141 before the verification window begins allows the system to revert to the state recorded in the snapshot when an anomaly is found during the pre-commit check, without having to perform complex recovery operations on the committed area 141. The candidate differential block status bitmap enables the NPU runtime 120 to quickly determine whether all candidate differential blocks in the window are ready, without having to traverse and roll back the index item by item.

[0018] At the level of cross-window key-value consistency maintenance, rollback index entries and window generation numbers are performed. The binding and generational consistency check of the verify execution graph 122 when reading candidate differential blocks enable the verify execution graph 122 to identify expired candidate differential blocks belonging to other verification windows in scenarios where the draft execution graph 121 and verify execution graph 122 pipelines overlap, thereby reducing the risk of the main KV cache being polluted. When inconsistency involves all candidate tokens, the window commit abort flag is set and the entire window is rolled back to the commit snapshot. This mechanism further ensures the consistency of the submitted status in Zone 141.

[0019] At the candidate KV transaction consistency level, the accept prefix adopts a two-phase commit process: a pre-commit check phase and a formal commit phase. In the pre-commit check phase, the status of rollback index entries, window generation number, target location, and KV page table version are checked item by item. Only when all checks pass is the index entry set to the pre-commit state. If any rollback index entry fails to meet the conditions, the entire window is rolled back to the state of the committed snapshot record. The purpose of this mechanism is to avoid inconsistencies in committed area 141 caused by partial commits when multi-level K / V data involved in the accept prefix needs to be merged into multiple target locations in committed area 141. After the formal commit is completed, the write pointer, committed sequence length, and KV page table version of committed area 141 are only atomically advanced after the commit completion event corresponding to commit_fence_id returns. Before advancement, the draft execution diagram 121 only reads the committed sequence length. Previously committed K / V data is not read from candidate differential blocks in the pre-commit state—this constraint ensures that even if draft execution diagram 121 and verify execution diagram 122 are executed concurrently in time, the KV cache read by draft execution diagram 121 always corresponds to a consistent snapshot state.

[0020] At the state management level of rollback index entries, the state of rollback index entries switches between idle, writing, ready, pre-commit, committed, and released according to explicit transition rules. Releasing candidate differential blocks corresponding to rejected suffixes follows specific state constraints: index entries not in the pre-commit state are directly set to the released state and added to the idle list; index entries in the pre-commit state are released only after the commit completion event returns. Release operations and prefix acceptance commits can be executed concurrently, but release does not cross the write pointer of committed area 141 before the current verification window begins. Furthermore, the entire release process does not read, write, or rebuild committed area 141 and its page tables. The combination of the above state constraints and release rules ensures that the release of suffix rejections is completed without interfering with prefix acceptance.

[0021] At the physical merging level of the accept prefix, the merging method is adaptively determined by the NPU runtime based on page alignment conditions, DMA queue congestion, and on-chip resource status, among page table remapping, pointer swapping, DMA scatter, and on-chip copy. There is no hard binding between the four methods. Zero-copy is preferred when page alignment conditions are met; DMA scatter is used when DMA is available; and on-chip copy is used when neither of the above conditions are met. This adaptive selection mechanism allows the system to complete the accept prefix commit even in DMA congestion scenarios by switching merging methods, helping to maintain the availability of the commit path under different NPU operating states.

[0022] At the dual execution graph pipeline scheduling level, descriptor queue 130 is organized using a double-buffering method (e.g., Figure 2 As shown, one set of verification window descriptors 131 is consumed by the verify execution graph 122, while another set is written by the draft execution graph 121. During NPU runtime, 120 synchronizes the candidate difference partitions into three states: readable, committed, and released, using the delta_ready signal, commit_fence_id, and release_fence_id. When the candidate difference partition 142 has insufficient space or the accept prefix is ​​not yet determined, the draft execution graph 121 enters a lightweight wait state without triggering host-side reconstruction of the execution graph. Compared to the token-by-token synchronization method, the pipelined scheduling mechanism of the descriptor queue and event flags described above helps improve the utilization of NPU computing units and DMA channels.

[0023] At the dynamic orchestration level of the verification window, the NPU window scheduling module dynamically adjusts at least one of the following for subsequent verification windows based on at least one of the following: acceptance rate, logit margin, context length, on-chip cache usage, and DMA queue status: number of candidate tokens, number of candidate branches, verification batch size, verification precision strategy, and capacity of candidate differential partition 142. This adjustment is made when the current free capacity of candidate differential partition 142 is... Smaller than the candidate difference partition size required by the current validation window At this time, the system proportionally converges the number of candidate tokens for the next verification window, enabling the system to continue operating within the limited capacity of the differential partition instead of directly degenerating into single-step decoding. When the acceptance rate of multiple consecutive verification windows falls below a threshold, the free block count of the candidate differential partition falls below a threshold, the output confidence falls below a threshold, or the power consumption or temperature of the NPU exceeds a threshold, the system switches to a conservative decoding strategy. The dynamic orchestration mechanism, in conjunction with the aforementioned KV three-partition and transaction consistency mechanism, ensures that the adjustment of window parameters is based on the actual capacity of the candidate differential partition 142 and the operating status of the rollback index area 143, thereby keeping the convergence and expansion of the verification window length consistent with the underlying KV cache management.

[0024] The technical effects at each of the above levels are not isolated from each other, but are interconnected through the unified scheduling data structure of the verification window descriptor 131: the address field in the descriptor enables the verify execution graph 122 to read K / V data from two different physical regions, the committed region 141 and the candidate differential region 142; the commit snapshot and window generation number enable the two-phase commit and generation verification to be performed at the window level rather than the token level; the commit_fence_id and release_fence_id connect the commit and release operations with the event synchronization mechanism of the dual execution graph pipeline; and the merge method field enables the formal commit phase to adaptively select the data migration path based on the current state of the NPU. It is the synergistic cooperation between the above fields and the KV three-partition physical layout, the rollback index state machine, and the NPU runtime event flags that enables the method of this application to manage the lifecycle of candidate KV data on the NPU in units of windows, which helps to reduce KV rollback costs, reduce execution graph synchronization overhead, and maintain the continuity of speculative inference pipeline under different operating states.

[0025] The specification of this application contains numerous technical features distributed across various technical solutions. Listing all possible combinations of these technical features (i.e., technical solutions) would make the specification excessively lengthy. To avoid this problem, the various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which are considered to have been described in this specification), unless such a combination of technical features is technically infeasible. For example, one example discloses feature A+B+C, and another example discloses feature A+B+D+E. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; they cannot be used simultaneously. Feature E can technically be combined with feature C. Therefore, the solution A+B+C+D should not be considered as described because it is technically infeasible, while the solution A+B+C+E should be considered as described. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the method for accelerating speculative inference of large language models for NPU according to an embodiment of this application. It shows the processing flow of execution graph construction and KV Cache three-partition initialization, verification window parameter determination, candidate token generation and candidate difference partition writing, verification window descriptor generation, target model verification, accepting prefix two-phase commit, rejecting suffix no-copy release, dynamic verification window orchestration and conservative backoff, dual execution graph pipeline scheduling, and iterative execution.

[0027] Figure 2This is a schematic diagram of the structure of a large language model speculative inference acceleration system for NPU according to an embodiment of this application. It shows the connection relationships between the NPU runtime, draft execution graph, verify execution graph, descriptor queue, verification window descriptor, KVCache, committed area, candidate difference area, rollback index area, model execution graph construction module, KV rollback cache management module, NPU window scheduling module, verification window descriptor generation module, fusion verification operator module, and commit and release module. Figure 2 It also schematically illustrates the correspondence between rollback index entry status, commit snapshot, window generation number, commit completion event identifier, release completion event identifier, and merge method. Detailed Implementation

[0028] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0029] Explanation of some concepts: KV Cache refers to the storage area used by Transformer-type large language models to cache the Key tensor and Value tensor in the attention mechanism during autoregressive decoding, so that when generating each new token, it is not necessary to recalculate the Key and Value corresponding to all previous tokens.

[0030] Speculative reasoning refers to a method of accelerating reasoning in the autoregressive decoding process of a large language model. It involves using a lightweight draft model or a lightweight branch of the target model to generate multiple candidate tokens in advance, and then having the target model verify the candidate tokens. If a candidate token is accepted, multiple tokens are advanced at once; if it is rejected, the tokens are regenerated from the rejection position. This method is also known as speculative decoding.

[0031] NPU Runtime 120 refers to the runtime scheduling layer on the NPU side or closely working with the NPU, used to manage execution graph startup, descriptor queues, event flags, DMA transfers, and cache status.

[0032] The draft execution graph 121 refers to the executable computation graph used to generate candidate tokens on the NPU. The corresponding model can come from an independent small model, a shallow branch of the target model, a quantized copy of the target model, or an early-retreat branch of the target model.

[0033] The verify execution graph 122 refers to the executable computation graph used to perform target model verification on candidate tokens on the NPU. This execution graph reads candidate tokens and KV Cache data and outputs the accept prefix length and rejection position.

[0034] Descriptor queue 130 refers to the queue structure used to pass verification window descriptors 131 between draft execution graph 121 and verify execution graph 122, and can be organized in a double-buffered manner.

[0035] The verification window descriptor 131 refers to the structured scheduling data structure designed for the NPU runtime 120. It includes at least the candidate token address, position offset, committed region base address, candidate differential region base address, and rollback index address. The NPU runtime 120 schedules draft generation, verify, KV commit, and release operations based on this descriptor.

[0036] KV Cache 140 refers to the general term for the storage area where key and value data are cached during the attention calculation process of a large language model.

[0037] The committed area 141 refers to the storage area in KV Cache 140 used to store the K / V data of historical tokens that have been accepted by the target model. It is a stable area that can be safely read by both draft execution graph 121 and verify execution graph 122.

[0038] Candidate differential partition 142 refers to the storage area in KV Cache 140 used to store the differential K / V data generated by the candidate tokens generated by the draft model in the current verification window. The data in this area will not enter the submitted area 141 before the target model is verified.

[0039] Rollback index area 143 refers to the index storage area in KV Cache 140 used to record the mapping relationship, space usage, and status information between candidate tokens and candidate differential blocks.

[0040] A candidate differential block refers to a storage block allocated in candidate differential partition 142 for a certain candidate token at a certain layer and in a certain KV head group, used to store differential K / V data.

[0041] A rollback index entry refers to an index record in the rollback index area 143, which at least includes the window generation number of the verification window to which it belongs, the token sequence number within the window, the layer number, the KV head group number, the candidate differential block address, the target location, and the status.

[0042] Acceptance prefix refers to the sequence of candidate tokens that are accepted by the target model consecutively, starting from the first candidate token, within a verification window.

[0043] The rejection position refers to the position of the first candidate token that is not accepted by the target model within a validation window.

[0044] A snapshot commit refers to the stable state of the committed area 141 recorded in the verification window descriptor 131 before the start of the verification window. It is represented by a triplet of the committed area write pointer, the committed sequence length, and the KV page table version.

[0045] The window generation number refers to the monotonically increasing number in the verification window descriptor 131, which is used to distinguish candidate difference blocks of different verification windows during the pipelined overlapping execution of draft execution diagram 121 and verify execution diagram 122.

[0046] The commit completion event flag (commit_fence_id) is an event flag used in NPU runtime 120 to indicate that the operation of merging the differential K / V data corresponding to the accepted prefix into the committed area 141 has been completed.

[0047] The release completion event flag (release_fence_id) is an event flag used in NPU runtime 120 to indicate that the release operation of the candidate differential block corresponding to the rejection suffix has been completed.

[0048] The delta_ready signal is an event signal issued by the NPU runtime 120 when the differential K / V data required for the current verification window in the candidate differential partition 142 is ready to be read by the verify execution (Figure 122).

[0049] The logit margin is the difference between the highest and second-highest logit values ​​output by the target model at a candidate token position, used to characterize the confidence level of the prediction at that position.

[0050] Conservative decoding strategy refers to the operational strategy of shortening the number of candidate tokens, reducing the number of candidate branches, improving verification accuracy, or switching to single-step decoding of the target model in situations such as continuous rejection, insufficient free blocks in candidate differential partitions, insufficient output confidence, abnormal temperature or power consumption.

[0051] The following is a brief summary of some of the innovative aspects of this application: In summary, when deploying large language model speculative inference on an NPU, the pipeline overlap between draft execution graph 121 and verify execution graph 122, along with the transactional consistency of candidate key-value data, constitutes a set of coupled constraints: increasing pipeline overlap requires that draft execution graph 121 accept the prefix length before verify execution graph 122 has confirmed it. At this point, candidate tokens for the next verification window are prepared, and candidate difference blocks from different windows coexist in candidate difference partition 142; if window generation numbers are lacking... The binding with rollback index entries and the generational consistency check during verify reads mean that verify execution (Figure 122) may misread candidate difference blocks belonging to adjacent windows and merge K / V data not belonging to the current window into committed area 141, potentially causing abnormal states in committed area 141. This application, based on the understanding of this coupling constraint, includes the verification window descriptor 131 (such as...) Figure 2 (As shown) The design is to simultaneously carry the commit snapshot triple. Window generation numbering Candidate Differential Block State Bitmap and submission completion event identifier The NPU runtime schedules the data structure 120 and makes it physically isolated from the three partitions of the KV Cache 140: the committed area 141, the candidate difference partition 142, and the rollback index area 143, forming an inseparable collaboration: Draft execution diagram 121 writes the differential K / V of the candidate token to the candidate difference partition 142 and generates a corresponding variable in the rollback index area 143. The bound versioned rollback index entry does not update the commit area write pointer. When the `verify` function reads candidate difference blocks as shown in Figure 122, it first checks intergenerational consistency. After verification, it performs a two-stage atomic commit of pre-commit check and formal commit on the accepted prefix. The length of the committed sequence is... Only Only after the event returns can atomic propagation proceed. The rollback index entries with rejected suffixes are only marked as released and added to the free list; no recovery copy is performed on the committed area 141. This coordination ensures that the recovery cost in rejection or abnormal scenarios primarily depends on the update of the rollback index entry status and the maintenance of the free list, avoiding the cost increasing with the number of rejected tokens or model layers. and KV bit width Linear growth; simultaneous merging method When DMA is congested, it can adaptively switch to page table remapping or on-chip copy, which helps maintain the availability of the commit path in congested scenarios.

[0052] Furthermore, through long-term research and engineering practice on speculative inference of large language models on NPU, the inventors of this application have discovered that the actual acceleration benefits of speculative inference depend not only on the inference speed of the draft model and the acceptance rate of candidate tokens, but also to a considerable extent on whether the temporary K / V data corresponding to the candidate tokens can be committed or discarded at low cost, and whether the verification process can form a continuous, batch-processable, and low-synchronization execution window on the NPU. Ordinary speculative inference focuses on how candidate tokens are generated and how they are accepted by the target model at the algorithm level. However, at the NPU runtime level, if the K / V data of each candidate token is immediately written to the main KV Cache, then once a candidate token is rejected, the system needs to restore the written pointer of the committed area, correct the position index, and process the cached content that has already been written. This will make the processing of the rejection suffix too deeply coupled with the main cache state.

[0053] After in-depth analysis, the inventors discovered that the management of candidate key-value (K / V) data in existing solutions faces a difficult-to-coordinate technical constraint: if the K / V data of candidate tokens is written to the main K / V cache at the time of generation, then once a candidate token is rejected by the target model, the data already written to the main K / V cache needs to be recovered through copying or reconstruction. The amount of data moved is positively correlated with the number of rejected tokens, the number of model layers, and the K / V bit width. In scenarios with a large number of layers or a high K / V bit width, the DMA bandwidth consumed by this recovery copy may exceed the computational overhead saved by speculative inference itself. However, if the K / V data of candidate tokens is not cached at all, the K / V of candidate tokens needs to be recalculated during target verification, which offsets the throughput gains that speculative inference should have. The inventors recognized that the root of the above constraint lies in the fact that existing technologies treat the K / V cache as a single contiguous storage area, lacking a physical isolation mechanism for accepted and unverified K / V data, and also lacking transactional management methods that allow the submission of the accept prefix and the release of the reject suffix to be completed through different cost paths.

[0054] Meanwhile, through in-depth observation of the NPU inference execution process, the inventors discovered that the draft execution graph and the verify execution graph are often compiled into two independent execution graphs in existing implementations. The two need to be synchronized with the host side after each candidate token or each group of candidate tokens is generated, including the transmission of candidate token sequences, the updating of position codes, and the coordination of KV Cache states. This per-token or per-group host-side synchronization makes the NPU's computing units and DMA channels idle during the synchronization waiting period. The inventors recognized that if candidate tokens are organized into verification windows and a descriptor carrying the KV physical address, rollback index, and commit policy is generated for each window, the NPU runtime can schedule draft generation, verify, KV commit, and release operations on a window-by-window basis according to the descriptor, thereby elevating per-token host-side synchronization to window-level scheduling. However, when attempting to further utilize the NPU's parallel capabilities by increasing the pipeline overlap between draft and verify, candidate K / V data from different verification windows will coexist in storage. In this case, if there is a lack of window-level identity identification and consistency verification mechanisms, the verify execution graph may misread candidate data belonging to other windows, thereby affecting the state of the verified KV cache area.

[0055] The inventors also noted that the length of the verification window in existing speculative inference schemes is usually determined by preset parameters and cannot be dynamically adapted to changes in runtime acceptance rate, on-chip SRAM remaining capacity, DMA queue congestion level, and device power consumption and temperature status. If the window arrangement is not linked with candidate differential partition occupancy, rollback index release status, and NPU running status, increasing the window may lead to more wasted erroneous candidates, while a window that is too small will reduce the gains of speculative inference.

[0056] Based on the above in-depth research, the inventors of this application propose a collaborative acceleration scheme for NPU-oriented KV differential rollback cache driven by verification window descriptors: Physical isolation between accepted and unverified K / V data is achieved by dividing the KV cache into a committed area, a candidate differential area, and a rollback index area; the physical address, differential KV address, rollback index, and verification strategy of candidate tokens are encapsulated into a window-level scheduling unit that can be directly consumed by the NPU runtime through a verification window descriptor; and low-cost merging of accepted prefixes and low-cost recovery of rejected suffixes are achieved through a two-phase atomic commit and copy-free release mechanism. The implementation process of this application is described in detail below through specific embodiments.

[0057] The technical solution of this application will be further described below with reference to embodiments. It should be understood that the following embodiments are used to illustrate this application and should not be construed as limiting the scope of protection of this application. Without departing from the concept of this application, those skilled in the art can make adaptive adjustments to the cache block organization method, descriptor field width, event identifier format, and submission method according to the on-chip SRAM capacity, DMA channel capability, execution graph mechanism, and runtime scheduling interface of the target NPU.

[0058] Example 1: A Method for Accelerating Speculative Inference Using Large Language Models for NPU This embodiment provides a method for accelerating speculative inference verification window orchestration and KV rollback caching in large language models for NPU. This method is executed by the NPU runtime on an NPU configured with on-chip SRAM, DMA channels, and an execution graph mechanism. It should be noted that the KV Cache referred to in this paper refers to the storage area used to cache Key and Value tensors in the attention mechanism during autoregressive decoding of Transformer-type large language models. The draft execution graph referred to in this paper refers to the NPU executable graph used to generate candidate tokens. Its corresponding model can come from independent small models, shallow branches of the target model, quantized copies of the target model, or early-retrieval branches of the target model. This application does not limit the specific source of the draft model. The verify execution graph referred to in this paper refers to the NPU executable graph used to verify candidate tokens against the target model. The NPU runtime referred to in this paper refers to the runtime scheduling layer on the NPU side or closely cooperating with the NPU, used to manage execution graph startup, descriptor queues, event flags, DMA transfers, and cache states.

[0059] Large language models generate only one token at a time during the autoregressive decoding phase, requiring layer-by-layer reading of model weights and access to the KV Cache for attention computation. Speculative inference pre-generates multiple candidate tokens using a lightweight draft model, which are then verified all at once by the target model. Multiple tokens can be advanced simultaneously when a candidate token is accepted, reducing the number of calls to the target model. However, when implementing speculative inference on the NPU, if the temporary KV data corresponding to the candidate token is directly written to the main KV Cache, a large-scale recovery copy of the written area is required after rejection. Furthermore, frequent synchronization between the draft execution graph and the verify execution graph can cause pipeline interruptions in the NPU computing unit and DMA channel. This embodiment reduces these overheads by verifying window descriptor-driven NPU window-level scheduling, three-partition management of the KV Cache, and a differential commit and release mechanism for candidate KV. Figure 1 As shown, the method in this embodiment includes the following steps.

[0060] Step 100: Construct the execution graph, divide the KV Cache into three partitions and initialize it. Step 110: Compile the target large language model into a verify execution graph and the draft path into a draft execution graph. Both types of execution graphs use a unified token buffer, position buffer, mask buffer, and KV descriptor format. The token buffer stores the candidate token sequence, the position buffer stores the position index or position offset of the candidate token, the mask buffer stores the attention mask required for target verification, and the KV descriptor describes the physical layout of the submitted KV and candidate differential KV. Through the unified format, the NPU runtime can share candidate token sequences, position codes, and KV cache address information between the draft execution graph and the verify execution graph without format conversion when switching execution graphs.

[0061] Step 120: Generate a key-value physical layout description for each attention module in the target model. This layout description records the layer number. KV head group number Dimensions of a single head Bit width of K / V data Structural parameters include block size, sequence dimension step size, page table entry size, and DMA alignment granularity. This indicates the number of independently stored KV head groups in the cache, which is equal to the number of attention heads in multi-head attention and equal to the number of KV groups in grouped query attention. This information will be referenced by the verification window descriptor in subsequent steps to determine the write location of candidate differential KV data in the candidate differential partition, as well as the target address calculation when accepting prefix commits to the committed area.

[0062] Step 130: Divide the KV Cache into three physical regions: committed region, candidate difference region, and rollback index region. Figure 2 The structural relationship between the execution graph, descriptor queue, and KV cache is illustrated. This three-partition design is one of the key features of this application. The committed area stores the K / V data of historical tokens that have been accepted by the target model, and is a stable area that can be safely read by both the draft and verify execution graphs. The candidate difference area stores the differential K / V data generated by candidate tokens generated by the draft model within the current verification window. This data does not enter the committed area before being verified by the target model. The rollback index area records the mapping relationship, space usage, and state information between candidate tokens and candidate difference blocks. The synergistic effect of the three partitions is that the K / V data of candidate tokens is written to the candidate difference area in isolation. If a candidate token is rejected by the target model, only the corresponding entry in the rollback index area needs to be marked as released, without performing any recovery copy on the committed area. Compared to the approach of directly writing candidate key-value pairs into the main key-value cache, in the case of rejection suffixes, the release operation of this application only involves the state update of the rollback index item and the maintenance of the free list, avoiding linear growth with the rejection length, model layer number and key-value bit width.

[0063] Candidate differential partitions can be organized using fixed-size blocks, variable-length pages, or circular logs; this application does not limit the specific organization method. The total capacity of the candidate differential partitions... It should meet the following requirements: Here, coefficient 2 indicates that it includes both K and V types of cache. The number of layers in the target model. For the number of KV head groups, For the dimensions of a single head, To verify the upper limit of the window length, This refers to the bit width (in bits) of the differential K / V data. This represents the number of parallel windows. If the actual candidate difference partition capacity is lower than the value calculated by Equation 1, the NPU will automatically reduce the capacity during runtime. Reduce the number of candidate branches or disable multi-branch candidates.

[0064] Step 140: Complete the pre-filling calculation for the input prompt, write the pre-filled key-value data into the committed area, and initialize the committed area write pointer. Length of submitted sequences and KV page table version Establish a free linked list for candidate difference partitions and initialize the rollback index table. This includes writing pointers. The write position of the committed area is recorded in units of token slots; when using byte addresses, the conversion is performed according to the byte step size of the target KV slot, so that in subsequent formula 6... The relevant departments were established to promote this initiative.

[0065] Step 150: Register the verification window descriptor format and descriptor queue in the NPU runtime, and pre-define several window strategy templates, including a short window low-latency template, a long window high-throughput template, a low-acceptance-rate conservative template, and a long context cache pressure template. Optionally, the NPU runtime also maintains runtime statistics for each verification window for subsequent dynamic window scheduling.

[0066] Step 200: Determine the verification window parameters, generate candidate tokens, and write them to the candidate difference partition. Step 210: The NPU window scheduling module determines the number of candidate tokens for the current verification window based on the current submitted sequence length, remaining capacity of candidate differential partitions, acceptance status of recent verification windows, logit margin, on-chip cache usage, and DMA queue status. The number of candidate branches and the verification precision strategy. The number of candidate branches is 1 in single-branch sequential speculation, and can be greater than 1 in multi-branch candidate verification scenarios. The verification precision strategy can adopt full-precision verification, mixed-precision verification, or low-bit verification plus local recalculation, etc. The dynamic adjustment mechanism of the window parameters will be detailed in step 700.

[0067] Step 220: The draft execution graph is generated based on the K / V data of the committed areas and the sequence of committed tokens. Candidate tokens to It should be noted that during the process of generating candidate tokens, the draft execution graph writes the key-value data corresponding to each candidate token to the candidate difference block allocated in the candidate difference partition, without updating the write pointer of the committed area. It also does not change the length of the submitted sequence. This means that the contents of the committed area remain unchanged until verification is complete, thus providing a basis for subsequent copy-free release.

[0068] Step 230: Simultaneously with the writing of candidate differential KV data, the rollback index area generates rollback index entries for each candidate token, each layer, and each KV head group. Each rollback index entry records at least the window generation number of its respective verification window, the token sequence number within the window, the layer number, the KV head group number, the candidate differential block address, the target location, and the status, which can be represented as: in, This is the window generation number of the current verification window, which monotonically increases between windows and is used to distinguish candidate difference blocks of different verification windows in the pipeline overlap of the draft execution graph and the verify execution graph; The sequence number of the candidate token within the current verification window; For floor number; KV head group number; This refers to the candidate differential block address corresponding to the candidate token in this layer and this KV head group; Once the candidate token is accepted, its differential key / value data should be written to the target location in the submitted area; This represents the current state of the index entry.

[0069] The binding of rollback index entries to window generation numbers is a key feature of this application. Through this binding relationship, the verify execution graph can check generational consistency when reading candidate differential blocks, thereby avoiding cross-window key-value misreads in scenarios with overlapping dual execution graph pipelines.

[0070] Furthermore, the rollback index entry's state switches between six states: FREE, WRITING, READY, PREPARRED, COMMITTED, and RELEASED. Additionally, there are two exception paths: VERIFIED_REJECT and ABORTED, both ultimately leading to the RELEASED state. When all differential K / V data for all layers and all KV head groups corresponding to a candidate token has been written, the corresponding rollback index entry's state changes from WRITING to READY. When all candidate differential blocks within the window are in the READY state, the NPU runtime triggers a candidate differential partition readable event. This event can be mapped to a native NPU runtime event, semaphore, or other equivalent synchronization mechanism to notify the verify execution graph that it can consume the data from that verification window.

[0071] Step 300: Construct the verification window descriptor and write it to the descriptor queue. Step 310: The verification window descriptor generation module constructs a verification window descriptor for the candidate tokens generated in this round of speculation. The verification window descriptor is a hardware scheduling data structure designed for the NPU runtime in this application. Based on this descriptor, the NPU runtime can schedule draft generation, verify, key-value commit, and release operations in one go, thereby elevating the per-token host-side synchronization to window-level scheduling.

[0072] The verification window descriptor includes at least the following fields: candidate token address, pointing to the starting position of the candidate tokens in the candidate token buffer; position offset, recording the position index of each candidate token in the complete sequence; committed zone base address, pointing to the starting position of the committed KV Cache; candidate differential zone base address, pointing to the starting position of the candidate differential block in this window; and rollback index address, pointing to the rollback index table corresponding to this window.

[0073] Step 320: In addition to the basic fields mentioned above, the validation window descriptor also includes the following extended fields: window number, window generation number, length of the committed sequence before the window starts, number of candidate tokens, number of candidate branches (1 for single-branch sequential speculation), validation strategy number, validation precision mode, on-chip cache budget, and window cutoff condition.

[0074] More specifically, to maintain transactional consistency of candidate key-value pairs in a dual-execution-graph pipeline overlap scenario, the verification window descriptor also includes: a window generation number, which is a monotonically increasing number used to distinguish different verification windows in pipeline overlap; a candidate differential block status bitmap, used to indicate the status of the rollback index entries corresponding to each candidate token in the current verification window. The NPU runtime can quickly determine whether all candidate differential blocks in the window are ready by checking this bitmap; commit completion event flags and release completion event flags, used to synchronize the completion status of commit and release operations; a window commit abort flag, used to mark whether an abnormal rollback has occurred in the current window; and a merge method, which records the specific method used when merging the accept prefix into the committed area, taken from one of DMA scatter, on-chip copy, page table remapping, and pointer swapping. When the target NPU does not support page table remapping or pointer swapping, the merge method can be downgraded to DMA scatter or on-chip copy.

[0075] In addition, the verification window descriptor also includes submitting snapshots. Used to record the first The stable state of the committed region before the start of the verification window. A commit snapshot is represented as a triple: in, This is the write pointer to the area that was committed before the start of this verification window. The length of the submitted sequence, in units of tokens, before the start of this verification window. This refers to the KV page table version before the start of the verification window. It should be noted that the page table version is not limited to the page table version in the virtual storage system; it can also be a version number or equivalent sequence number maintained by the NPU runtime to characterize the stable state of the committed KV physical layout. The purpose of the commit snapshot is that when an anomaly is detected during the pre-commit check, the system can revert entirely to the state recorded in the commit snapshot without requiring complex recovery operations on the committed areas.

[0076] Step 330: Write the constructed verification window descriptors to the descriptor queue. The descriptor queue is organized using a double-buffering method: one set of verification window descriptors is consumed by the verify execution graph, and the other set is continued to be written by the draft execution graph. This double-buffering mechanism ensures that the verify execution graph consumes the first set of descriptors. When the descriptor of the first verification window is ready, the draft execution graph can simultaneously prepare the second one. The input of a verification window is partially verified, thereby enabling pipeline overlap between the draft and verify execution graphs.

[0077] Step 400: Verify the candidate tokens using the verify execution graph to validate the target model. Step 410: The verify execution graph reads the current verification window descriptor from the descriptor queue. Based on the address information recorded in the descriptor, it reads the candidate token sequence, position offset, K / V data of the committed area, and differential K / V data of the candidate differential block. During attention calculation, the verify execution graph needs to use K / V data from two different physical areas simultaneously: the committed area and the candidate differential block. The K / V data in the committed area corresponds to the attention context of historically accepted tokens, and the K / V data in the candidate differential block corresponds to the temporary attention state generated by the current candidate token. Logically, the two constitute a complete context sequence accessible to the verify execution graph.

[0078] Specifically, the verify execution graph executes the first... When calculating layer attention, the sequence length in the submitted region is taken into account. The previous key-value data and the candidate difference partition within the current validation window The candidate token corresponding to the first Layered key-value (K / V) data is concatenated along the sequence dimension to form a sequence of length [length missing]. The complete key-value sequence is used by the attention operator to calculate attention scores for candidate tokens. This concatenation can be achieved by DMA moving data from two physically discontinuous regions into a contiguous buffer in on-chip SRAM, or by an attention operator supporting multi-source address reads directly reading data from the submitted region and the candidate difference region respectively. Correspondingly, the first... The position is marked as visible; for the current verification window, the position is marked as visible. There are 1 candidate tokens, and their visibility range is the first 10 ... This allows for the establishment of specific positions, thereby maintaining the constraint of causal attention.

[0079] Step 420: When reading candidate differential blocks in the verify execution graph, verify whether the window generation number in the corresponding rollback index entry matches the window generation number in the verification window descriptor. If the window generation number corresponding to a candidate token is inconsistent, it indicates that the candidate differential block does not belong to the current verification window or has expired. The candidate token is considered rejected, and the acceptance prefix length is truncated to before the candidate token. When the inconsistency involves all candidate tokens in the current verification window, the window commit abort flag is set, and the current verification window is rolled back to the commit snapshot. The record state is then released, along with all candidate differential blocks for the current verification window. The purpose of window generational consistency checking is to prevent the verify execution graph from misreading candidate differential blocks belonging to other verification windows in scenarios where the draft and verify execution graphs overlap, thereby reducing the risk of the main KV cache being polluted.

[0080] Step 430: Verify the target model verification on the candidate tokens using the execution graph, and output the accepted prefix length. and rejection position And alternative tokens if necessary. Accepts prefix length. This represents the number of candidate tokens consecutively accepted by the target model starting from the beginning of the validation window. When This indicates that all candidate tokens in the current verification window have been accepted.

[0081] In one implementation, the verify execution graph calculates the output probability distribution of the target model for each candidate token position. It then compares the token with the highest output probability from the target model at that position with the candidate token generated by the draft model. If they match, the candidate token at that position is accepted; otherwise, that position is rejected, and the token with the highest output probability from the target model is used as the replacement token. (Accepted prefix length) This represents the number of tokens that are consecutively accepted starting from the first candidate token in the current validation window. In another implementation, the validation decision uses a probabilistic acceptance criterion, which calculates the ratio of the probability of the target model for the candidate token at that position to the probability of the draft model for that candidate token, and accepts or rejects the token according to the corresponding probability, thereby ensuring that the distribution of the final output is consistent with the distribution of the target model.

[0082] NPU runtime based on Generate submission mask and release mask : in, The sequence number of the candidate token within the current verification window. This represents the number of candidate tokens in the current verification window. When... When all candidate tokens are rejected, skip step 500 and directly execute the full release operation in step 600; when If all candidate tokens are accepted, skip step 600 and only accept the prefix submission.

[0083] Step 500: Accepting the two-phase commit of the prefix and water level atomic advancement This step is the core of the candidate KV transaction consistency mechanism in this application. The differential K / V data in the candidate differential blocks corresponding to the accepted prefix are merged into the committed area, employing a two-phase commit process consisting of a pre-commit check phase and a formal commit phase. The purpose of the two-phase commit design is to avoid intermediate state pollution in the committed area when partial failures occur during the commit process of multi-level K / V data.

[0084] Step 510 (Pre-commit Check Phase): Perform the following checks on each rollback index entry corresponding to the accept prefix in token order. First, check if the status is ready; second, check if the window generation number recorded in the rollback index entry matches the window generation number in the verification window descriptor; third, check if the target position recorded in the rollback index entry matches the length of the committed sequence based on the commit snapshot. The target location of the committed area is consistent with the token sequence number within the window; fourth, the KV page table version in the commit snapshot. Fifth, whether the candidate differential blocks of each layer and each KV head group of the corresponding candidate token are all in a commitable state.

[0085] When all the above conditions are met, the state of the corresponding rollback index entry is changed from ready to pre-commit, the corresponding bit is set in the candidate differential block state bitmap, and the corresponding candidate differential block is locked so that it cannot be reused by the free list before the release completion event flag returns. When any rollback index entry does not meet the above conditions, the rollback index entry that has been set to the pre-commit state is restored to the ready state, the window commit abort flag is set, and the current verification window is rolled back to the commit snapshot. The status of the record. The significance of the pre-commit check is that the merge operation is only actually performed after all index items to be committed have entered the pre-commit state, thereby avoiding inconsistencies in the committed area caused by partial commits.

[0086] Step 520 (Formal Submission Phase): For those in the pre-submission state and The NPU runtime merges the candidate differential blocks into the target location of the committed area according to the token order and the merging method.

[0087] Furthermore, the merging method is adaptively determined by the NPU runtime based on the following conditions: When the candidate differential block corresponding to the accept prefix meets the page alignment condition with the target location of the submitted area, and the target location is not occupied by other verification windows, page table remapping or pointer swapping is preferred. In this case, the physical location of the differential K / V data is not moved; only the page table mapping is updated or the pointer is swapped, which is a zero-copy operation. Otherwise, when the number of fragments of the candidate differential block does not exceed the NPU DMA scatter descriptor queue capacity, and the DMA queue congestion does not exceed the congestion boundary set in the runtime policy template, DMA scatter is used to complete the data migration. When none of the above conditions are met, on-chip copy is used. It should be noted that there is no hard binding relationship between the above four merging methods; the NPU runtime selects according to the actual state at runtime. Optionally, when the target NPU does not support page table remapping or pointer swapping, the merging method is downgraded to a choice between DMA scatter and on-chip copy. In a degraded scenario of continuous DMA congestion, the system can still complete the submission of the accept prefix by switching to on-chip copy mode, thus maintaining the normal operation of the inference process.

[0088] Once the formal submission is complete, the status of the corresponding rollback index item will be changed from pre-submitted to submitted.

[0089] Step 530 (Water Level Advancement after Commit Completion Event): After the commit completion event corresponding to the commit completion event identifier returns, the NPU runtime atomically advances the write pointer of the committed area, the length of the committed sequence, and the KV page table version. in, For the first The length of the accept prefix for each verification window. , , These are the write pointer to the committed area after the push, the length of the committed sequence, and the KV page table version. This is a version progression function that ensures the KV page table version is monotonically increasing. It should be noted that in Formula 6... and All are advanced in units of token slots; when When using byte address representation, the byte step size of the target KV slot is used. Converted to the corresponding byte increment.

[0090] It should be noted that before the commit completion event is returned, the draft execution graph only reads the length of the committed sequence when generating candidate tokens for the next verification window. Previously committed key-value (K / V) data is not read from candidate differential blocks in the pre-commit state. This constraint ensures that even if the draft and verify execution graphs overlap in time, the K / V cache read by the draft execution graph remains a consistent snapshot state, thus preventing the quality of subsequent candidate token generation from being affected by reading unstable K / V data.

[0091] Step 600: Reject uncopyable releases with suffixes Based on the rejection location The rejected candidate token and its subsequent candidate tokens will be released, along with their corresponding candidate differential blocks. For rollback index entries, if their state is not pre-committed, they are directly set to the released state, and the corresponding candidate differential block is added to the free list of the candidate differential partition, allowing it to be reused by subsequent verification windows. If the rollback index entry's state is pre-committed (this may occur in the exception handling path), the release operation is performed after the commit completion event flag is returned. Simultaneously, the rejection position and its subsequent token buffer, position buffer, and mask buffer entries are synchronously marked as invalid. After release, the release completion event flag is set.

[0092] Furthermore, the release of a rejected suffix and the commit of an accepted prefix can be executed in parallel, but the release must not cross the write pointer of a committed area before the start of the current verification window. Furthermore, the release process must not release candidate differential blocks that are in a pre-commit state. The entire release process does not read, write, or rebuild committed regions and their page tables.

[0093] This is the key advantage of this application compared to the scheme of directly writing candidate KV to the main KV cache. In the direct write scheme, the more rejected candidate tokens, the more model layers, and the higher the KV bit width, the greater the cost of recovery copying, and the larger the migration amount. The number of bytes is positively correlated with the rejection length, model layer number, and KV bit width. However, in this application, regardless of the number of rejected candidate tokens and the model size, the release operation only involves rolling back the state update of the index item and maintaining the free list, thereby avoiding the aforementioned migration amount from increasing linearly with the rejection length, model layer number, and KV bit width; the recovery of the abnormal window or full rejection window is simply equivalent to discarding a descriptor and restoring the commit snapshot.

[0094] Step 700: Dynamic Validation Window Arrangement and Conservative Rollback Step 710: Dynamically adjust the parameters of subsequent verification windows based on the runtime status of the most recent verification windows. Adjustable status information includes, but is not limited to: acceptance rate, logit margin, context length, on-chip cache usage, and DMA queue status. Adjustable window parameters include, but are not limited to: number of candidate tokens, number of candidate branches, verification batch size, verification precision strategy, and candidate difference partition capacity. For example, when the acceptance rate is high and the on-chip cache is sufficient, increase the window length to improve the benefit of a single verification; when the acceptance rate decreases, the logit margin approaches zero, or the DMA queue is congested, shorten the window length or switch to a more conservative verification strategy.

[0095] Step 720: When the current free capacity of the candidate difference partition Smaller than the candidate difference partition size required by the current validation window At that time, the number of candidate tokens for the next verification window is converged using the following formula: in, This is the floor function. For the first The required candidate difference partition capacity for each verification window is calculated as follows: The meanings of the symbols are consistent with those in Formula 1. For the first The number of candidate tokens in the verification window. In a multi-branch candidate verification scenario, the number of candidate tokens in the verification window. The required candidate difference partition capacity for each verification window is calculated based on the number of candidate branches. Enlarge accordingly, that is When speculating in a single-branch sequence The purpose of Formula 7 is to reduce the number of candidate tokens in the next window proportionally when the remaining space of the candidate difference partition is insufficient to accommodate the original number of candidate tokens, so that the system can continue to operate with the limited capacity of the difference partition, instead of directly degenerating into single-step decoding.

[0096] Furthermore, to prevent the number of candidate tokens from converging to zero and causing speculative reasoning to completely stop, a lower limit is set for the verification window length. ( When the result of formula 7 is less than Time to take: When the current free capacity is insufficient to meet the requirements When the required capacity is reached, switch to single-step decoding of the target model as per step 730.

[0097] Optionally, the NPU window scheduling module can utilize window benefit scoring to assist in adjusting the window length. The basic window benefit scoring can be expressed as: in, This represents the average number of tokens accepted over the most recent few windows. For DMA queue congestion, The candidate difference partition occupancy rate, For power consumption or temperature penalty items, , , , These are the weights configured by the runtime policy template. The above items can be obtained using simple moving averages, exponential moving averages, or other statistical methods supported by the NPU runtime.

[0098] In the preferred implementation that uses submission snapshots and event receipts, the window benefit score can be further incorporated with submission-related penalties to form an extended score: in, This represents the congestion caused by waiting for completed submissions within the most recent statistical period. This refers to the number or proportion of abnormal windows. To pre-submit rollback rate, , , These are configurable runtime weights. When the above penalty terms increase, the system can shorten the next window length, reduce the number of parallel windows, or decrease the number of candidate branches; when multiple consecutive windows complete commits without errors, a larger window length can be gradually restored.

[0099] The window length is updated accordingly between the upper and lower limits: in, The function truncates the calculation results to... Within the interval, This is the rounding function. and These are the lower and upper limits of the verification window length, respectively, determined by the session context length and the candidate differential partition capacity.

[0100] Step 730: Switch to a conservative decoding strategy when the system meets any of the following conditions: the acceptance rate of multiple consecutive verification windows is lower than a preset threshold; the number of free blocks in the candidate difference partition is lower than a preset threshold; the output confidence is lower than a preset threshold; the power consumption or temperature of the NPU exceeds a preset threshold; the number of windows that continuously trigger the window submission abort flag reaches a threshold; or the submission completion event times out. The conservative decoding strategy includes at least one of the following: shortening the number of candidate tokens, reducing the number of candidate branches to 1, improving verification accuracy, clearing candidate difference partitions, and switching to single-step decoding of the target model. Once the system re-enters a stable state, a larger window length can be gradually restored.

[0101] Furthermore, if the commit completion event does not return within the waiting boundary, the NPU runtime will lock the candidate differential blocks in the pre-commit state to a non-reusable state and make the next verification window based solely on the length of the committed sequence. The previous KV Cache generated candidate tokens and switched to a conservative decoding strategy. After the commit completion event returned, the locked candidate differential blocks were committed according to step 520 or released according to step 600. If no response was received within the further timeout boundary and the confirmation event was lost, the NPU runtime performed exception cleanup, adding the locked candidate differential blocks to the free list and reverting to the state of the commit snapshot record. Through the above processing, the long-term occupation of pre-committed blocks, which could lead to the candidate differential block resource being locked, was avoided.

[0102] Step 800: Dual Execution Graph Pipeline Scheduling The draft and verify execution graphs are piped through a descriptor queue in the NPU runtime. Specifically, the draft execution graph is executed in a window. After candidate generation is complete, the verification window descriptor is written to one buffer of the descriptor queue. The verify execution graph consumes windows from the other buffer of the descriptor queue. When using descriptors, the draft execution graph can be prepared based on known committed KV snapshots. The NPU runtime uses event flags to synchronize the following three states: candidate differential partition readable state, commit completed state, and release completed state. These three event flags are mapped to the NPU runtime's native events, semaphores, or other equivalent hardware synchronization mechanisms, and are not limited to instruction names from a specific vendor.

[0103] When the candidate difference partition space is insufficient or the accept prefix is ​​not yet determined, the draft execution graph enters a lightweight wait state without triggering host-side reconstruction of the execution graph. Through the pipelined scheduling mechanism of the descriptor queue and event flags described above, draft generation, target verification, accept prefix submission, and reject suffix release can be scheduled by the NPU runtime in window units, reducing the number of interactions between the host and the NPU. Compared to the method of returning to host-side synchronization token by token, this method helps improve the utilization of NPU computing units and DMA channels.

[0104] Step 900: Iterative Execution After completing steps 500 and 600, the system returns to step 210 to continue the next round of candidate token generation and verification. Before returning, if step 500 outputs a replacement token, that replacement token is used as the starting point for the next round of generation. The loop from steps 200 to 800 continues until the generation termination condition is met.

[0105] Example 2: A Large Language Model Speculative Inference Acceleration System for NPU This embodiment provides a large language model speculative inference acceleration system for NPUs, running on an NPU configured with on-chip SRAM, DMA channels, and an execution graph mechanism, for executing the method described in Embodiment 1. Figure 2 As shown, the system includes the following modules.

[0106] The model execution graph construction module is used to compile the draft model and the target model into NPU-executable draft execution graphs and verify execution graphs, respectively, and to establish a shared token buffer, position buffer, mask buffer, and KV descriptor format for both.

[0107] The KV rollback cache management module is used to divide the KV Cache into the committed area, the candidate differential partition, and the rollback index area. It manages the allocation and reclamation of differential blocks in the candidate differential partition, maintains the state transition of each rollback index entry in the rollback index area, and triggers window length convergence when the capacity of the candidate differential partition is insufficient.

[0108] The NPU window scheduling module reads the current system running status, determines the number of candidate tokens, candidate branches, verification batch, verification precision strategy, and candidate difference partition capacity for subsequent verification windows, and switches to a conservative decoding strategy when the backoff conditions are met. Optionally, the running status used includes acceptance rate, logit margin, context length, on-chip cache usage, and DMA queue status.

[0109] The verification window descriptor generation module is used to construct verification window descriptors for candidate tokens generated in each round of speculation and write the descriptors to the descriptor queue.

[0110] The fusion verification operator module is used by the verify execution graph to read candidate tokens, position offsets, K / V data of the committed area and candidate difference blocks from the verification window descriptors in the descriptor queue, perform target model verification on the candidate tokens, verify the intergenerational consistency of the verification window, and output the accept prefix length and rejection position.

[0111] The commit and release module is used to perform a two-phase commit on the candidate differential blocks corresponding to the accept prefix based on the accept prefix length. After the commit completion event returns, it atomically advances the write pointer of the committed area and the length of the committed sequence. It also marks the candidate differential blocks corresponding to the rejected token and its subsequent tokens as released based on the rejection position to release the differential blocks, and does not perform a recovery copy on the committed area.

[0112] The six modules described above work collaboratively under the scheduling of the NPU runtime. The draft execution graph and the verify execution graph achieve double-buffered pipelined execution through a descriptor queue. Each of these modules can be implemented by software programs, the NPU runtime, firmware, drivers, hardware state machines, or a combination thereof.

[0113] Example 3: Electronic Devices and Computer-Readable Storage Media This embodiment provides an electronic device, including an NPU, a memory, and a processor. The memory stores program instructions, which, when executed by the processor, the NPU, or when the NPU runs, cause the electronic device to perform the method described in Embodiment 1. The NPU is configured with on-chip SRAM, DMA channels, and an execution graph mechanism. This electronic device can be a server, an edge computing device, an inference acceleration device, a terminal device, or other computing devices configured with an NPU.

[0114] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in Embodiment 1. The computer-readable storage medium includes, but is not limited to, non-volatile memory, magnetic disk, optical disk, flash memory, solid-state drive, or other media capable of storing program instructions.

[0115] Other optional implementation methods To further illustrate the implementation of this application, several optional implementations are given below. These optional implementations are optional implementation methods and do not constitute a limitation on the above embodiments, nor do they limit specific NPU manufacturers, specific event register names, DMA instruction names, or execution graph compilation interfaces.

[0116] In one optional implementation, the NPU runtime synchronizes event flags through hardware event registers, semaphores, or native event objects. A candidate differential block readable event notifies the verify execution graph to read the candidate differential block; a commit complete event notifies that the committed area write pointer and the committed sequence length can be atomically advanced; and a release complete event notifies that the candidate differential block can be added to the free list. On NPUs with a limited number of event channels, the commit complete event and release complete event can be merged into a single event flag, but the NPU runtime still maintains separate commit complete and release complete states.

[0117] In one alternative implementation, the draft model uses the first few layers of the target model as shallow branches, sharing the word embedding layers and weight parameters of the first few Transformer layers of the target model. The shallow branches are appended with a lightweight linear mapping head to the output of the last layer, projecting the hidden states onto the vocabulary space to generate the probability distribution of candidate tokens. Because the shallow branches share some weights from the target model, the generated candidate tokens have a certain degree of semantic coherence, while the computational cost is lower than that of the full target model. In another alternative implementation, the draft model is a small, independently trained language model with a vocabulary identical to the target model.

[0118] In one alternative implementation, candidate differential partitions are organized into fixed-size blocks, with each differential block being the size of the K or V data of a candidate token's KV head group in a layer. The candidate differential partition maintains a linked list of free blocks. When the draft execution graph writes one layer of key-value data for a candidate token, a block is retrieved from the free list and the data is written. In another optional implementation, the candidate differential partition uses a page-based organization, where each page can hold single-layer key-value data for multiple tokens, suitable for scenarios with a large number of candidate tokens. In yet another optional implementation, the candidate differential partition uses a circular log approach, allocating space contiguously in the order of writing and releasing it entirely at the window level.

[0119] In one alternative implementation, the beneficial effects of the proposed solution can be evaluated in terms of tokens / s, average single-token decoding latency, KV migration amount in rejection suffix scenarios, candidate differential partition space reuse rate, and DMA bandwidth usage. Furthermore, the changes in acceptance rate and end-to-end throughput before and after dynamic adjustment of the verification window can be observed under different context lengths, different task types, and different on-chip SRAM capacity configurations.

[0120] In one optional implementation, parameters such as the commit completion event wait boundary, DMA congestion boundary, and number of consecutive exception windows are set by the NPU runtime based on device configuration, session concurrency, and policy template. When the commit completion event does not return within the wait boundary, the next verification window is based solely on the length of the committed sequence. The previous KV Cache generated candidate tokens and did not reuse candidate differential blocks in the pre-commit state; when the DMA congestion exceeds the congestion boundary, the NPU runtime will switch the merging method to the available methods among on-chip copy, page table remapping or pointer swapping; when the number of windows that continuously trigger the window commit abort flag reaches the threshold, the NPU runtime reduces the number of candidate tokens, reduces the number of candidate branches to 1, or switches to single-step decoding of the target model.

[0121] The above embodiments have the following technical effects First, it reduces the cost of KV rollback. By dividing the KV Cache into a committed area, a candidate differential area, and a rollback index area, the K / V data of candidate tokens is written to the candidate differential area in isolation without directly overwriting the committed area. When a candidate token is rejected, only the rollback index entry needs to be marked as released and the candidate differential block needs to be added to the free list; there is no need to perform a recovery copy on the committed area. Compared to the corresponding scheme where candidate KV is directly written to the main KV Cache, in the recovery scenario of rejected suffixes, the amount of K / V recovery migration for rejected suffixes in the committed area is zero, and the release cost mainly depends on the index entry status update and free list maintenance, avoiding linear growth with the rejection length, model layer number, and KV bit width.

[0122] Second, it improves NPU pipeline efficiency. The verification window descriptor enables draft generation, verifying, key-value commit, and release operations to be scheduled by the NPU runtime based on the descriptor queue, reducing per-token host-side synchronization. The draft execution graph and the verify execution graph achieve pipeline overlap through a double-buffered descriptor queue, which helps improve the utilization of NPU computing units and DMA channels.

[0123] Third, by binding window generation numbers and using a snapshot commit mechanism, the risk of cross-window key-value misreading and cross-layer intermediate state pollution is reduced in scenarios with overlapping dual execution graph pipelines. Binding rollback index entries to window generation numbers allows the verify execution graph to identify expired data when reading candidate differential blocks. Two-phase commit ensures that the commit area write pointer only advances atomically after all pre-commit checks have passed and the commit completion event has returned. Recovery of failed commit windows only requires rolling back to the commit snapshot, avoiding a linear increase in recovery cost with rejection length, model layer number, and key-value bit width.

[0124] Fourth, the prefix merging method can adaptively select from multiple merging methods based on page alignment conditions, DMA queue congestion, and on-chip resource status, which helps maintain the availability of the commit path under different NPU operating states. In DMA congestion scenarios, the commit can still be completed by switching merging methods.

[0125] Fifth, the number of candidate tokens, candidate branches, and verification accuracy strategy of the verification window can be dynamically adjusted based on the acceptance rate, candidate difference partition capacity, and DMA queue status. When the idle capacity of the candidate difference partition is insufficient, the system proportionally shortens the convergence window length; in the event of consecutive low acceptance rates or abnormal power consumption, the system can switch to a conservative decoding strategy, which helps reduce the waste of computing resources in low-yield scenarios.

[0126] It should be noted that in this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this application, if a reference is made to performing an action based on an element, it means performing the action at least based on that element, including two cases: performing the action only based on that element, and performing the action based on that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.

[0127] It should be understood that after reading the above disclosure of this application, those skilled in the art can make various alterations or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. A method for accelerating speculative reasoning in large language models for NPUs, characterized in that, The method, executed by NPU runtime scheduling on an NPU configured with on-chip SRAM, DMA channels, and an execution graph mechanism, includes: Construct the draft execution graph for generating candidate tokens and the verify execution graph for verifying the candidate tokens using the target model; The KV Cache is divided into a committed area, a candidate difference area, and a rollback index area. The committed area stores the K / V data of accepted tokens, the candidate difference area stores the differential K / V data generated by candidate tokens, and the rollback index area records the mapping relationship between candidate tokens and candidate difference blocks. The draft execution graph generates candidate tokens based on the K / V data of the committed area, writes the differential K / V data corresponding to the candidate tokens into the candidate differential block of the candidate differential partition, does not update the write pointer of the committed area, and generates the corresponding rollback index entry in the rollback index area. A verification window descriptor is constructed for the candidate token and written to the descriptor queue. The verification window descriptor includes at least the candidate token address, the position offset, the submitted zone base address, the candidate difference zone base address, and the rollback index address. The verify execution graph reads the verification window descriptor from the descriptor queue, reads the candidate token, the K / V data of the submitted area and the candidate difference block according to the verification window descriptor, performs target model verification on the candidate token, and outputs the accept prefix length and rejection position; Based on the accepted prefix length, the differential K / V data of the candidate differential blocks corresponding to the accepted prefix are merged into the committed area, and the write pointer and committed sequence length of the committed area are updated; Based on the rejection position, the rollback index entries corresponding to the rejected candidate token and its subsequent candidate tokens are marked as released to release the corresponding candidate differential blocks, and no recovery copy is performed on the committed area; The draft execution graph and the verify execution graph are executed in a pipelined manner during the NPU runtime through the descriptor queue.

2. The method according to claim 1, characterized in that, The verification window descriptor also includes: window number, length of the committed sequence before the window starts, number of candidate tokens, number of candidate branches, verification strategy number, verification precision mode, on-chip cache budget, window cutoff condition, window generation number, commit snapshot, candidate differential block status bitmap, commit completion event identifier, release completion event identifier, window commit abort flag, and merging method; The window generation number is a monotonically increasing number used to distinguish different verification windows in pipeline overlap; the candidate differential block state bitmap is used to indicate the state of the rollback index item corresponding to each candidate token in the current verification window; the merging method records the merging method of the accepted prefix, which is taken from one of DMA scatter, on-chip copy, page table remapping, and pointer swapping. The submitted snapshot Record number The submitted area status before the start of each verification window is represented as a triplet: in, This is the write pointer to the committed area mentioned before the start of this verification window. The length of the submitted sequence, in units of tokens, before the start of this verification window. This is the KV page table version before the start of this verification window.

3. The method according to claim 2, characterized in that, The rollback index entry is bound to the window generation number. The rollback index entry records at least the window generation number of the verification window, the token sequence number within the window, the layer number, the KVhead group number, the candidate differential block address, the target position, and the status. The target position is the position where the differential K / V data of the corresponding candidate token should be written to the submitted area after the candidate token is accepted. When the candidate token is validated by the verify execution graph, the window generation number in the corresponding rollback index item is checked to see if it matches the window generation number in the verification window descriptor when the candidate differential block is read. If they do not match, the corresponding candidate token is considered rejected. When the inconsistency involves all candidate tokens in the current verification window, the window submission abort flag is set, the current verification window is rolled back to the state of the submission snapshot record, and all candidate differential blocks of the current verification window are released.

4. The method according to claim 3, characterized in that, The differential K / V data in the candidate differential blocks corresponding to the accepted prefix are merged into the committed area, using a two-stage submission process consisting of a pre-submission check phase and a formal submission phase. During the pre-commit check phase, the rollback index entries corresponding to the accept prefix are checked one by one in token order, and the following conditions are met: the status is ready; the window generation number matches the window generation number in the verification window descriptor; the target position matches the length of the committed sequence according to the commit snapshot. The target location of the committed area is consistent with the token sequence number within the window; the KV page table version in the committed snapshot. The submission actions of other verification windows have not changed the status; the candidate differential blocks of each layer and each KV head group of the corresponding candidate token are all in the committable state; when all conditions are met, the corresponding rollback index item is set from the ready state to the pre-commit state, the corresponding bit is set in the candidate differential block state bitmap, and the corresponding candidate differential block is locked so that it is not reused by the free list before the release is completed; when any rollback index item does not meet the conditions, the rollback index item that has been set to the pre-commit state is restored to the ready state, the window submission abort flag is set, and the current verification window as a whole reverts to the state of the submission snapshot record; During the formal submission phase, the candidate differential blocks corresponding to the rollback index items in the pre-submission state are merged into the target position of the submitted area according to the merging method, based on the token order, and the corresponding rollback index items are set to the submitted state. After the commit completion event corresponding to the commit completion event identifier is returned, the write pointer of the committed area, the length of the committed sequence, and the KV page table version are atomically advanced: in, For the first The length of the accept prefix for each verification window. , , These are the write pointer to the committed area after the push, the length of the committed sequence, and the KV page table version. A version advance function to ensure that the KV page table version is monotonically incremented; Before the submission completion event returns, when the draft execution graph generates candidate tokens for the next verification window, it only reads the length of the submitted sequence. Previously committed key-value data is not read from candidate differential blocks that are in the pre-commit state.

5. The method according to claim 4, characterized in that, The state of the rollback index entry switches between idle, writing, ready, pre-commit, committed, and released; When releasing the rejected candidate token and the candidate differential block corresponding to the subsequent candidate token, if the state of the corresponding rollback index item is not in the pre-commit state, it is directly set to the released state, and the corresponding candidate differential block is added to the free list of the candidate differential partition; if the state of the corresponding rollback index item is in the pre-commit state, it is released after the commit completion event returns. After the release is completed, set the release completion event flag. The release and acceptance prefix submission are executed in parallel, releasing write pointers that do not cross the submitted area before the current verification window begins. Furthermore, the release process does not read, write, or rebuild the committed area and its page tables.

6. The method according to claim 2, characterized in that, The merging method is adaptively determined by the NPU runtime based on the page alignment condition between the candidate differential block corresponding to the accept prefix and the target location of the submitted region, the queue congestion of the DMA channel, and the on-chip resource status. When the page alignment condition is met and the target position is not occupied by other verification windows, page table remapping or pointer swapping is used. Otherwise, DMA scatter is used when the number of fragments in the candidate differential block does not exceed the capacity of the DMA scatter descriptor queue and the DMA queue congestion does not exceed the congestion boundary. If none of the above conditions are met, on-chip copying will be used.

7. The method according to claim 1, characterized in that, The descriptor queue uses a double-buffered approach: one set of verification window descriptors is consumed by the verify execution graph, and the other set is continued to be written by the draft execution graph; the verify execution graph consumes the first set of descriptors... When the verification window descriptor of the first verification window is prepared, the draft execution graph is ready for the first verification window. The NPU runtime synchronizes the candidate difference partition readability, submission completion, and release completion status through event flags; when the candidate difference partition space is insufficient or the accept prefix is ​​not determined, the draft execution graph enters a waiting state without triggering host-side reconstruction of the execution graph; Based on at least one of the following factors in recent verification windows: acceptance rate, logit margin, context length, on-chip cache usage, and DMA queue status, dynamically adjust at least one of the following factors in subsequent verification windows: number of candidate tokens, number of candidate branches, verification batch size, verification precision strategy, and candidate differential partition capacity. No. The required candidate difference partition capacity for each verification window satisfy: in, For the first The number of candidate tokens for each verification window. The number of layers in the target model. For the number of KVheads, For the dimension of a single head, The bit width of the differential K / V data is expressed in bits, and a coefficient of 2 indicates that it includes both K and V types of cache. When the current free capacity of the candidate difference partition Less than When the number of candidate tokens for the next verification window is converged, the following formula is used: : in, This is the floor function; When the acceptance rate of multiple consecutive verification windows is lower than the threshold, the number of idle blocks in the candidate differential partition is lower than the threshold, the output confidence is lower than the threshold, or the power consumption or temperature of the NPU exceeds the threshold, the conservative decoding strategy is switched to. The conservative decoding strategy includes at least one of shortening the number of candidate tokens, reducing the number of candidate branches, improving verification accuracy, and switching to single-step decoding of the target model.

8. A large language model speculative inference acceleration system for NPU, running on an NPU equipped with on-chip SRAM, DMA channels, and an execution graph mechanism, characterized in that, The system includes: The model execution graph construction module is used to construct the draft execution graph for generating candidate tokens and the verify execution graph for verifying the candidate tokens using the target model. The KV rollback cache management module is used to divide the KV Cache into a committed area, a candidate difference area, and a rollback index area, and to manage the candidate difference blocks and corresponding rollback index entries written by the draft execution graph into the candidate difference area. The NPU window scheduling module is used to determine the number of candidate tokens, the number of candidate branches, the verification batch, the verification precision strategy, and the candidate difference partition capacity of the verification window, for use by the draft execution graph to generate candidate tokens; The verification window descriptor generation module is used to construct verification window descriptors for candidate tokens and write them into the descriptor queue; The fusion verification operator module is used to read candidate tokens, position offsets, K / V data of the submitted area and candidate difference blocks from the verification window descriptors in the descriptor queue by the verify execution graph, perform target model verification on the candidate tokens, and output the accept prefix length and rejection position; The submission and release module is used to merge the differential K / V data in the candidate differential block corresponding to the acceptance prefix into the submitted area according to the acceptance prefix length, and release the rejected candidate token and the candidate differential block corresponding to its subsequent candidate token according to the rejection position, without performing a recovery copy on the submitted area; The system is used to perform the method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, The device includes an NPU, a memory, and a processor, wherein the memory stores program instructions that, when executed, cause the electronic device to perform the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images