Speculative decoding method, apparatus, device, and storage medium

Abstract

The application relates to the technical field of model reasoning, and discloses a speculation decoding method and device, equipment and a storage medium, which comprise the following steps: based on a first model, first candidate word elements corresponding to each current candidate word element are generated, and the first candidate word elements are stored in a buffer area; when it is detected that a preset synchronization condition is met, second candidate word elements in the buffer area are synchronized to a second model; based on the second model, a second check result is determined according to the second candidate word elements and a first check result; based on the first model, the buffer area is updated according to the second check result, and current candidate word elements of the next round are determined based on the updated buffer area, so that the next round of reasoning is performed. The generation process of the first model and the check process of the second model can be decoupled in time, the idle waiting time between processes is minimized, parallel work of the two models is realized, and the overall reasoning delay of the speculation decoding is effectively reduced.

Description

Technical Field

[0001] This application relates to the field of model reasoning technology, and in particular to a speculative decoding method, apparatus, device, and storage medium. Background Technology

[0002] In the speculative decoding process, the inference phase of the draft model and the verification phase of the target model are sequentially dependent. The draft model must first generate a draft before the target model can be verified, and only after the target model has been verified can the draft model proceed to the next round of inference based on the verification results. In traditional solutions, the inference processes of the draft model and the target model are executed alternately in a serial manner, with both waiting for each other. This results in underutilization of resources, high inference latency, and limits the acceleration effect of speculative decoding. Summary of the Invention

[0003] The main objective of this application is to provide a speculative decoding method, apparatus, device, and storage medium, which aims to solve the technical problem that the inference process of the draft model and the target model is executed alternately in a serial manner, which limits the acceleration effect of speculative decoding.

[0004] To achieve the above objectives, this application proposes a speculative decoding method, which includes: Based on the first model, generate first candidate word elements corresponding to each current candidate word element, and store the first candidate word elements in the buffer; When a preset synchronization condition is detected, the second candidate word in the buffer is synchronized to the second model; Based on the second model, a second verification result is determined according to the second candidate word and the first verification result. The first verification result is generated by verifying the candidate word of the previous round of synchronization through the second model. Based on the first model, the buffer is updated according to the second verification result, and the current candidate word for the next round is determined based on the updated buffer to perform the next round of inference.

[0005] Optionally, the buffer includes a first cache and a second cache; The step of generating first candidate lexical units corresponding to each current candidate lexical unit based on the first model, and storing the first candidate lexical units in a buffer, includes: Based on the first model, extended reasoning is performed on each of the current candidate word elements to generate multiple first candidate word elements, and the generation probability of each first candidate word element is obtained; Based on the generation probability and the path weight of the current candidate word, a second candidate word is determined from the first candidate word. The second candidate word is stored in the first cache, and the first candidate word, as well as the association between each first candidate word and the current candidate word, is stored in the second cache.

[0006] Optionally, the step of determining the second candidate lexical unit from the first candidate lexical unit based on the generation probability and the path weight of the current candidate lexical unit includes: The target path weight of the first candidate word is determined based on the generation probability and the path weight of the current candidate word. The second candidate word is determined from the first candidate word based on the target path weight.

[0007] Optionally, before the step of synchronizing the second candidate lexical unit in the buffer to the second model when a preset synchronization condition is detected, the method further includes: Get the current reasoning round; When the current inference round is an integer multiple of the preset communication period, it is determined that the buffer satisfies the preset synchronization condition; When the current inference round is not an integer multiple of the preset communication period, return to the step of generating a first candidate word corresponding to each current candidate word based on the first model, and storing the first candidate word in the buffer.

[0008] Optionally, the step of determining the second verification result based on the second model, according to the second candidate lexical unit and the first verification result, includes: Based on the first verification result, the target candidate path is determined from the candidate paths corresponding to the second candidate word element; Based on the second model, reason about the third candidate word in the target candidate path and output the target probability of the third candidate word; The second verification result is determined based on the target probability and the generation probability of each of the third candidate words.

[0009] Optionally, the step of determining the second verification result based on the target probability and the generation probability of each of the third candidate lexical units includes: When the target probability is greater than or equal to the generation probability, the third candidate word is set to pass the verification. When the target probability is less than the generation probability, the third candidate word is set as failing the verification, and the position information of the third candidate word that failed the verification in the target candidate path and the candidate word output by the second model at the position information are recorded. The second verification result is determined based on the verification result of the third candidate word.

[0010] Optionally, the step of updating the buffer based on the first model and the second verification result includes: When the second verification result indicates that there is a fourth candidate word element that failed the verification in the target candidate path, the position information of the fourth candidate word element in the target candidate path is obtained; Based on the location information, the fifth candidate word element is searched in the buffer. The fourth candidate word in the target candidate path is replaced with the fifth candidate word to backtrack the candidate words in the buffer.

[0011] Furthermore, to achieve the above objectives, this application also proposes a speculative decoding apparatus, which includes: The generation module is used to generate first candidate word elements corresponding to each current candidate word element based on the first model, and store the first candidate word elements in the buffer; A synchronization module is used to synchronize the second candidate word in the buffer to the second model when a preset synchronization condition is detected. The verification module is used to determine a second verification result based on the second model, the second candidate word, and the first verification result, wherein the first verification result is generated by verifying the candidate word of the previous round of synchronization through the second model. An update module is used to update the buffer based on the first model and the second verification result, and to determine the current candidate word for the next round based on the updated buffer, so as to perform the next round of inference.

[0012] In addition, to achieve the above objectives, this application also proposes a speculative decoding device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the speculative decoding method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the speculative decoding method described above.

[0014] In this application, the generated candidate lexical units are stored in a buffer using a first model, and the candidate lexical units in the buffer are synchronized to the second model when a preset synchronization condition is met. This decouples the generation process of the first model from the verification process of the second model in time. The first model can generate candidate lexical units based on the updated buffer without waiting for the second model to complete its verification before starting. This minimizes the idle waiting time between processes, enables the two models to work in parallel, significantly improves the utilization of GPU resources, and effectively reduces the overall inference latency of speculative decoding. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the inferential decoding method of this application; Figure 2 This is a schematic diagram of the parallel speculative decoding method in the buffer of this application; Figure 3 This is a flowchart illustrating the second embodiment of the inferential decoding method of this application; Figure 4 This is a schematic diagram illustrating the extended reasoning of this application; Figure 5 This is a schematic diagram illustrating the construction of the multipath cache in this application; Figure 6 This is a flowchart illustrating the third embodiment of the inferential decoding method of this application; Figure 7 This is a schematic diagram illustrating the path search for the draft of this application; Figure 8 This is a schematic diagram illustrating buffer update or rollback in two scenarios of this application; Figure 9 This is a schematic diagram of the module structure of the inferred decoding device according to an embodiment of this application; Figure 10 This is a schematic diagram of the device structure of the hardware operating environment involved in the speculative decoding method in this application embodiment.

[0018] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0020] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0021] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, model inference, and program execution functions, such as a computer, or an electronic device capable of performing the above functions. The following description uses a speculative decoding device as an example to illustrate this embodiment and the subsequent embodiments.

[0022] Based on this, embodiments of this application provide a speculative decoding method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the inferential decoding method of this application.

[0023] In this embodiment, the speculative decoding method includes: Step S10: Based on the first model, generate first candidate word elements corresponding to each current candidate word element, and store the first candidate word elements in the buffer.

[0024] It should be noted that the first model can be a sequence generation model with a small number of parameters, used to quickly generate candidate word sequences. For example, it could be a lightweight Transformer decoder or a dedicated inference model that has undergone knowledge distillation and compression. In inference decoding, the first model can be a decoding model. The current candidate word can be a word determined after the previous round of inference, the last word in the candidate sequence that serves as the starting point for this round of generation, or multiple words at the end of the candidate sequence. Optionally, the current candidate word can also be the last word in the initial input sequence.

[0025] The first candidate lexical unit can be generated by the first model after expanding and reasoning on each current candidate lexical unit, reflecting the first model's prediction of subsequent output units given a prefix. The buffer is used to store candidate lexical units and their relationships.

[0026] Specifically, as an optional implementation, the first model takes a single current candidate word as input, performs forward reasoning, outputs a probability distribution on the vocabulary, and samples or selects the word with the highest probability according to the probability distribution as the first candidate word corresponding to the current candidate word. The first model repeats the above process until a corresponding first candidate word is generated for each current candidate word. For example, if the current candidate word is "I", the first model takes "I" as input for reasoning, outputs the probability distribution of each word in the vocabulary, and selects the word "love" with the highest probability as the first candidate word.

[0027] In another optional implementation, the first model takes each current candidate word as input and performs extended reasoning in parallel to generate multiple first candidate words for each current candidate word. For example, the first model takes each word in the current candidate word set as input, performs forward reasoning, and selects the top K words with the highest probabilities from the output probability distribution as the K first candidate words corresponding to that current candidate word, where K is an integer greater than 1.

[0028] Optionally, after the first model generates the first candidate word, the first candidate word and its corresponding generation probability are stored together in the buffer.

[0029] Understandably, if the first model fails to generate a valid first candidate word, for example, if the probability distribution of the word list is all zero or the generation is interrupted, then the current round of inference will stop and an error message will be output.

[0030] Step S20: When the preset synchronization condition is detected, the second candidate word in the buffer is synchronized to the second model.

[0031] It should be noted that the preset synchronization condition can be a preset judgment rule that triggers data synchronization between the process where the first model is located and the process where the second model is located. It is used to control the timing and frequency at which the first model sends candidate words to the second model.

[0032] The second model can be a target generation model with a large number of parameters, used to perform parallel verification of candidate lexical units generated by the first model. For example, the second model can be a large language model with the same vocabulary as the first model. The second candidate lexical units are candidate lexical units in the buffer that have been filtered by path weights and synchronized to the second model for verification. The second candidate lexical units are stored in the first cache of the buffer, forming one or more candidate paths.

[0033] Specifically, when the preset synchronization conditions are detected, the first model pauses the generation operation and sends the second candidate lexical units in the buffer to the process where the second model is located.

[0034] Optionally, the preset synchronization conditions can be set based on the amount of data in the buffer or a time threshold. For example, a synchronization operation is triggered when the number of second candidate words stored in the buffer reaches a preset threshold; or a synchronization operation is triggered when the time interval since the last synchronization operation reaches a preset time threshold.

[0035] When synchronizing the second candidate word to the second model, the process containing the first model packages the second candidate word into a synchronization data packet and sends it to the process containing the second model. It then switches its own state to a waiting state until it receives the verification result returned by the second model.

[0036] In the specific implementation process, to avoid the communication overhead caused by excessively frequent synchronization, and to prevent excessively long synchronization intervals from leading to excessively long candidate paths in the buffer and causing a large amount of invalid calculations when verification fails, the following can be included before step S20: Get the current reasoning round; When the current inference round is an integer multiple of the preset communication period, it is determined that the buffer satisfies the preset synchronization condition; When the current inference round is not an integer multiple of the preset communication period, return to the step of generating a first candidate word corresponding to each current candidate word based on the first model, and storing the first candidate word in the buffer.

[0037] It should be noted that the current inference round can be the number of generation iterations already performed by the first model. The preset communication period is a pre-defined positive integer representing the number of inference rounds between two adjacent synchronization operations, used to control the communication frequency between the first and second models. Optionally, the preset communication period can be dynamically adjusted according to the difference in inference speed between the first and second models to balance the waiting time between the two processes.

[0038] Specifically, as an optional implementation, after each generation iteration, the first model increments the number of generation iterations by one and updates the current inference round. The first model performs a modulo operation between the current inference round and a preset communication period to determine if the current inference round is an integer multiple of the preset communication period. If the modulo result of the current inference round versus the preset communication period is zero, it is determined that the preset synchronization condition is met, and a synchronization operation is performed, synchronizing the second candidate lexical unit in the buffer to the second model. If the modulo result is not zero, it is determined that the preset synchronization condition is not met, and the first model does not perform a synchronization operation but returns to the generation step to continue the next round of candidate lexical unit expansion inference.

[0039] In one example, with a preset communication cycle of 4, the first model triggers synchronization when the inference rounds reach multiples of 4 (e.g., rounds 4, 8, 12, etc.), while only performing generation operations without synchronization in rounds 1-3 and 5-7 (non-integer multiples). This allows the first model to continuously generate multiple candidate words between synchronizations, accumulating sufficient data before synchronizing to the second model all at once. This reduces the number of communication cycles while ensuring the second model has enough candidate words for parallel verification.

[0040] Step S30: Based on the second model, determine the second verification result according to the second candidate lexical and the first verification result. The first verification result is generated by verifying the candidate lexical in the previous round of synchronization through the second model.

[0041] It should be noted that the first verification result can be the verification information generated by the second model after performing parallel verification on the candidate words synchronized from the buffer in the previous round. The first verification result includes the pass or fail status of each candidate word in the candidate path in the previous round, as well as the correct candidate word output by the second model at the fail position. By using the first verification result as reference information for this round of verification, the continuity and consistency of path selection between multiple rounds of verification can be guaranteed.

[0042] The second verification result can be the verification information generated by the second model after verifying the second candidate word in this round of synchronization. The second verification result includes the verification status of each candidate word in this round of verification, the location information of the verification failure, and the correct candidate word output by the second model at the failure position.

[0043] Specifically, as an optional implementation, after receiving the second candidate lexical unit synchronized by the first model, the second model obtains a first verification result. The first verification result records the position of the last candidate lexical unit that passed verification in the previous round of synchronization. The second model determines a second verification result based on the second candidate lexical unit and the first verification result.

[0044] Step S40: Based on the first model, update the buffer according to the second verification result, and determine the current candidate word for the next round based on the updated buffer, so as to perform the next round of inference.

[0045] Specifically, as an optional implementation, after receiving the second verification result from the second model, the first model parses the second verification result and determines whether all the second candidate words synchronized in this round have passed verification. When the second verification result indicates that all the second candidate words synchronized in this round have passed verification, the candidate words in the buffer remain unchanged, and the first model directly determines the current candidate word as the input for the next round of inference and continues to perform extended inference at the end of the current candidate path.

[0046] In one example, reference Figure 2 , Figure 2This is a schematic diagram of the parallel speculation decoding method based on the buffer in this application. The diagram sets up two processes, speculation and verification, to isolate the reasoning process of the draft model (first model) and the target model (second model) and realize parallel reasoning. The area with the dashed border in the middle is the interval for communication between the two processes, used to synchronize buffer data (draft sequence) and verification results. In the speculation process, the draft path generated by the draft model reasoning will be stored in the buffer and sent to the verification process during synchronous communication. Specifically, it includes the following steps: (1) Draft model reasoning: The draft model reasoning generates draft tokens. (2) Update the buffer: The draft tokens are stored in the buffer, where the sequence of all tokens in the buffer is a draft path. (3) Communication judgment: Used to control the frequency of process communication and balance the length of the draft path. If the draft path is too long, too many tokens will be rolled back when the verification fails, which will lead to a waste of computing resources; while if the path is too short, it will affect the efficiency of parallel verification. A "communication ratio" hyperparameter (an integer greater than 0) is set to control the communication frequency. When the number of draft inference rounds reaches a multiple of the communication ratio, the process enters the communication interval; otherwise, inference continues. The communication ratio setting needs to balance the difference in inference speed between the two models, aiming to allow both processes to enter the synchronous communication zone simultaneously as much as possible, reducing the waiting time between processes. Sending and receiving verification results: The sending buffer is given to the verification process, which blocks and waits for the verification result of the previous draft path. In parallel inference, draft inference will be one round "faster" than verification inference, meaning that it receives the verification result from the previous buffer. Updating / rolling back the buffer: The buffer is updated / rolled back based on the verification result. While waiting for the verification result, the verification process first obtains the verification basis through the target model inference, then performs the verification draft operation to verify the candidate words. After verification, the verification result is fed back to the inference process by sending the verification result. After receiving the verification result, the inference process performs the update / rollback operation on the buffer according to whether the verification passes or fails, thus completing a complete round of parallel inference decoding iteration.

[0047] In this embodiment, the generated candidate lexical units are stored in a buffer by the first model, and the candidate lexical units in the buffer are synchronized to the second model when the preset synchronization conditions are met. This decouples the generation process of the first model from the verification process of the second model in time. The first model can generate candidate lexical units based on the updated buffer without waiting for the second model to complete the verification before starting. This minimizes the idle waiting time between processes, enables the parallel operation of the two models, significantly improves the utilization of GPU resources, and effectively reduces the overall inference latency of speculative decoding.

[0048] Reference Figure 3 , Figure 3This is a flowchart illustrating the second embodiment of the speculative decoding method of this application. Based on the first embodiment described above, a second embodiment of the speculative decoding method of this application is proposed.

[0049] In the second embodiment, the buffer includes a first buffer and a second buffer, and step S10 includes: Step S101: Based on the first model, perform extended reasoning on each of the current candidate words to generate multiple first candidate words, and obtain the generation probability of each first candidate word.

[0050] It should be noted that the first cache is a storage area in the buffer used to store high-quality candidate words (i.e., second candidate words) after path weight filtering. The candidate words stored in the first cache serve as input for subsequent synchronization to the second model and the next round of extended inference. The second cache is a storage area in the buffer used to store all first candidate words and their relationships. The complete candidate word metadata and parent-child relationships stored in the second cache are used for path backtracking and alternative path search when validation fails.

[0051] The first cache has a smaller storage capacity than the second cache. The first cache only stores a portion of the high-quality candidate words from all candidate words to avoid the input size growing exponentially with the number of inference rounds, which would affect inference efficiency.

[0052] The first candidate word can be a candidate word generated by the first model after expanding and inferring from each current candidate word. Each current candidate word corresponds to multiple first candidate words. The generation probability can be the probability value of the word in the output probability distribution when the first model generates each first candidate word, and is used to measure the confidence of the first model in the prediction.

[0053] Specifically, as an optional implementation, the first model performs independent forward inference with each current candidate word as input. For each current candidate word, the first model inputs it into the model network. After calculation by the embedding layer, attention layer, and feedforward network layer, it outputs the original logical value in the vocabulary dimension. Then, the Softmax function is used to convert the logical value into a probability distribution, which covers the predicted probability of each word in the vocabulary. The first model selects the top K words with the highest probabilities from this probability distribution as the K first candidate words corresponding to the current candidate word, and records the probability value corresponding to each first candidate word in the probability distribution as the generation probability.

[0054] In one example, reference Figure 4 , Figure 4This is a schematic diagram of the extended inference process in this application. Each token sequence connected by an arrow represents a draft path. The inference decoding process starts with the draft model. The prompt text "Tell me story about Card" in the prompt box is sent to the draft model via the input arrow. The draft model performs extended inference along the direction indicated by the inference arrow, writing the candidate words generated layer by layer into the multi-path cache marked by the dotted box on the right. Inside the multi-path cache, candidate words are organized and expanded level by level in the form of hierarchical nodes. The first-level nodes include "There, Card, That, It". The first-level nodes are further expanded to obtain the second-level nodes "is, are, was, no", and then the second-level nodes are further expanded to obtain the third-level nodes "ordinary, longer, simple, match". The nodes at each level are connected according to their corresponding relationships at the time of generation, thus forming multiple branch candidate paths from the first-level "There, Card, That, It", through the second-level "is, are, was, no", and then to the third-level "ordinary, longer, simple, match", for subsequent synchronization to the target model for parallel verification and path filtering.

[0055] Step S102: Determine a second candidate word from the first candidate word based on the generation probability and the path weight of the current candidate word.

[0056] It should be noted that path weights are used to measure the quality or confidence of the candidate path containing a candidate word, reflecting the cumulative confidence of the entire path from the starting position to the current candidate word. The path weight of the current candidate word can be the global weight corresponding to the candidate word when it was determined after the previous round of inference; a higher value indicates a better candidate path. For the initial round, the path weight of the current candidate word can be set to an initial value. The second candidate word is selected from the first candidate words and is used to store in the first cache and await synchronization to the second model.

[0057] In the specific implementation process, the target path weight of the first candidate word can be determined by combining the generation probability of each first candidate word with the path weight of the corresponding parent candidate word. Based on this, second candidate words are selected from all first candidate words, ensuring that the selected second candidate words consider not only the generation probability of the current step but also the cumulative confidence of the entire path. At this time, step S102 may include: The target path weight of the first candidate word is determined based on the generation probability and the path weight of the current candidate word. The second candidate word is determined from the first candidate word based on the target path weight.

[0058] Specifically, as an optional implementation, after the first model obtains the generation probability of each first candidate word, for each first candidate word, the generation probability is multiplied by the path weight of the corresponding current candidate word to obtain the target path weight of the first candidate word. The target path weight reflects both the prediction confidence (i.e., generation probability) of the first model for the candidate word in the current step and inherits the historical information of the path in which the candidate word is located (i.e., the path weight of the parent candidate word), which can more comprehensively evaluate the quality of the path in which the candidate word is located.

[0059] After calculating the target path weight of each first candidate word, the first model determines the second candidate word from all first candidate words based on the target path weight. Specifically, the first model sorts all first candidate words in descending order of target path weight and selects a specified number of first candidate words with the highest target path weight as second candidate words. The specified number is a preset positive integer, and its specific value can be set according to the size of the first cache and the actual application scenario.

[0060] In another alternative implementation, the first model can identify all first candidate lexical units whose target path weight exceeds a preset threshold as second candidate lexical units.

[0061] In one example, reference Figure 5 , Figure 5 This diagram illustrates the multi-path caching construction of this application. In the diagram, TopK is 3. The draft model needs to infer from the three cached tokens selected in the previous round, generating three tokens with the highest probabilities, and using each probability as the weight for this round of inference. These weights are then multiplied by the parent weights to calculate the corresponding global weights. These tokens are stored in the L2 cache, while the three tokens with the highest global weights are also stored in the L1 cache. Only the L1 cache is used as input for subsequent inference; the L2 cache is only used during the inference process, combining the verification results to backtrack the correct draft path. This is because if all tokens in L2 were used for the next round of inference, the model's input would increase exponentially with the number of rounds, affecting the efficiency of draft inference.

[0062] The diagram shows three candidate expansion trees rooted at There, Card, and That. Each root node carries the inference weight for the current round. The candidate words Is, Are, Has, Was, catalog, Game, Evening, and Old, derived from each root node, are labeled with their respective generation probabilities. The generation probability of each candidate word is multiplied by the inference weight of its root node for the current round, resulting in the global weights of each candidate word, indicated by red numbers in the middle row. The candidate words are then sorted according to their global weights. Under the constraint of TopK = 3 in the previous L1 cache (marked in the upper left), each candidate word and its global weight are written to the L2 cache (represented by the green rounded box at the bottom), so that the L2 cache contains multiple words and their corresponding global weights. Further, from the words stored in the L2 cache, the top few words are selected according to their global weights from highest to lowest and written to the L1 cache (marked by the red dashed box on the left), so that the L1 cache contains "Is 0.25, Are 0.20, Was". The three items, 0.18, serve as the current candidate words for the next round of extended reasoning. Among them, the L1 cache is the first cache, with a capacity of the TopK value (3 in the figure), which stores only the three candidate words with the highest global weight; the L2 cache is the second cache, which stores all nine first candidate words and their parent-child relationships with the current candidate words.

[0063] Step S103: Store the second candidate word element in the first cache, and store the first candidate word element, as well as the association between each first candidate word element and the current candidate word element, in the second cache.

[0064] Specifically, as an optional implementation, after determining the second candidate word, the first model writes the second candidate word into the first cache. During writing, the first model stores the second candidate word in descending order of target path weight, so that the candidate word with the highest path weight can be quickly retrieved as inference input in subsequent steps. Simultaneously, the first model writes all first candidate words (including the second candidate words selected for the first cache and other first candidate words not selected for the first cache) into the second cache, and records the parent-child relationship between each first candidate word and the current candidate word that generated it in the second cache.

[0065] In this embodiment, by further setting a first cache and a second cache in the buffer, the first cache is used to store high-quality candidate words after being filtered by path weight, which ensures synchronization efficiency. The second cache is used to store all first candidate words and their parent-child relationships, which provides a complete association data foundation for subsequent path backtracking and cache updates, ensuring that alternative paths can be quickly located when verification fails, and effectively reducing repeated reasoning.

[0066] Reference Figure 6 , Figure 6 This is a flowchart illustrating the third embodiment of the speculative decoding method of this application. Based on the first embodiment described above, a third embodiment of the speculative decoding method of this application is proposed.

[0067] In the third embodiment, step S30 includes: Step S301: Based on the first verification result, determine the target candidate path from the candidate paths corresponding to the second candidate word.

[0068] Specifically, as an optional implementation, after receiving the second candidate lexical unit synchronized by the first model, the second model obtains a first verification result. The first verification result records the position of the last candidate lexical unit that passed verification in the previous round of synchronization, as well as the candidate lexical unit identifier corresponding to that position. Based on this position information, the second model selects the candidate path following the corresponding position from among the multiple candidate paths corresponding to the second candidate lexical unit synchronized in this round as the target candidate path.

[0069] If the first verification result indicates that all candidate words synchronized in the previous round have passed verification (i.e., there are no candidate words that have failed verification), then the second model determines the end position of the candidate path in the previous round as the last position that passed verification. Based on this, the second model selects the candidate path with the highest global weight from the second candidate words synchronized in this round, and determines this candidate path as the target candidate path.

[0070] If the first verification result indicates that there are candidate words that failed verification in the candidate words synchronized in the previous round, the second model extracts the position information of the failed verification from the first verification result and determines the last candidate word that passed verification before that position. Then, the second model searches for candidate paths with the prefix of the candidate word that passed verification in the second candidate words synchronized in this round. If multiple candidate paths with the prefix of the candidate word that passed verification are found, the one with the highest global weight is selected as the target candidate path.

[0071] Step S302: Based on the second model, reason about the third candidate word in the target candidate path and output the target probability of the third candidate word.

[0072] It should be noted that the third candidate lexical is any candidate lexical in the target candidate path.

[0073] Specifically, as an optional implementation, the second model takes the prefix sequence corresponding to each third candidate word in the target candidate path as input and performs one forward inference. After performing embedding, encoding, and decoding calculations on the input sequence, the second model outputs the original logical value in the vocabulary dimension, and then converts the logical value into a probability distribution using the Softmax function. The probability distribution reflects the second model's prediction of the next word given the prefix. The second model extracts the probability value corresponding to each third candidate word from the probability distribution as the target probability of each third candidate word. Since the target candidate path contains multiple third candidate words, the second model only needs to perform one forward inference to simultaneously obtain the target probabilities of all third candidate words, achieving parallel verification.

[0074] In one example, if the target candidate path is "Once upon a time", the second model takes the prefix sequence corresponding to each third candidate word in the path (such as "Once", "Once upon", "Once upon a", etc.) as input, performs forward inference, outputs the word probability distribution at each position, and extracts the probability values ​​of each third candidate word such as "upon", "a", "time" in the corresponding probability distribution as their respective target probabilities.

[0075] Step S303: Determine the second verification result based on the target probability and the generation probability of each of the third candidate words.

[0076] In the specific implementation process, the target probability output by the second model can be compared one by one with the generation probability of the first model to determine whether each candidate word has passed the verification. This ensures that the statistical distribution of the candidate words generated by the first model is consistent with the output distribution of the second model, thereby ensuring the lossless output quality of the inference decoding. At this time, step S303 may include: When the target probability is greater than or equal to the generation probability, the third candidate word is set to pass the verification. When the target probability is less than the generation probability, the third candidate word is set as failing the verification, and the position information of the third candidate word that failed the verification in the target candidate path and the candidate word output by the second model at the position information are recorded. The second verification result is determined based on the verification result of the third candidate word.

[0077] It should be noted that the target probability is the probability value predicted by the second model for the third candidate word given the prefix, while the generation probability is the probability value when the first model generates the third candidate word. By comparing the target probability and the generation probability, it can be determined whether the candidate word generated by the first model is consistent with the predicted distribution of the second model, thereby deciding whether to accept the candidate word.

[0078] Location information is the sequence number or index of the third candidate word in the target candidate path, used to uniquely identify the position of the verification object in the path.

[0079] Specifically, as an optional implementation, after obtaining the target probability of each third candidate word in the target candidate path, the second model sequentially compares the target probability of each third candidate word with its corresponding generation probability. For each third candidate word in the target candidate path, if the target probability is greater than or equal to the generation probability, the third candidate word is set to pass the verification, indicating that the candidate word generated by the first model at that position is consistent with the prediction of the second model and can be accepted as a valid output. If the target probability is less than the generation probability, the third candidate word is set to fail the verification, indicating that the candidate word generated by the first model at that position is inconsistent with the prediction of the second model, the candidate word is rejected, and all candidate words after that position are also rejected.

[0080] When the third candidate word fails the verification, the second model records the position information of the third candidate word in the target candidate path. At the same time, it selects the word with the highest probability from the probability distribution output by the second model at that position as the correct candidate word and records the correct candidate word in association with the position information.

[0081] After the second model completes the comparison of all third candidate words, it generates a second verification result based on the verification results of each third candidate word. The second verification result includes verification pass information and verification fail information. The verification pass information includes all verified third candidate words and their position information in the target candidate path, as well as the position of the last verified third candidate word. The verification fail information includes the position information of the first verified third candidate word, as well as the correct candidate word output by the second model at that position.

[0082] In one example, reference Figure 7 , Figure 7This is a schematic diagram of the path search in the draft of this application. The dashed box above shows a weighted word graph composed of nodes such as "There, is, a, Card, are, cards, That, was, no" and their associated probability weights. The nodes are connected by edges in the order of generation to form multiple candidate paths. After calculating the global weight of each candidate path according to the probability weights marked on each edge, the path with the highest global weight is marked with a thick line and used as the target candidate path for validation in this round. Subsequently, each node in this target candidate path is validated in parallel, word by word. In another set of word node paths given in the dashed box below, the following words are considered: ordinary, souvenir, piece, of, longest, simple, the, bright. The nodes are connected according to their probability weights to form candidate paths. Nodes on the verification sequence are verified one by one. When a node is verified, it is marked as a failed verification and the failure is indicated in text, thus terminating the continued verification on that branch in this round. Then, based on the verification results and the upper-level cache, candidate paths are searched again in the word graph according to the location information of the failed verification and the candidate nodes retained in the upper-level cache. Other branches composed of ordinary, souvenir, piece, of, longest, simple, the, and bright, or the corresponding nodes in the upper-level cache, are used as continuation paths for the next round of extended reasoning and verification.

[0083] In this embodiment, the target candidate path is determined from the candidate paths corresponding to the second candidate word based on the first verification result, so that the second model can inherit the verification state of the previous round during verification. It can continue to advance at the position where the verification is passed and switch paths at the position where the verification fails, thus ensuring the continuity and rationality of path selection between multiple rounds of verification.

[0084] Furthermore, when a word in a certain path fails to be verified, in order to replace it with another candidate word at the same layer and continue reasoning, avoiding the computational overhead of regeneration, reducing the number of backtracking steps and reasoning delay, in the third embodiment, step S40 further includes: Step S401: When the second verification result indicates that there is a fourth candidate word that has failed verification in the target candidate path, obtain the position information of the fourth candidate word in the target candidate path.

[0085] It should be noted that the fourth candidate word is the first candidate word in the target candidate path that fails the verification, and its verification status is "failed". All the candidate words before it have passed the verification.

[0086] Specifically, as an optional implementation, after receiving the second verification result from the second model, the first model parses the second verification result and extracts the verification failure information. The second verification result contains an overall flag indicating whether the verification passed or failed. When this flag indicates that there is a candidate word that failed in this round of verification, the first model extracts the position information field from the verification failure information in the second verification result. The position information is the sequence number of the fourth candidate word in the target candidate path. Since the second model compares each third candidate word sequentially during the verification process, it stops subsequent verifications when it encounters the first third candidate word that failed verification. Therefore, the second verification result records at most one verification failure position information, namely the position information of the fourth candidate word.

[0087] Step S402: Based on the location information, search for the fifth candidate word in the buffer.

[0088] It should be noted that the fifth candidate lexical is any other candidate lexical that shares the same parent as the fourth candidate lexical and can be used to replace the fourth candidate lexical. By searching for the fifth candidate lexical in the buffer, alternative paths can be attempted from other branches at the same level when verification fails, thus avoiding a direct rollback and regeneration.

[0089] Specifically, as an optional implementation, after obtaining the position information of the fourth candidate word in the target candidate path, the first model determines the parent candidate word of the fourth candidate word in the target candidate path based on this position information. The parent candidate word refers to the candidate word in the target candidate path that precedes the fourth candidate word and has a direct parent-child relationship with it; that is, the word preceding the fourth candidate word in the path. If the fourth candidate word is the first word in the target candidate path, its parent candidate word is the last word that passed verification in the previous round or the last word of the initial input sequence. The first model uses this parent candidate word as the query key to search in the second cache for all extended candidate words that have a direct relationship with this parent candidate word. The second cache records the parent-child relationships between each candidate word; each relationship indicates that a parent candidate word has been extended to generate a child candidate word. The first model searches in the second cache for all child candidate words associated with the parent candidate word and selects other candidate words besides the fourth candidate word.

[0090] The first model filters from the other candidate words found based on the target path weight of each candidate word, selecting the candidate word with the highest target path weight as the fifth candidate word. If there are multiple candidate words with the same parent as the fourth candidate word and a higher path weight than the fourth candidate word, then the one with the highest path weight is selected as the fifth candidate word.

[0091] Step S403: Replace the fourth candidate word in the target candidate path with the fifth candidate word to backtrack the candidate words in the buffer.

[0092] Understandably, after finding the fifth candidate word, the first model replaces the fourth candidate word in the target candidate path with the fifth candidate word.

[0093] Specifically, the first model obtains the position information of the fourth candidate word in the target candidate path, marks the fourth candidate word at that position as invalid and removes it from the first cache, and then inserts the fifth candidate word at that position, inheriting the positional relationship of the fourth candidate word in the path. After the replacement is completed, all candidate words in the target candidate path that are after the fourth candidate word are deleted, because these candidate words are generated based on the fourth candidate word, and after the fourth candidate word is replaced, these subsequent candidate words are no longer valid.

[0094] In one example, reference Figure 8 , Figure 8 This is a schematic diagram illustrating buffer update or rollback in two scenarios of this application. The target model receives candidate word sequence (such as "you", "like", "eat", "what", "I", "like", "apple", "you", "ne", etc.) from the draft model, performs parallel verification of these candidate words through one forward inference, outputs the probability distribution at each position, and determines whether each candidate word is accepted.

[0095] The target model at the top of the diagram outputs "I like apples,". If all "I like apples" passes the validation, it becomes the buffer for the previous round. If the next correct token in the valid sequence is ",", then the updated buffer is ", what about you".

[0096] The target model outputs "I like strawberries". All instances of "I like" pass validation, forming the buffer for the previous round. Tokens that fail validation (such as "strawberries") are rejected, and the correct token output by the target model at that position (such as "apple") is used for correction. Therefore, "I like strawberries" in the previous buffer is backtracked, and the next correct token in the validated sequence is "apple".

[0097] The diagram uses the contrast between "I like apples" and "I like strawberries" to illustrate the two scenarios of successful and unsuccessful validation. When the candidate word "strawberry" does not match the probability distribution of the target model at that position, the candidate word is marked as unsuccessful, and its subsequent word "and pear" is also rejected. The target model then outputs the correct word "apple" with the highest probability at the failed position as a replacement, thus ensuring that the final output is consistent with the result independently generated by the target model.

[0098] In this embodiment, when the second verification result indicates that there is a fourth candidate word that failed verification in the target candidate path, the position information of the word is obtained and a fifth candidate word is searched in the buffer. The fourth candidate word in the target candidate path is replaced with the fifth candidate word to achieve buffer rollback. When a word in a certain path fails verification, it can be replaced with another candidate word in the same layer to continue reasoning, avoiding the computational overhead of regeneration, significantly reducing the number of rollbacks and reasoning delays, and effectively improving the overall acceleration effect of speculative decoding.

[0099] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the deductive decoding method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0100] This application also provides a speculative decoding device, please refer to... Figure 9 The speculative decoding device includes: The generation module 10 is used to generate first candidate word elements corresponding to each current candidate word element based on the first model, and store the first candidate word elements in a buffer. Synchronization module 20 is used to synchronize the second candidate word in the buffer to the second model when a preset synchronization condition is detected. The verification module 30 is used to determine a second verification result based on the second model, the second candidate word, and the first verification result, wherein the first verification result is generated by verifying the candidate word of the previous round of synchronization through the second model. The update module 40 is used to update the buffer based on the first model according to the second verification result, and to determine the current candidate word in the next round based on the updated buffer, so as to perform the next round of inference.

[0101] The speculative decoding apparatus provided in this application, employing the speculative decoding method in the above embodiments, can solve the technical problem that the alternating execution of the inference processes of the draft model and the target model in a serial manner limits the acceleration effect of speculative decoding. Compared with the prior art, the beneficial effects of the speculative decoding apparatus provided in this application are the same as those of the speculative decoding method provided in the above embodiments, and other technical features in the speculative decoding apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0102] This application provides a speculative decoding device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the speculative decoding method in Embodiment 1 above.

[0103] The following is for reference. Figure 10 The diagram illustrates a structural schematic suitable for implementing the speculative decoding device of the embodiments of this application. The speculative decoding device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The speculative decoding device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0104] like Figure 10 As shown, the speculative decoding device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the speculative decoding device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the speculation decoding device to communicate wirelessly or wiredly with other devices to exchange data. Although speculation decoding devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0105] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0106] The speculative decoding device provided in this application, employing the speculative decoding method in the above embodiments, can solve the technical problem that the alternating execution of the inference processes of the draft model and the target model in a serial manner limits the acceleration effect of speculative decoding. Compared with the prior art, the beneficial effects of the speculative decoding device provided in this application are the same as those of the speculative decoding method provided in the above embodiments, and other technical features in this speculative decoding device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0107] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0109] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the speculative decoding method in the above embodiments.

[0110] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0111] The aforementioned computer-readable storage medium may be included in the speculative decoding device; or it may exist independently and not assembled into the speculative decoding device.

[0112] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the speculative decoding device, cause the speculative decoding device to perform the speculative decoding method described above.

[0113] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0116] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described speculative decoding method. This solves the technical problem that the sequential execution of the inference processes of the draft model and the target model limits the acceleration effect of speculative decoding. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the speculative decoding method provided in the above embodiments, and will not be repeated here.

[0117] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

[0118] It should be noted that the data collection, tag management, rule setting, and push decision-making processes involved in this application are designed to work with other technical features to solve technical problems. They do not involve or support any illegal activities. Any data processing that may violate laws and regulations (such as unauthorized collection of privacy data, generation of discriminatory tags, setting unfair rules, or pushing illegal information) is not within the scope of protection of this application's technical solution. Of course, the user data in this application will be encrypted, anonymized, or de-identified before storage to ensure user data security.

Claims

1. A speculative decoding method, characterized in that, The speculative decoding method includes: Based on the first model, generate first candidate word elements corresponding to each current candidate word element, and store the first candidate word elements in the buffer; When a preset synchronization condition is detected, the second candidate word in the buffer is synchronized to the second model; Based on the second model, a second verification result is determined according to the second candidate word and the first verification result. The first verification result is generated by verifying the candidate word of the previous round of synchronization through the second model. Based on the first model, the buffer is updated according to the second verification result, and the current candidate word for the next round is determined based on the updated buffer to perform the next round of inference.

2. The speculative decoding method as described in claim 1, characterized in that, The buffer includes a first cache and a second cache; The step of generating first candidate lexical units corresponding to each current candidate lexical unit based on the first model, and storing the first candidate lexical units in a buffer, includes: Based on the first model, extended reasoning is performed on each of the current candidate word elements to generate multiple first candidate word elements, and the generation probability of each first candidate word element is obtained; Based on the generation probability and the path weight of the current candidate word, a second candidate word is determined from the first candidate word. The second candidate word is stored in the first cache, and the first candidate word, as well as the association between each first candidate word and the current candidate word, is stored in the second cache.

3. The speculative decoding method as described in claim 2, characterized in that, The step of determining the second candidate word from the first candidate word based on the generation probability and the path weight of the current candidate word includes: The target path weight of the first candidate word is determined based on the generation probability and the path weight of the current candidate word. The second candidate word is determined from the first candidate word based on the target path weight.

4. The speculative decoding method as described in claim 1, characterized in that, Before the step of synchronizing the second candidate lexical unit in the buffer to the second model when a preset synchronization condition is detected, the method further includes: Get the current reasoning round; When the current inference round is an integer multiple of the preset communication period, it is determined that the buffer satisfies the preset synchronization condition; When the current inference round is not an integer multiple of the preset communication period, return to the step of generating a first candidate word corresponding to each current candidate word based on the first model, and storing the first candidate word in the buffer.

5. The speculative decoding method as described in claim 1, characterized in that, The step of determining the second verification result based on the second model, according to the second candidate lexical unit and the first verification result, includes: Based on the first verification result, the target candidate path is determined from the candidate paths corresponding to the second candidate word element; Based on the second model, reason about the third candidate word in the target candidate path and output the target probability of the third candidate word; The second verification result is determined based on the target probability and the generation probability of each of the third candidate words.

6. The speculative decoding method as described in claim 5, characterized in that, The step of determining the second verification result based on the target probability and the generation probability of each of the third candidate words includes: When the target probability is greater than or equal to the generation probability, the third candidate word is set to pass the verification. When the target probability is less than the generation probability, the third candidate word is set as failing the verification, and the position information of the third candidate word that failed the verification in the target candidate path and the candidate word output by the second model at the position information are recorded. The second verification result is determined based on the verification result of the third candidate word.

7. The speculative decoding method as described in claim 1, characterized in that, The step of updating the buffer based on the first model and the second verification result includes: When the second verification result indicates that there is a fourth candidate word element that failed the verification in the target candidate path, the position information of the fourth candidate word element in the target candidate path is obtained; Based on the location information, the fifth candidate word element is searched in the buffer. The fourth candidate word in the target candidate path is replaced with the fifth candidate word to backtrack the candidate words in the buffer.

8. A speculative decoding device, characterized in that, The device includes: The generation module is used to generate first candidate word elements corresponding to each current candidate word element based on the first model, and store the first candidate word elements in the buffer; A synchronization module is used to synchronize the second candidate word in the buffer to the second model when a preset synchronization condition is detected. The verification module is used to determine a second verification result based on the second model, the second candidate word, and the first verification result, wherein the first verification result is generated by verifying the candidate word of the previous round of synchronization through the second model. An update module is used to update the buffer based on the first model and the second verification result, and to determine the current candidate word for the next round based on the updated buffer, so as to perform the next round of inference.

9. A speculative decoding device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the speculative decoding method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the speculative decoding method as described in any one of claims 1 to 7.

Images