A speculative decoding method and system based on static and dynamic vocabulary cooperation

Abstract

The application discloses a kind of speculative decoding method and system based on static and dynamic vocabulary cooperation, comprising: initialization static vocabulary and dynamic vocabulary;Based on inference request, draft model generates draft on vocabulary;Utilize target model on its complete vocabulary to verify draft, output the probability distribution of each position of draft on the complete vocabulary of target model;Utilize probability distribution, from each draft position respectively select the first few words of highest probability to form candidate word set, remove the word contained in, obtain remaining candidate word set;Utilize remaining candidate word set to update and maintain data thereof;Determine the token that is verified by using probability distribution, and append it to output sequence;Repeat the above steps until sequence termination token is generated, obtain inference result.The application can effectively improve the quality of draft under the cutting paradigm of vocabulary, improve the inference acceleration performance of speculative decoding.

Description

Technical Field

[0001] This invention relates to the field of large language reasoning technology, and in particular to a speculative decoding method and system based on the collaboration of static and dynamic word lists. Background Technology

[0002] Speculative decoding significantly reduces the number of decoding steps for the target model without changing the output distribution by providing predictions through a draft model and verifying the target model in parallel. It is one of the mainstream technologies in the industry for accelerating LLM inference.

[0003] For example, Chinese patent document CN120012932A discloses a method for rapidly generating tokens for a large language model on the client side.

[0004] Multi-Token Prediction (MTP) can be used in the draft generation process of speculative decoding. It directly predicts multiple future tokens from their hidden states at the same time, rather than just predicting the next token. MTP modules typically share a backbone with the main model and have a very lightweight structure. For example, the EAGLE module consists of only a single-layer Transformer module, along with word embeddings and a language model head shared with the backbone. Because of its lightweight structure and ability to fully reuse existing semantic information from the main model, MTP achieves high-quality multi-step predictions and is therefore widely used.

[0005] However, the ever-expanding vocabulary size of large models poses challenges to the generation of drafts for speculative decoding. Llama-2 used a 32k vocabulary, Llama-3 expanded the vocabulary to 128k, and Qwen-2.5 further expanded the vocabulary to 152k... Larger vocabularies bring finer-grained language representation and better model performance, but also increase the computational overhead of the language model head (LM head) and probability normalization (softmax). The output processing of large vocabularies has become a computational bottleneck for small draft structures, especially popular single-layer Transformer draft models like EAGLE.

[0006] FR-Spec (Frequency-Ranked Speculative Sampling) proposes a method to accelerate the speculative decoding draft process by pruning the vocabulary of a draft model. FR-Spec pre-obtains a static vocabulary based on statistical analysis of a large-scale corpus. And restrict the draft stage to only The probability distribution of each word is calculated, thus reducing the computational scale from the size of the complete vocabulary. Reduced to the size of a high-frequency vocabulary ( This significantly reduced the computational cost of drafts. However, Once fixed, it cannot perceive the real-time context, which has the following drawbacks:

[0007] 1. Low hit rate in long-tail scenarios: Many technical terms in professional texts are not included. This causes the draft model to sample other words, resulting in a high rejection rate and a decrease in speedup ratio for the target model.

[0008] 2. Sensitive to distribution drift: Even if the vocabulary is specially tailored for various scenarios, the distribution of the same model will change in different requests. FR-Spec's static vocabulary pruning method cannot adapt to this, and its speedup effect is suboptimal.

[0009] If simply expand The output processing overhead of the draft model increases accordingly, while excessive scaling... Furthermore, the limited candidate space can lead to a decrease in draft acceptance rate, thereby weakening the overall acceleration effect. Therefore, how to enhance the coverage of high-output-probability words while maintaining a lightweight draft vocabulary has become a problem that needs to be solved to further improve speculative decoding efficiency. Summary of the Invention

[0010] This invention provides a speculative decoding method and system based on the collaboration of static and dynamic vocabularies, which solves the problem in the prior art that static vocabularies cannot adapt to different reasoning scenarios, resulting in low quality of draft generation during the reasoning process.

[0011] A speculative decoding method based on the collaboration of static and dynamic vocabularies includes:

[0012] S1. Initialize the static vocabulary. With dynamic vocabulary ;

[0013] S2. Based on the user's reasoning request, use the draft model in the vocabulary. Generate a draft;

[0014] S3. Utilize the target model in its complete vocabulary. The generated draft is validated, and the positions of the draft in the target model's complete vocabulary are output. The probability distribution on;

[0015] S4. Using the probability distribution obtained in step S3, select the position with the highest probability from each draft position. A candidate word set is composed of 10 words. Remove those already included in the static vocabulary. The words are used to obtain the remaining candidate word set. ;

[0016] S5. Utilize the remaining candidate word set For dynamic vocabulary Update its maintenance data;

[0017] S6. Use the probability distribution obtained in step S3 to determine the tokens that have passed verification and append them to the output sequence.

[0018] S7. Repeat steps S2 to S6 until a sequence termination token is generated. The output complete sequence is the reasoning result of this request.

[0019] Preferably, in step S1, the static vocabulary... word count for and Integers between [a certain range].

[0020] Preferably, in step S1, the dynamic vocabulary... Its initial size is 0, and it has a maximum word count. 128 and Integers between [a certain range].

[0021] Furthermore, in step S1, the static vocabulary... The method for obtaining it is as follows:

[0022] Choose a large dataset Random sampling of it The sampled data was segmented using the target model's word segmenter, and word frequencies were counted. The data was then sorted by frequency, and the most frequent word was selected. A static vocabulary list is composed of 10 words. .

[0023] Preferably, in step S4, The value is a positive integer less than or equal to 1024.

[0024] The specific process of step S5 is as follows:

[0025] Traverse the set of remaining candidate words For each word in the list, if the word already exists in the dynamic vocabulary... If the word does not exist in the dynamic thesaurus, then only its maintenance data will be updated; if the word does not exist in the dynamic thesaurus... And word count Word count limit not reached Then add the word And its maintenance data will be updated synchronously; if the word does not exist in the dynamic thesaurus. And word count Word limit reached Then, according to the preset replacement strategy, After eliminating at least one existing word, add that word. And update its maintenance data simultaneously.

[0026] The LRU mechanism is used when updating and maintaining data and removing existing words, specifically:

[0027] A linked list queue is used to maintain the dynamic vocabulary. The order of most recently used elements in the queue: the first element represents the least recently used element, and the last element represents the most recently used element.

[0028] When updating and maintaining data, if the current word already exists in the dynamic thesaurus before this access... If it is in the middle, move it to the end of the queue; otherwise, insert it into the end of the queue.

[0029] When discarding an existing word, remove the corresponding word at the head of the queue and add it to the dynamic vocabulary list. Removed from the middle.

[0030] A speculative decoding system based on the collaboration of static and dynamic vocabulary is characterized by comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the above-mentioned speculative decoding method.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. This invention introduces a dynamic vocabulary. The static vocabulary problem has been solved. The inability to adapt to different reasoning scenarios has improved the quality and acceptance rate of draft generation.

[0033] 2. This invention uses the probability distribution information generated during the target model verification process as the basis for updating the dynamic vocabulary, ensuring the effectiveness and relevance of the auxiliary vocabulary content without introducing too much additional overhead.

[0034] 3. This invention uses the LRU mechanism to manage the dynamic vocabulary space, ensuring the validity of the vocabulary content. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of a speculative decoding method based on the collaboration of static and dynamic word lists, according to an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the various functional modules in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0040] like Figure 1 As shown, a speculative decoding method based on the collaboration of static and dynamic vocabularies includes the following steps:

[0041] S1. Initialize the static vocabulary. With dynamic vocabulary .in The word count is , Has a word limit The initial size is 0.

[0042] In this embodiment, The selected number is 16384. The value was selected as 1024.

[0043] Static vocabulary The method for obtaining it is: select a large dataset. 1,000,000 data points were randomly sampled, and the target model's word segmenter was used to segment the data and count the word frequencies. After sorting by word frequency, the 16,384 words with the highest frequency were selected to form a sequence. .

[0044] In this embodiment, a large dataset The SlimPajama-627B dataset was selected.

[0045] S2. Based on the user's reasoning request, use the draft model in the vocabulary. A draft is generated. In this embodiment, the draft model uses the EAGLE-2 module of the Llama-3-8B-Instruct model.

[0046] S3. Utilize the target model in its complete vocabulary. The generated draft is validated, and the probability distribution of each position in the draft on the complete vocabulary of the target model is output. In this embodiment, the target model is the Llama-3-8B-Instruct model.

[0047] S4. Using the probability distribution obtained in step S3, select the position with the highest probability from each draft position. A candidate word set is composed of 10 words. Remove those already included in the static vocabulary. The words are used to obtain the remaining candidate word set. In this embodiment, It is 16.

[0048] S5. Utilize the remaining candidate word set For dynamic vocabulary And update its maintenance data.

[0049] Traverse the set of remaining candidate words For each word in the list, if the word already exists in the dynamic vocabulary... If the word does not exist in the dynamic thesaurus, then only its maintenance data will be updated; if the word does not exist in the dynamic thesaurus... And word count Word count limit not reached Then add the word And its maintenance data will be updated synchronously; if the word does not exist in the dynamic thesaurus. And word count Word limit reached Then, according to the preset replacement strategy, After eliminating at least one existing word, add that word. And update its maintenance data simultaneously.

[0050] In this embodiment, the LRU mechanism is used when updating maintenance data and removing existing words, specifically as follows:

[0051] A linked list queue is used to maintain the dynamic vocabulary. The order of most recently used elements in the queue: the first element represents the least recently used element, and the last element represents the most recently used element.

[0052] When updating and maintaining data, if the current word already exists in the dynamic thesaurus before this access... If it is in the middle, move it to the end of the queue; otherwise, insert it into the end of the queue.

[0053] When discarding an existing word, remove the corresponding word at the head of the queue and add it to the dynamic vocabulary list. Removed from the middle.

[0054] S6. Use the probability distribution obtained in step S3 to determine the tokens that have passed verification and append them to the output sequence.

[0055] S7. Repeat steps S2 to S6 until a sequence termination token is generated. The output complete sequence is the reasoning result of this request.

[0056] like Figure 2As shown, the functional modules in this embodiment of the invention operate as follows: ① First, the dataset is randomly sampled and word frequencies are counted, and high-frequency words are selected to obtain a static word list. Dynamic vocabulary It is then initialized to empty. ② After the user submits a request to the system, ③ the draft model is based on a static vocabulary. With dynamic vocabulary ④ Generate a draft. ⑤ The target model calculates the probability distribution of the draft. ⑥ The system adjusts the dynamic vocabulary based on the analysis results. The system then updates the sequence. ⑥ Subsequently, the system verifies the generated result. ⑦ Acceptable tokens are confirmed. ⑧ The system executes the entire process in a loop until the sequence generation is complete, and ⑨ finally outputs the response. This process, through the collaboration of prior pruning of the static vocabulary and adaptive updating of the dynamic vocabulary, improves draft quality while maintaining the low overhead of the draft vocabulary pruning paradigm, thereby reducing the number of "draft-verification" rounds and increasing inference speed.

[0057] To verify the effectiveness of this invention, experiments were conducted on a server equipped with an NVIDIA GeForce RTX 4090 graphics card using the Spec-Bench and HumanEval datasets. Spec-Bench is a widely used dataset in the field of speculative decoding. It extracts 80 data points from each subtask of multiple datasets, including WMT14 DE-EN (translation task), MT-Bench (multi-turn dialogue task), Natural Questions (retrieval and question answering task), GSM8K (mathematical task), and CNN / Daily Mail (summarizing task), to form a multi-task dataset. The HumanEval dataset focuses on code generation tasks, containing 164 data points, thus addressing the lack of a code generation task in the Spec-Bench dataset. Following the settings of the Spec-Bench and FR-Spec papers, the experiments set a maximum generation length of 1024 for each subtask in the Spec-Bench dataset and 512 for the HumanEval dataset. The search depth of the EAGLE module was fixed at 6, the final number of draft nodes selected was 60, and the sampling temperature was set to 0. The final average draft acceptance length of the Llama-3-8B-Instruct model under different decoding methods is shown in Table 1, and the number of tokens generated per second under different decoding methods is shown in Table 2. The average draft acceptance length reflects the quality of the draft; with the same number of drafts, a longer average acceptance length indicates higher draft quality. The number of tokens generated per second reflects the inference speed of the inference system; with other conditions being equal, a higher number of tokens generated per second indicates faster inference. The unit k used in the tables represents a base of 1024.

[0058] Experiments show that the embodiments of the present invention can effectively improve draft quality and increase the inference speed of the system. Table 1 shows that this embodiment, using a total draft vocabulary of 17k (16k static vocabulary and 1k dynamic vocabulary), surpasses the FR-Spec method using a 32k draft vocabulary in terms of average draft acceptance length. Table 2 shows that this embodiment achieves the highest value among the group of methods in terms of tokens generated per second. This is achieved by further reducing the vocabulary size to reduce draft overhead, while using a dynamic vocabulary to ensure draft quality.

[0059] Table 1. Average Draft Acceptance Length of Llama-3-8B-Instruct under Different Decoding Methods

[0060]

[0061] Table 2. Number of tokens generated per second under different decoding methods in Llama-3-8B-Instruct

[0062]

[0063] This invention introduces a dynamic vocabulary. The static vocabulary problem has been solved. This method addresses the issue of adaptability to different inference scenarios, improves the quality and acceptance rate of draft generation, utilizes the probability distribution information generated during the target model validation process as the basis for updating the dynamic vocabulary, and ensures the effectiveness and relevance of the auxiliary vocabulary content without introducing excessive additional overhead. The LRU mechanism is used to manage the dynamic vocabulary space, guaranteeing the effectiveness of the vocabulary content. This method has good versatility and can be widely applied to various scenarios for accelerating large language model inference based on speculative decoding.

[0064] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A speculative decoding method based on static and dynamic vocabulary cooperation, characterized in that, include: S1, initialize static lexicon with dynamic lexicon ; S2, based on the user's reasoning request, generate a draft using the draft model on the vocabulary above; S3. Utilize the target model in its complete vocabulary. The generated draft is validated, and the positions of the draft in the target model's complete vocabulary are output. The probability distribution on; S4. Using the probability distribution obtained in step S3, select the position with the highest probability from each draft position. A candidate word set is composed of 10 words. Remove those already included in the static vocabulary. The words are used to obtain the remaining candidate word set. ; S5. Utilizing the remaining candidate word set updating the dynamic word list and its maintenance data; S6. Use the probability distribution obtained in step S3 to determine the tokens that have passed verification and append them to the output sequence. S7. Repeat steps S2 to S6 until a sequence termination token is generated. The output complete sequence is the reasoning result of the corresponding request.

2. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 1, characterized in that, In step S1, the static vocabulary The number of words is between and an integer.

3. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 1, characterized in that, In step S1, the dynamic vocabulary Its initial size is 0, and it has a maximum word count. 128 and Integers between [a certain range].

4. The speculative decoding method based on the collaboration of static and dynamic vocabulary as described in claim 1, characterized in that, In step S1, the static lexicon The acquisition method is as follows: Choose a large dataset Random sampling of it The sampled data was segmented using the target model's word segmenter, and word frequencies were counted. The data was then sorted by frequency, and the most frequent word was selected. A static vocabulary list is composed of 10 words. .

5. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 1, characterized in that, In step S4, the value of the parameter is a positive integer less than or equal to 1024.

6. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 1, characterized in that, The specific process of step S5 is as follows: Traverse the set of remaining candidate words For each word in the list, if the word already exists in the dynamic vocabulary... If the word does not exist in the dynamic thesaurus, then only its maintenance data will be updated; if the word does not exist in the dynamic thesaurus... And word count Word limit not reached Then add the word And its maintenance data will be updated synchronously; if the word does not exist in the dynamic thesaurus. And word count Word limit reached Then, according to the preset replacement strategy, After eliminating at least one existing word, add that word. And update its maintenance data simultaneously.

7. The speculative decoding method based on the collaboration of static and dynamic vocabulary as described in claim 6, characterized in that, When updating maintenance data and deprecating existing words, an LRU replacement strategy is used, specifically: A linked list queue is used to maintain the dynamic vocabulary. The order of most recently used elements in the queue: the first element represents the least recently used element, and the last element represents the most recently used element. When updating and maintaining data, if the current word already exists in the dynamic thesaurus before this access... If it is in the middle, move it to the back of the line; Otherwise, insert it at the end of the queue; When a word is retired, the head of the queue is removed and the word is removed from the dynamic vocabulary in the middle.

8. A speculative decoding system based on static and dynamic vocabulary collaboration, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the speculative decoding method according to any one of claims 1-7.

Images