Prospective decoding method and system based on static and dynamic word list collaboration

Abstract

The invention discloses a speculation decoding method and system based on static and dynamic vocabulary collaboration. The speculation decoding method comprises the following steps: initializing a static vocabulary and a dynamic vocabulary; generating a draft on the word list based on the reasoning request and the draft model; verifying the draft on the complete vocabulary by using the target model, and outputting probability distribution of each position of the draft on the complete vocabulary of the target model; by means of probability distribution, first several words with the highest probability are selected from each draft position to form a candidate word set, included words are removed, and a remaining candidate word set is obtained; updating the remaining candidate word set and the maintenance data thereof by using the remaining candidate word set; determining a token passing verification by utilizing probability distribution, and adding the token to an output sequence; and repeating the steps until a sequence termination token is generated, and obtaining a reasoning result. According to the method, the draft quality under the word list clipping normal form can be effectively improved, and the reasoning acceleration performance of speculation decoding is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language reasoning, in particular to a speculative decoding method and system based on static and dynamic vocabulary cooperation. BACKGROUND

[0002] Speculative decoding provides prediction through a draft model, and target model parallel verification, significantly reduces the decoding steps of the target model without changing the output distribution, and is one of the mainstream technologies in the industry to accelerate LLM reasoning.

[0003] A method for quickly generating tokens for an end-side large language model is disclosed in Chinese patent document CN120012932A.

[0004] MTP (Multi-Token Prediction) can be used in the draft generation process of speculative decoding, which directly predicts multiple tokens in the future on the same hidden state at the same time, rather than just predicting the next token. The MTP module usually shares the main trunk with the main model and has a very lightweight structure, such as the EAGLE module, which is composed of only a single layer of Transformer module and shares word embedding and language model head with the main trunk. MTP can fully reuse the existing semantic information of the main model due to its lightweight structure, and achieve high-quality multi-step prediction, so it is widely used.

[0005] However, the continuous expansion of the vocabulary size of large models poses a challenge to the draft generation of speculative decoding. Llama-2 uses a 32k vocabulary, Llama-3 expands the vocabulary to 128k, and Qwen-2.5 further expands the vocabulary to 152k……A larger vocabulary brings more fine-grained language expression and better model performance, but also increases the computational overhead of the language model head (LM head) and the probability normalization (softmax). The output processing of the large vocabulary becomes a computational bottleneck for small draft structures, especially for popular single-layer Transformer draft models such as EAGLE.

[0006] FR-Spec (Frequency-Ranked Speculative Sampling) proposes a method to speed up the speculative decoding draft process by pruning the vocabulary of the draft model. FR-Spec obtains a static vocabulary according to the statistics of large-scale corpus in advance , and limits the calculation of the probability distribution of each word in the draft stage to , thereby reducing the computational scale from the size of the complete vocabulary to the size of the high-frequency vocabulary , significantly reducing the computational overhead of the draft. However, Once fixed, it cannot perceive real-time context, with the following shortcomings: 1. Low long-tail scenario hit rate: a large number of professional terms in professional text are not in the internal, resulting in the draft model only sampling other words, causing high target model rejection rate and declining speed ratio.

[0007] 2. Distribution drift sensitive: even if the word table is specially tailored for multiple scenarios, the same model will have distribution changes in different requests, and the static word table tailoring method of FR-Spec cannot be self-adaptive, and the acceleration effect is suboptimal.

[0008] If the is simply expanded , the output processing overhead of the draft model will increase, and excessive reduction will cause the draft acceptance rate to decline due to limited candidate space, thereby weakening the overall acceleration effect. Therefore, how to maintain the lightweight of the draft word table while enhancing its coverage of high output probability words has become a problem to be solved to further improve the speculative decoding efficiency.

[0009] The present application provides a speculative decoding method and system based on static and dynamic word table cooperation, which solves the problem that the static word table in the prior art cannot adapt to different reasoning scenarios, resulting in low quality of draft generation in the reasoning process.

[0010] A speculative decoding method based on static and dynamic word table cooperation, comprising: S1, initializing a static word table and a dynamic word table ; S2, based on the reasoning request of the user, generating a draft on the word table using the draft model; S3, verifying the generated draft using the target model on its complete word table , outputting the probability distribution of each position of the draft on the target model complete word table ; S4, using the probability distribution obtained in step S3, selecting the top word from each draft position to form a candidate word set , removing the words already contained in the static word table to obtain the remaining candidate word set ; S5, updating the dynamic word table and its maintenance data using the remaining candidate word set ; S6, using the probability distribution obtained in step S3 to determine the token that passes the verification and appending it to the output sequence; S7, repeat steps S2 to S6 until a sequence termination token is generated, and the complete sequence output is the inference result of the current request.

[0011] Preferably, in step S1, the static vocabulary has a number of words between 128 and 1024.

[0012] Preferably, in step S1, the dynamic vocabulary has an initial size of 0 and an upper limit of a number of words between 128 and 1024.

[0013] Further, in step S1, the static vocabulary is obtained by: selecting a large data set , randomly sampling pieces of data from the data set, using a tokenizer of the target model to tokenize the sampled data and count word frequencies, sorting the words by frequency, and selecting the top words with the highest frequency to form the static vocabulary .

[0014] Preferably, in step S4, the value of is a positive integer less than or equal to 1024.

[0015] The specific process of step S5 is as follows: traverse each word in the remaining candidate word set , if the word already exists in the dynamic vocabulary , only update its maintenance data; if the word does not exist in the dynamic vocabulary and the number of words has not reached the upper limit of the number of words , add the word to and update its maintenance data synchronously; if the word does not exist in the dynamic vocabulary and the number of words has reached the upper limit of the number of words , according to the preset replacement strategy, eliminate at least one existing word in and add the word to and update its maintenance data synchronously.

[0016] When updating the maintenance data and eliminating existing words, the LRU mechanism is adopted, specifically: use a linked list type queue to maintain the order of recent use of elements in the dynamic vocabulary , the head element represents the least recently used element, and the tail element represents the most recently used element. ​​​When updating the maintenance data, if the current word has existed in the dynamic word table before the current visit If the current word has existed in the dynamic word table before the current visit If the current word has existed in the dynamic word table before the current visit If the current word has existed in the dynamic word table before the current visit

[0017] A speculative decoding system based on cooperation of static and dynamic word tables, characterized by comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the above speculative decoding method.

[0018] Compared with the prior art, the present application has the following beneficial effects: 1. The present application introduces a dynamic word table to solve the problem that the static word table cannot adapt to different reasoning scenarios, improving the quality and acceptance rate of draft generation.

[0019] 2. The present application uses the probability distribution information generated during the target model verification process as the basis for updating the dynamic word table, ensuring the effectiveness and relevance of the auxiliary word table content without introducing excessive additional overhead.

[0020] 3. The present application uses the LRU mechanism to manage the dynamic word table space, ensuring the effectiveness of the word table content. DETAILED DESCRIPTION

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A flowchart of a speculative decoding method based on cooperation of static and dynamic word tables according to an embodiment of the present application.

[0023] Figure 2 A schematic diagram of each functional module in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0025] It should be noted that the features in the following embodiments and implementation examples can be combined with each other without conflict.

[0026] As shown in the accompanying drawings, a speculative decoding method based on static and dynamic vocabulary cooperation comprises the following steps: Figure 1 S1, initializing a static vocabulary and a dynamic vocabulary . Among them the number of words is , , has a word number limit , and the initial size is 0.

[0027] In this embodiment, 16384 is selected, 1024 is selected.

[0028] The acquisition method of the static vocabulary is to select a large data set , randomly sample 1,000,000 data from it, use the tokenizer of the target model to tokenize the data and count the word frequency, and take the 16,384 words with the highest frequency after sorting by word frequency to form the static vocabulary .

[0029] In this embodiment, the large data set is selected as the SlimPajama-627B data set.

[0030] S2, based on the user's reasoning request, use the draft model to generate a draft on the vocabulary . In this embodiment, the draft model uses the EAGLE-2 module of the Llama-3-8B-Instruct model.

[0031] S3, use the target model to verify the generated draft on its complete vocabulary , output the probability distribution of each position of the draft on the complete vocabulary of the target model. In this embodiment, the target model uses the Llama-3-8B-Instruct model.

[0032] S4, use the probability distribution obtained in step S3 to select the top word from each draft position to form a candidate word set , remove the words already included in the static vocabulary , and obtain the remaining candidate word set . In this embodiment, 16.

[0033] S5, use the remaining candidate word set For dynamic vocabulary And update its maintenance data.

[0034] Traverse the set of remaining candidate words For each word in the list, if the word already exists in the dynamic vocabulary... If the word does not exist in the dynamic thesaurus, then only its maintenance data will be updated; if the word does not exist in the dynamic thesaurus... And word count Word count limit not reached Then add the word And its maintenance data will be updated synchronously; if the word does not exist in the dynamic thesaurus. And word count Word limit reached Then, according to the preset replacement strategy, After eliminating at least one existing word, add that word. And update its maintenance data simultaneously.

[0035] In this embodiment, the LRU mechanism is used when updating maintenance data and removing existing words, specifically as follows: A linked list queue is used to maintain the dynamic vocabulary. The order of most recently used elements in the queue: the first element represents the least recently used element, and the last element represents the most recently used element. When updating and maintaining data, if the current word already exists in the dynamic thesaurus before this access... If it is in the middle, move it to the end of the queue; otherwise, insert it into the end of the queue. When discarding an existing word, remove the corresponding word at the head of the queue and add it to the dynamic vocabulary list. Removed from the middle.

[0036] S6. Use the probability distribution obtained in step S3 to determine the tokens that have passed verification and append them to the output sequence.

[0037] S7. Repeat steps S2 to S6 until a sequence termination token is generated. The output complete sequence is the reasoning result of this request.

[0038] like Figure 2 As shown, the functional modules in this embodiment of the invention operate as follows: ① First, the dataset is randomly sampled and word frequencies are counted, and high-frequency words are selected to obtain a static word list. Dynamic vocabulary It is then initialized to empty. ② After the user submits a request to the system, ③ the draft model is based on a static vocabulary. With dynamic vocabulary ④ Generate a draft. ⑤ The target model calculates the probability distribution of the draft. ⑥ The system adjusts the dynamic vocabulary based on the analysis results. The updating is performed. 6) Then, the system verifies the generated result. 7) The token that can be accepted is confirmed. 8) The system performs the entire process in a loop until the sequence generation is completed, and 9) finally outputs the response. The flow cooperates the prior pruning of the static vocabulary and the adaptive updating of the dynamic vocabulary, improves the draft quality while maintaining the low overhead of the draft vocabulary pruning paradigm, thereby reducing the “draft-verification” round and improving the reasoning speed.

[0039] To verify the effect of the present application, the embodiment of the present application uses Spec-Bench and HumanEval two datasets for experiments on a server equipped with an NVIDIA GeForce RTX 4090 graphics card. Spec-Bench is a widely used dataset in the field of speculative decoding, which extracts 80 data for each subtask from multiple datasets such as WMT14 DE-EN (translation task), MT-Bench (multi-turn dialogue task), Natural Questions (retrieval and question answering task), GSM8K (mathematics task), CNN / Daily Mail (summary task), etc. to form a multi-task dataset; the HumanEval dataset focuses on code generation tasks and contains 164 data, which makes up for the short board of the Spec-Bench dataset without code tasks. According to the settings of Spec-Bench and FR-Spec papers, the experiment sets the maximum generation length for each subtask in the Spec-Bench dataset to 1024, sets the maximum generation length for the HumanEval dataset to 512, fixes the search depth of the EAGLE module to 6, finally selects the number of draft nodes to 60, and sets the sampling temperature to 0. The average accepted length of the draft of the Llama-3-8B-Instruct model in the embodiment under different decoding methods is shown in Table 1, and the number of tokens generated per second under different decoding methods is shown in Table 2. Among them, the average accepted length of the draft reflects the quality of the draft, and when the number of drafts is the same, the longer the average accepted length, the higher the quality of the draft; the number of tokens generated per second reflects the reasoning speed of the reasoning system, and when other conditions are the same, the larger the number of tokens generated per second, the faster the reasoning. The unit k used in the table represents the base of 1024.

[0040] Experiments show that the embodiment of the present application can effectively improve the draft quality and improve the reasoning speed of the system. Table 1 shows that the embodiment of the present application uses a total of 17k size of the total draft vocabulary (static vocabulary 16k, dynamic vocabulary 1k), which exceeds the FR-Spec method using a 32k draft vocabulary in the average accepted length of the draft. Table 2 shows that the embodiment of the present application reaches the highest value of the same group method in the number of tokens generated per second, which is achieved by continuously reducing the size of the vocabulary to reduce the draft overhead, and using the dynamic vocabulary to ensure the quality of the draft.

[0041] Table 1. Average Draft Acceptance Length of Llama-3-8B-Instruct under Different Decoding Methods

[0042] Table 2. Number of tokens generated per second under different decoding methods in Llama-3-8B-Instruct

[0043] This invention introduces a dynamic vocabulary. The static vocabulary problem has been solved. This method addresses the issue of adaptability to different inference scenarios, improves the quality and acceptance rate of draft generation, utilizes the probability distribution information generated during the target model validation process as the basis for updating the dynamic vocabulary, and ensures the effectiveness and relevance of the auxiliary vocabulary content without introducing excessive additional overhead. The LRU mechanism is used to manage the dynamic vocabulary space, guaranteeing the effectiveness of the vocabulary content. This method has good versatility and can be widely applied to various scenarios for accelerating large language model inference based on speculative decoding.

[0044] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A speculative decoding method based on static and dynamic vocabulary cooperation, characterized in that, Comprising: S1, initialize static lexicon with dynamic lexicon ; S2, based on the user's reasoning request, utilize the draft model to generate a draft in the vocabulary on the generated draft; S3, utilize the target model to generate a probability distribution over its complete vocabulary S4, validate the generated draft by comparing the probability distribution of each position in the draft against the probability distribution of each position in the target model's complete vocabulary S4, validate the generated draft by comparing the probability distribution of each position in the draft against the probability distribution of each position in the target model's complete vocabulary S4, using the probability distribution obtained in step S3, selecting from each draft position the top word with the highest probability to form a candidate word set ; removing words that are already contained in the static word list to obtain a remaining candidate word set ; and ; S5. Utilizing the remaining candidate word set updating the dynamic word list and its maintenance data; S6, determining the token of verification pass by using the probability distribution obtained in step S3, and appending it to the output sequence; S7, repeating steps S2 to S6 until the sequence termination token is generated, and the complete sequence output is the inference result corresponding to the request.

2. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 1, characterized in that, In step S1, the static vocabulary The number of words is between and an integer.

3. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 1, characterized in that, In step S1, the dynamic vocabulary has an initial size of 0, with an upper limit of the number of words being an integer between 128 and 128.

4. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 1, characterized in that, In step S1, the static lexicon The acquisition method is as follows: Select a large dataset , randomly sample it , use the tokenizer of the target model to tokenize the sampled data and count the word frequency, sort by word frequency and take the top word groups to form a static word table .

5. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 1, characterized in that, In step S4, the value of the parameter is a positive integer less than or equal to 1024.

6. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 1, characterized in that, The specific process of step S5 is: Traverse the set of remaining candidate words For each word in the list, if the word already exists in the dynamic vocabulary... If the word does not exist in the dynamic thesaurus, then only its maintenance data will be updated; if the word does not exist in the dynamic thesaurus... And word count Word count limit not reached Then add the word And its maintenance data will be updated synchronously; if the word does not exist in the dynamic thesaurus. And word count Word limit reached Then, according to the preset replacement strategy, After eliminating at least one existing word, add that word. And update its maintenance data simultaneously.

7. The speculative decoding method based on static and dynamic vocabulary cooperation according to claim 6, characterized in that, When updating the maintenance data and eliminating the existing words, the LRU replacement strategy is adopted, specifically: Maintaining a dynamic vocabulary using a linked list type queue the order of most recent use of the elements, with the head element representing the least recently used element and the tail element representing the most recently used element; When updating the maintenance data, if the current word existed in the dynamic word list before the current visit If the current word is not in the queue, it is added to the head of the queue. Otherwise, it is inserted at the tail; When a word is retired, the head of the queue is removed and the word is removed from the dynamic vocabulary in the middle.

8. A speculative decoding system based on static and dynamic vocabulary collaboration, characterized in that, The computer device comprises a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the speculative decoding method in any one of claims 1-7.

Images