A method and device for speculative decoding of large models on board space based on pre-generated tokens
Fast guessing and parallel verification of input sequences is performed through pre-generated token dictionary, which solves the problem of slow inference speed of large models on satellites, and efficient text generation is achieved, suitable for satellite-borne environments with limited resources.
Patent Information
- Application Number
- CN202411017441.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-29
AI Technical Summary
When deploying large models on satellites, due to limited resources, the traditional serial text generation method is limited by IO bandwidth and the inference speed is affected. The existing methods need to retrain the decoder or additionally train the approximate model, which increases complexity and resource requirements .
The strategy of pre-generated tokens is adopted to pre-generate the successor tokens of each token through the text generation model, and a pre-generated token dictionary is constructed to quickly guess the input sequence, and the accuracy and coherence of the generated token sequences are ensured through a parallel verification mechanism.
It significantly improves the speed of token generation, reduces dependence on computing resources, adapts to resource limitations in the satellite-borne environment, optimizes the model inference process, and improves decoding efficiency.
Smart Images

Figure CN118569207B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for speculative decoding of a large satellite model based on pre-generated tokens. Background Art
[0002] In text generation technology, the use of large models has been widely studied and applied. The current mainstream method is to predict the next token based on the input and generated tokens. If a large model is deployed on a satellite, due to limited resources, this serial text generation method will have a great impact on the inference speed under the IO bandwidth limit.
[0003] Reference 1 (Cai T, Li Y, Geng Z, et al. Medusa: Simple llm inference acceleration framework with multiple decoding heads[J]. arXiv preprint arXiv:2401.10774, 2024.) proposes a new framework called medusa, which overcomes the serial limitation of the autoregressive decoding process by adding multiple decoding heads to predict multiple subsequent tags in parallel. This method requires retraining the added decoding heads for prediction, and requires special training strategies to maintain the quality of generation.
[0004] Reference 2 (Leviathan Y, Kalman M, Matias Y. Fast inference from transformers via speculative decoding[C] / / International Conference on Machine Learning. PMLR, 2023: 19274-19286.) proposed a new algorithm called "speculative decoding", which uses a more efficient approximate model to generate multiple predictions in parallel, and then the target model evaluates and accepts those predictions that conform to the original distribution, thereby improving the decoding speed while keeping the output unchanged. However, this method requires additional training of an efficient approximate model, and the acceleration effect also depends on the quality of the approximate model.
[0005] Reference 3 (Ankner Z, Parthasarathy R, Nrusimha A, et al. Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding[J]. arXiv preprintarXiv:2402.05109, 2024.) proposed a new decoding framework called Hydra, which improves the prediction accuracy of draft heads in traditional speculative decoding by introducing sequentially dependent draft heads (Hydra heads). However, since the Hydra head needs to consider more contextual information, more training data or longer training time is required to achieve ideal performance, and the sequential dependency of the Hydra head increases the complexity of model design. Summary of the invention
[0006] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a method and device for speculative decoding of a large satellite model based on pre-generated tokens.
[0007] Compared with traditional speculative decoding techniques, this paper proposes a novel on-board large model decoding strategy, which involves using a text generation model to pre-generate the successor tokens for each token to build a set of pre-generated token dictionary pairs. During the inference phase of the model, the method uses the pre-generated token dictionary pairs to quickly guess the subsequent tokens for each token in the input sequence. Subsequently, a parallel processing mechanism is used to batch verify these guessed tokens through the large model to ensure the accuracy and coherence of the generated token sequence. This process significantly improves the speed of token generation while maintaining a high quality standard for the generated text.
[0008] The objective of the present invention is achieved through the following technical solutions:
[0009] The first aspect of the present invention relates to a method for speculative decoding of a large model on board a satellite based on a pre-generated token, comprising the following steps:
[0010] Step 1: Before inference, input all tokens into the large text model, pre-generate the corresponding next token, and combine all tokens and their corresponding next tokens to form a pre-generated token dictionary pair;
[0011] Step 2: During inference, guess the M tokens after the input token sequence based on the pre-generated token dictionary obtained in step 1, and use the large model to verify and generate the M guessed tokens in parallel, and then match the verification token with the guessed token until there is no match.
[0012] The step 1 is implemented by the following sub-steps:
[0013] (1.1) Input all tokens into the text generation model separately and pre-generate the corresponding next token. This method uses a trained text generation model or an existing text generation model network. Input all N (N is the total number of tokens) tokens of the model into the model separately and only generate the next token.
[0014] (1.2) All tokens and their corresponding next tokens are combined to form a pre-generated token dictionary and stored in a file. According to (1.1), all N tokens are combined to form the next token obtained by the model, and the pre-generated token dictionary pair {token i: next token i} (i∈[0,N]) is formed.
[0015] Preferably, step (1.1) specifically includes: taking each token in the token sequence as input and processing it using a trained text generation model; the text generation model encodes the input token through its internal neural network structure, especially the self-attention mechanism; on this basis, the text generation model predicts and generates the next token of the token, and this process involves sampling of probability distribution to determine the most likely next token.
[0016] Preferably, step (1.2) specifically includes: through step (1.1), using the trained text generation model to predict the token that is most likely to follow each token, this process generates a mapping relationship, called a pre-generated token dictionary pair, each token i is mapped to its predicted next token i, forming a key-value pair {token i: next token i}, where i ranges from 0 to N, and N represents the total number of tokens in the sequence.
[0017] The step 2 is implemented by the following sub-steps:
[0018] (2.1) Guess the M tokens after the input token sequence based on the pre-generated token dictionary obtained in step 1. First, tokenize the input text. For the input token sequence, take the last token in the sequence as the key value, perform a key value search based on the pre-generated token dictionary obtained in step 1, take the key value found by the key value as the next token of the input token sequence, and concatenate it to the last bit of the input token sequence. Repeat this guessing step M times to obtain the next M tokens of the input token sequence.
[0019] (2.2) Use the large model to verify and generate the M guessed tokens in parallel, and then match the verification token with the guessed token until they do not match. Input the input token sequence and the next token concatenated after each guess, that is, a total of M+1 token sequences, into the text generation large model to generate the next token in parallel for verification. During verification, the guessed token and the verification token are verified in sequence, that is, to determine whether the guessed token is equal to the verification token. When the guessed token j and the verification token j (j=[1,2,…,j,…,M]) do not match, the acceptable token is (verification token 1, verification token 2,…, verification token j); when the guessed token and the verification token from 1 to M all match, the acceptable token is (verification token 1, verification token 2,…, verification token M+1), that is, all generated verification tokens are acceptable.
[0020] (2.3) Concatenate the acceptable token obtained in step (2.2) to the initial input token as the next input token, and then repeat steps (2.1) and (2.2) until the acceptable token appears to signal the end of generation, thus terminating the text generation process.
[0021] Preferably, step (2.1) specifically includes: using the token mapping dictionary pre-generated in step (1.2) to perform continuous key value lookup and prediction operations on the input sequence. Specifically, for the last token in the input token sequence, first search in the pre-generated token mapping dictionary to determine the predicted token i+1 corresponding to the token i; this predicted token i+1 is regarded as a guess of the input token sequence, and then spliced to the end of the input sequence to form a new sequence; it expands the input sequence to token 1, token 2, ..., token i, token i+1; in order to further predict subsequent tokens in the sequence, the above key value lookup and splicing steps are repeated M times, and in each step, the token at the new sequence endpoint is used as the input for the next search, and so on, until M predicted tokens are generated, and a guess sequence of M tokens after the input token sequence is obtained: token 1, token 2, ..., token i, token i+1, ..., token i+M.
[0022] Preferably, step (2.2) specifically includes: taking the M+1 token sequences generated in step (2.1) as input and inputting them into the text generation model, the text generation model processes these sequences in parallel and attempts to generate the next token of each sequence, the generated token is called a verification token, which is used to compare with the predicted token obtained by key value lookup; the verification process is intended to evaluate the accuracy of the predicted tokens, that is, to check whether they match the verification tokens generated by the model; for each sequence, starting from the first predicted token, verify whether it is equal to the corresponding verification token one by one; if the predicted token does not match the verification token at the jth position of the sequence (where j ranges from 1 to M, including M), then in the sequence, all verification tokens at positions 1 to j (i.e., verification token 1, verification token 2, ..., verification token j) are considered acceptable. This means that although the subsequent predictions of the sequence fail to be consistent with the results generated by the model, the first j parts of the sequence are accurate. If all predicted tokens match the corresponding verification tokens during the entire M prediction process, then all generated verification tokens in the entire sequence (i.e., verification token 1, verification token 2, ..., verification token M+1) are considered acceptable.
[0023] Step (2.3) specifically includes: splicing the verified acceptable token sequence in step (2.2) back to the original input sequence to form an extended sequence; the extended sequence will be used as the input token sequence in the next iteration to drive the text generation model to continue prediction; in each iteration, the text generation model will predict the next token based on the current input sequence using its internal language model knowledge; then, the text generation model will compare the generated predicted token with the expected token obtained by key-value lookup to verify the accuracy of the prediction. This verification process will be repeated until a specific termination condition is met, such as a predetermined number of tokens being generated, or a clear end signal is received, such as a specific end token or a threshold of semantic completeness is reached.
[0024] The second aspect of the present invention relates to a satellite-borne large model speculative decoding device based on pre-generated tokens, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a satellite-borne large model speculative decoding method based on pre-generated tokens of the present invention.
[0025] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a method of speculative decoding of a large satellite model based on pre-generated tokens according to the present invention.
[0026] A fourth aspect of the present invention relates to a spaceborne large model speculative decoding system based on pre-generated tokens, comprising a token pre-generation module and a token guessing and verification module;
[0027] The Token pre-generation module is used to generate the next token of all tokens in the current model offline, thus forming a token dictionary pair;
[0028] The token guessing and verification module is used to perform token queries based on the token dictionary when the model generates tokens through online inference, guess the generated token, and verify the correctness of the token guessed by the token pre-generation module online. When the verification is successful, the guessed token is accepted, thereby accelerating the generation of token decoding for large models.
[0029] Aiming at the current text generation method of large models, the present invention proposes a space-borne large model speculative decoding method based on pre-generated tokens. All tokens are generated by the large model to form a pre-generated token dictionary pair. During model reasoning, the token sequence after the input token is guessed according to the pre-generated token dictionary pair, and the large model is used for parallel verification to receive the accurately predicted token, thereby accelerating the reasoning speed.
[0030] The advantages of the present invention are: good adaptability to the resource limitations of the onboard environment and effective relief of the IO bandwidth limitation; through the strategy of pre-generating tokens, the demand for real-time token generation is reduced, thereby reducing the dependence on computing resources; in addition, through the parallel verification mechanism, the reasoning process of the model is optimized and the decoding efficiency is improved, which has important practical significance and application value for onboard application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a flow chart of the method of the present invention.
[0032] Figure 2 It is a schematic diagram of the next token pre-generation module of the present invention.
[0033] Figure 3 It is a schematic diagram of the token guessing and verification module of the present invention.
[0034] Figure 4 It is a system structure diagram of the present invention.
[0035] Figure 5 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] Example 1
[0038] Reference Figure 1 This embodiment relates to a method for speculative decoding of a large model on board a satellite based on a pre-generated token, comprising the following steps:
[0039] Step 1: Before inference, input all tokens into the large text model, pre-generate the corresponding next token, and combine all tokens and their corresponding next tokens to form a pre-generated token dictionary pair;
[0040] Step 2: During inference, guess the M tokens after the input token sequence based on the pre-generated token dictionary obtained in step 1, and use the large model to verify and generate the M guessed tokens in parallel, and then match the verification token with the guessed token until there is no match.
[0041] The step 1 is implemented by the following sub-steps: Figure 2 As shown:
[0042] (1.1) Input all tokens into the text generation model and pre-generate the corresponding next token. Initialize the text generation model, which can be a trained text generation model or an existing text generation model network. Figure 1 As shown in the figure, each independent token of all N (N-bit tokens in total) in the sequence is input to the model, and the model is asked to predict the next token after each token. The model encodes the input token through its internal neural network structure, especially the self-attention mechanism. On this basis, the model predicts and generates the next token of the token, which usually involves sampling the probability distribution to determine the most likely next token.
[0043] (1.2) Pair all tokens generated in step (1.1) with their corresponding successor tokens to form a set of pre-generated token dictionary pairs. This set is represented as {token i: next token i} (i∈[0,N]). This mapping relationship is called a pre-generated token dictionary pair. In this mapping, each token i is mapped to its predicted next token i, forming a key-value pair {token i: next token i}, where i ranges from 0 to N, and N represents the total number of tokens in the sequence.
[0044] The step 2 is implemented by the following sub-steps: Figure 3 As shown:
[0045] (2.1) Guess the M tokens after the input token sequence based on the pre-generated token dictionary obtained in step 1. Tokenize the input text sequence and extract the last token of the sequence as the key value. Use the pre-generated token dictionary obtained in step 1 to search and use the key value as the index to obtain the corresponding subsequent token. Concatenate this token to the end of the input sequence and repeat this process M times to generate the M tokens after the input sequence.
[0046] (2.2) Use the large model to verify the M guessed tokens in parallel, and then match the verification tokens with the guessed tokens until they do not match. Through M key-value lookups and concatenation operations in (2.1), M+1 token sequences are generated. These sequences consist of the original input sequence plus M consecutive predicted tokens. Next, these sequences are used as input into the text generation model. The model will process these sequences in parallel and try to generate the next token for each sequence. The generated tokens are called verification tokens and are used to compare with the predicted tokens obtained by key-value lookup. The verification process aims to evaluate the accuracy of the predicted tokens, that is, to check whether they match the verification tokens generated by the model. For each sequence, starting from the first predicted token, verify whether it is equal to the corresponding verification token one by one. If at the jth position of a sequence (where j ranges from 1 to M, inclusive), the predicted token does not match the verification token, then all verification tokens at positions 1 to j in the sequence (i.e., verification token 1, verification token 2, ..., verification token j) are considered acceptable. This means that, although subsequent predictions of the sequence failed to agree with the results generated by the model, the first j parts of the sequence were accurate. If all predicted tokens match the corresponding verification tokens throughout the M predictions, then all generated verification tokens in the entire sequence (i.e., verification token 1, verification token 2, ..., verification token M+1) are considered acceptable.
[0047] (2.3) The acceptable token obtained in (2.2) is concatenated to the initial input token as the next input token, and then steps (2.1) and (2.2) are repeated until the acceptable token appears to generate an end signal, and the text generation process is terminated. The verified acceptable token sequence is concatenated back to the original input sequence to form an extended sequence. This extended sequence will be used as the input token sequence in the next iteration to drive the model to continue predicting. In each iteration, the model will predict the next token based on the current input sequence and use its internal language model knowledge. Subsequently, the model compares the generated predicted token with the expected token obtained by key-value lookup to verify the accuracy of the prediction. This verification process will be repeated until a specific termination condition is met, such as a predetermined number of tokens being generated, or a clear end signal is received, such as a specific end token or a threshold of semantic completeness is reached.
[0048] Example 2
[0049] This embodiment relates to a large-scale speculative decoding system based on pre-generated tokens. Figure 5 As shown, it includes a token pre-generation module and a token guessing and verification module;
[0050] The Token pre-generation module is used to generate the next token of all tokens in the current model offline, thus forming a token dictionary pair;
[0051] The token guessing and verification module is used to perform token queries based on the token dictionary when the model generates tokens through online inference, guess the generated token, and verify the correctness of the token guessed by the token pre-generation module online. When the verification is successful, the guessed token is accepted, thereby accelerating the generation of token decoding for large models.
[0052] This embodiment is used to implement a method for speculative decoding of a large satellite model based on a pre-generated token in Embodiment 1:
[0053] Step 1: Before inference, input all tokens into the large text model, pre-generate the corresponding next token, and combine all tokens and their corresponding next tokens to form a pre-generated token dictionary pair;
[0054] Step 2: During inference, guess the M tokens after the input token sequence based on the pre-generated token dictionary obtained in step 1, and use the large model to verify and generate the M guessed tokens in parallel, and then match the verification token with the guessed token until there is no match.
[0055] The step 1 is implemented by the following sub-steps: Figure 2 As shown:
[0056] (1.3) Input all tokens into the text generation model and pre-generate the corresponding next token. Initialize the text generation model, which can be a trained text generation model or an existing text generation model network. Figure 1 As shown in the figure, each independent token of all N (N-bit tokens in total) in the sequence is input to the model, and the model is asked to predict the next token after each token. The model encodes the input token through its internal neural network structure, especially the self-attention mechanism. On this basis, the model predicts and generates the next token of the token, which usually involves sampling the probability distribution to determine the most likely next token.
[0057] (1.4) Pair all tokens generated in step (1.1) with their corresponding successor tokens to form a set of pre-generated token dictionary pairs. This set is represented as {token i: next token i} (i∈[0,N]). This mapping relationship is called a pre-generated token dictionary pair. In this mapping, each token i is mapped to its predicted next token i, forming a key-value pair {token i: next token i}, where i ranges from 0 to N, and N represents the total number of tokens in the sequence.
[0058] The step 2 is implemented by the following sub-steps: Figure 3 As shown:
[0059] (2.4) Guess the M tokens after the input token sequence based on the pre-generated token dictionary obtained in step 1. Tokenize the input text sequence and extract the last token of the sequence as the key value. Use the pre-generated token dictionary obtained in step 1 to search and use the key value as the index to obtain the corresponding subsequent token. Concatenate this token to the end of the input sequence and repeat this process M times to generate the M tokens after the input sequence.
[0060] (2.5) Use the large model to verify the M guessed tokens in parallel, and then match the verification tokens with the guessed tokens until they do not match. Through M key-value lookups and concatenation operations in (2.1), M+1 token sequences are generated. These sequences consist of the original input sequence plus M consecutive predicted tokens. Next, these sequences are used as input into the text generation model. The model will process these sequences in parallel and try to generate the next token for each sequence. The generated tokens are called verification tokens and are used to compare with the predicted tokens obtained by key-value lookup. The verification process aims to evaluate the accuracy of the predicted tokens, that is, to check whether they match the verification tokens generated by the model. For each sequence, starting from the first predicted token, verify whether it is equal to the corresponding verification token one by one. If at the jth position of a sequence (where j ranges from 1 to M, inclusive), the predicted token does not match the verification token, then all verification tokens at positions 1 to j in the sequence (i.e., verification token 1, verification token 2, ..., verification token j) are considered acceptable. This means that, although subsequent predictions of the sequence failed to agree with the results generated by the model, the first j parts of the sequence were accurate. If all predicted tokens match the corresponding verification tokens throughout the M predictions, then all generated verification tokens in the entire sequence (i.e., verification token 1, verification token 2, ..., verification token M+1) are considered acceptable.
[0061] (2.6) The acceptable token obtained in (2.2) is concatenated to the initial input token as the next input token, and then steps (2.1) and (2.2) are repeated until the acceptable token appears to generate an end signal, and the text generation process is terminated. The verified acceptable token sequence is concatenated back to the original input sequence to form an extended sequence. This extended sequence will be used as the input token sequence in the next iteration to drive the model to continue predicting. In each iteration, the model will predict the next token based on the current input sequence and use its internal language model knowledge. Subsequently, the model compares the generated predicted token with the expected token obtained by key-value lookup to verify the accuracy of the prediction. This verification process will be repeated until a specific termination condition is met, such as a predetermined number of tokens being generated, or a clear end signal is received, such as a specific end token or a certain semantic completeness threshold is reached.
[0062] Example 3
[0063] This example relates to a large-scale speculative decoding system for satellite-borne models based on pre-generated tokens, which is used to implement a large-scale speculative decoding method for satellite-borne models based on pre-generated tokens in Experimental Example 1. Figure 5 As shown, it includes a token pre-generation module and a token guessing and verification module;
[0064] The Token pre-generation module is used to generate the next token of all tokens in the current model offline, thus forming a token dictionary pair;
[0065] The token guessing and verification module is used to perform token queries based on the token dictionary when the model generates tokens through online inference, guess the generated token, and verify the correctness of the token guessed by the token pre-generation module online. When the verification is successful, the guessed token is accepted, thereby accelerating the generation of token decoding for large models.
[0066] The text generation model uses a mainstream open source large model for experiments, and the results are shown in the following table:
[0067]
[0068] Example 4
[0069] The present embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for speculative decoding of a large satellite model based on a pre-generated token in Embodiment 1 is implemented.
[0070] Example 5
[0071] The present embodiment relates to a device for speculative decoding of large satellite models based on pre-generated tokens, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a method for speculative decoding of large satellite models based on pre-generated tokens of embodiment 1.
[0072] The present invention also provides Figure 4 The one shown corresponds to Figure 1 A schematic structural diagram of a large-scale speculative decoding device based on a pre-generated token. Figure 4 As mentioned above, at the hardware level, the onboard large model speculative decoding based on pre-generated tokens includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1The above-mentioned method for speculative decoding of large models on board space based on pre-generated tokens. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0073] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0074] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.
[0075] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0076] For the convenience of description, the above device is described as being divided into various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0077] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0078] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0079] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0081] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0082] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0083] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0084] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0085] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0086] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0087] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0088] So far, the technical solution of the present invention has been described in conjunction with an embodiment shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.
Claims
1. A method for speculative decoding of large models on board space based on pre-generated tokens, characterized in that: The following steps are involved: Step 1: Offline pre-generate the next token of all tokens to obtain a pre-generated token dictionary pair; the pre-generated token dictionary pair is implemented through the following sub-steps: (1.1) Input all tokens into the text generation model to pre-generate the next token; (1.2) Combine all tokens and the next token obtained in step (1.1) to form a pre-generated token dictionary pair {token i: next token i} (i∈[0,N]); Step 2: During inference, guess the token sequence after step 1 and use the text generation model to verify the guessed sequence; this is achieved through the following sub-steps: (2.1) Take the last token of the input token sequence as the key value, perform a key value search based on the pre-generated dictionary obtained in step (1.2), take the found key value as the guessed token of the input token sequence and concatenate it to the input token sequence, repeat this guessing step M times, and guess the M tokens after the input token sequence; (2.2) Through step (2.1), add M concatenations from the input token, and there are a total of M+1 token sequences. Input them into the text generation model and generate the next token verification in parallel to verify whether the guessed token is equal to the verification token. When there is no match at j (j=[1,2,…,j,…,M]), the acceptable token is (verification token 1, verification token 2,…, verification token j). When all of them match, the acceptable token is (verification token 1, verification token 2,…, verification token M+1), that is, all generated verification tokens are acceptable. (2.3) Concatenate the acceptable token obtained in step (2.2) into the input sequence as the next input token, and repeat steps (2.1) and (2.2) to generate acceptable tokens until the generation end signal appears.
2. A method for speculative decoding of a large model on board based on pre-generated tokens as claimed in claim 1, characterized in that: Step (1.1) specifically includes: taking each token in the token sequence as input and processing it using a trained text generation model; the text generation model encodes the input token through its internal neural network structure, especially the self-attention mechanism; on this basis, the text generation model predicts and generates the next token of the token. This process involves sampling the probability distribution to determine the most likely next token.
3. The method for speculative decoding of a large model on board based on pre-generated tokens as claimed in claim 1, characterized in that: Step (1.2) specifically includes: through step (1.1), the trained text generation model can be used to predict the token that is most likely to follow each token. This process generates a mapping relationship, called a pre-generated token dictionary pair, where each token i is mapped to its predicted next token i, forming a key-value pair {token i: next token i}, where i ranges from 0 to N, and N represents the total number of tokens in the sequence.
4. The method for speculative decoding of a large model on board based on pre-generated tokens as claimed in claim 1, characterized in that: Step (2.1) specifically includes: using the token mapping dictionary pre-generated in step (1.2), performing continuous key value lookup and prediction operations on the input sequence. Specifically, for the last token in the input token sequence, first search in the pre-generated token mapping dictionary to determine the predicted token i+1 corresponding to the token i; this predicted token tokeni+1 is regarded as a guess of the input token sequence, and then spliced to the end of the input sequence to form a new sequence; it expands the input sequence to token 1, token 2, ..., token i, tokeni+1; in order to further predict subsequent tokens in the sequence, the above key value lookup and splicing steps are repeated M times. In each step, the token at the new sequence endpoint is used as the input for the next search, and so on, until M predicted tokens are generated, and the guess sequence of M tokens after the input token sequence is obtained: token 1, token 2, ..., token i, token i+1, ..., token i+M.
5. The method for speculative decoding of a large model on board based on pre-generated tokens as claimed in claim 1, characterized in that: Step (2.2) specifically includes: taking the M+1 token sequences generated in step (2.1) as input and inputting them into the text generation model, the text generation model processes these sequences in parallel and tries to generate the next token of each sequence. The generated token is called a verification token and is used to compare with the predicted token obtained by key-value lookup; the verification process aims to evaluate the accuracy of the predicted tokens, that is, to check whether they match the verification tokens generated by the model; for each sequence, starting from the first predicted token, verify one by one whether it is equal to the corresponding verification token; if at the jth position of the sequence, j ranges from 1 to M, the predicted token does not match the verification token, then in the sequence, all verification tokens at positions 1 to j are considered acceptable. If all predicted tokens match the corresponding verification tokens during the entire M prediction process, then all generated verification tokens in the entire sequence are considered acceptable.
6. A method for speculative decoding of a large model on board based on pre-generated tokens as claimed in claim 1, characterized in that: Step (2.3) specifically includes: splicing the verified acceptable token sequence in step (2.2) back to the original input sequence to form an extended sequence; the extended sequence will be used as the input token sequence in the next iteration to drive the text generation model to continue prediction; in each iteration, the text generation model will predict the next token based on the current input sequence using its internal language model knowledge; then, the text generation model will compare the generated predicted token with the expected token obtained by key-value lookup to verify the accuracy of the prediction; this verification process will be repeated until a specific termination condition is met.
7. A large-scale speculative decoding device based on pre-generated tokens, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement a method for speculative decoding of a large satellite model based on a pre-generated token as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a method for speculative decoding of a large satellite model based on a pre-generated token as described in any one of claims 1-6 is implemented.
9. A large-scale speculative decoding system based on pre-generated tokens, characterized in that: Includes a token pre-generation module and a token guessing and verification module; The token pre-generation module is used to generate the next token of all tokens of the current model offline, thereby forming a token dictionary pair; the pre-generated token dictionary pair is implemented through the following sub-steps: (1.1) Input all tokens into the text generation model to pre-generate the next token; (1.2) Combine all tokens and the next token obtained in step (1.1) to form a pre-generated token dictionary pair {token i: next token i} (i∈[0,N]); The token guessing and verification module is used to perform token queries based on the token dictionary when the model generates tokens through online inference, guess the generated tokens, and verify the correctness of the tokens guessed by the token pre-generation module online. When the verification is successful, the guessed token is accepted, thereby realizing the accelerated generation of token decoding for large models. This is achieved through the following sub-steps: (2.1) Take the last token of the input token sequence as the key value, perform a key value search based on the pre-generated dictionary obtained in step (1.2), take the found key value as the guessed token of the input token sequence and concatenate it to the input token sequence, repeat this guessing step M times, and guess the M tokens after the input token sequence; (2.2) Through step (2.1), add M concatenations from the input token, and there are a total of M+1 token sequences. Input them into the text generation model and generate the next token verification in parallel to verify whether the guessed token is equal to the verification token. When there is no match at j (j=[1,2,…,j,…,M]), the acceptable token is (verification token 1, verification token 2,…, verification token j). When all of them match, the acceptable token is (verification token 1, verification token 2,…, verification token M+1), that is, all generated verification tokens are acceptable. (2.3) Concatenate the acceptable token obtained in step (2.2) into the input sequence as the next input token, and repeat steps (2.1) and (2.2) to generate acceptable tokens until the generation end signal appears.
Citation Information
Patent Citations
Model training method and device, code generation method and device, storage medium and equipment
CN117828360A
Dynamic guess decoding method and device for large language model, equipment and medium
CN118095209A
Network blasting attack detection method and system based on langchain large model
CN118138343A
Model decoding method and device, electronic equipment, storage medium and program product
CN118297060A
Large language model acceleration processing method and device based on intermediate layer decoding
CN118396044A
