Speculation decoding method and device for large language model and medium

Abstract

The embodiment of the invention discloses a speculation decoding method and device for a large language model and a medium, and relates to the technical field of large language models, and the method comprises the steps: obtaining the current task information of a target large language model; the method comprises the steps of inputting original text prompt information into a target large language model, determining initial token information, constructing a plurality of patch data of a specified patch level according to the initial token information and the original text prompt information, generating a plurality of candidate patch sequences in parallel according to each patch data and a draft model, mapping each candidate patch sequence, determining initial draft information, and outputting the initial draft information. The preliminary draft information comprises a token sequence corresponding to each candidate patch sequence; and performing parallel verification on the token sequence corresponding to each candidate patch sequence through the target large language model, screening the token sequences corresponding to the candidate patch sequences, determining the next round of input of the draft model, and performing speculative decoding.

Description

Technical Field

[0001] The present specification relates to the technical field of large language models, and in particular to a method, device and medium for speculative decoding of large language models. Background Art

[0002] With the rapid development and continuous iteration of Large Language Models (LLMs), the emergence ability of models has been continuously enhanced, which enables models to complete complex tasks without explicit programming guidance. However, as the number of model parameters increases exponentially, the reasoning time also increases accordingly. To alleviate this problem, speculative decoding technology has emerged. Speculative decoding technology is a method to improve the efficiency of reasoning of large-scale language models. In the speculative decoding process, a smaller "draft" model of the same source is first used to quickly generate multiple possible subsequent words (tokens). This draft model needs to share the same tokenizer and similar training data set as the target large language model, but the number of parameters is much smaller than the target language model. These generated candidate words are then passed to the full target language model for parallel verification.

[0003] Since a longer token sequence may reject more tokens, the overall reasoning time may be higher than the time of directly reasoning the target language model. Therefore, the length of the token sequence generated by the draft model in each draft model in the traditional speculative decoding process is required to be shorter. However, the shorter speculation length is not applicable to all situations, and the target language model can only decide whether to accept based on local information (the current few tokens generated), which may lead to suboptimal global decisions and a lack of full understanding of the overall context. Therefore, in terms of long text generation, when it comes to tasks that require capturing long-distance dependencies, the model in the traditional speculative decoding process may not fully consider the impact of subsequent words in the sequence on the previously generated parts, which may result in incomplete or illogical sentence structure, thus affecting the overall coherence and readability of the text. Summary of the invention

[0004] One or more embodiments of the present specification provide a speculative decoding method, device and medium for a large language model, which are used to solve the following technical problems: In the generation of long texts, when it comes to tasks that require capturing long-distance dependencies, the model in the traditional speculative decoding process may not be able to fully consider the impact of subsequent words in the sequence on the previously generated parts, which may cause the generated sentence structure to be incomplete or logically incoherent, thereby affecting the coherence and readability of the overall text.

[0005] One or more embodiments of this specification adopt the following technical solutions:

[0006] One or more embodiments of the present specification provide a speculative decoding method for a large language model, the method comprising: obtaining current task information of a target large language model, wherein the current task information comprises original text prompt information and task type information; inputting the original text prompt information into the target large language model to determine initial token information, and constructing multiple patch data of a specified patch level according to the initial token information and the original text prompt information, wherein the specified patch level is determined based on the task type information; generating multiple candidate patch sequences in parallel according to each of the patch data and a pre-trained draft model, so as to map each of the candidate patch sequences through a probability distribution to determine preliminary draft information, wherein the preliminary draft information comprises a token sequence corresponding to each of the candidate patch sequences; performing parallel verification on the token sequence corresponding to each of the candidate patch sequences through the target large language model, screening the token sequence corresponding to the candidate patch sequence, and determining the next round of input of the draft model for speculative decoding.

[0007] Furthermore, based on the initial token information and the original text prompt information, multiple patch data of the specified patch level are constructed, specifically including: splicing the initial token information and the original text prompt information to determine the spliced ​​text information; matching the corresponding specified patch level through the task type information; based on the specified patch level, converting the spliced ​​text information to determine multiple patch data.

[0008] Furthermore, based on the specified patch level, the concatenated text information is converted to determine a plurality of patch data, specifically including: based on the specified patch level, dividing the concatenated text information into corresponding dimensions to determine a plurality of token sets; determining a first token vector corresponding to each first token in each of the token sets; and determining a weighted average value corresponding to each of the token sets based on each of the first token vectors, so as to determine the patch data corresponding to each of the token sets based on the weighted average value.

[0009] Furthermore, before generating multiple candidate patch sequences in parallel according to each of the patch data and the pre-trained draft model, the method also includes: obtaining a source data set corresponding to the target large language model to process the source data set and determine a patch-level source data set; and training a predetermined initial draft model through the patch-level source data set to determine a trained draft model.

[0010] Furthermore, the target large language model is used to verify the token sequence corresponding to each of the candidate patch sequences in parallel, so as to screen the token sequence corresponding to the candidate patch sequence and determine the next round of input of the draft model for speculative decoding, which specifically includes: determining the mapping probability distribution data corresponding to each of the token sequences in the preliminary draft information; verifying the token sequence corresponding to each of the candidate patch sequences in parallel through the target large language model, and outputting the verification probability distribution data corresponding to each of the token sequences; and screening the token sequence corresponding to the candidate patch sequence according to the mapping probability distribution data and the mapping probability distribution data, and determining the next round of input of the draft model for speculative decoding.

[0011] Further, according to the mapping probability distribution data and the mapping probability distribution data, the token sequence corresponding to the candidate patch sequence is screened to determine the next round of input of the draft model for speculative decoding, specifically including: determining the mapping probability data of each token in the mapping probability distribution data corresponding to each of the token sequences, and determining the verification probability data of each token in the verification probability distribution data corresponding to each of the token sequences; comparing the mapping probability data of each token and the verification probability data of each token in sequence according to a preset order to determine the token probability comparison result corresponding to each of the token sequences, wherein the token probability comparison result includes all the same probabilities, partially the same probabilities and all the different probabilities; determining the patch verification results of the target large language model for multiple candidate patch sequences through the token probability comparison results corresponding to the token sequences, so as to screen the token sequences corresponding to the candidate patch sequences based on the patch verification results, and determine the next round of input of the draft model for speculative decoding, wherein the patch verification result includes any one of full sequence acceptance, partial acceptance and full sequence rejection.

[0012] Furthermore, the patch verification results of the target large language model for multiple candidate patch sequences are determined by comparing the token probabilities corresponding to the token sequences, specifically including: when there are token probability comparison results corresponding to a specified token sequence that the probabilities are all the same, determining that the patch verification results of the multiple candidate patch sequences are fully accepted; when there are no token probability comparison results corresponding to multiple specified token sequences that the probabilities are all the same, and there are token probability comparison results corresponding to token sequences that are partially the same in probabilities, determining that the patch verification results of the multiple candidate patch sequences are partially accepted; when the token probability comparison results corresponding to multiple token sequences are all probabilities that are all different, determining that the patch verification results of the multiple candidate patch sequences are fully rejected.

[0013] Further, based on the patch verification result, the token sequence corresponding to the candidate patch sequence is screened to determine the next round of input of the draft model for speculative decoding, specifically including: when the patch verification result is full sequence acceptance, determining the designated candidate patch sequence corresponding to the designated token sequence; deleting the remaining candidate patch sequences except the designated candidate patch sequence from multiple candidate patch sequences to perform subsequent reasoning based on the designated token sequence corresponding to the designated candidate patch sequence; when the patch verification result is partial acceptance, determining each of the token sequences according to the token probability comparison result corresponding to each of the token sequences The method comprises the following steps: determining the number of tokens with the same probability in a column; screening multiple token sequences according to the number of tokens to determine the preset token sequence with the largest number of tokens, so as to determine the probability consistency result corresponding to each preset token in the preset token sequence, wherein the probability consistency result includes any one of the same probability and different probability; according to the probability consistency result corresponding to each preset token, tokens are removed from the preset token sequence to determine a first preset token sequence, so as to perform subsequent reasoning according to the first preset token sequence, wherein the probability consistency results of multiple first preset tokens in the first preset token sequence are all the same probability.

[0014] One or more embodiments of the present specification provide a speculative decoding device for a large language model, including:

[0015] at least one processor; and,

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0018] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0019] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: through the above technical solution, by introducing the patch-level data processing method, it is possible to process long texts more flexibly, and determine the specified patch level based on the task type information, so that the model can adjust the size and number of patches for different task types during processing, thereby more effectively capturing long-distance dependencies in the sequence, which is especially important for complex tasks that require understanding the full text context, such as story continuation and long text writing; multiple candidate patch sequences are generated in parallel, and mapped using probability distribution to determine preliminary draft information. This method significantly improves the processing speed. Traditional sequence generation methods often need to generate each token step by step, while this method reduces the overall calculation time and improves generation efficiency by processing multiple candidate patches in parallel; through the target large language model The model verifies and screens the token sequences corresponding to the candidate patch sequences in parallel, which can ensure that the generated text is more coherent and smooth in structure and logic. Multiple rounds of verification and screening processes help reduce errors and irrationalities in the generated text and improve the overall quality of the text. It not only relies on the target large language model, but also combines the pre-trained draft model. The draft model is used to preliminarily generate candidate patch sequences, while the target large language model is used for subsequent verification and screening. This combination makes the entire system more flexible and adaptable, and can cope with different types of generation tasks. Through parallel processing and phased verification, this method can make more efficient use of computing resources. In the draft generation stage, multiple candidate patches can be processed in parallel, and in the verification stage, candidate sequences can be gradually screened and optimized as needed, avoiding unnecessary repeated calculations and waste of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0021] Figure 1 A flowchart of a method for inferential decoding of a large language model provided in an embodiment of this specification;

[0022] Figure 2 A schematic diagram of the structure of a speculative decoding device for a large language model provided in an embodiment of this specification. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0024] The embodiments of this specification provide a speculative decoding method for a large language model. It should be noted that the execution subject in the embodiments of this specification can be a server or any device with data processing capabilities. Figure 1 A flowchart of a method for inferential decoding of a large language model provided in an embodiment of this specification is shown in FIG. Figure 1 As shown, it mainly includes the following steps:

[0025] Step S101, obtaining current task information of the target large language model.

[0026] In one embodiment of the present specification, the current task information corresponding to the target language large model is obtained. It should be noted that the current task information includes the original text prompt information and the task type information. This can be achieved by designing a meta-controller, which is used to determine the patch size to be used according to the original text prompt information input, that is, the original prompt, during reasoning. It mainly includes a feature extraction module, a strategy selection module, a dynamic division module and an output module. The feature extraction module extracts content features from the input data, such as text type, content complexity, sentence length, etc. The task of the feature extraction module is to extract features from the input text for determining the patch size, including text type features, and may also include text length features, sentence structure features and text vocabulary features.

[0027] In one embodiment of the present specification, when extracting text type features, the type of text can be determined by a classification model or rule. The text type here includes but is not limited to descriptive document types, dialogue content types, high-precision content types and code types. The descriptive document types here include background information of news articles, general explanations of popular science articles, etc., dialogue content types include scripts, dialogue parts in novels, chat records, high-precision content types include legal documents, medical reports, scientific and technological specifications, etc., and code types such as code snippets. In addition, in addition to determining the task type information corresponding to the current task, the text length of the input text can also be counted to determine the text length feature in order to evaluate whether a more fine-grained division is needed. Through dependency parsing or other grammatical analysis methods, the complexity of the sentence and the relationship between the sentences are judged to obtain sentence structure features. The vocabulary density and vocabulary difficulty of the text are analyzed to obtain text vocabulary features to provide support for content with high precision requirements.

[0028] Step S102: input the original text prompt information into the target large language model to determine the initial token information, and construct multiple patch data of the specified patch level according to the initial token information and the original text prompt information.

[0029] In one embodiment of the present specification, the original text prompt information, i.e., the original prompt, is input into the target large language model to determine the initial token information. After obtaining the initial token, a patch level conversion is performed through the initial token and the original prompt to construct multiple patch data of a specified patch level, wherein the specified patch level is determined based on the task type information.

[0030] According to the initial token information and the original text prompt information, multiple patch data of the specified patch level are constructed, specifically including: splicing the initial token information and the original text prompt information to determine the spliced ​​text information; matching the corresponding specified patch level through the task type information; based on the specified patch level, converting the spliced ​​text information to determine multiple patch data.

[0031] Based on the specified patch level, the concatenated text information is converted to determine multiple patch data, specifically including: based on the specified patch level, the concatenated text information is divided into corresponding dimensions to determine multiple token sets; a first token vector corresponding to each first token in each of the token sets is determined; based on each of the first token vectors, a weighted average value corresponding to each of the token sets is determined, so as to determine the patch data corresponding to each of the token sets based on the weighted average value.

[0032] In one embodiment of the present specification, the initial token information and the original text prompt information are spliced, the initial token is spliced ​​after the original prompt, the spliced ​​text is determined, and multiple reference tokens corresponding to the spliced ​​text are determined. The strategy selection module of the meta-controller selects a suitable patch division strategy according to the extracted content features and predefined rules. The strategy selection module determines the specific patch division strategy according to the output of the feature extraction module. The strategy selection module adopts the method of the rule engine to perform patch division using predefined rules. The specific rule example is as follows: When the text type is a description document type and the length exceeds the preset length threshold, the patch level is determined to be a large patch between 50-100 tokens, and the specific corresponding token number can be randomly set in the range of 50-100 tokens according to the requirements. When the text type is a dialogue content type, the patch level is determined to be 2-3 sentence tokens. When the text type is a high-precision required content type, the patch level is determined to be a minimum patch of 5-10 tokens. When the text type is a code type, token-level division is adopted.

[0033] According to the task type information and the above rules, the corresponding patch level is determined, and the token compression range corresponding to the patch level is determined. The dynamic division module of the meta-controller divides the input text into patches of specified sizes according to the pre-set division strategies corresponding to different patch levels. The specific implementation method is as follows: When the patch level is a large patch, that is, it is within the range of 50-100 tokens, multiple patches are generated. For small patches corresponding to the dialogue content type, they are divided according to sentences, and then 2-3 sentences are combined to obtain a patch. When the text type is a high-precision content type, the minimum patch with a patch level of 5-10 tokens is determined. For code snippets, they are strictly divided according to the number of tokens to ensure that the granularity of each patch meets the requirements.

[0034] In one embodiment of the present specification, in accordance with the above method, multiple token sets corresponding to multiple reference tokens are determined, wherein the number of reference tokens in each of the token sets corresponds to the specified patch level; each of the token sets is compressed, and the patch data corresponding to each of the token sets is determined to determine multiple patch data.

[0035] In one embodiment of the present specification, each first token in the token set is converted into a vector form to obtain a corresponding first token vector, and a weighted average of multiple first token vectors is calculated to achieve a process of averaging the vector representations of multiple adjacent characters (tokens) and compressing them into a single "patch" to determine the corresponding patch data.

[0036] When determining the weight of a token, the equal weight method can be used to set the weight of all tokens to the same, that is, the weight of each token is 1, which is suitable for situations where there is no additional information or limited computing resources. In addition, the TF-IDF weight method can be used. TF-IDF (Term Frequency-Inverse Document Frequency) is a commonly used text feature weighting method that combines two factors: term frequency (TF) and inverse document frequency (IDF). TF represents the frequency of a token in the token set corresponding to the patch data. For example, the frequency of occurrence can be determined by sentence structure features, while IDF represents the general importance of the token in the original prompt. By calculating the product of TF and IDF, the TF-IDF weight of the token can be obtained. It takes into account both the local importance of the token in the document (TF) and its global importance (IDF) in the entire document collection, and can effectively distinguish important and unimportant tokens. The weight method based on the attention mechanism can also be used to automatically learn the weight relationship between tokens based on the attention mechanism. In text processing, the attention mechanism can be introduced to allow the model to automatically assign different weights to different tokens. By averaging the vector representations of multiple adjacent characters (tokens) and compressing them into a single patch, it can process long texts more efficiently and better capture long-range dependencies.

[0037] Step S103, generating multiple candidate patch sequences in parallel according to each patch data and the pre-trained draft model, so as to map each candidate patch sequence through probability distribution and determine preliminary draft information.

[0038] The preliminary draft information includes the token sequence corresponding to each candidate patch sequence and the mapping probability distribution data corresponding to each token sequence.

[0039] Traditional speculative decoding technology operates at the token level, that is, the prompt is input into the draft model and the target model for pre-filling, and the intermediate state (KV Cache) is generated and saved to avoid repeated calculations, thereby improving the inference efficiency. The specific speculative decoding process is as follows: the target model and the draft model generate a new token based on the pre-filling, and replace the token generated by the draft model with the target model, so that it can continue to generate several tokens (usually 5) based on the token. Then pass it to the target model in parallel to determine whether it is accepted. If the target model accepts all tokens generated by the draft model, a new token is generated and spliced ​​to the end of the sequence, and then passed to the draft model to continue generating. If the target model rejects all tokens generated by the draft model, a new token is generated and passed to the draft model to generate a new token sequence again. If the target model only accepts some of the tokens generated by the draft model, the unaccepted tokens and all tokens after them are deleted, and the token generated by the target model at that position is spliced ​​to the end of the accepted tokens, and then passed to the draft model to generate a new token sequence. The above process is repeated until the model generates an end symbol or reaches the maximum generation length.

[0040] In the above-mentioned inference decoding process, the length of the token sequence generated by the draft model each time is required to be short. This is because a longer token sequence may reject more tokens, and the overall reasoning time may be higher than the time of directly reasoning the target model. However, the shorter inference length is not applicable to all situations, and the target model can only decide whether to accept based on local information (the few tokens currently generated), which may lead to suboptimal global decisions and a lack of full understanding of the overall context. Secondly, although the target model performs quality assessment on the output sequence of the draft model in parallel, the overall process is still relatively linear and does not fully utilize the parallel computing capabilities of the hardware.

[0041] Before generating multiple candidate patch sequences in parallel according to each of the patch data and the pre-trained draft model, the method also includes: obtaining a source data set corresponding to the target large language model to process the source data set and determine a patch-level source data set; training a predetermined initial draft model through the patch-level source data set to determine a trained draft model.

[0042] In one embodiment of the present specification, in order to enable the draft model (e.g., Qwen2-1.5B-Instruct) to effectively predict the next Patch, the same source data set as the target model (e.g., Qwen2-72B-Instruct) is used. According to the content characteristics and complexity of the data set, the training data is re-divided into patch-level granularities of different lengths. According to different data set types, a variety of patch division strategies are adopted. For general descriptive documents, large patches (such as 50-100 tokens) are used to quickly process large amounts of information, which is suitable for background information of news articles, general explanations of popular science articles, etc. The advantage is that it can improve the processing speed and is suitable for processing large-scale data. For dialogue content, small patches (such as 2-3 sentences or 10-20 tokens) are used to ensure the coherence of the dialogue and the accurate expression of character characteristics. It is suitable for dialogue parts in scripts and novels, chat records, etc. For content that requires high-precision processing (legal documents, medical reports, scientific and technological specifications, etc.), smaller patches (5-10 tokens) are used to ensure accuracy when processing key information, maximize accuracy, and reduce the risk of errors. For code snippets, etc., the token level is continued to be used to ensure the accuracy of each character, symbol, and keyword, and avoid grammatical errors or logical errors caused by improper patch division. Retrain the draft model at the patch level to ensure that the draft model is consistent with the target model in terms of knowledge system and language style, thereby laying a solid foundation for subsequent precise reasoning.

[0043] In one embodiment of the present specification, a pre-trained draft model generates multiple possible candidate patch sequences in parallel based on patch data. Unlike traditional Token-level reasoning, the output of the draft model is no longer a specific Token sequence, but an abstract semantic representation in units of Patch. The draft model generates the original output values ​​logits of multiple patch-level sequences, and uses the softmax function to convert the logits into a probability distribution. The softmax function ensures that the sum of all possible token probabilities is 1, so that the output at each position can be interpreted as the probability of selecting the token. At each position, the token with the highest probability is selected according to the probability distribution, so that the logits of the patch sequence are mapped back to a specific token sequence to determine the preliminary draft information. The preliminary draft information includes the token sequence corresponding to each candidate patch sequence and the mapping probability distribution data corresponding to each token sequence.

[0044] Step S104, the token sequence corresponding to each candidate patch sequence is verified in parallel through the target large language model to screen the token sequence corresponding to the candidate patch sequence and determine the next round of input of the draft model for speculative decoding.

[0045] The target large language model is used to verify the token sequence corresponding to each candidate patch sequence in parallel, the token sequence corresponding to the candidate patch sequence is screened, and the next round of input of the draft model is determined for speculative decoding, specifically including: determining the mapping probability distribution data corresponding to each token sequence in the preliminary draft information; the target large language model is used to verify the token sequence corresponding to each candidate patch sequence in parallel, and the verification probability distribution data corresponding to each token sequence is output; according to the mapping probability distribution data and the mapping probability distribution data, the token sequence corresponding to the candidate patch sequence is screened, and the next round of input of the draft model is determined for speculative decoding.

[0046] In one embodiment of the present specification, a plurality of candidate patch sequences, that is, token sequences corresponding to the candidate patch sequences, are received in parallel by a target large language model, and each token sequence is verified in parallel to determine the verification probability distribution data corresponding to each token sequence. The mapping probability distribution data corresponding to each of the token sequences in the preliminary draft information is determined, and the token sequences corresponding to the candidate patch sequences are screened according to the mapping probability distribution data and the mapping probability distribution data to determine the next round of input of the draft model for speculative decoding. The target model is able to evaluate the quality of each generated candidate patch in parallel. This method not only increases the diversity of the output, but also increases the probability of finding the optimal solution. Through this innovative Patch-level reasoning technology, not only the reasoning speed of the large language model is greatly improved, but also the model's ability to handle long texts and complex tasks is enhanced, opening up new possibilities for the practical application of large-scale language models.

[0047] According to the mapping probability distribution data and the mapping probability distribution data, the token sequence corresponding to the candidate patch sequence is screened, and the next round of input of the draft model is determined for speculative decoding, specifically including: determining the mapping probability data of each token in the mapping probability distribution data corresponding to each of the token sequences, and determining the verification probability data of each token in the verification probability distribution data corresponding to each of the token sequences; comparing the mapping probability data of each token and the verification probability data of each token in sequence according to a preset order to determine the token probability comparison result corresponding to each of the token sequences, wherein the token probability comparison result includes all the same probabilities, partially the same probabilities and all the different probabilities; determining the patch verification results of the target large language model for multiple candidate patch sequences through the token probability comparison results corresponding to the token sequence, so as to screen the token sequence corresponding to the candidate patch sequence based on the patch verification results, and determine the next round of input of the draft model for speculative decoding, wherein the patch verification result includes any one of full sequence acceptance, partial acceptance and full sequence rejection.

[0048] In one embodiment of the present specification, the mapping probability data of each token in the mapping probability distribution data corresponding to each token sequence is determined, and the verification probability data of each token in the verification probability distribution data corresponding to each token sequence is determined; the mapping probability data of each token and the verification probability data of each token are compared in sequence according to the token generation order, and the token probability comparison result corresponding to each token sequence is determined. It should be noted that the token probability comparison result here is the probability comparison result at the sequence level of the entire token sequence, including all the same probabilities, partially the same probabilities, and all different probabilities. Through the token probability comparison result corresponding to the oken sequence, the patch verification result of the target large language model for multiple candidate patch sequences is obtained, and the patch verification result includes any one of full sequence acceptance, partial acceptance, and full sequence rejection. It should be noted that if the tokens at all positions of a sequence are consistent with those before verification, it is considered that the target model fully accepts this sequence, that is, the corresponding full sequence acceptance. According to the patch verification result, the token sequence corresponding to the candidate patch sequence is screened to determine the next round of input of the draft model for speculative decoding.

[0049] The patch verification results of the target large language model for multiple candidate patch sequences are determined by comparing the token probabilities corresponding to the token sequence, specifically including: when there is a token probability comparison result corresponding to a specified token sequence that the probabilities are all the same, determining that the patch verification results of the multiple candidate patch sequences are fully accepted; when there are no token probability comparison results corresponding to multiple specified token sequences that the probabilities are all the same, and there are token probability comparison results corresponding to token sequences that are partially the same in probabilities, determining that the patch verification results of the multiple candidate patch sequences are partially accepted; when the token probability comparison results corresponding to multiple token sequences are all probabilities that are all different, determining that the patch verification results of the multiple candidate patch sequences are fully rejected.

[0050] In one embodiment of the present specification, when there is a token probability comparison result corresponding to a specified token sequence that all probabilities are the same, the patch verification results of multiple candidate patch sequences are determined to be full sequence acceptance. Generally, since the token generated by the target large language model is determined, if there is a full sequence acceptance, only one patch sequence has a patch verification result that is full sequence acceptance. When there is no token probability comparison result corresponding to a specified token sequence that all probabilities are the same, and there are token probability comparison results corresponding to token sequences that are partially the same in probability, the patch verification results of multiple candidate patch sequences are determined to be partially accepted. When the token probability comparison results corresponding to multiple token sequences are all different in probability, it means that all tokens mapped by the candidate patch sequences are rejected by the target model, and the patch verification results of multiple candidate patch sequences are determined to be full sequence rejection. For example, suppose there are 5 candidate patch sequences. The first token corresponding to the first candidate patch sequence has the same probability, that is, the token is accepted. The second token has different probabilities, that is, the token is not accepted. The third one is generally not accepted. In this case, the probability of this candidate patch sequence is partially the same. The first token and the second token corresponding to the second candidate patch sequence have the same probability, that is, the two tokens are accepted. The third token has different probabilities. In this case, the probability of this candidate patch sequence is partially the same. If all tokens in the fifth candidate patch sequence have the same probability, that is, the target model accepts all tokens, then the fifth one is accepted in its entirety.

[0051] Based on the patch verification result, the token sequence corresponding to the candidate patch sequence is screened to determine the next round of input of the draft model for speculative decoding, which specifically includes: when the patch verification result is full sequence acceptance, determining the designated candidate patch sequence corresponding to the designated token sequence; deleting the remaining candidate patch sequences except the designated candidate patch sequence from multiple candidate patch sequences to perform subsequent reasoning based on the designated token sequence corresponding to the designated candidate patch sequence; when the patch verification result is partially accepted, determining the probability of each token sequence in each token sequence according to the token probability comparison result. The method comprises the following steps: determining the number of tokens with the same probability; screening multiple token sequences according to the number of tokens to determine a preset token sequence with the largest number of tokens, so as to determine a probability consistency result corresponding to each preset token in the preset token sequence, wherein the probability consistency result includes any one of the same probability and different probability; according to the probability consistency result corresponding to each preset token, tokens are removed from the preset token sequence to determine a first preset token sequence, so as to perform subsequent reasoning according to the first preset token sequence, wherein the probability consistency results of multiple first preset tokens in the first preset token sequence are all the same probability.

[0052] In one embodiment of the present specification, when the patch verification result is full sequence acceptance, that is, the tokens at all positions of a sequence are consistent with those before verification, the target model fully accepts this sequence and determines the designated candidate patch sequence corresponding to the designated token sequence. The draft model deletes the remaining candidate patch sequences except the designated candidate patch sequence in multiple candidate patch sequences, and continues to generate after this patch sequence. When the patch verification result is partial acceptance, that is, when the token sequences mapped by all patch sequences are not fully accepted, the number of tokens with the same probability in each token sequence is determined according to the token probability comparison result corresponding to each token sequence, and the preset token sequence with the largest number of tokens is selected from multiple token sequences. The probability consistency result corresponding to each preset token in the preset token sequence is determined, and the rejected tokens and their subsequent contents are deleted in the preset token sequence, and the high-quality part is retained, that is, the multiple accepted tokens are retained to determine the first preset token sequence. The target model generates a new token and splices it to the back of the retained first preset token sequence. The concatenated token sequence is converted back to a patch representation and passed to the draft model for further generation. When the patch verification result is a full sequence rejection, the recovery mechanism is initiated when all tokens mapped to the patch sequence are rejected by the target model. The target model independently generates a new token as a seed. This new token is passed to the draft model to guide it to regenerate multiple patch sequences.

[0053] Through this multi-level sequence verification and optimization mechanism, an organic combination of patch sequence generation and precise reasoning is achieved, which not only significantly improves the reasoning efficiency of large language models, but also ensures the stability and reliability of output quality. It is suitable for tasks that require long-term consistency and high-quality output, such as long text generation, complex problem answering and other scenarios. At the same time, its adaptive characteristics enable it to flexibly respond to language tasks of different difficulty and characteristics, showing broad application potential.

[0054] Through the above technical solution, by introducing a patch-level data processing method, it is possible to process long texts more flexibly, determine the specified patch level based on the task type information, and enable the model to adjust the size and number of patches for different task types during processing, thereby more effectively capturing long-distance dependencies in the sequence, which is particularly important for complex tasks that require understanding the context of the entire text, such as story continuation and long text writing; generate multiple candidate patch sequences in parallel, and use probability distribution for mapping to determine preliminary draft information. This method significantly improves the processing speed. Traditional sequence generation methods often need to generate each token step by step, while this method reduces the overall calculation time and improves generation efficiency by processing multiple candidate patches in parallel; the target large language model is used to map the candidate patch sequences to the t Parallel verification and screening of oken sequences can ensure that the generated text is more coherent and smooth in structure and logic. Multiple rounds of verification and screening processes can help reduce errors and irrationalities in the generated text and improve the overall quality of the text. It not only relies on the target large language model, but also combines the pre-trained draft model. The draft model is used to preliminarily generate candidate patch sequences, while the target large language model is used for subsequent verification and screening. This combination makes the entire system more flexible and adaptable, and can cope with different types of generation tasks. Through parallel processing and phased verification, this method can make more efficient use of computing resources. In the draft generation stage, multiple candidate patches can be processed in parallel, and in the verification stage, the candidate sequences can be gradually screened and optimized as needed, avoiding unnecessary repeated calculations and waste of resources.

[0055] The embodiment of this specification also provides a speculative decoding device for a large language model, such as Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0056] The embodiments of the present specification also provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0057] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0058] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0059] The devices and media provided in the embodiments of this specification correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0060] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0061] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0064] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0065] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0066] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0067] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0068] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A speculative decoding method for large language models, It is characterized in that The method comprises: Acquire current task information of the target large language model, wherein the current task information includes original text prompt information and task type information; Inputting the original text prompt information into the target large language model to determine initial token information, and constructing a plurality of patch data of a specified patch level according to the initial token information and the original text prompt information, wherein the specified patch level is determined based on the task type information; Generate multiple candidate patch sequences in parallel according to each of the patch data and the pre-trained draft model, so as to map each of the candidate patch sequences through probability distribution and determine preliminary draft information, wherein the preliminary draft information includes a token sequence corresponding to each of the candidate patch sequences; The token sequence corresponding to each of the candidate patch sequences is verified in parallel through the target large language model to screen the token sequence corresponding to the candidate patch sequence and determine the next round of input of the draft model for speculative decoding.

2. The method for inferential decoding of a large language model according to claim 1, It is characterized in that According to the initial token information and the original text prompt information, multiple patch data of the specified patch level are constructed, specifically including: Splicing the initial token information and the original text prompt information to determine spliced ​​text information; Match the corresponding specified patch level according to the task type information; Based on the specified patch level, the spliced ​​text information is converted to determine a plurality of patch data.

3. The method for inferential decoding of a large language model according to claim 2, It is characterized in that Based on the specified patch level, the spliced ​​text information is converted to determine a plurality of patch data, specifically including: Based on the specified patch level, dividing the spliced ​​text information into corresponding dimensions to determine multiple token sets; Determine a first token vector corresponding to each first token in each of the token sets; According to each of the first token vectors, a weighted average value corresponding to each of the token sets is determined, so as to determine the patch data corresponding to each of the token sets based on the weighted average value.

4. The method for inferential decoding of a large language model according to claim 1, It is characterized in that Before generating a plurality of candidate patch sequences in parallel according to each of the patch data and the pre-trained draft model, the method further comprises: Acquire a source data set corresponding to the target large language model to process the source data set and determine a patch-level source data set; The predetermined initial draft model is trained using the patch-level source data set to determine a trained draft model.

5. The method for inferential decoding of a large language model according to claim 1, It is characterized in that The target large language model is used to perform parallel verification on the token sequence corresponding to each candidate patch sequence to screen the token sequence corresponding to the candidate patch sequence and determine the next round of input of the draft model for speculative decoding, specifically including: Determine mapping probability distribution data corresponding to each of the token sequences in the preliminary draft information; Verifying the token sequence corresponding to each of the candidate patch sequences in parallel through the target large language model, and outputting verification probability distribution data corresponding to each of the token sequences; According to the mapping probability distribution data and the mapping probability distribution data, the token sequence corresponding to the candidate patch sequence is screened to determine the next round of input of the draft model for speculative decoding.

6. The method for inferential decoding of a large language model according to claim 5, It is characterized in that According to the mapping probability distribution data and the mapping probability distribution data, the token sequence corresponding to the candidate patch sequence is screened to determine the next round of input of the draft model for speculative decoding, specifically including: Determine the mapping probability data of each token in the mapping probability distribution data corresponding to each of the token sequences, and determine the verification probability data of each token in the verification probability distribution data corresponding to each of the token sequences; Comparing the mapping probability data of each token and the verification probability data of each token in sequence according to a preset order, and determining a token probability comparison result corresponding to each token sequence, wherein the token probability comparison result includes all the same probabilities, partially the same probabilities, and all the different probabilities; The patch verification results of the target large language model for multiple candidate patch sequences are determined by comparing the token probabilities corresponding to the token sequence, so as to screen the token sequences corresponding to the candidate patch sequences based on the patch verification results, and determine the next round of input of the draft model for speculative decoding, wherein the patch verification result includes any one of full sequence acceptance, partial acceptance and full sequence rejection.

7. The method for inferential decoding of a large language model according to claim 6, It is characterized in that Determining the patch verification results of the target large language model for multiple candidate patch sequences through the token probability comparison results corresponding to the token sequence, specifically including: When the token probability comparison results corresponding to the specified token sequence are all the same, the patch verification results of multiple candidate patch sequences are determined to be all sequence acceptance; When there are no token probability comparison results corresponding to multiple specified token sequences whose probabilities are all the same, and there are token probability comparison results corresponding to token sequences whose probabilities are all partially the same, the patch verification results of the multiple candidate patch sequences are determined to be partially accepted; When the token probability comparison results corresponding to the multiple token sequences are all different probabilities, it is determined that the patch verification results of the multiple candidate patch sequences are all sequence rejections.

8. The method for inferential decoding of a large language model according to claim 7, It is characterized in that Based on the patch verification result, the token sequence corresponding to the candidate patch sequence is screened to determine the next round of input of the draft model for speculative decoding, specifically including: When the patch verification result is full sequence acceptance, determining a designated candidate patch sequence corresponding to the designated token sequence; Deleting the remaining candidate patch sequences except the specified candidate patch sequence from the multiple candidate patch sequences, so as to perform subsequent reasoning based on the specified token sequence corresponding to the specified candidate patch sequence; When the patch verification result is partially accepted, determining the number of tokens with the same probability in each token sequence according to the token probability comparison result corresponding to each token sequence; According to the number of tokens, screening is performed in the plurality of token sequences to determine the preset token sequence with the largest number of tokens, so as to determine the probability consistency result corresponding to each preset token in the preset token sequence, wherein the probability consistency result includes any one of the same probability and different probability; According to the probability consistency results corresponding to each preset token, tokens are eliminated in the preset token sequence to determine a first preset token sequence, so as to perform subsequent reasoning based on the first preset token sequence, wherein the probability consistency results of multiple first preset tokens in the first preset token sequence all have the same probability.

9. A speculative decoding device for a large language model, It is characterized in that The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, It is characterized in that The computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images