General speculative decoding method and device for large language model and computer equipment

Abstract

The invention relates to a general speculation decoding method and device for a large language model and computer equipment, and the method comprises the steps: inputting a cue word prefix into a draft model, and outputting a draft sequence and the original probability of each lexical element in the draft sequence; encoding the draft sequence into an agent sequence of the target model; in the alignment layer, many-to-many alignment mapping between each lexical element in the draft sequence and each lexical element in the agent sequence is established; probability copying is carried out according to many-to-many alignment mapping and the original probability, and a first conditional probability is obtained; calculating a second conditional probability of each lexical element in the proxy sequence through the target model; and performing speculative sampling, and verifying the result of the speculative sampling under a verification condition to output a target sequence. Through application of the method and the device, the problems that the lossless acceleration principle of speculative decoding is violated and the acceleration effect is limited in the prior art are solved, and cross-vocabulary mathematical lossless acceleration is realized.

Description

Technical Field

[0001] This application relates to the field of speculative decoding technology, and in particular to general speculative decoding methods, apparatus and computer equipment for large language models. Background Technology

[0002] Current speculative decoding methods (such as Medusa, EAGLE-1 / 2 / 3, etc.) achieve a 2-3x speedup by integrating an auxiliary decoder head into the target model or using a dedicated draft model. However, these methods require the draft model to have a completely identical vocabulary to the target model, and the draft model needs to be trained separately for each target model. Based on this, the Lexical Intersection Mapping (TLI) algorithm is introduced to attempt to solve the vocabulary mismatch problem. Its steps are: 1. Calculate the intersection Vd of the draft vocabulary Vd and the target vocabulary Vt, Vd∩Vt. 2. Renormalize the probability distribution of the draft model on the intersection, setting the probability of words outside the intersection to zero. 3. Perform standard speculative sampling within the limited intersection space. 4. String-Level Exact Matching (SLEM): Accept words only when the strings are completely matched, preventing probabilistic sampling and resulting in information loss. The drawback of this approach is that a large number of draft words may be forcibly discarded, making probabilistic sampling impossible, violating the lossless speedup principle of speculative decoding, and thus limiting the speedup effect.

[0003] There are currently no effective solutions to the problem that the related technologies violate the lossless acceleration principle of speculative decoding and have limited acceleration effects. Summary of the Invention

[0004] This embodiment provides a general speculative decoding method, apparatus, and computer device for large language models to solve the problem that the speedup effect is limited due to violations of the lossless acceleration principle of speculative decoding in related technologies.

[0005] Firstly, this embodiment provides a general speculative decoding method for large language models, applicable to a general speculative decoding system for large language models; the system includes a draft model, an alignment layer, and a target model connected in sequence; the method includes:

[0006] Input the prompt word prefix into the draft model, and output the draft sequence and the original probability of each word element in the draft sequence;

[0007] The draft sequence is encoded into a proxy sequence for the target model;

[0008] In the alignment layer, a many-to-many alignment mapping is established between each word in the draft sequence and each word in the proxy sequence; based on the many-to-many alignment mapping and the original probability, a probability copy is performed to obtain the first conditional probability;

[0009] The second conditional probability of each word in the proxy sequence is calculated using the target model.

[0010] Perform speculative sampling, verify the results of speculative sampling under verification conditions, and output the target sequence; the verification conditions are determined by the first conditional probability and the second conditional probability.

[0011] In some embodiments, establishing a many-to-many alignment mapping between each word in the draft sequence and each word in the surrogate sequence includes:

[0012] Determine the lengths of the draft sequence and the proxy sequence respectively;

[0013] Dynamic time warping is performed based on the length of the draft sequence, the length of the proxy sequence, preset window constraints, and preset edit distance function to obtain a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence.

[0014] In some embodiments, dynamic time warping is performed based on the length of the draft sequence, the length of the proxy sequence, a preset window constraint, and a preset edit distance function to obtain a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence, including:

[0015] Construct and initialize the cumulative cost matrix based on the length of the draft sequence and the length of the proxy sequence;

[0016] Based on the window constraints and the edit distance function, the cumulative cost matrix is ​​filled using a dynamic programming recursive formula.

[0017] By backtracking from the filled cumulative cost matrix to the initialized cumulative cost matrix, the target cost path is extracted from the backtracking process to obtain the many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence.

[0018] In some embodiments, the method further includes:

[0019] During the process of filling the cumulative cost matrix using a dynamic programming recursive approach, the distance is calculated in character units.

[0020] In some embodiments, the method further includes:

[0021] During dynamic time warping, the markers used to control text boundaries or task instructions in the draft sequence and the proxy sequence are removed.

[0022] In some embodiments, the method further includes:

[0023] The quality of the draft sequence output by the draft model is controlled according to the preset anti-repetition mechanism.

[0024] In some embodiments, speculative sampling is performed, and the results of the speculative sampling are verified under verification conditions to output a target sequence, including:

[0025] Perform speculative sampling sequentially to generate random numbers for each word in the proxy sequence;

[0026] If the random number meets the verification condition, the word at the current position is accepted to perform verification at the next position;

[0027] If the random number does not meet the verification conditions, the word at the current position is rejected, the current verification round is terminated, and the accepted word is output as the target sequence.

[0028] In some embodiments, the method further includes:

[0029] Update the prompt word prefix based on the target sequence;

[0030] The next round of speculative decoding will be performed based on the updated prompt word prefix.

[0031] Secondly, this embodiment provides a general inferential decoding device for large language models, suitable for a general inferential decoding system for large language models; the system includes a draft model, an alignment layer, and a target model connected in sequence; the device includes:

[0032] The first processing module is used to input the prompt word prefix into the draft model and output the draft sequence and the original probability of each word element in the draft sequence;

[0033] An encoding module is used to encode the draft sequence into a proxy sequence for the target model;

[0034] An alignment module is used to establish a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence in the alignment layer; and to perform probability copying based on the many-to-many alignment mapping and the original probability to obtain a first conditional probability.

[0035] The second processing module is used to calculate the second conditional probability of each word in the proxy sequence through the target model;

[0036] The sampling verification module is used to perform speculative sampling, verify the results of speculative sampling under verification conditions, and output the target sequence; the verification conditions are determined by the first conditional probability and the second conditional probability.

[0037] Thirdly, this embodiment provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the general speculative decoding method for large language models described in the first aspect above.

[0038] Compared with related technologies, the general speculative decoding method, apparatus, and computer device for large language models provided in this embodiment, by inputting prompt word prefixes into a draft model, outputting a draft sequence and the original probabilities of each word element in the draft sequence; encoding the draft sequence into a proxy sequence for the target model; establishing a many-to-many alignment mapping between each word element in the draft sequence and each word element in the proxy sequence in the alignment layer; performing probability replication based on the many-to-many alignment mapping and the original probabilities to obtain a first conditional probability; calculating a second conditional probability of each word element in the proxy sequence through the target model; performing speculative sampling, verifying the results of speculative sampling under verification conditions to output the target sequence; the verification conditions are determined by the first and second conditional probabilities; solving the problem of limited acceleration effect in related technologies due to violation of the lossless acceleration principle of speculative decoding, by using many-to-many alignment mapping to accurately capture the correspondence of inherent semantics, and losslessly transferring the original probabilities from word elements in the draft sequence to word elements in the proxy sequence; providing an accurate baseline distribution for the verification conditions, thereby achieving mathematical lossless acceleration across the vocabulary.

[0039] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0041] Figure 1 This is a hardware structure block diagram of a terminal device for a general speculative decoding method for large language models provided in an embodiment of this application;

[0042] Figure 2 This is a flowchart of a general speculative decoding method for large language models provided in an embodiment of this application;

[0043] Figure 3 This is a schematic diagram of the structure of a general speculative decoding system for large language models provided in an embodiment of this application;

[0044] Figure 4 This is a flowchart of step S232;

[0045] Figure 5This is a schematic diagram of performance indicators under different window constraints provided in an embodiment of this application;

[0046] Figure 6 This is a schematic diagram illustrating the word matching effect of DTW under different model pairs according to an embodiment of this application;

[0047] Figure 7 This is a structural block diagram of a general speculative decoding device for large language models provided in an embodiment of this application.

[0048] In the diagram: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 210, first processing module; 220, encoding module; 230, alignment module; 240, second processing module; 250, sampling verification module. Detailed Implementation

[0049] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0051] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1This is a hardware structure block diagram of the terminal for the general speculative decoding method for large language models in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0052] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the general speculative decoding method for large language models in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0053] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0054] This embodiment provides a general speculative decoding method for large language models. Figure 2 This is a flowchart of the general speculative decoding method for large language models in this embodiment, as follows: Figure 2 As shown, the process includes the following steps:

[0055] Step S210: Input the prompt word prefix into the draft model and output the draft sequence and the original probability of each word element in the draft sequence;

[0056] Step S220: Encode the draft sequence into a surrogate sequence for the target model;

[0057] Step S230: In the alignment layer, establish a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence; perform probability copying based on the many-to-many alignment mapping and the original probability to obtain the first conditional probability;

[0058] Step S240: Calculate the second conditional probability of each word in the proxy sequence using the target model;

[0059] Step S250: Perform speculative sampling, verify the results of speculative sampling under verification conditions, and output the target sequence; the verification conditions are determined by the first conditional probability and the second conditional probability.

[0060] This method embodiment can be applied to a general inferential decoding system for large language models, the architecture of which is as follows: Figure 3 As shown, it includes a draft model, an alignment layer, and a target model connected in sequence. The draft model refers to an accelerated model capable of quickly generating candidate draft sequences for the target model to validate, thus accelerating the target model's operation; these include, but are not limited to, Qwen2.5-0.5B, Llama-3.1-7B, GPT-2-small, DeepSeek-0.5B, and OPT-350M. The target model refers to the main model that ultimately outputs a high-quality target sequence; these include, but are not limited to, Llama-3.1-70B, GPT-3.5-turbo, DeepSeek-R1, Mixtral8x22B, and Claude-3. In this embodiment, the problem of completely identical vocabularies between the draft model and the target model in related technologies can be solved. Therefore, the draft model and the target model can be heterogeneous models or models from the same family. That is, even for heterogeneous models, this method embodiment can achieve lossless mathematical acceleration across vocabularies and output high-quality target sequences.

[0061] The prompt prefix is ​​a structured instruction at the beginning of the input text, used to guide the model to generate results with a specific format, style, or content. After being input into the draft model, the draft model quickly outputs a draft sequence; each term in the draft sequence is accompanied by its original probability. This process leverages the fast inference speed of the draft model, transforming serial generation into parallel verification, providing a prerequisite for acceleration.

[0062] This method utilizes the segmenters of both the draft model and the target model to encode the draft sequence into a proxy sequence for the target model. Specifically, the draft model's segmenter decodes the draft sequence into an intermediate string, restoring discrete word IDs to a continuous text space. Then, the target model's segmenter re-encodes the intermediate string into a proxy sequence for the target model. This process achieves sequence encoding, enabling direct communication between heterogeneous models. Since different segmenters have different granularities, it is common for word lengths in the proxy sequence and the draft sequence to differ. In other embodiments, other methods, such as encoding algorithms, can also be used to complete the sequence encoding; there are no restrictions on this.

[0063] This study employs algorithms such as Dynamic Time Warping (DTW), Embedding spatial nearest neighbor matching, and pre-learned alignment matrices to establish many-to-many alignment mappings between words in the draft sequence and words in the proxy sequence. These many-to-many alignment mappings can correspond to words in one draft sequence, multiple draft sequences, or multiple draft sequences, allowing for flexible mapping and accurate capture of inherent semantic correspondences. Furthermore, the many-to-many alignment mapping enables probability replication of the original probabilities to obtain the first conditional probability; that is, it seamlessly transfers the original probabilities from words in the draft sequence to words in the proxy sequence, achieving a lossless transfer of probability quality. During probability replication, the probability of each word in the proxy sequence is equal to the probability of all words mapped to that position in the latest draft sequence, ensuring consistent overall probabilities and providing a correct baseline distribution for subsequent inference sampling.

[0064] In this process, steps S240 and S250 are both executed within the target model; the second conditional probability of each lexical in the proxy sequence is calculated; at this point, the verification condition can be determined by the first and second conditional probabilities; speculative sampling is then performed to verify whether the result of speculative sampling is accepted, thereby outputting the target sequence; this achieves lossless mathematical acceleration across the vocabulary. In other embodiments, the target model can be further accelerated by using a single forward computation combined with parallel verification, and there are no restrictions on this.

[0065] In related technologies, the Tlexical Intersection Mapping (TLI) algorithm is used to attempt to solve the vocabulary mismatch problem. Its approach is as follows: 1. Calculate the intersection Vd and the target vocabulary Vt, Vd∩Vt. 2. Renormalize the probability distribution of the draft model on the intersection, setting the probability of words outside the intersection to zero. 3. Perform standard speculative sampling within the limited intersection space. 4. String-Level Exact Matching (SLEM): Accepts words only when the strings are completely matched, preventing probabilistic sampling and resulting in information loss. The drawbacks of this scheme are: the possibility of forcibly discarding a large number of draft words, the inability to perform probabilistic sampling, violating the lossless acceleration principle of speculative decoding, and thus limiting the speedup effect. In this embodiment, by inputting the prompt word prefix into the draft model, the draft sequence and the original probabilities of each word in the draft sequence are output; the draft sequence is encoded into a proxy sequence for the target model; in the alignment layer, a many-to-many alignment mapping is established between each word in the draft sequence and each word in the proxy sequence; probability replication is performed based on the many-to-many alignment mapping and the original probabilities to obtain the first conditional probability; the second conditional probability of each word in the proxy sequence is calculated through the target model; speculative sampling is performed, and the results of speculative sampling are verified under verification conditions to output the target sequence; the verification conditions are determined by the first and second conditional probabilities; this solves the problem in related technologies that violate the lossless acceleration principle of speculative decoding and have limited acceleration effect. It uses the many-to-many alignment mapping to accurately capture the correspondence of the inherent semantics, and losslessly transfers the original probabilities from the words in the draft sequence to the words in the proxy sequence; it provides an accurate baseline distribution for the verification conditions, thereby achieving mathematical lossless acceleration across the vocabulary.

[0066] The steps described above are explained in detail below:

[0067] In some embodiments, establishing a many-to-many alignment mapping between each word in the draft sequence and each word in the surrogate sequence in step S230 includes the following steps:

[0068] Step S231: Determine the lengths of the draft sequence and the surrogate sequence respectively;

[0069] Step S232: Perform dynamic time warping based on the length of the draft sequence, the length of the proxy sequence, the preset window constraints, and the preset edit distance function to obtain the many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence.

[0070] In this embodiment, the length of the draft sequence is calculated as the total length of each word in the draft sequence, using characters as the unit; the length of the surrogate sequence can be calculated using the same method. Using characters instead of bytes can adapt to languages ​​without spaces and improve alignment robustness. The window constraint and edit distance function are pre-set; the window constraint is used to balance alignment accuracy and computational cost; the edit distance function is used to reflect differences at the word unit level.

[0071] After the above data (length of the draft sequence, length of the surrogate sequence, window constraints, and edit distance function) are prepared, dynamic time warping is performed on these data to obtain a many-to-many alignment mapping between each word in the draft sequence and each word in the surrogate sequence.

[0072] This embodiment supports pluggable alignment layers without requiring any training or model modification, enabling zero-cost adaptation to new target models; it also allows for free combination of heterogeneous models, achieving flexible arrangement; thus reducing switching costs.

[0073] In some of these embodiments, such as Figure 4 As shown, step S232 involves dynamic time warping based on the length of the draft sequence, the length of the surrogate sequence, preset window constraints, and a preset edit distance function to obtain a many-to-many alignment mapping between each word in the draft sequence and each word in the surrogate sequence. This includes the following steps:

[0074] Step S2321: Construct and initialize the cumulative cost matrix based on the length of the draft sequence and the length of the proxy sequence;

[0075] Step S2322: Fill the cumulative cost matrix according to the window constraints and the edit distance function using a dynamic programming recursive formula;

[0076] Step S2323: Backtrack from the filled cumulative cost matrix to the initialized cumulative cost matrix, extract the target cost path from the backtracking process, and obtain the many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence.

[0077] In this embodiment, a dynamic time warping (DTW) algorithm is used for label mapping to ensure lossless speculative sampling; wherein, the length of the draft sequence D is k; the length of the surrogate sequence T is m; the edit distance function is the edit distance d; the window constraint is w; the specific implementation process is as follows:

[0078] 1. Based on the length k of the draft sequence D and the length m of the proxy sequence T, construct a k×m cumulative cost matrix C[i,j]; initialize C[0,0]=0.

[0079] 2. Under the alignment window constraint (|ij|≤w), the cumulative cost moments are filled recursively using dynamic programming: C[i,j]=d(Di+Tj)+min(C[i-1,j],C[i,j-1],C[i-1,j-1]). Di is the i-th word in the draft sequence D, and Tj is the j-th word in the surrogate sequence T. The constraint |ij|≤w limits the search space to a diagonal band of width w, eliminating extreme offsets and reducing the computational complexity from 0 (km) to 0 (w.max(k,m)); this balances alignment accuracy with computational burden. During this process, edit distance can be calculated in character units.

[0080] 3. Backtrack from the filled cumulative cost matrix C[k,m] to the initialized cumulative cost matrix C[0,0], extract the minimum cost path as the target cost path, and obtain the many-to-many alignment mapping between k words in the draft sequence D and m words in the proxy sequence T as π'.

[0081] In this embodiment, DTW is applied to construct a many-to-many alignment mapping between draft sequences and surrogate sequences, thereby accurately transferring the probability distribution from the lexical units of the draft model to the lexical units of the target model. The alignment process is performed instantaneously in each decoding step without any retraining or model modification. DTW many-to-many mapping remains effective even with an intersection of only 6.4%. It possesses universal compatibility without the need for shared vocabulary: allowing seamless integration of any existing draft model without strict word matching. Furthermore, it supports many-to-many alignment and adapts to different word segmentation granularities.

[0082] Furthermore, the standard edit distance has a cost of 1 for insertion, deletion, and replacement, which can accurately reflect the differences at the metacharacter level.

[0083] A window constraint w of 8 can balance alignment accuracy and computational cost. Figure 5 This is a diagram illustrating performance metrics under different window constraints; for example... Figure 5 The data shows the performance metrics for w=4; w=8; w=16; w=∞; the transaction processing rate (TPS) and acceptance rate both reach their peak at w=8. Figure 6 This diagram illustrates the word matching performance of DTW under different model pairs; its window constraint is ∞; DTW word matching converges to the upper limit. Figure 6 This indicates that the identifier deviation between the matched proxies and the draft sequences is not exactly zero. Here, the identifier represents the sequence number within the sequence. If the vocabularies of the two models were perfectly aligned, the DTW matching result would converge precisely to the diagonal position. Therefore, the existence of this deviation indicates that the method embodiments of this application can effectively perform matching (…). Figure 6 95% of the lexical offsets are within the range of w.

[0084] In some embodiments, the general speculative decoding method for large language models further includes the following steps:

[0085] During dynamic time warping, remove markers used to control text boundaries or task instructions from draft sequences and proxy sequences.

[0086] In this embodiment, the marker used to control text boundaries or task instructions is... <bos> 、 <eos>These markers are removed before alignment during the dynamic time warping process to prevent them from interfering with the alignment process.

[0087] In some embodiments, the general speculative decoding method for large language models further includes the following steps:

[0088] Based on the preset anti-repetition mechanism, the quality of the draft sequence output by the draft model is controlled.

[0089] Specifically, the anti-repetition mechanism triggers a probabilistic penalty in the draft model when the repetition rate of words in the draft sequence within a unit window exceeds a repetition rate threshold, thereby controlling the repetition rate of words in the draft sequence. The repetition rate threshold can be set according to the application scenario and is not subject to any restrictions.

[0090] This embodiment utilizes an anti-repetition mechanism to control the quality of the draft sequence output by the draft model, preventing the draft model from having an artificially high speedup ratio.

[0091] In some embodiments, in step S240, the target model can calculate the second conditional probability of each word in the proxy sequence by performing a single forward propagation on the proxy sequence. ;in, . This represents the second conditional probability of the j-th word.

[0092] In some embodiments, the execution of speculative sampling in step S250, and the verification of the speculative sampling results under verification conditions to output the target sequence, includes the following steps:

[0093] Perform speculative sampling sequentially to generate random numbers for each word in the proxy sequence;

[0094] If the random number meets the verification condition, the word at the current position is accepted to perform verification at the next position;

[0095] If the random number does not meet the verification conditions, the word at the current position is rejected, the current verification round is terminated, and the accepted word is output as the target sequence.

[0096] Specifically, using the obtained first conditional probability Second conditional probability To determine the verification conditions, the verification conditions are as follows: ; r is the random number of each word in the proxy sequence generated by speculative sampling, where r ~ U(0,1).

[0097] Based on the acceptance criteria of speculative sampling, each word is connected in sequence: if If the random number satisfies the verification condition, the word at the current position is accepted for verification at the next position; otherwise, the word at the current position is rejected, the current verification is terminated, and the accepted word is output as the target sequence.

[0098] This embodiment achieves mathematically lossless acceleration across vocabularies by maintaining probability propagation through DTW's many-to-many alignment mapping and completing verification by combining standard speculative sampling, thus possessing the mathematical rigor of lossless acceleration.

[0099] In some embodiments, the general speculative decoding method for large language models further includes the following steps:

[0100] Update the prompt word prefix based on the target sequence;

[0101] The next round of inference decoding will be based on the updated prompt word prefix.

[0102] Specifically, upon obtaining the target sequence, the current round of speculative decoding can be considered complete. The target sequence can then be updated with the cue word prefix to proceed to the next round of speculative decoding; this continues until a sentence termination word is encountered, such as: <eos>At this point, the iteration ends, and the target sequence obtained at this point is output as the complete target sequence. In other embodiments, the iteration termination condition can also be the number of iterations, and there is no limitation on this.

[0103] Regarding the target sequence, it should be noted that after receiving γ tokens, since the input of the target model's forward propagation includes the prefix + γ tokens from the draft model, the next token (γ+1) will be generated as an additional token and fully utilized. That is, the target sequence will use a total of γ+1 tokens to update the cue word prefix and enter the next round of inference decoding.

[0104] This embodiment enables lossless, rapid iteration, with an output distribution consistent with that achieved when the target model is used alone.

[0105] The following example illustrates the overall implementation process described above:

[0106] The draft model is: Qwen2.5-0.5B accelerated target model Llama-3.1-70B, with the prompt word prefix "AIresearch follows".

[0107] The execution process of the general inference decoding method for large language models is as follows:

[0108] Draft sequence generation: The prompt word prefix "AI research follows" is input into the draft model Qwen2.5-0.5B, and the output draft sequence D=["Scal","ing","law","s","are"] (k=5) is generated. The original probabilities of each word are p'("Scal")=0.6, p'("ing")=0.5, p'("law")=0.7, p'("s")=0.4, and p'("are")=0.8.

[0109] Encoding: The word segmenter of the draft model decodes the draft sequence D into the string "Scaling laws are", and the word segmenter of the target model re-encodes the string into the surrogate sequence T=["Scale","ing","law","s","are"] (m=5).

[0110] Alignment: The optimal path is calculated using the DTW algorithm, resulting in a many-to-many alignment mapping: π'=[(1,1),(2,1),(2,2),(3,3),(4,4),(5,5)];

[0111] Here, (x,y) means that the x-th word in the draft sequence D is mapped to the y-th word in the proxy sequence T, that is, "Scal"+"ing" is mapped to "Scale", "ing" is mapped to "ing", and the rest are aligned in turn.

[0112] Based on the many-to-many mapping and the original probability, the probability is replicated: p("Scale")=p'("Scal")=0.6, p("Scale")=p'("ing")=0.5, p("ing")=p'("ing")=0.5, p("law")=p'("law")=0.7, p("s")=p'("s")=0.4, p("are")=p'("are")=0.8.

[0113] Speculative sampling and parallel validation: The target model performs a single forward propagation on the proxy sequence and calculates the second conditional probability of each word in the proxy sequence. We get q("Scale")=0.9, q("ing")=0.3, q("law")=0.6, q("s")=0.5, q("are")=0.7.

[0114] Perform speculative sampling sequentially to generate random numbers r for each word in the proxy sequence; the verification condition is: For example, if the random number r for "Scale" is 1.2 < {min(1,0.9 / 0.5)=1.4}, then "Scale" is accepted; if the random number r for "ing" is 0.7 > {min(1,0.3 / 0.5)=0.6}, then "ing" is rejected, the current round of verification is terminated, and the accepted tokens are used as the target sequence to adjust the prefix of the prompt word to enter the next round.

[0115] Iteration: Assuming the first word is accepted, the prefix is ​​updated to "AI research follows Scale", and the decoding continues; until a word indicating the end of the sentence is encountered, for example: <eos>At this point, the iteration ends, and the target sequence obtained at this point is output as the complete target sequence.

[0116] Through the above preferred embodiments, no training or model modification is required, thus supporting pluggable alignment layers and achieving zero-cost adaptation to new target models; under dynamic time warping, fine-grained semantic alignment of heterogeneous vocabularies is achieved through editing distance metrics, enabling free combination between heterogeneous models and achieving flexible deployment; reducing switching costs and achieving zero-cost migration.

[0117] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0118] This embodiment also provides a general speculative decoding device for large language models, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as described previously. The terms "module," "unit," "subunit," etc., used below can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0119] Figure 7 This is a structural block diagram of the general speculative decoding device for large language models in this embodiment, as shown below. Figure 7 As shown, the device includes: a general-purpose speculative decoding system suitable for large language models; the system includes a draft model, an alignment layer, and a target model connected in sequence; the device includes:

[0120] The first processing module 210 is used to input the prompt word prefix into the draft model and output the draft sequence and the original probability of each word element in the draft sequence;

[0121] Encoding module 220 is used to encode the draft sequence into a surrogate sequence for the target model;

[0122] Alignment module 230 is used to establish a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence in the alignment layer; perform probability copying based on the many-to-many alignment mapping and the original probability to obtain the first conditional probability; and obtain the probability of transferring the draft word to the target word without loss of quality.

[0123] The second processing module 240 is used to calculate the second conditional probability of each word in the proxy sequence through the target model;

[0124] The sampling verification module 250 is used to perform speculative sampling, verify the results of speculative sampling under verification conditions, and output the target sequence; the verification conditions are determined by the first conditional probability and the second conditional probability.

[0125] The above-mentioned device solves the problem of limited acceleration effect in related technologies that violate the lossless acceleration principle of speculative decoding. It uses many-to-many alignment mapping to accurately capture the correspondence of the inherent semantics and losslessly transfers the original probability from the lexicon of the draft sequence to the lexicon of the proxy sequence. It provides an accurate benchmark distribution for the verification conditions, thereby achieving mathematical lossless acceleration across the vocabulary.

[0126] In some embodiments, the alignment module 230 is also used to determine the lengths of the draft sequence and the surrogate sequence, respectively;

[0127] Dynamic time warping is performed based on the length of the draft sequence, the length of the surrogate sequence, preset window constraints, and preset edit distance function to obtain a many-to-many alignment mapping between each word in the draft sequence and each word in the surrogate sequence.

[0128] In some embodiments, the alignment module 230 is further configured to construct and initialize a cumulative cost matrix based on the length of the draft sequence and the length of the proxy sequence;

[0129] Based on the window constraints and the edit distance function, the cumulative cost matrix is ​​filled using a dynamic programming recursive formula.

[0130] By backtracking from the filled cumulative cost matrix to the initialized cumulative cost matrix, the target cost path is extracted from the backtracking process, resulting in a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence.

[0131] In some embodiments, the general inference decoding apparatus for large language models further includes:

[0132] The settings module is used to edit distance calculations in character units during the process of recursively filling the cumulative cost matrix according to dynamic programming.

[0133] In some embodiments, the general inference decoding apparatus for large language models further includes:

[0134] The removal module is used to remove markers used to control text boundaries or task instructions in draft sequences and proxy sequences during dynamic time warping.

[0135] In some embodiments, the general inference decoding apparatus for large language models further includes:

[0136] The control module is used to control the quality of the draft sequence output by the draft model according to the preset anti-repetition mechanism.

[0137] In some embodiments, the second processing module 240 is further configured to perform speculative sampling sequentially to generate random numbers for each word in the proxy sequence;

[0138] If the random number meets the verification condition, the word at the current position is accepted to perform verification at the next position;

[0139] If the random number does not meet the verification conditions, the word at the current position is rejected, the current verification round is terminated, and the accepted word is output as the target sequence.

[0140] In some embodiments, the general inference decoding apparatus for large language models further includes:

[0141] The iterative module is used to update the prompt word prefix based on the target sequence;

[0142] The next round of inference decoding will be based on the updated prompt word prefix.

[0143] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0144] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0145] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0146] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0147] S1, input the prompt word prefix into the draft model, and output the draft sequence and the original probability of each word element in the draft sequence;

[0148] S2, encode the draft sequence into a surrogate sequence for the target model;

[0149] S3, In the alignment layer, establish a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence; perform probability copying based on the many-to-many alignment mapping and the original probability to obtain the first conditional probability;

[0150] S4, calculate the second conditional probability of each word in the proxy sequence using the target model;

[0151] S5, perform speculative sampling, verify the results of speculative sampling under the verification conditions, and output the target sequence; the verification conditions are determined by the first conditional probability and the second conditional probability.

[0152] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0153] Furthermore, in conjunction with the general speculative decoding method for large language models provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the general speculative decoding methods for large language models described in the above embodiments.

[0154] It should be noted that all information and data involved in this application are authorized by the user or fully authorized by all parties and will be used legally.

[0155] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0156] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0157] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0158] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.< / eos> < / eos> < / eos> < / bos>

Claims

1. A general inferential decoding method for large language models, characterized in that, A general-purpose inferential decoding system for large language models; the system comprises a draft model, an alignment layer, and a target model connected in sequence; the method includes: Input the prompt word prefix into the draft model, and output the draft sequence and the original probability of each word element in the draft sequence; The draft sequence is encoded into a proxy sequence for the target model; In the alignment layer, a many-to-many alignment mapping is established between each word in the draft sequence and each word in the proxy sequence; based on the many-to-many alignment mapping and the original probability, a probability copy is performed to obtain the first conditional probability; The second conditional probability of each word in the proxy sequence is calculated using the target model. Perform speculative sampling, verify the results of speculative sampling under verification conditions, and output the target sequence; the verification conditions are determined by the first conditional probability and the second conditional probability.

2. The general inferential decoding method for large language models according to claim 1, characterized in that, Establishing a many-to-many alignment mapping between each word in the draft sequence and each word in the surrogate sequence includes: Determine the lengths of the draft sequence and the proxy sequence respectively; Dynamic time warping is performed based on the length of the draft sequence, the length of the proxy sequence, preset window constraints, and preset edit distance function to obtain a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence.

3. The general inferential decoding method for large language models according to claim 2, characterized in that, Dynamic time warping is performed based on the length of the draft sequence, the length of the proxy sequence, preset window constraints, and a preset edit distance function to obtain a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence, including: Construct and initialize the cumulative cost matrix based on the length of the draft sequence and the length of the proxy sequence; Based on the window constraints and the edit distance function, the cumulative cost matrix is ​​filled using a dynamic programming recursive formula. By backtracking from the filled cumulative cost matrix to the initialized cumulative cost matrix, the target cost path is extracted from the backtracking process to obtain the many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence.

4. The general inferential decoding method for large language models according to claim 3, characterized in that, The method further includes: During the process of filling the cumulative cost matrix using a dynamic programming recursive approach, the distance is calculated in character units.

5. The general inferential decoding method for large language models according to claim 1, characterized in that, The method further includes: During dynamic time warping, the markers used to control text boundaries or task instructions in the draft sequence and the proxy sequence are removed.

6. The general inferential decoding method for large language models according to claim 1, characterized in that, The method further includes: The quality of the draft sequence output by the draft model is controlled according to the preset anti-repetition mechanism.

7. The general speculative decoding method for large language models according to claim 1, characterized in that, Perform speculative sampling, validate the results of speculative sampling under validation conditions, and output the target sequence, including: Perform speculative sampling sequentially to generate random numbers for each word in the proxy sequence; If the random number meets the verification condition, the word at the current position is accepted to perform verification at the next position; If the random number does not meet the verification conditions, the word at the current position is rejected, the current verification round is terminated, and the accepted word is output as the target sequence.

8. The general inferential decoding method for large language models according to claim 1, characterized in that, The method further includes: Update the prompt word prefix based on the target sequence; The next round of speculative decoding will be performed based on the updated prompt word prefix.

9. A general-purpose speculative decoding device for large language models, characterized in that, A general-purpose inferential decoding system for large language models; the system includes a draft model, an alignment layer, and a target model connected in sequence; the apparatus includes: The first processing module is used to input the prompt word prefix into the draft model and output the draft sequence and the original probability of each word element in the draft sequence; An encoding module is used to encode the draft sequence into a proxy sequence for the target model; An alignment module is used to establish a many-to-many alignment mapping between each word in the draft sequence and each word in the proxy sequence in the alignment layer; and to perform probability copying based on the many-to-many alignment mapping and the original probability to obtain a first conditional probability. The second processing module is used to calculate the second conditional probability of each word in the proxy sequence through the target model; The sampling verification module is used to perform speculative sampling, verify the results of speculative sampling under verification conditions, and output the target sequence; the verification conditions are determined by the first conditional probability and the second conditional probability.

10. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the general speculative decoding method for large language models as described in any one of claims 1 to 8.

Images