Large language model reasoning acceleration method based on speculative decoding

Abstract

The invention discloses a large language model reasoning acceleration method based on speculative decoding, and relates to the technical field of natural language processing, and the method comprises the following steps: S100, freezing a basic language model; s200, extracting a hidden state of the key layer; s300, constructing a de-wharf network set; s400, training a de-wharf network; and S500, deploying the basic language model. According to the method, a multi-solution wharf network structure is adopted, and a parallel draft and verification mechanism is combined, so that the reasoning efficiency of the basic language model is improved, the resource consumption is reduced, and meanwhile, the consistency and the accuracy of a generated result are considered.

Description

Technical Field

[0001] This invention relates to the field of natural language processing (NLP) technology in artificial intelligence, and in particular to a method for accelerating reasoning in large language models based on speculative decoding. Background Technology

[0002] In recent years, large language models have demonstrated remarkable performance in natural language processing, with their applications in various fields such as text generation, machine translation, and dialogue systems driving the development of artificial intelligence. However, existing large language models often rely on step-by-step generation reasoning. In long text generation and real-time interaction, due to the sequential and dependent nature of the generation steps, the inference latency increases linearly with the text length, limiting the inference speed and making it difficult to meet the demands of real-time and large-scale applications.

[0003] Existing inference mechanisms, such as autoregressive generation based on Transformers, rely on the results of the previous generation step to predict the next word after each word is generated. While this step-by-step generation method has advantages in terms of generation coherence and consistency, it suffers from significant bottlenecks in inference speed and computational resources. Its completely serial computational nature makes it difficult to parallelize the inference process, resulting in low throughput and difficulty in meeting the real-time requirements of large-scale online services.

[0004] To accelerate the inference process, speculative decoding has emerged as a novel inference acceleration method. The core idea is to introduce a lightweight module to predict multiple subsequent lexical units as candidate sequences at once, while maintaining the output quality of the base language model. These are then validated by the base language model, with only prefix-consistent parts retained as the final output. Speculative decoding reduces the effective steps in inference and improves inference speed by generating multiple candidate lexical units in parallel during the drafting phase and performing parallel validation during the verification phase.

[0005] However, existing speculative decoding still faces significant bottlenecks in practical applications. On the one hand, candidate sequences are typically predicted in parallel by a lightweight drafting module or a multi-decoder network in a single forward computation. This prediction essentially relies on an approximate representation of the current prefix semantics. When this representation is insufficient to stably characterize long-distance semantic constraints, syntactic dependencies, and contextual consistency, candidate lexical units are prone to semantic drift, contextual inconsistency, or biases with locally high confidence but overall irrationality. This leads to a decrease in the acceptance rate of the base language model during the acceptability verification stage, making it difficult to effectively translate the reduction in steps achieved through parallel drafting into actual speedup benefits. On the other hand, speculative decoding still requires additional model forward computation on candidate sequences to complete acceptability verification, generating corresponding intermediate activation and caching overhead. Simultaneously, the multi-decoder network itself introduces additional parameters and parallel scheduling costs. In large-scale language models and long-context scenarios, these additional computational and storage overheads are further amplified, resulting in a significant increase in GPU memory usage, bandwidth pressure, and computational power consumption.

[0006] Therefore, those skilled in the art are dedicated to developing a method for accelerating reasoning in large language models based on speculative decoding. Summary of the Invention

[0007] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to improve the efficiency of model inference and reduce resource consumption, while taking into account the consistency and accuracy of the generated results.

[0008] The applicant argues that existing speculative decoding methods struggle to simultaneously balance inference speed, generation quality, and resource consumption under varying model sizes and workloads. The applicant employs a multi-decoder-head network structure, combined with parallel drafting and a basic language model acceptance verification mechanism, to improve model inference efficiency and reduce resource consumption, while maintaining consistency and accuracy in the generated results. In one embodiment of the present invention, a method for accelerating reasoning in large language models based on speculative decoding is provided, comprising: S100. Freeze the basic language model. Obtain the pre-trained basic language model and its corresponding weight file and configuration file. Freeze the pre-trained basic language model to obtain the frozen basic language model. S200. Extract the hidden states of the key layers, inputting a word sequence. From the frozen base language model, extract the hidden state matrix of the key layers, and concatenate the hidden state matrices according to the hidden state dimension to form a unified context matrix. Through context projection matrix Obtain the dimensionality reduction context matrix Take the reduced-dimensional context matrix With input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. ; S300, Construct a decoder head network set, using semantic vectors As input, the decoder head network set is initialized, and an intermediate feature transformation layer is constructed for each decoder head network to transform the semantic vector. Mapping to the intermediate hidden space yields the intermediate representation vector. Construct the output mapping layer of each decoder network to convert the intermediate representation vectors. Mapping to the vocabulary space completes the decoding head network set. Build; S400. Train the decoder network, construct training sample pairs, extract semantic vectors, calculate the predicted output of the decoder network, construct the loss function, update the parameters of the decoder network, and complete the training of the decoder network. S500: Deploy the basic language model, load the frozen basic language model into the deep learning framework, and input the word sequence. Input the frozen base language model, extract the hidden state matrix of the key layers, and concatenate the hidden state matrices along the hidden state dimensions to obtain the unified context matrix. Using context projection matrix Obtain the dimensionality reduction context matrix Take the reduced-dimensional context matrix In and input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. Input the decoder head network set and generate candidate word sequences in parallel. , and the input word sequence The sequences are concatenated and then input back into the frozen base language model to generate valid candidate sequences. , and the input word sequence The extended prefix sequence is generated by concatenating the prefixes, and iterative reasoning is performed to generate the final extended prefix sequence.

[0009] Optionally, in the large language model inference acceleration method based on speculative decoding in the above embodiments, step S100 includes: S110. Obtain the pre-trained model parameters. Obtain the pre-trained basic language model and its corresponding weight file and configuration file from the official website of the basic language model. S120. Freeze the base language model. Freeze the pre-trained base language model to obtain the frozen base language model. During the inference process, the parameters are not updated, and it only participates in the generation of hidden states.

[0010] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, the basic language model adopts one of the LLaMA (Large Language Model Meta AI) series models.

[0011] Preferably, in the large language model reasoning acceleration method based on speculative decoding in the above embodiments, the basic language model is selected as the LLaMA3.2 3B model, and a Transformer decoder structure with causal self-attention mechanism is adopted.

[0012] Furthermore, in the large language model inference acceleration method based on speculative decoding in the above embodiments, the structural hyperparameters of the basic language model include at least the number of decoder layers, the dimension of hidden states, and the size of the vocabulary logits.

[0013] Preferably, in the large language model inference acceleration method based on speculative decoding in the above embodiments, the number of decoder layers is 28, and the hidden state dimension is... vocabulary logits size .

[0014] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, step S200 includes: S210. Hidden state matrix extraction, input word sequence Forward inference is performed on the frozen base language model to extract the hidden state matrices of key layers, including L-1, L-6, and L-12. L-1 represents the last layer of the decoder, L-6 represents the sixth layer counting backwards from the last layer, and L-12 represents the twelfth layer counting backwards from the last layer. The corresponding hidden state matrices are denoted as follows: , and , The hidden state dimension of the base language model for a single word output is represented by the dimension of the hidden state, and the _th_ ... Behavioral location The hidden state vector, Represents the real number field; S220. Hidden State Matrix Concatenation: Concatenate the hidden state matrices of the key layers along the hidden state dimension to obtain a unified context matrix. ,in This represents the dimension of the hidden state vectors of the three key layers concatenated sequentially at the same word element position; S230, Unified context matrix dimensionality reduction, using context projection matrix For the unified context matrix A linear projection is performed on the hidden state dimension to obtain the dimensionality-reduced context matrix. The formula is as follows: ; S240. Semantic vector calculation, obtaining the reduced-dimensional context matrix. In and input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. , where semantic vector Represents the input word sequence Middle and last word elements The contextual semantic representation.

[0015] Preferably, in the large language model inference acceleration method based on speculative decoding in the above embodiments, the input word sequence length =128.

[0016] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, step S300 includes: S310. Initialize the decoder head network set, using semantic vectors. As input, the prediction window size is set to... , A positive integer representing the number of tokens predicted in parallel, equivalent to the number of decoder network heads, used to construct the decoder network set. , No. A decoder network The corresponding predicted position is ,in Each decoder network receives the same semantic vector. Each decoder network has the same structure, containing an intermediate feature transformation layer and an output mapping layer, and the parameters are not shared between the decoder networks. S320. Construct intermediate feature transformation layers. Set up intermediate feature transformation layers for each decoder head network to transform semantic vectors. Mapped to the intermediate hidden space, the first A decoder network Set a set of linear mapping parameters ,in , , = With a fixed intermediate dimension, the linear transformation result is processed by the nonlinear activation function ReLU to obtain the intermediate representation vector. The formula is as follows: ; S330. Construct the output mapping layer, set the output mapping layer for each decoder head network, and transfer the intermediate representation vector... Mapped to the vocabulary space, the first A decoder network Set a set of output layer parameters ,in , , Complete the decoding head network set with a fixed vocabulary size of logits. Build.

[0017] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, the prediction window size... The range is 2≤K≤5.

[0018] Preferably, in the large language model inference acceleration method based on speculative decoding in the above embodiments, the prediction window size... =4.

[0019] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, step S400 includes: S410. Construct training sample pairs and select input word sequences. The following series A sequence of words As a supervisory label; S420. Calculate the dimensionality-reduced context matrix, and transform the input word sequence... The input is fed into the frozen base language model for forward inference, and the hidden state matrix of the key layer is read. The hidden state matrix of the key layer has a dimension of 1. t H = The hidden state matrix is ​​concatenated along the hidden state dimension to obtain the unified context matrix. Using context projection matrix For the unified context matrix The formula for linear projection is as follows: ; Obtain the dimensionality reduction context matrix ; S430. Calculate the semantic vector and obtain the reduced-dimensional context matrix. In and input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. ; S440, Calculate the decoder head network prediction output and convert the semantic vector... Input to each decoder network respectively After intermediate feature transformation layer, weight matrix Bias vector To obtain the intermediate representation vector The formula is as follows: , intermediate representation vector Mapped to the vocabulary space, weight matrix bias Calculate the vocabulary logits vector The formula is as follows: , get K A set of logits vectors for each vocabulary The vocabulary of candidate nouns is logits; S450. Construct the loss function and use supervision labels. As a monitoring signal, K Each decoder network has a cross-entropy loss function, and the loss function for each decoder network is defined as follows: , The overall loss function is the arithmetic mean of all decoder network layers, and the formula is as follows: ; Select the training optimizer, set the learning rate, do not use weight decay, set the batch size and epochs, do not introduce early stopping, disable mixed precision during all training processes, set the computational precision, and perform forward computation and backpropagation. S460. Decoder network parameter update: Perform backpropagation and parameter update, train according to the training epoch, and complete the parameter update for all decoder network sets. Optimization; S470, Parameter Saving: Save the network parameters of all decoder network sets. Save, among .

[0020] Preferably, in the large language model inference acceleration method based on speculative decoding in the above embodiments, the training optimizer is the Adam optimizer.

[0021] Preferably, in the large language model inference acceleration method based on speculative decoding in the above embodiments, the first-order moment decay rate of the Adam optimizer is... Set to 0.9, second moment decay rate Set it to 0.999.

[0022] Preferably, in the large language model inference acceleration method based on speculative decoding in the above embodiments, the learning rate is set to... .

[0023] Preferably, in the large language model inference acceleration method based on speculative decoding in the above embodiments, the training batch size is 64.

[0024] Preferably, in the large language model reasoning acceleration method based on speculative decoding in the above embodiments, the training epoch is 3.

[0025] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, the network parameters of the decoding head network set are saved in .pt format.

[0026] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, the computational precision is set to full precision FP32.

[0027] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, step S500 includes: S510, Construct a decoder head network set, load the frozen basic language model into the deep learning framework, and K Each decoder network model file is loaded as an independent inference module to construct a decoder network ensemble. ; S520. Perform context encoding to convert the input word sequence into a given word sequence. Input the frozen base language model, perform forward inference, extract the hidden state matrix of the key layers, and concatenate the hidden state matrices along the hidden state dimensions to obtain the unified context matrix. Using context projection matrix Obtain the dimensionality reduction context matrix Take the reduced-dimensional context matrix In and input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. ; S530. Generate candidate word sequences in parallel, input the semantic vectors into the decoder head network set, and perform forward propagation in parallel to calculate the vocabulary logits vector of the candidate words at position t+k. For each vocabulary logits vector Applying the softmax operation yields the candidate word distribution. Each decoder network performs greedy sampling based on the candidate word distribution, selecting the candidate word with the highest probability. , obtain a length ofK candidate word sequence ; S540. Perform acceptability verification by inputting the word sequence. With candidate lexical sequences splicing together to form a complete sequence Re-enter the frozen base language model at each position. Above, read the vocabulary logits vector. And calculate the probability distribution. The formula is as follows: ; If satisfied Then the candidate word element is considered Accepted; otherwise rejected. This indicates that the base language model is frozen at the position. For candidate lexical elements The assigned probability, To meet the acceptance threshold, the largest consecutively accepted subsequence is considered a valid candidate sequence for the current inference. Specifically, from Start by calculating and comparing in sequence. and When the first occurrence is less than Stop verification at that time, and Defined as ,like That is, to refuse, An empty sequence; S550, Output the inference results, including the valid candidate sequences. Concatenate to the input word sequence At the end, an extended prefix sequence is formed, which serves as the input for the next round of inference. Steps S520 to S550 are repeated until the inference result of the frozen basic language model is a stop symbol. <eos>When the condition is met, stop the inference and output the final extended prefix sequence.

[0028] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, the deep learning framework adopts the PyTorch inference framework.

[0029] Optionally, in the large language model inference acceleration method based on speculative decoding in any of the above embodiments, the acceptance threshold range is [range missing]. .

[0030] Preferably, in the large language model reasoning acceleration method based on speculative decoding in the above embodiments, the threshold τ = 0.9 is set.

[0031] This invention employs a multi-decoder network structure, combined with a parallel drafting and verification mechanism, which improves the inference efficiency of the basic language model and reduces resource consumption, while also ensuring the consistency and accuracy of the generated results.

[0032] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0033] Figure 1 This is a flowchart of an exemplary embodiment of a method for accelerating reasoning in large language models based on speculative decoding; Figure 2 This is a flowchart illustrating the extraction of the hidden state of the key layer in an exemplary embodiment; Figure 3 This is a flowchart illustrating the training of the decoding head network in an exemplary embodiment; Figure 4 This is a flowchart illustrating the deployment of the underlying language model in an exemplary embodiment. Detailed Implementation

[0034] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0035] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of components is schematically exaggerated in some places in the drawings.

[0036] The inventors designed a method to accelerate reasoning in large language models based on speculative decoding, such as... Figure 1 As shown, it includes the following steps: S100. Freeze the base language model. Obtain the pre-trained base language model and its corresponding weight and configuration files. Freeze the pre-trained base language model to obtain the frozen base language model. The base language model uses the LLaMA 3.2 3B model, employing a Transformer decoder structure with a causal self-attention mechanism. The structural hyperparameters include at least the number of decoder layers, the hidden state dimension, and the vocabulary logits size. The number of decoder layers is 28, and the hidden state dimension... vocabulary logits size Specifically, this includes: S110. Obtain the pre-trained model parameters. Obtain the pre-trained basic language model and its corresponding weight file and configuration file from the official website of the basic language model. S120. Freeze the base language model. Freeze the pre-trained base language model to obtain the frozen base language model. During the inference process, the parameters are not updated, and it only participates in the generation of hidden states.

[0037] S200. Extract the hidden states of the key layers, inputting a word sequence. To freeze the underlying language model, input word sequence length =128, extract the hidden state matrix of the key layer, and concatenate the hidden state matrices according to the hidden state dimension to form a unified context matrix. Through context projection matrix Obtain the dimensionality reduction context matrix Take the reduced-dimensional context matrix With input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. Specifically, this includes: S210. Hidden state matrix extraction, input word sequence Forward inference is performed on the frozen base language model to extract the hidden state matrices of key layers, including L-1, L-6, and L-12. L-1 represents the last layer of the decoder, L-6 represents the sixth layer counting backwards from the last layer, and L-12 represents the twelfth layer counting backwards from the last layer. The corresponding hidden state matrices are denoted as follows: , and , The hidden state dimension of the base language model for a single word output is represented by the dimension of the hidden state, and the _th_ ... Behavioral location The hidden state vector, Represents the real number field; S220. Hidden State Matrix Concatenation: Concatenate the hidden state matrices of the key layers along the hidden state dimension to obtain a unified context matrix. ,in This represents the dimension of the hidden state vectors of the three key layers concatenated sequentially at the same word element position; S230, Unified context matrix dimensionality reduction, using context projection matrix For the unified context matrix A linear projection is performed on the hidden state dimension to obtain the dimensionality-reduced context matrix. The formula is as follows: ; S240. Semantic vector calculation, obtaining the reduced-dimensional context matrix. In and input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. , where semantic vector Represents the input word sequence Middle and last word elements The contextual semantic representation.

[0038] S300, Construct a decoder head network set, using semantic vectors As input, the decoder head network set is initialized, and an intermediate feature transformation layer is constructed for each decoder head network to transform the semantic vector. Mapping to the intermediate hidden space yields the intermediate representation vector. Construct the output mapping layer of each decoder network to convert the intermediate representation vectors. Mapping to the vocabulary space completes the decoding head network set. Construction; specifically including: S310. Initialize the decoder head network set, using semantic vectors. As input, the prediction window size is set to... , =4, indicating the number of tokens predicted in parallel, which is equivalent to the number of decoder head networks used to construct the decoder head network set. , No. A decoder network The corresponding predicted position is ,in Each decoder network receives the same semantic vector. Each decoder network has the same structure, containing an intermediate feature transformation layer and an output mapping layer, and the parameters are not shared between the decoder networks. S320. Construct intermediate feature transformation layers. Set up intermediate feature transformation layers for each decoder head network to transform semantic vectors. Mapped to the intermediate hidden space, the first A decoder network Set a set of linear mapping parameters ,in , , = With a fixed intermediate dimension, the linear transformation result is processed by the nonlinear activation function ReLU to obtain the intermediate representation vector. The formula is as follows: ; S330. Construct the output mapping layer, set the output mapping layer for each decoder head network, and transfer the intermediate representation vector... Mapped to the vocabulary space, the first A decoder network Set a set of output layer parameters ,in , , Complete the decoding head network set with a fixed vocabulary size of logits. Build.

[0039] S400. Train the decoder network, construct training sample pairs, extract semantic vectors, calculate the decoder network's predicted output, construct the loss function, update the decoder network parameters, and complete the decoder network training; specifically including: S410. Construct training sample pairs and select input word sequences. The following series A sequence of words As a supervisory label; S420. Calculate the dimensionality-reduced context matrix, and transform the input word sequence... The input is fed into the frozen base language model for forward inference, and the hidden state matrix of the key layer is read. The hidden state matrix of the key layer has a dimension of 1. t H = The hidden state matrix is ​​concatenated along the hidden state dimension to obtain the unified context matrix. Using context projection matrix For the unified context matrix The formula for linear projection is as follows: ; Obtain the dimensionality reduction context matrix ; S430. Calculate the semantic vector and obtain the reduced-dimensional context matrix. In and input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. ; S440, Calculate the decoder head network prediction output and convert the semantic vector... Input to each decoder network respectively After intermediate feature transformation layer, weight matrix Bias vector To obtain the intermediate representation vector The formula is as follows: , intermediate representation vector Mapped to the vocabulary space, weight matrix bias Calculate the vocabulary logits vector The formula is as follows: , get K A set of logits vectors for each vocabulary The vocabulary of candidate nouns is logits; S450. Construct the loss function and use supervision labels. As a monitoring signal, K Each decoder network has a cross-entropy loss function, and the loss function for each decoder network is defined as follows: , The overall loss function is the arithmetic mean of all decoder network layers, and the formula is as follows: ; The training optimizer is selected as the Adam optimizer, and the first-order moment decay rate of the Adam optimizer is... Set to 0.9, second moment decay rate Set to 0.999, set the learning rate to... We do not use weight decay, set the training batch size to 64 and the training epoch to 3, do not introduce early stopping mechanism, disable mixed precision in all training processes, set the computation precision to full precision FP32, and perform forward computation and backpropagation. S460. Decoder network parameter update: Perform backpropagation and parameter update, train according to the training epoch, and complete the parameter update for all decoder network sets. Optimization; S470, Parameter Saving: Save the network parameters of all decoder network sets. Save as a .pt file, in which .

[0040] S500: Deploy the basic language model, load the frozen basic language model into the deep learning framework, and use the PyTorch inference framework to input the word sequence. Input the frozen base language model, extract the hidden state matrix of the key layers, and concatenate the hidden state matrices along the hidden state dimensions to obtain the unified context matrix. Using context projection matrix Obtain the dimensionality reduction context matrix Take the reduced-dimensional context matrix In and input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. Input the decoder head network set and generate candidate word sequences in parallel. , and the input word sequence The sequences are concatenated and then input back into the frozen base language model to generate valid candidate sequences. , and the input word sequence The extended prefix sequence is generated by concatenation, and iterative reasoning is performed to generate the final extended prefix sequence; specifically, this includes: S510, Construct a decoder head network set, load the frozen basic language model into the deep learning framework, and K Each decoder network model file is in .pt format and is loaded as an independent inference module to construct a decoder network ensemble. ; S520. Perform context encoding to convert the input word sequence into a given word sequence. Input the frozen base language model, perform forward inference, extract the hidden state matrix of the key layers, and concatenate the hidden state matrices along the hidden state dimensions to obtain the unified context matrix. Using context projection matrix Obtain the dimensionality reduction context matrix Take the reduced-dimensional context matrix In and input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. ; S530. Generate candidate word sequences in parallel, input the semantic vectors into the decoder head network set, and perform forward propagation in parallel to calculate the vocabulary logits vector of the candidate words at position t+k. For each vocabulary logits vector Applying the softmax operation yields the candidate word distribution. Each decoder network performs greedy sampling based on the candidate word distribution, selecting the candidate word with the highest probability. , obtain a length of K candidate word sequence ; S540. Perform acceptability verification by inputting the word sequence. With candidate lexical sequences splicing together to form a complete sequence Re-enter the frozen base language model at each position. Above, read the vocabulary logits vector. And calculate the probability distribution. The formula is as follows: ; If satisfied Then the candidate word element is considered Accepted; otherwise rejected. This indicates that the base language model is frozen at the position. For candidate lexical elements The assigned probability, Assuming an acceptance threshold of τ = 0.9, the largest consecutively accepted subsequence is considered a valid candidate sequence for the current inference. Specifically, from Start by calculating and comparing in sequence. and When the first occurrence is less than Stop verification at that time, and Defined as ,like That is, to refuse, An empty sequence; S550, Output the inference results, including the valid candidate sequences. Concatenate to the input word sequence At the end, an extended prefix sequence is formed, which serves as the input for the next round of inference. Steps S520 to S550 are repeated until the inference result of the frozen basic language model is a stop symbol. <eos>When the condition is met, stop the inference and output the final extended prefix sequence.

[0041] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.< / eos> < / eos>

Claims

1. A large language model inference acceleration method based on speculative decoding, characterized in that, The method comprises the following steps: S100, freezing a basic language model, obtaining a pre-trained basic language model and corresponding weight files and configuration files, freezing the pre-trained basic language model to obtain a frozen basic language model; S200. Extract the hidden states of the key layers, inputting a word sequence. Based on the frozen base language model, the hidden state matrix of the key layer is extracted, and the hidden state matrix is ​​concatenated according to the hidden state dimension to form a unified context matrix. Through context projection matrix Obtain the dimensionality reduction context matrix Take the dimensionality reduction context matrix With the input lexical sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. ; S300, construct a decoding head network set, map the semantic vector to an intermediate hidden space to obtain an intermediate representation vector As input, initialize the decoding head network set, construct the intermediate feature transformation layer of each decoding head network, map the semantic vector to the intermediate hidden space to obtain the intermediate representation vector As input, initialize the decoding head network set, construct the intermediate feature transformation layer of each decoding head network, map the semantic vector to the intermediate hidden space to obtain the intermediate representation vector , construct the output mapping layer of each decoding head network, map the intermediate representation vector to the vocabulary space, complete the decoding head network set , construct the output mapping layer of each decoding head network, map the intermediate representation vector to the vocabulary space, complete the decoding head network set construct; S400, training a decoding head network, constructing a training sample pair, extracting a semantic vector, calculating a decoding head network prediction output, constructing a loss function, updating the decoding head network parameters, and completing the decoding head network training; S500: Deploy the basic language model, load the frozen basic language model into the deep learning framework, and input the word sequence. Input the frozen base language model, extract the hidden state matrix of the key layers, and concatenate the hidden state matrix along the hidden state dimension to obtain the unified context matrix. Using context projection matrix Obtain the dimensionality reduction context matrix Take the dimensionality reduction context matrix In and input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. Input the decoder head network set and generate candidate word sequences in parallel. , and the input word sequence The sequences are then concatenated and input again into the frozen base language model to generate valid candidate sequences. , and the input word sequence The extended prefix sequence is generated by concatenating the prefixes, and iterative reasoning is performed to generate the final extended prefix sequence.

2. The speculative-decoding-based large language model inference acceleration method of claim 1, wherein, The step 100 comprises: S110, obtaining pre-trained model parameters, obtaining a pre-trained basic language model and corresponding weight files and configuration files from a basic language model official website; S120, freezing the basic language model, freezing the pre-trained basic language model to obtain a frozen basic language model, and not updating parameters in an inference process, but only participating in generating a hidden state.

3. The speculative-decoding-based large language model inference acceleration method of claim 1 or 2, wherein, The basic language model adopts one of LLaMA series models.

4. The speculative-decoding-based large language model inference acceleration method of claim 3, wherein, The basic language model selects an LLaMA3.2 3B model, and adopts a causal self-attention mechanism Transformer decoder structure.

5. The speculative-decoding-based large language model inference acceleration method of claim 1, wherein, The step S200 comprises: S210, hidden state matrix extraction, input the word sequence forward inference to the frozen base language model, extract the hidden state matrix of the key layers, the key layers include L-1, L-6 and L-12, wherein L-1 represents the last layer of the decoder, L-6 represents the sixth layer from the last layer, and L-12 represents the twelfth layer from the last layer; the corresponding hidden state matrix is respectively denoted as 、 and , represents the hidden state dimension output by the frozen base language model for a single word, and the first behavior position hidden state vector, represents the real number field; S220, hidden state matrix splicing, concatenating the hidden state matrix of the key layer in the hidden state dimension to obtain a unified context matrix wherein represents concatenating the hidden state vectors of the three key layers at the same token position in sequence. S230, dimension reduction of the unified context matrix, using a context projection matrix to the unified context matrix linear projection on the hidden state dimension, to obtain a reduced dimension context matrix , the formula is as follows: ； S240, semantic vector calculation, taking the reduced dimension context matrix in the input token sequence in the input token sequence as a semantic vector wherein the semantic vector represents the input token sequence in the input token sequence contextual semantic representation.

6. The speculative-decoding-based large language model inference acceleration method of claim 1, wherein, The step S300 comprises: S310. Initialize the decoder head network set, using the semantic vector. As input, the prediction window size is set to... , A positive integer representing the number of tokens predicted in parallel, equivalent to the number of decoder network heads, used to construct the decoder network set. , No. A decoder network The corresponding predicted position is ,in Each decoder network receives the same semantic vector. Each decoder network has the same structure, containing an intermediate feature transformation layer and an output mapping layer, and the parameters are not shared between the decoder networks. S320, construct an intermediate feature transformation layer, set the intermediate feature transformation layer of each decoding head network, map the semantic vector to an intermediate hidden space, the first decoding head network sets a set of linear mapping parameters , wherein , , = is a fixed intermediate dimension, and the linear transformation result is processed through a nonlinear activation function ReLU to obtain an intermediate representation vector , and the formula is as follows: ； S330. Construct an output mapping layer, set the output mapping layer for each decoder head network, and transfer the intermediate representation vector... Mapped to the vocabulary space, the first A decoder network Set a set of output layer parameters ,in , , To complete the decoding head network set with a fixed vocabulary size of logits. Build.

7. The speculative-decoding-based large language model inference acceleration method of claim 1, wherein, The prediction window size The range is 2 ≤ K ≤ 5.

8. The speculative-decoding-based large language model inference acceleration method of claim 6, wherein, The step S400 comprises: S410, construct a training sample pair, select the input token sequence post-continuous a token sequence as a supervised label; S420, calculating a reduced dimension context matrix, inputting the input token sequence to the frozen base language model for forward inference, reading the hidden state matrix of the key layer, the dimension of the hidden state matrix of the key layer is t H = , splicing the hidden state matrix in the hidden state dimension direction to obtain a unified context matrix , using a context projection matrix to perform linear projection on the unified context matrix , and the formula is as follows: ； obtaining a reduced dimension context matrix ; S430, computing a semantic vector, taking the reduced dimension context matrix in the input token sequence in the input token sequence as the semantic vector ; S440, computing a decoding head network prediction output, said semantic vector is input to each decoding head network , through the intermediate feature transformation layer, the weight matrix , the bias vector , to obtain an intermediate representation vector , as follows: ， mapping the intermediate representation vector to a vocabulary space, a weight matrix , a bias , computing a vocabulary logits vector , as follows: ， obtained K a set of vocabulary logits vectors vocabulary logits as candidate tokens; S450, construct a loss function using the supervised label As a supervised signal, the loss function is constructed as follows: K The cross-entropy loss function is constructed for each decoding head network, and the loss function of the decoding head network is defined as follows: ， The overall loss function is an arithmetic average of all decoding head networks, and the formula is as follows: ； An training optimizer is selected, a learning rate is set, weight decay is not adopted, a training batch size and a training round number epoch are set, an early stopping mechanism is not introduced, mixed precision is disabled in all training processes, a calculation precision is set, forward calculation and back propagation are executed; S460, decoding head network parameter updating, performing back propagation and parameter updating, training according to the training round epoch, completing the optimization of network parameters of all decoding head network sets ; S470, parameter saving, saving network parameters of the network set of all decoding head networks wherein .

9. The speculative-decoding-based large language model inference acceleration method of claim 8, wherein, The training optimizer is an Adam optimizer, a first moment decay rate is set to 0.9, a second moment decay rate is set to 0.

999.

10. The speculative-decoding-based large language model inference acceleration method of claim 8, wherein, The step S500 comprises: S510, construct a decoding head network set, load the frozen base language model into a deep learning framework, and load one decoding head network model file as an independent inference module respectively to construct the decoding head network set K ;​ S520. Perform context encoding to convert the input word sequence into a given word sequence. Input the frozen base language model, perform forward inference, extract the hidden state matrix of the key layer, and concatenate the hidden state matrix along the hidden state dimension to obtain the unified context matrix. Using context projection matrix Obtain the dimensionality reduction context matrix Take the dimensionality reduction context matrix The input word sequence Middle and last word elements The corresponding row vectors serve as semantic vectors. ; S530, generating candidate token sequences in parallel, inputting the semantic vector into the decoding head network set, the decoding head networks performing forward propagation in parallel to calculate vocabulary logits vectors of the candidate tokens corresponding to position t+k , for each vocabulary logits vector , applying a softmax operation to obtain a candidate token distribution , each decoding head network performs greedy sampling based on the candidate token distribution to select the candidate token with the highest probability , obtaining a candidate token sequence with a length of K ;​ S540. Perform acceptability verification and convert the input lexical sequence. With the candidate lexical sequence splicing together to form a complete sequence Then input it again into the frozen base language model, at each position. Above, read the vocabulary logits vector. And calculate the probability distribution. The formula is as follows: ； If , the candidate wordpiece is considered accepted, otherwise rejected, where denotes the probability assigned by the frozen base language model at position to the candidate wordpiece , is an acceptance threshold, and the maximum subsequence of consecutive acceptances is considered the valid candidate sequence for the current inference , specifically, starting from , one computes and compares to , and stops verification when the first occurs, and defines as , if , i.e. rejected, then is the empty sequence; S550, Output the inference results, including the valid candidate sequences. Concatenate to the input word sequence At the end, an extended prefix sequence is formed, which serves as the input for the next round of inference. Steps S520 to S550 are repeated until the inference result of the frozen basic language model is a stop symbol. <eos>When the number of iterations reaches a preset number of iterations, the inference is stopped, and a final extended prefix sequence is output.< / eos>

Images