Accelerated reasoning method and system for large-scale autoregression language model

Abstract

The invention relates to an accelerated reasoning method and system for a large-scale autoregression language model. The method comprises the following steps: a decoding layer in a target model component inputs a last hidden state vector output by a current round into a language model output layer of the target model component, so that the language model output layer generates an output token of the current round; a word embedding layer of the draft model component executes word embedding processing on the token sequence containing the output token of the current round to obtain a word embedding vector of the token sequence containing the output token of the current round; a normalization layer of the draft model component performs root mean square layer normalization processing on the final hidden state vector and the word embedding vector, and performs one-to-one correspondence splicing on the final hidden state vector and the word embedding vector after normalization processing on the feature dimension to form a joint vector; performing dimension reduction mapping processing on the joint vector by a full connection layer of the draft model component to form an intermediate representation vector; a Res Block nonlinear decoding layer of the draft model component converts the intermediate representation vector into a prediction input vector; and a language model output layer of the draft model component calculates and obtains predicted score distribution of the next token based on the predicted input vector, so that the target model performs verification based on the predicted score distribution to obtain the next token.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computer information processing, and in particular, to an accelerated inference method and system for large-scale autoregressive language models in artificial intelligence, natural language processing and computational linguistics. BACKGROUND

[0002] With the rapid development of natural language processing (NLP) technology, large-scale language models (such as GPT, Qwen, etc.) have shown excellent performance in text generation, question answering, translation and other tasks. However, the high performance of these models is often accompanied by huge computational overhead and resource requirements. In this context, in order to balance performance and efficiency, speculative sampling technology has emerged.

[0003] Autoregressive decoding is the de facto standard for current large language models (LLMs) to generate text. Its basic principle is to generate word tokens one by one, and each newly generated word token depends on all previous word tokens. This inherent sequence dependency leads to a serious performance bottleneck. At each generation of a single word token, a complete forward propagation calculation through the entire model containing tens of billions or even hundreds of billions of parameters is required. This process not only has a huge amount of computation, but is also limited by memory bandwidth, making the inference speed of the model slow and the latency high.

[0004] With the continuous increase in model size, this problem becomes more and more prominent, seriously restricting the application of LLMs in real-time dialogue, interactive applications and large-scale service deployment scenarios. Therefore, the current technical field faces a core challenge: how to effectively reduce the inference latency and computational cost of LLMs without sacrificing the quality of generated text, and avoid excessive dependence on expensive and professional computing hardware.

[0005] In order to solve this bottleneck, in recent years, the method of speculative sampling (also known as speculative decoding) has been proposed. The core idea is to use a small draft model with lower computational cost to pre-generate a candidate word token sequence (draft), and then perform a parallel verification by the original large "target model", so as to accept multiple word tokens in a single forward propagation, achieving acceleration. However, existing speculative sampling methods still have many limitations.

[0006] Speculative sampling is a strategy to optimize generation efficiency. Before sampling with the target model, a draft model with much smaller parameter size is used to predict one or more sequences of draft tokens, which are then input to the target model. The target model then accepts multiple tokens based on the overlap ratio in the sampling stage, and multiple tokens are generated at a time.

[0007] Conventional speculative sampling is divided into two steps: speculation first, then sampling. That is, before sampling with the target model, a draft model with much smaller parameter size is used to predict n consecutive sequences of draft tokens (the parameter n is determined in the system initialization stage and will not be modified in the subsequent system running process). The target model then accepts multiple tokens based on the overlap ratio in the sampling stage, and multiple tokens are generated at a time.

[0008] There are currently various speculative sampling schemes, each with advantages and disadvantages. Medusa predicts multiple future word pieces by adding multiple parallel “decoding heads” to the base LLM. Therefore, it does not need to maintain a completely independent draft model, but its prediction only depends on the hidden state of the target model before generating the current word piece (e.g., the nth word piece), which makes the decoding head completely unaware of the actual state of the n+1th word piece when predicting the n+2th word piece, resulting in a very low word piece acceptance rate. DeepSeek MTP predicts multiple future word pieces by introducing multiple prediction heads. This method maintains the causal chain of the generation process and can significantly improve sampling efficiency, but it requires additional prediction steps and model maintenance, and the integration implementation has certain complexity.

[0009] Therefore, existing methods either need to maintain an additional draft model, increasing system complexity, or are integrated into the main model but have the problem of low acceptance rate or excessive overhead. Therefore, it is still necessary to find a more lightweight and efficient speculative sampling scheme that can improve the acceptance rate while reducing the additional overhead.

[0010] Therefore, how to balance the speed advantage brought by parallel prediction and the accuracy required by maintaining autoregressive dependence, that is, while maintaining lightweight and high parallelism, a novel feature fusion mechanism is required to reintroduce key, adjacent autoregressive information into the prediction process, which is a problem that users and technical personnel in the field hope to solve.

[0011] The above information disclosed in the Background section is only for enhancing the understanding of the background of the present application, and therefore it can include information that does not constitute the prior art that is already known to those of ordinary skill in the art. SUMMARY

[0012] Therefore, the present disclosure provides an accelerated inference method for a large-scale autoregressive language model, comprising: inputting the last hidden state vector output by the current round into the language model output layer of the target model component, so that the language model output layer generates the output token of the current round; the word embedding layer of the draft model component performs word embedding processing on the token sequence containing the output token of the current round to obtain the word embedding vector of the token sequence containing the output token of the current round; the normalization layer of the draft model component respectively performs root mean square layer normalization processing on the last hidden state vector and the word embedding vector, and splices the normalized last hidden state vector and the word embedding vector in a one-to-one corresponding manner to form a joint vector; the fully connected layer of the draft model component performs dimension reduction mapping processing on the joint vector to form an intermediate representation vector; the Res Block nonlinear decoding layer of the draft model component transforms the intermediate representation vector into a predicted input vector; and the language model output layer of the draft model component calculates the predicted score distribution of the next token based on the predicted input vector, so that the target model verifies the predicted score distribution to obtain the next token.

[0013] According to the inference acceleration method for a large-scale autoregressive language model of the present disclosure, the word embedding layer of the draft model is shared with the target model component.

[0014] According to the accelerated inference method for a large-scale autoregressive language model of the present disclosure, the transformation performed by the Res Block nonlinear decoding layer comprises a linear transformation, a SiLU activation, and a residual connection.

[0015] According to the accelerated inference method for a large-scale autoregressive language model of the present disclosure, it further comprises: comparing the predicted result of the language model output layer of the draft model component with the output result of the target model through a cross-entropy loss function, calculating the loss, and updating the model parameters.

[0016] According to the accelerated inference method for a large-scale autoregressive language model of the present disclosure, it further comprises: splitting the coefficient matrix of the fully connected layer by row for parallel processing.

[0017] According to the accelerated inference method for a large-scale autoregressive language model of the present disclosure, it further comprises: splitting the coefficient matrix and the bias of the Res Block nonlinear decoding layer by row for parallel processing.

[0018] According to the acceleration inference method for a large-scale autoregressive language model of the present disclosure, the coefficient matrix of the language model output layer is further split by column for parallel processing.

[0019] According to another aspect of the present disclosure, a target model component and a draft model component are also provided, wherein the decoding layer in the target model component inputs the last hidden state vector output in the current round into the language model output layer of the target model component, so that the language model output layer generates an output token in the current round, the word embedding layer of the draft model component performs word embedding processing on a token sequence containing the output token in the current round to obtain a word embedding vector of the token sequence containing the output token in the current round, the normalization layer of the draft model component respectively performs root mean square layer normalization processing on the last hidden state vector and the word embedding vector, and the normalized last hidden state vector and the word embedding vector are one-to-one spliced in the feature dimension to form a joint vector, the fully connected layer of the draft model component performs dimension reduction mapping processing on the joint vector to form an intermediate representation vector, the ResBlock nonlinear decoding layer of the draft model component transforms the intermediate representation vector into a predicted input vector, and the language model output layer of the draft model component calculates a predicted score distribution of the next token based on the predicted input vector, so that the target model verifies based on the predicted score distribution to obtain the next token.

[0020] According to the acceleration inference system for a large-scale autoregressive language model of the present disclosure, the result predicted by the language model output layer of the draft model component is compared with the result of the output of the target model through a cross-entropy loss function to calculate a loss, so as to update the model parameters.

[0021] According to the acceleration inference system for a large-scale autoregressive language model of the present disclosure, the coefficient matrix of the language model output layer is further split by column for parallel processing.

[0022] By employing the accelerated inference method and system for large-scale autoregressive language models of the present disclosure, twice as many tokens can be generated in one round of inference than existing inference schemes, and under the same evaluation conditions, the acceptance rate of inference results will be more than 50% higher than that of existing inference schemes. Under low concurrency, the output throughput is maximally improved (26%-65%), while under high concurrency, the throughput also reaches 14%; the short text acceptance rate reaches 52%, and the long text acceptance rate reaches 57%. The draft model component of the accelerated inference method and system for large-scale autoregressive language models of the present disclosure, for any model, whether it is a standard Transformer, a variant with MoE / long context modification, or even a new sequence model that is not a Transformer, the training and inference of the draft model component of the present disclosure only rely on a unified and minimalist interface: the previous hidden state vector H and the word embedding vector E of the target next token, and the training code and the inference code can remain consistent, without the need to redevelop for different models. The evolution of the number of layers, attention patterns, position encoding, KV Cache organization, and other aspects of the upstream model Target Model will not "affect" the implementation of the draft model component of the present disclosure, which makes it possible for the downstream inference system to quickly adapt to new models at a very low cost, improving research and online efficiency.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0025] Figure 1 is a schematic diagram of a first embodiment of an accelerated inference system for large-scale autoregressive language models according to an exemplary embodiment.

[0026] Figure 2 is a schematic diagram of an example of an accelerated inference system for large-scale autoregressive language models according to an exemplary embodiment.

[0027] Figure 3 is a flowchart of an accelerated inference method for large-scale autoregressive language models according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the several views and, thus, description of the same elements will not be repeated.

[0029] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the

[0030] The block diagrams in the drawings show only the functionality of the example embodiments and do not imply any particular physical or architectural arrangement of the example embodiments. No inference should be made regarding the architecture or configuration of a device implementing functionality indicated by any of the diagrams based on an example shown and described herein. Also, well-known structures, devices, and operations have not been shown or described in detail in order to avoid obscuring the disclosure.

[0031] The flow diagrams depicted herein are examples of sequences of operations that can be performed by some of the example embodiments. The depicted examples do not limit the scope of the example embodiments. One of ordinary skill in the art would recognize from the above description and associated drawings of examples that the examples are not limited by the illustrated ordering of operations. For example, operations can occur in various orders and / or concurrently, and also other components can be used instead of or in addition to the components illustrated.

[0032] It should be understood that although the terms first, second, third, etc. can be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the present disclosure. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0033] Those skilled in the art will understand that the drawings described herein are merely illustrative and should not be construed as limiting the scope of the present disclosure. The modules or flows in the drawings are not necessarily essential to implement the present disclosure, and thus should not be used to limit the scope of protection of the present disclosure.

[0034] Figure 1 is a schematic diagram of a first embodiment of an accelerated inference system for large-scale autoregressive language models, in accordance with an example embodiment. As Figure 1As shown, an accelerated inference system 100 for large-scale autoregressive language models includes a target model component and a draft model component. The target model component is basically the same as the existing LLM target model component sampling, but the difference is that the transform layer (transformer layer) used in the speculative sampling method in the existing target model is replaced by the decoding layer (decoder layer) disclosed in the present disclosure, that is, the decoding layer in the target model component inputs the last hidden state vector output by the current round into the language model output layer of the target model component, that is Figure 1 the LM head in the present disclosure, so that the language model output layer verifies the candidate dimension sequence (also called draft token) to generate the output token of the current round. Here, the decoding layer (decoder layer) is used to replace the transformer layer (transformer layer) at the same layer position of the conventional LLM model, which can only provide two parameters, i.e., the hidden state parameter output by the current round and the word vector output by the current round, for the draft model mentioned later in the present disclosure, without the need for updating the existing dynamic positionid, input description, KV Cache. It should be noted that the replacement here does not mean that there is no transformer layer (transformer layer) in the target model component of the present disclosure, but emphasizes that only the hidden state output by the decoding layer (decoder layer) included in the transformer layer (transformer layer) is used in the draft model in the present disclosure. The decoding layer is the basic unit of the Transformer decoder, which usually contains a masked multi-head self-attention mechanism to ensure that only generated words can be seen when generating each new word, thereby maintaining the causality of sequence generation. Therefore, in the present disclosure, the decoding layer can be regarded as a special type of transformer layer.

[0035] Specifically, the hidden state sequence H = [h[1], h[2], … h[L]] output by the last layer of the decoding layer (decoder layer) of the target model is taken first (representing the semantics before the current position, the hidden state sequence is composed of the hidden state of each time step, represented as h1, h2,..., hT where T is the sequence length. Each ht contains the context information up to time step t).

[0036] As Figure 1As shown, the word embedding layer of the draft model component performs word embedding processing on the token sequence containing the output token of the current turn, to obtain the word embedding vector of the token sequence containing the output token of the current turn. This is equivalent to shifting the word embedding vector E = [e[1], e[2],…e[L]] obtained by embedding the input token sequence of the output token of the current turn by one position to E = [e[2], e[3],…e[L+1]] for the next round of the dialogue, so that each e[i+1] corresponds to the “next token” of h[i]. As shown in the token sequence t2, t3, t4, t4, t5, t6 displayed outside the draft model component in 1, the token sequence is formed by shifting the token sequence after word embedding, to show the above process.

[0037] As shown in 1, the draft model component includes a word embedding layer, a normalization layer, a fully connected layer, and an output layer. Figure 1 As shown, the normalization layer (RMS normalization, or RMSnorm) of the draft model component respectively performs root mean square layer normalization processing on the last hidden state vector and the word embedding vector. For example, RMSNorm processing is applied to E and H mentioned above. RMSNorm is a commonly used normalization technique for deep learning models, which improves training stability and efficiency by scaling and normalizing the input of the neural network layer, and is scale-invariant.

[0038] As shown in 1, the draft model component includes a word embedding layer, a normalization layer, a fully connected layer, and an output layer. Figure 1 As shown, the normalization layer of the draft model component concatenates the normalized last hidden state vector and the word embedding vector in a one-to-one manner in the feature dimension to form a joint vector. Feature concatenation: multiple input vectors (such as from different neural network layers or feature sources) are connected to form a higher-dimensional joint vector. For example, (h[i], e[i+1]) of E and H are concatenated in the feature dimension to form a joint vector. For example, assuming H = [h1, h2, h3, h4] and E = [e2, e3, e4, e5], the joint vector = [[h1, e2],[h2, e3], [h3, e4], [h4,e5]]. This concatenation jointly represents the current time feature and the next time feature.

[0039] As shown in 1, the draft model component includes a word embedding layer, a normalization layer, a fully connected layer, and an output layer. Figure 1As shown, the full connection layer of the draft model component performs dimension reduction mapping processing on the joint vector to form an intermediate representation vector. Dimension reduction mapping is an operation commonly used for feature integration and dimension compression in deep learning. The Fully Connected Layer Mapping takes the joint vector as input (denoted as vector x with dimension inDin) and inputs it into the full connection layer. The full connection layer realizes mapping through the linear transformation formula y=f(Wx+b), and by setting the output dimension out<inDout<Din, the weight matrix compresses the input information to realize dimension reduction. This is similar to learning a mapping function f:x→y that projects high-dimensional features into a low-dimensional space, preserving key information and removing noise. After dimension reduction, the output dimension is lower, reducing computational overhead and memory requirements and improving model efficiency. In the present disclosure, the joint vector after splicing is input into a full connection layer for dimension reduction mapping to obtain an intermediate representation vector I that integrates the semantic information of the above text and the candidate token information. This full connection transformation can be represented as: I = [H, E] * W^T Where W is the projection matrix, and "^T" represents matrix transposition. At this time, I contains the mixed representation of the current context and the content of the next token. The projection matrix is essentially a mathematical tool for "projecting" data from one space to another (usually a low-dimensional space) while preserving key information. It is widely used in machine learning, computer vision, natural language processing, etc., for example, to achieve dimension reduction compression, such as reducing 1000-dimensional data to 100-dimensional data to reduce computational load (e.g., PCA principal component analysis); complete feature extraction, such as projecting word vectors into different spaces (Query / Key / Value space) in Transformers to extract different features; perform spatial transformation, such as projecting three-dimensional object coordinates to a two-dimensional screen in 3D graphics rendering.

[0040] Subsequently, the Res Block nonlinear decoding layer of the draft model component transforms the intermediate representation vector into a predicted input vector. The Res Block nonlinear transformation takes the vector I as input into the lightweight Res Block module, performs linear transformation, SiLU activation function, and residual connection to calculate the updated vector X. Formally: X = I + SiLU(I * V^T + B) where V and B are the weights and biases inside the Snake Head of the present disclosure. This nonlinear transformation enables the model to capture more complex interactions between the context and the candidate token.

[0041] Res Block The nonlinear transformation of Res Block (Residual Block) is the core component of ResNet (Residual Network), which introduces nonlinearity inside the block through convolution operations, activation functions, and normalization methods, enabling the model to learn complex patterns. In the Res Block nonlinear transformation, V and B are the weight matrix and bias vector inside the Snake Head (draft model), which are learned through the model training process. The shape of V is (output_dim, input_dim), and the shape of B is (output_dim,). They together constitute the parameters of the linear transformation. The specific content of the linear transformation is to multiply the input vector I with the transpose of the weight matrix V and add the bias B, i.e., I·V^T + B. This transformation maps the input vector to a new feature space through matrix multiplication, achieving feature recombination and dimension transformation. The SiLU activation function (also known as the Swish function) is defined as f(x) = x·σ(x), where σ(x) is the sigmoid function. Its role is to introduce nonlinearity into the model, enabling the network to learn more complex patterns. The SiLU function combines the characteristics of linear functions and sigmoid functions, maintaining gradient smoothness and achieving a gating mechanism-like effect. The residual connection forms the structure of X = I + SiLU(I·V^T + B) by directly adding the original input I to the transformed result. This connection effectively alleviates the gradient vanishing problem in deep networks, ensuring that information can be effectively propagated in the network. In specific implementation, the residual connection requires that the input I and the dimension of the transformed output must be consistent. The capture of this interaction is reflected in that the linear transformation is responsible for feature combination, the SiLU activation introduces nonlinear interaction, and the residual connection maintains the integrity of the information. When the input I undergoes linear transformation by V and B, the features of the candidate token interact in the transformed space, and the SiLU function further enhances the complexity of this interaction. The interaction is finally reflected in the updated vector X, which retains the original information and contains new feature relationships learned through transformation.

[0042] Finally, as Figure 1As shown, the language model output layer of the draft model component calculates a predicted score distribution of the next token based on the predicted input vector, so that the target model verifies the next token based on the predicted score distribution. Specifically, the language model output layer of the draft model component outputs the predicted logits of the next token by inputting X into the LM Head (language model output layer) of the original model: L = X * U^T where L is the predicted logits score distribution of the next token, and U is the weight of the language model output layer (LM Head).

[0043] Alternatively, in order to enhance the prediction ability of the system, the present disclosure further adds a cross-entropy loss function after the draft model component, compares the output predicted logits with the result token selected after verification of the target model component, calculates the loss Loss between the predicted logits and the real next token, and through such training, the draft model component (SnakeHead) of the present disclosure has the ability to predict the next token given the current context and a candidate token.

[0044] Because the present disclosure adopts a decoding layer instead of a transform layer in the target model, i.e., discards the transform layer, and adopts a lightweight ResBlock (1-layer linear transformation + SiLU activation + residual connection) instead of a transform layer in the draft model component, the defects of high computation and high memory requirement of the traditional LLM inference system in the training and inference stages are eliminated. Specifically, as mentioned earlier, because there is no transformer structure, the draft model component (snake head) only needs to provide 2 parameters, i.e., hidden states H and embedding vector E of nexttoken, without dynamic position id, input description, KV Cache update. Alternatively, the draft model component uses the original model's embedding layer when doing embedding, i.e., Snake and the original model share embedding weights, without additional overhead. In traditional LLM systems, such as DeepSeek and EAGLE LLM systems with transformer structure, position id (position encoding information), kv chace (including kv caches corresponding request length, request page id, kv caches tiling, and mapping, length, page, position pointer, etc. information), inputdescription (including request length, request offset, etc. information), and return logitsindices (return which position token) need to be provided. This inevitably leads to high computation and memory requirement. And the preparation steps of the above data of these traditional systems are all on the CPU, which is very cumbersome, and because GPU information is needed, information synchronization is needed, which will cause GPU operation interruption and cuda graph break (dynamic batchlength / position id will change kernel parameters, which is easy to cause cuda graph break, graph break will cause performance degradation, change a large graph into 2 small graphs, which is not conducive to graph memory reuse, and there will be graph checking before the graph enters, and one more check will waste a lot of CPU time, while the GPU is idle, in addition, there is the risk of memory overflow) and other consequences. These consequences do not exist in the system of the present disclosure.

[0045] In comparison, the draft model component (snake head) of the present disclosure only needs to provide 2 parameters. Therefore, the input parameters are less, not only saving the time of preparing parameters, but also without the need of preparing kvcaches due to the absence of a transformer, thus not requiring synchronization between the cpu and the gpu. In addition to saving synchronization time, further, it will not cause a broken graph, thus saving more time.

[0046] The actual test comparison results show that, in the training + inference stage, compared with the QwQ-32B-fp8 model of the present disclosure, the newly added parameters introduced by the draft model component (snake head) only account for 2% of the original model, the parameter overhead is extremely low (if it is a transformer structure, more than 4% of the parameters will be added), and if it is a larger model, the proportion of the newly added parameters will be smaller, within 0.5%. Since the draft model component replaces a transformer block with a snake head, compared with a transformer layer, the newly added calculation amount is extremely low and can be ignored. In terms of time, due to the reduction of the calculation time of a transformer layer, the preparation time of parameters, the synchronization time between the cpu and the gpu, and the overhead of a broken graph, in addition, in terms of memory, not only the parameter memory of a transformer layer is reduced, but also the kv cache memory of the transformer layer is reduced.

[0047] Compared with the traditional LLM system, such as Medusa Head, the input of the draft model of the present disclosure additionally introduces semantic information of the next token, i.e., the vector Hidden States (H) output at the end of the transformer layers (decoding layers in the present disclosure) of the target model, which contains the semantic information of the entire previous text. On the one hand, the target model component will generate the token output of the current round according to the Hidden States. The token of the current round is input into the word embedding layer to obtain the word embedding vector (E) representing the current semantic information after RMSNorm (maintaining the numerical value in a stable scale to ensure the numerical stability of subsequent calculations and suppress the problems of gradient explosion / disappearance, etc.). The word embedding vector (E) is combined with the Hidden States (also processed by RMSNorm) and passed through a full connection layer (Full Connection Layer) for dimension reduction, so as to obtain a vector with previous semantic + current semantic information, and the next token is predicted based on the vector, and the accuracy will be greatly improved (about 45% to 62%). The traditional Medusa method does not know the token information generated in the current round, so the accuracy of the predicted next token is very poor, usually between 20% and 40%.

[0048] In summary, by adding the hidden state of the nth round token and the information of the n+1 round token in the input of the draft model component of the present disclosure, the draft model can obtain more abundant prediction conditions, thereby increasing the acceptance rate of the prediction of the n+2 round token. More importantly, since the draft model component of the present disclosure only uses the hidden state output by the last decoding layer of the target model and the word embedding vector of the predetermined number of token sequences of the current round, it is not necessary to dynamically update the position id, input description, and KV Cache as in the existing LLM text generation system, which greatly reduces the computational load.

[0049] Alternatively, in order to solve the problem of insufficient GPU memory to support long text and high batch training during training when using the acceleration inference system for large-scale autoregressive language model of the present disclosure, tensor parallelism can be implemented in the training stage.

[0050] Specifically, different layers of the Snake Head component are respectively subjected to different splitting of the coefficient matrix. For example, the full connection layer coefficient matrix is split by row, the ResBlock coefficient matrix and bias are split by row, and the LM Head coefficient matrix is split by column.

[0051] By splitting the full connection layer by row, the update logic of the weight of the RMSNorm block during training is simpler, and the TP size (the number of split blocks) is more flexible. For example, now it is [H, E] splicing. If it is split by column, the vector of the splicing is preferably set to 2, so that two TP blocks can respectively obtain [H] and [E] to calculate, and the update of the RMSNorm is simple. However, if the TP size is 4, it will be divided into [H1], [H2], [E1], [E2] 4 blocks, and each TP block takes one. The update logic of the RMSNorm weight is complex, and the training difficulty of the RMSNorm weight will be higher if the TP size is set to a non-2 exponential (3, 5, 6, etc.). In contrast, by splitting by row, each TP block will save all the [H, E] vectors, and the update of the RMSNorm weight is more direct.

[0052] By splitting the ResBlock layer by column, only one communication transmission (Communication) is required from the full connection layer to the ResBlock stage, which reduces one transmission compared to other splitting methods, from twice AllGather to once AllReduce. Although the communication amount is the same, the memory occupation and bandwidth overhead of calling AllReduce only once are greatly reduced, and the communication waiting time and delay time are reduced by communicating only once, and the calculation is more efficient. In addition, it also ensures that each TP block from the ResBlock block to the LM block is independent during the training process, that is, each TP block only needs the local calculation result to correct the locally stored weight parameters, and the logic is simple when doing back propagation.

[0053] By splitting the LM Head layer by row, the memory occupation of each TP device can be reduced, and the data communication transmission (Communication) is minimized. In this way, when using the parallel Parallel Cross Entropy Loss to calculate the loss, local gradient calculation can be performed to ensure the smooth progress of back propagation during training.

[0054] Alternatively, as Figure 1It is shown that long text and high batch training can be further supported by flow level row (PP) and multi-processing. Specifically, multi-processing is used in the target model component (producer) and the draft model component (consumer) respectively, and the target model component and the draft model component transmit data asynchronously through a queue, i.e., the hidden state of the previous round and the token sequence word embedding vector of the current round.

[0055] In this way, by using two-stage pipeline parallelism and queue asynchronous transmission, the target model component can immediately jump to the next round of data generation after generating one round of data without waiting for the draft model component to complete subsequent training of the data, which can improve the utilization of the graphics card and improve the training efficiency, and also reduce the waiting time of the draft model component, because the target model component has already generated the next round of data when the draft model component starts processing the data.

[0056] In addition, the present disclosure can use multi-processing to bind each process to a GPU graphics card, which makes the algorithm design difficulty low and easy to achieve uniform allocation of memory and computing capacity, and can use the NCCL communication library to speed up the transmission of cross-GPU data. For example, tests show that, compared with a traditional inference system, the target model component uses TP size = 4, and the draft model component uses TP size = 2, i.e., the layers of the draft model component are divided into two graphics cards, which can increase the length of the processable text from 4k to 8k, and the batch size can be increased from 1 to 2. Further, under the same batch size, the peak memory usage is reduced by more than 45%. In terms of training of the draft model component, the queue asynchronous transmission brought by PP and the additional throughput improvement make the training speed increase by 10%. At the same time, the applicant found that the larger the TP size of the draft model component, the larger the batch size that can be processed at the same time.

[0057] Figure 2 is a schematic diagram of an example of an accelerated inference system for a large-scale autoregressive language model according to an example embodiment. As shown Figure 2 As shown, when a text inference request is received, for example, "What did you do today?" In the case that the entire model has been inferred to generate "I today", it will continue to generate two possibilities: "I ate an apple today." and "I went to the park today.", where tokens are "I", "today", "ate" / "went", "apple" / "park", ".".

[0058] In this disclosure, in the current round, the Target Model decides what to generate from all possible answers based on its defined decoding strategy (Greddy, top-k, top-p, …). The model doesn't know which token it will generate beforehand; simply put, in the current round, the model has a 60% probability of generating "eaten" and a 40% probability of generating "went". Therefore, if the generation of the token in the current round is unknown, predicting the token for the next round becomes very difficult. For example, the traditional Medusa might generate "park", but the generation for this round would be "ate the park" (this token will not be received; the actual output is still "ate"). However, using a method like… Figure 2 The system disclosed herein, when making predictions in the next round, informs the draft model (i.e., Snake Head) of the token selected in the current round (assuming "eaten" is generated here). This gives the draft model a higher probability of generating the answer "apple" instead of "park" because the draft model component receives the additional information about "eaten" and concatenates it with its previous hidden states. Since "eaten" followed by "apple" is more logical, its acceptance probability is higher. The actual output for this round is "eaten apple". Compared to Medusa's speculative sampling, which generates only one token per round, this system generates two tokens per round.

[0059] Compared to Medusa Head under the same request conditions, the new parameters in this disclosure account for only 0.12% of the original model, having almost no impact on inference efficiency. Regarding acceptance rate, it is significantly improved compared to Medusa: on four typical benchmarks (Chinese dialogue, English dialogue, and code generation), the original Medusa Head's candidate token acceptance rate was only 22% / 36% / 40%. After introducing the draft model component of this disclosure, the acceptance rate under the same evaluation conditions increased to 52% / 64% / 66%, respectively. Furthermore, it achieved a 64% acceptance rate for mathematical problems, showing a significant improvement in acceptance rate across all environments. Specifically, the improvement was as high as 2.36 times in Chinese scenarios and approximately 1.77 times in English and code scenarios. Compared to no speculative sampling, the draft model component disclosed herein reduces the average generation latency per token (TPOT) by 8-25% overall; first-byte latency (TTFT) has no negative impact; output throughput is improved by the most (26%-65%) under low concurrency, while throughput under high concurrency reaches 14%; short text reception rate reaches 52%, and long text reception rate reaches 57%.

[0060] Figure 3 is a flowchart of an accelerated inference method for a large-scale autoregressive language model according to an example embodiment. As shown in Figure 1 the accelerated inference method for a large-scale autoregressive language model, first, at step S110, the decoding layer in the target model component inputs the last hidden state vector output by the current round into the language model output layer of the target model component, so that the language model output layer generates the output token of the current round. Subsequently at step S120, the word embedding layer of the draft model component performs word embedding processing on the token sequence containing the output token of the current round to obtain the word embedding vector of the token sequence containing the output token of the current round. At step S130, the normalization layer of the draft model component respectively performs root mean square layer normalization processing on the last hidden state vector and the word embedding vector. Subsequently at step S140, the normalized last hidden state vector and the word embedding vector are spliced in a one-to-one correspondence in the feature dimension to form a joint vector. Next, at step S150, the fully connected layer of the draft model component performs dimension reduction mapping processing on the joint vector to form an intermediate representation vector. Subsequently, at step S160, the Res Block nonlinear decoding layer of the draft model component transforms the intermediate representation vector into a predicted input vector. Finally, at step S170, the language model output layer of the draft model component calculates the predicted score distribution of the next token based on the predicted input vector, so that the target model verifies based on the predicted score distribution to obtain the next token.

[0061] In summary, compared with the conventional speculative sampling method and system, by adopting the accelerated inference method and system for large-scale autoregressive language model of the present disclosure, twice as many tokens as the existing inference scheme can be generated in one round of inference, and under the same evaluation conditions, the acceptance rate of the inference result will be increased by more than 50% compared with the existing inference scheme. In low concurrency, the output throughput is maximized (26%-65%), and in high concurrency, the throughput also reaches 14%; the short text acceptance rate reaches 52%, and the long text reaches 57%. The draft model component of the accelerated inference method and system for large-scale autoregressive language model of the present disclosure is applicable to any model, whether it is a standard Transformer, a variant with MoE / long context modification, or even a new sequence model that is not a Transformer. The training and inference of the draft model component of the present disclosure only rely on a unified and minimalist interface: the previous hidden state vector H and the word embedding vector E of the target next token, and the training code and inference code can remain consistent, without the need to redevelop for different models. The evolution of the upstream model Target Model in terms of the number of layers, attention form, position encoding, KV Cache organization, etc. will not "affect" the implementation of the draft model component of the present disclosure, which makes it possible for the downstream inference system to quickly adapt to new models at a very low cost, improving the research and development and online efficiency.

[0062] Those skilled in the art can understand that each of the above modules can be distributed in the device as described in the embodiments, or can be changed in one or more devices different from the embodiments. The modules of the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0063] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a plurality of instructions to make a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

[0064] The example embodiments of the present disclosure are specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present disclosure is intended to cover various modifications and equivalent arrangements within the spirit and scope of the appended claims.

Claims

1. An accelerated inference method for large-scale autoregressive language models, comprising: The decoding layer in the target model component inputs the last hidden state vector of the current round into the language model output layer of the target model component so that the language model output layer can generate the output token for the current round. The word embedding layer of the draft model component performs word embedding processing on the token sequence containing the output token of the current round to obtain the word embedding vector of the token sequence containing the output token of the current round. The normalization layer of the draft model component performs root mean square normalization on the final hidden state vector and the word embedding vector respectively, and then concatenates the normalized final hidden state vector and the word embedding vector in a one-to-one correspondence on the feature dimension to form a joint vector. The fully connected layer of the draft model component performs dimensionality reduction mapping on the joint vector to form an intermediate representation vector; The Res Block nonlinear decoding layer of the draft model component transforms the intermediate representation vector into a prediction input vector; The language model output layer of the draft model component calculates the predicted score distribution of the next token based on the predicted input vector, so that the target model can verify the next token based on the predicted score distribution.

2. The method for accelerating inference in large-scale autoregressive language models as described in claim 1, wherein the word embedding layer of the draft model is shared with the target model components.

3. The accelerated inference method for large-scale autoregressive language models as described in claim 1, wherein, The transformations performed by the ResBlock nonlinear decoding layer include: a linear transformation, SiLU activation, and residual connections.

4. The accelerated inference method for large-scale autoregressive language models as described in claim 1, further comprising: The loss is calculated by comparing the predictions of the language model output layer of the draft model component with the output of the target model using the cross-entropy loss function, in order to update the model parameters.

5. The accelerated inference method for large-scale autoregressive language models as described in claim 1, further comprising: The coefficient matrix of the fully connected layer is split row by row and processed in parallel.

6. The accelerated inference method for large-scale autoregressive language models as described in claim 1, further comprising: The coefficient matrix and bias of the Res Block nonlinear decoding layer are split row-wise and processed in parallel.

7. The accelerated inference method for large-scale autoregressive language models as described in claim 1, further comprising: The coefficient matrix of the language model output layer is split column-wise for parallel processing.

8. An accelerated inference system for large-scale autoregressive language models, comprising: The system comprises a target model component and a draft model component. The decoding layer in the target model component inputs the last hidden state vector of the current round's output into the language model output layer of the target model component, enabling the language model output layer to generate the output token for the current round. The word embedding layer in the draft model component performs word embedding processing on the token sequence containing the output token of the current round, obtaining a word embedding vector for the token sequence containing the output token of the current round. The normalization layer in the draft model component performs root mean square normalization on both the last hidden state vector and the word embedding vector, and then concatenates the normalized last hidden state vector and word embedding vector in a one-to-one correspondence along the feature dimension to form a joint vector. The fully connected layer in the draft model component performs dimensionality reduction mapping on the joint vector to form an intermediate representation vector. The Res Block nonlinear decoding layer in the draft model component transforms the intermediate representation vector into a prediction input vector. Finally, the language model output layer in the draft model component calculates the prediction score distribution for the next token based on the prediction input vector, allowing the target model to verify the prediction score distribution and obtain the next token.

9. The accelerated inference system for large-scale autoregressive language models as described in claim 8, wherein the results predicted by the language model output layer of the draft model component are compared with the results of the target model output by a cross-entropy loss function to calculate the loss, thereby updating the model parameters.

10. The accelerated inference system for large-scale autoregressive language models as described in claim 8, wherein the coefficient matrix of the fully connected layer is partitioned by rows, the coefficient matrix and bias of the Res Block nonlinear decoding layer are partitioned by rows, or the coefficient matrix of the language model output layer is partitioned by columns.

Images