Decoding acceleration method, device, equipment, computer-readable storage medium and product

Abstract

The present disclosure provides a decoding acceleration method, apparatus, device, computer-readable storage medium and product, which relate to the field of artificial intelligence, and in particular to the field of deep learning. The specific implementation scheme is as follows: obtaining the text to be predicted sent by the terminal device; inputting the text to be predicted into the encoding layer preset in the large language model, and obtaining the first hidden state corresponding to the text to be predicted output by the encoding layer; performing decoding operations based on the first hidden state through the language model head in the large language model and the preset multi-head model to generate multiple predicted texts, wherein the multi-head model includes multiple decoding modules, the decoding module includes a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head; performing verification operations on multiple predicted texts based on the large language model, and sending the multiple target predicted texts that have passed the verification to the terminal device. This improves the speed of the large language model in the decoding stage and improves the efficiency of the large language model in data processing.

Description

Technical Field

[0001] The present disclosure relates to deep learning in artificial intelligence, and more particularly to a decoding acceleration method, apparatus, device, computer-readable storage medium, and product. Background Art

[0002] The current mainstream large language model prediction method is the autoregressive decoding scheme. When predicting a text, this scheme mainly includes two stages: the first stage is the prefill stage (encoding stage), and the second stage is the decode stage (decoding stage).

[0003] After acquiring the hidden state generated during the encoding phase, the large language model performs a forward pass during the decoding phase, outputting only one predicted text unit (token). This process continues until all tokens have been output. However, this decoding method is often inefficient, resulting in a lengthy prediction process. Summary of the Invention

[0004] The present disclosure provides a decoding acceleration method, apparatus, device, computer-readable storage medium, and product for accelerating the decoding stage of a large language model.

[0005] According to a first aspect of the present disclosure, a decoding acceleration method is provided, comprising:

[0006] Obtain the text to be predicted sent by the terminal device;

[0007] Inputting the text to be predicted into a coding layer preset in a large language model, and obtaining a first hidden state corresponding to the text to be predicted output by the coding layer;

[0008] Generate multiple predicted texts by performing a decoding operation based on the first hidden state using a language model head in a large language model and a preset multi-head model, wherein the multi-head model includes multiple decoding modules, each of which includes a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head;

[0009] A verification operation is performed on the multiple predicted texts based on the large language model, and multiple target predicted texts that pass the verification are sent to the terminal device.

[0010] According to a second aspect of the present disclosure, a decoding acceleration device is provided, comprising:

[0011] An acquisition module, used to acquire the text to be predicted sent by the terminal device;

[0012] an encoding module, configured to input the text to be predicted into an encoding layer preset in a large language model, and obtain a first hidden state corresponding to the text to be predicted output by the encoding layer;

[0013] a decoding module, configured to generate a plurality of predicted texts by performing a decoding operation based on the first hidden state using a language model head in a large language model and a preset multi-head model, wherein the multi-head model includes a plurality of decoding modules, each of which includes a multi-layer perceptron and a language model head, and the plurality of decoding modules reuse the language model head;

[0014] A verification module is used to perform a verification operation on the multiple predicted texts based on the large language model, and send the multiple target predicted texts that pass the verification to the terminal device.

[0015] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0021] The technology disclosed herein improves the speed of a large language model in the decoding stage and improves the efficiency of data processing by the large language model.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0024] Figure 1 This is a diagram of the system architecture on which the embodiments of the present disclosure are based;

[0025] Figure 2 A flowchart of a decoding acceleration method provided in an embodiment of the present disclosure;

[0026] Figure 3 A flowchart of a decoding acceleration method provided in yet another embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of a model architecture provided for an embodiment of the present disclosure;

[0028] Figure 5 A flowchart of a decoding acceleration method provided in yet another embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of another model architecture provided in an embodiment of the present disclosure;

[0030] Figure 7 A flowchart of a decoding acceleration method provided in yet another embodiment of the present disclosure;

[0031] Figure 8 A schematic diagram of the structure of a decoding acceleration device provided in an embodiment of the present disclosure;

[0032] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] Glossary:

[0035] Large Language Model (LLM): A large-scale neural network model based on deep learning technology, specifically designed for processing and generating natural language text. Large language models typically have very large parameter sizes, typically containing hundreds of millions to hundreds of billions of parameters. This scale enables them to capture complex language patterns and semantic relationships.

[0036] Token: During the decoding process of large language models (such as GPT), the generated "token" is generally the basic unit of text generated by the model. In the Chinese context, a token can be a Chinese character, a word, or a subword (i.e., part of a word), depending on the word segmentation or encoding method used by the model.

[0037] Multi-layer perceptron (MLP): A classic feed-forward artificial neural network architecture. It consists of at least three layers: an input layer, one or more hidden layers, and an output layer. Each layer consists of several neurons (or nodes), and neurons between layers are fully connected. It can be used for a variety of tasks, including classification, regression, and pattern recognition.

[0038] Language Model Head (LM Head): Usually refers to the output layer of the model, which can map the hidden state of the model to the probability distribution of each word in the vocabulary.

[0039] Hidden state refers to the intermediate representation generated by the model's internal layers (such as hidden layers) when processing input data. Hidden state is passed between layers of the model to capture the characteristics and context of the input data.

[0040] The present disclosure provides a decoding acceleration method, apparatus, device, computer-readable storage medium, and product, which are applied to the field of deep learning in the field of artificial intelligence to achieve an acceleration effect in the decoding stage of large language models.

[0041] The current mainstream large language model prediction method is autoregressive decoding. This method involves two phases when predicting a text: the pre-filter phase (encoding phase) and the decode phase (decoding phase). During the decoding phase, the large language model performs a forward calculation during inference, outputting only one token. This process continues until all tokens have been output. This decoding method is relatively inefficient.

[0042] To improve the decoding efficiency of large language models during prediction, a parallel decoding solution has been proposed. For example, draft tokens can be generated by using smaller models with the same vocabulary or series. During inference with a large model, larger models often take longer to decode a single token (one decoding step), while smaller models can decode a single token much faster.

[0043] However, using this method for parallel decoding requires loading a small model during actual inference deployment of a large language model, significantly increasing graphics memory overhead. Furthermore, large batch sizes can easily trigger computational bottlenecks, resulting in no speedup compared to the original method.

[0044] In the process of solving the above technical problems, the inventors discovered through research that in order to improve decoding speed, a multi-head model can be designed based on the original large language model. In this way, during a forward data processing, multiple tokens can be quickly generated at once based on this multi-head model.

[0045] Furthermore, the multi-head model can include multiple decoding modules, each of which reuses the language model head from the larger language model. This saves a significant amount of additional graphics memory overhead associated with the multi-head model, further expanding the application scope of accelerated decoding based on the multi-head model.

[0046] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0047] In order to enable readers to have a deeper understanding of the implementation principle of this disclosure, the following Figures 1-9 The embodiments of the present disclosure are further refined.

[0048] Figure 1 This is a system architecture diagram based on the embodiment of the present disclosure, such as Figure 1 As shown, the system architecture based on which the present disclosure is based includes at least a terminal device 11 and a server 12. The server 12 is provided with a decoding acceleration device, which can be written in languages ​​such as C / C++, Java, Shell or Python.

[0049] Based on the above system architecture, a user can input text to be predicted through a terminal device 11. After obtaining the text to be predicted, the server 12 can use the large language model to extract the hidden state associated with the text to be predicted. Using the preset language model head and the multi-head model, multiple predicted texts are generated simultaneously based on the hidden state, accelerating the decoding phase. Furthermore, the large language model can verify multiple predicted texts simultaneously, further improving the data processing speed of the large language model.

[0050] Figure 2 A flowchart of a decoding acceleration method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the method includes:

[0051] Step 201: Obtain the text to be predicted sent by the terminal device.

[0052] The embodiment of the present invention is implemented by a decoding acceleration device, which can be coupled to a server. The server can be connected to a terminal device for communication and thus can exchange information with the terminal device.

[0053] The Big Language Model is a large-scale neural network model based on deep learning technology, specifically designed for processing and generating natural language text. Users can input text to be predicted into the Big Language Model based on their needs. The Big Language Model then processes the text and generates predictions.

[0054] Optionally, the user may input the text to be predicted on the terminal device, so the decoding acceleration device may obtain the text to be predicted sent by the terminal device.

[0055] For example, users can input a question text into the large language model based on actual needs. The large language model can extract the hidden state based on the question text and generate predicted text based on the decoding operation of the hidden state. The predicted text can then be used to implement the answer operation to the question text.

[0056] Step 202: Input the text to be predicted into a coding layer preset in a large language model, and obtain a first hidden state corresponding to the text to be predicted output by the coding layer.

[0057] In this embodiment, the large language model includes a preset encoding layer, which can extract hidden features of the text to be predicted input by the terminal device to generate a first hidden state corresponding to the text to be predicted.

[0058] Optionally, the encoding layer can perform data processing on the predicted text through self-attention mechanism, multi-head attention mechanism, feedforward neural network, layer normalization, residual connection and position encoding, so that the encoding layer can effectively process and convert the input data, capture complex semantic and grammatical relationships, and generate the first hidden state corresponding to the text to be predicted.

[0059] Step 203: Perform decoding operations based on the first hidden state through the language model head in the large language model and the preset multi-head model to generate multiple predicted texts, wherein the multi-head model includes multiple decoding modules, and the decoding modules include a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head.

[0060] The current prediction method for large language models is an autoregressive decoding scheme. Generally, only one prediction is generated during each decoding process, and this process is repeated until all predictions are generated. However, autoregressive decoding schemes are often inefficient.

[0061] In this embodiment, to accelerate decoding, a multi-head model capable of decoding can be pre-trained. This multi-head model can include multiple decoding modules, each capable of decoding and generating a predicted text. Therefore, multiple predicted texts can be generated during a single decoding process.

[0062] The decoding module can include a multi-layer perceptron and a language model head. Multiple decoding modules in a multi-head model can reuse the language model head of the large language model. This can save the memory overhead of the multi-head model. In addition, the multi-layer perceptron can be composed of a residual network. The additional multi-layer perceptron uses the unnormalized prediction values ​​(logits) output by the large model for self-distillation training, so that each decoding module learns the ability to predict the next predicted text.

[0063] Therefore, after obtaining the hidden state associated with the text to be predicted, a decoding operation can be performed based on the first hidden state by using the language model head in the large language model and the preset multi-head model to generate multiple predicted texts.

[0064] Among them, the first predicted text can be generated by the language model head in the large language model, and subsequently multiple predicted texts can be generated based on the previous predicted text and the hidden state by each decoding module in the multi-head model.

[0065] Step 204: Verify the multiple predicted texts based on the large language model, and send the multiple target predicted texts that pass the verification to the terminal device.

[0066] In this embodiment, after a plurality of predicted texts are generated by using the language model head and the multi-head model, further verification operations may be performed on the predicted texts.

[0067] Therefore, the multiple predicted texts and the text to be predicted can be spliced ​​and input into the large language model, so that the large language model can perform a verification operation on the multiple predicted texts to determine whether the predicted text is usable.

[0068] Furthermore, multiple target predicted texts that have passed verification may be determined from the multiple predicted texts, and the multiple target predicted texts may be sent to the terminal device, so that the user can view the multiple target predicted texts on the terminal device.

[0069] For example, after receiving a question text sent by a user via a terminal device, the large language model can be used to predict the response text. To improve prediction speed, the language model head and multi-head model can perform multi-head parallel decoding on the hidden state associated with the question text, generating multiple predicted texts in a single forward data processing pass. Among the multiple predicted texts, the predicted text that passes the large language model verification is selected as the response text. The response text is then sent to the terminal device for the user to view.

[0070] The decoding acceleration method provided in this embodiment pre-trains a multi-head model, enabling simultaneous generation of multiple predicted texts during the decoding phase based on the multiple decoding modules within the multi-head model. Furthermore, by enabling multiple decoding modules to reuse the language model heads within a large language model, the significant additional graphics memory overhead associated with the multi-head model can be effectively reduced.

[0071] Optionally, based on any of the above embodiments, the method further includes:

[0072] A preset training data set is obtained, wherein the training data set includes multiple training data, the training data includes original text and annotation information, and the annotation information is multiple texts generated by the large language model through autoregressive decoding.

[0073] The training data is input into the multi-head model to obtain a prediction result output by the multi-head model.

[0074] The loss value corresponding to the multi-head model is calculated based on the prediction results and the annotation information.

[0075] Determine whether the multi-head model meets a preset convergence condition based on the loss value.

[0076] If so, it is determined that the multi-head model has converged, and the training operation of the multi-head model is completed.

[0077] If not, reverse gradient adjustment is performed on the parameters associated with the multi-head model based on the loss value, and the step of inputting the training data into the multi-head model to obtain the prediction result output by the multi-head model is returned to, until the multi-head model meets the preset convergence conditions, and the training operation of the multi-head model is completed.

[0078] In this embodiment, before performing a decoding operation based on the multi-head model, it is first necessary to perform a training operation on the multi-head model so that the multi-head model has decoding capabilities.

[0079] Optionally, a preset training data set may be obtained, wherein the training data set includes a plurality of training data, each training data includes original text and annotation information, and the annotation information is a plurality of texts generated by the large language model through autoregressive decoding.

[0080] After obtaining the training dataset, the multi-head model can be iteratively trained using the training dataset.

[0081] Optionally, for each training data, the original text in the training data can be input into the multi-head model to obtain the prediction result output by the multi-head model. The prediction result is obtained by decoding the hidden state associated with the original text by the multi-head model.

[0082] The loss value corresponding to the multi-head model is calculated based on the prediction results and the annotation information. Among them, any loss value algorithm can be used to calculate the loss value corresponding to the multi-head model, and this disclosure does not limit this.

[0083] Furthermore, the loss value can be used to determine whether the multi-head model meets a preset convergence condition. This convergence condition includes, but is not limited to, the loss value being less than a preset loss value threshold, the difference between the loss values ​​of two rounds of training being less than a preset difference threshold, etc. Alternatively, the convergence condition can also include the training duration reaching a preset duration threshold, the number of training rounds reaching a preset number threshold, etc. This disclosure is not limited to this.

[0084] If so, the multi-head model is determined to have converged, indicating that it can accurately perform decoding operations, and the training operation for the multi-head model can be completed. Otherwise, it indicates that the multi-head model does not currently have the ability to accurately decode. Based on the loss value, the parameters associated with the multi-head model are reversely adjusted. The process returns to the step of inputting training data into the multi-head model and obtaining the prediction results output by the multi-head model, until the multi-head model meets the preset convergence conditions, and the training operation for the multi-head model is completed.

[0085] The decoding acceleration method provided in this embodiment pre-acquires a training dataset containing original text and annotation information, thereby enabling iterative training of a multi-head model based on this training dataset. This enables the multi-head model to have decoding capabilities similar to those of a large language model. When a preset convergence condition is met, iterative training of the multi-head model is terminated. This enables the multi-head model to accurately generate predicted text based on the input data that is similar to the predicted text generated by the large language model using autoregressive decoding. This improves the data processing accuracy of the multi-head model.

[0086] Further, based on any of the above embodiments, step 203 includes:

[0087] A decoding operation is performed on the first hidden state through the language model head to obtain a first predicted text obtained by a first round of decoding.

[0088] Multiple decoding modules in the multi-head model sequentially generate multiple second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state.

[0089] The first predicted text and the plurality of second predicted texts are determined as the plurality of predicted texts.

[0090] In this embodiment, the first predicted text can be generated by the language model head in the large language model, and subsequently multiple predicted texts can be generated based on the previous predicted text and the hidden state by each decoding module in the multi-head model.

[0091] Therefore, after obtaining the first hidden state, the first hidden state can be decoded by the language model head using an autoregressive decoding method to obtain the first predicted text obtained by the first round of decoding.

[0092] The first predicted text is used as the predicted text for the previous decoding operation. Therefore, each decoding module in the multi-head model can sequentially generate multiple second predicted texts based on the predicted text for the previous decoding operation and the first hidden state. The first predicted text and the multiple second predicted texts are determined as multiple predicted texts.

[0093] The decoding acceleration method provided in this embodiment improves the accuracy of the generated predicted text by performing the next decoding operation based on the predicted text obtained in the previous decoding operation and the first hidden state. Furthermore, multiple predicted texts can be quickly generated during a single forward data processing step, thereby improving decoding speed.

[0094] Furthermore, based on any of the above embodiments, the decoding operation on the first hidden state through the language model head to obtain the first predicted text obtained by the first round of decoding includes:

[0095] The first hidden state is input into the language model head to obtain an unnormalized prediction value output by the language model head.

[0096] A first predicted text predicted by the large language model is determined based on the unnormalized predicted value and a preset vocabulary.

[0097] In this embodiment, after the first hidden state is input into the language model head, the language model head can perform a linear transformation on the first hidden state to generate an unnormalized prediction value. The unnormalized prediction value (logits) is a vector of the same size as the vocabulary in the preset vocabulary, where each element corresponds to the score of a word in the vocabulary. Logits can be positive, negative, or zero, representing the relative importance of each word in a given context.

[0098] Therefore, after obtaining the unnormalized prediction value, the first predicted text predicted by the large language model can be determined based on the unnormalized prediction value and the preset vocabulary. Multiple candidate words that match the unnormalized prediction value can be determined in the preset vocabulary, and the candidate word with the highest probability is determined as the first predicted text.

[0099] The decoding acceleration method provided in this embodiment pre-builds a preset vocabulary, so that after obtaining an unnormalized prediction value based on the language model header prediction, the first predicted text can be accurately determined based on the unnormalized prediction value and the preset vocabulary, thereby improving the accuracy and efficiency of text prediction.

[0100] Figure 3 A flowchart of a decoding acceleration method provided by another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 3 As shown, the multiple decoding modules in the multi-head model sequentially generate multiple second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state, including:

[0101] Step 301: Obtain a historical prediction text generated by a previous round of decoding operation, wherein the historical prediction text includes any one of the first prediction text and the second prediction text generated by a previous decoding module.

[0102] Step 302: Perform a vectorization operation on the historical prediction text to obtain vector information corresponding to the historical prediction text.

[0103] Step 303: Input the vector information and the first hidden state into the next decoding module in the multi-head model to obtain an unnormalized prediction value output by the next decoding module.

[0104] Step 304: Determine a second predicted text predicted by the next decoding module based on the unnormalized prediction value output by the next decoding module and a preset vocabulary.

[0105] Step 305: Determine the confidence associated with the unnormalized prediction value output by the next decoding module, and determine whether a preset termination condition is currently met based on the confidence.

[0106] Step 306: If the conditions are met, the second predicted texts generated by each decoding module in the multi-head model are determined as the multiple second predicted texts.

[0107] Step 307: If not satisfied, the second predicted text predicted by the next decoding module is determined as the current historical predicted text, and the process returns to the step of performing the vectorization operation on the historical predicted text to obtain the vector information corresponding to the historical predicted text, until the termination condition is met, and the second predicted text generated by each decoding module in the multi-head model is determined as the multiple second predicted texts.

[0108] In this embodiment, each decoding module in the multi-head model can sequentially generate a plurality of second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state.

[0109] Optionally, the historical prediction text generated by the previous round of decoding operation can be obtained, wherein the historical prediction text can be the first prediction text generated by an autoregressive method, or the historical prediction text can also be the second prediction text generated by the previous decoding module. This disclosure does not impose any restrictions on this.

[0110] Furthermore, the decoding module can also include an embedding layer, so that the historical prediction text can be vectorized through the embedding layer to obtain the vector information corresponding to the historical prediction text. The vector information and the first hidden state are input into the next decoding module in the multi-head model to obtain the unnormalized prediction value output by the next decoding module.

[0111] Based on the unnormalized prediction value output by the next decoding module, multiple matching candidate words are determined in the preset word list, and the candidate words whose probabilities meet the preset conditions are determined as the second predicted text predicted by the next decoding module.

[0112] It should be noted that since the multi-head model includes multiple decoding modules, each decoding module performs decoding operations based on the predicted text generated by the previous decoding operation and the first hidden state. Therefore, decoding modules with later rankings have a higher difficulty in generating predicted text, and the accuracy of the generated predicted text is lower than that of the predicted texts generated by the first N decoding modules. Therefore, in order to improve the content quality of the generated predicted text and avoid the problems of long prediction times and computing resource consumption caused by the continuous generation of low-quality predicted text, a termination condition can be pre-set.

[0113] Therefore, after the next decoding module completes decoding, the confidence associated with the next decoding module can be determined based on the unnormalized prediction value output by the next decoding module, and whether the preset termination condition is currently met can be determined based on the confidence.

[0114] If the condition is satisfied, it indicates that the quality of the content of the currently generated prediction text is not high, so the subsequent decoding operation can be stopped, and the second prediction text generated by each decoding module in the multi-head model is determined as multiple second prediction texts.

[0115] On the contrary, it indicates that the quality of the currently generated predicted text is good and the next round of decoding operations can be continued. Therefore, the second predicted text predicted by the next decoding module can be determined as the current historical predicted text, and the process returns to the step of vectorizing the historical predicted text to obtain the vector information corresponding to the historical predicted text until the termination condition is met. The second predicted text generated by each decoding module in the multi-head model is determined as multiple second predicted texts.

[0116] Figure 4 A schematic diagram of the model architecture provided in the embodiment of the present disclosure, such as Figure 4 As shown, the model architecture includes at least a base model 41 associated with a large language model and a multi-head model 42, wherein the multi-head model 42 includes multiple decoding modules, wherein the decoding module includes a multi-layer perceptron 43 and a language model head 44. The multiple decoding modules reuse the language model head in the large language model.

[0117] Based on the above model architecture, after obtaining the text to be predicted, an encoding operation can be performed through the basic model 41 to obtain a hidden state 45. The hidden state 45 is input into the language model head of the large language model to obtain the first predicted text 46 predicted by the large language model. This first predicted text 46 is determined as the current historical predicted text, and a vectorization operation is performed on this historical predicted text. The vector information and the hidden state 45 are input into the first decoding module in the multi-head model 42, so that the multi-layer perceptron 43 and language model head 44 in the decoding module process the input data to obtain the second predicted text 47. After each decoding module outputs the predicted text, it can be determined whether the termination condition is met. If the termination condition is met, the subsequent decoding operation is stopped.

[0118] The decoding acceleration method provided in this embodiment improves the accuracy of the generated predicted text by performing the next decoding round based on the predicted text obtained from the previous decoding round and the first hidden state. Furthermore, a termination condition is pre-set, allowing the decoding operation to be accurately terminated based on the termination condition, avoiding the continuous generation of low-quality predicted text and improving decoding efficiency and accuracy.

[0119] Further, based on any of the above embodiments, step 305 includes:

[0120] Determine the probabilities of multiple candidate words in the preset word list that match the unnormalized prediction value output by the next decoding module.

[0121] The probability of the candidate word meeting the preset condition is determined as the confidence level.

[0122] Determine whether the confidence level is greater than a preset threshold associated with the next decoding module.

[0123] If the confidence level is greater than or equal to the preset threshold, it is determined that the termination condition is not currently met.

[0124] If the confidence level is less than the preset threshold, it is determined that the termination condition is currently met.

[0125] In this embodiment, since each decoding module performs a current round of decoding based on the predicted text output in the previous round and the first hidden state, a preset threshold associated with each decoding module can be set. The termination condition can be that when the confidence level associated with the decoding module is less than the preset threshold, subsequent decoding operations are stopped.

[0126] Optionally, the probabilities of multiple candidate words that match the unnormalized prediction value output by the next decoding module can be determined in a preset word list. The probabilities of the candidate words that meet the preset conditions are determined as the confidence level. Among them, the candidate word with the highest probability can be determined, and the highest probability is determined as the confidence level.

[0127] Determine whether the confidence level is greater than a preset threshold associated with the next decoding module. If so, determine that the termination condition is not currently met. If less than, determine that the termination condition is currently met.

[0128] Wherein, multiple decoding modules in the multi-head model are connected in sequence, and the size of the preset threshold associated with each decoding module is proportional to the size of the sequence number of the decoding module.

[0129] It should be noted that since each decoding module performs decoding based on the predicted text generated by the previous decoding operation and the first hidden state, the later the decoding module is ranked, the more difficult it is to generate predicted text and the lower the accuracy is.

[0130] Therefore, in a multi-head model, multiple decoding modules are sequentially connected, and the preset threshold associated with each decoding module is proportional to the sequence number of the decoding module. That is, the later the decoding module is ranked, the larger the preset threshold is associated with. For example, in a three-head model, the preset thresholds of 0.4, 0.45, and 0.5 can be set for the three decoding modules, respectively.

[0131] The decoding acceleration method provided in this embodiment pre-sets a corresponding preset threshold for each decoding module, thereby determining the content quality of the predicted text generated by that decoding module based on the corresponding confidence level and the preset threshold. Furthermore, since each decoding module performs its current decoding operation based on the predicted text output from the previous round and the first hidden state, the preset threshold allows subsequent decoding operations to be stopped if the content quality of the predicted text generated by the decoding module is low, thus preventing the continuous generation of low-quality predicted text.

[0132] Figure 5 A flowchart of a decoding acceleration method provided by another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 5 As shown, after step 204, the following steps are further included:

[0133] Step 501: Determine a first number of the plurality of target predicted texts that have passed verification.

[0134] Step 502: Dynamically adjust a preset threshold associated with each decoding module based on a ratio of the first number to a second number of the plurality of predicted texts, wherein the value of the preset threshold is inversely proportional to the value of the ratio.

[0135] In this embodiment, the preset threshold value can be dynamically adjusted based on the computational efficiency associated with the previous round of forward data processing operations. The computational efficiency can be obtained by calculating a ratio of a first number of the plurality of target predicted texts that have passed verification to a second number of the plurality of predicted texts.

[0136] Therefore, after obtaining multiple preset texts in each round of forward data processing, the predicted texts can be verified using the large language model to obtain multiple verified target predicted texts. A first number of the multiple verified target predicted texts is determined. A ratio of the first number to a second number of the multiple predicted texts is calculated to obtain a computational efficiency.

[0137] Furthermore, the preset threshold can be dynamically adjusted based on computational efficiency. The threshold is inversely proportional to the ratio. Since a higher computational efficiency indicates more accurate predictions generated by the multi-head model, a lower threshold can be set. This allows the multi-head model, with its higher accuracy, to generate more predicted text in a single forward data processing pass.

[0138] As an implementable approach, during the predicted text verification phase, a set of thresholds can be set for each decoding module. The confidence associated with each decoding module can be compared against the threshold in order of the modules. If the confidence score exceeds the threshold, the text is sent to the large language model for verification. Otherwise, the text is discarded and no further verification is performed.

[0139] The decoding acceleration method provided in this embodiment determines the computational efficiency corresponding to the multi-head model, thereby determining the current prediction accuracy of the multi-head model. Furthermore, by determining the preset threshold associated with each decoding module in the next decoding operation based on the computational efficiency, a high computational efficiency can be ensured. Improving computational efficiency in both the generation and verification stages of the preset text further increases decoding speed.

[0140] Furthermore, based on any of the above embodiments, the decoding module further includes a context information integration layer, wherein the context information integration layer is used to integrate context information of input data of the decoding module. The method includes:

[0141] Input data is obtained, and context information of the input data of the decoding module is integrated through the context information integration layer, wherein the input data includes vector information associated with the predicted text generated by the last decoding operation and the first hidden state.

[0142] The input data is input to the multilayer perceptron, so that the multilayer perceptron performs weighted linear combination on the input data and then performs nonlinear transformation through a preset activation function to obtain output data.

[0143] The output data is input into a language model head, so that the language model head generates an unnormalized prediction value based on the output data.

[0144] In this embodiment, the decoding module also includes a context information integration layer (transformer layer), which is used to integrate the context information of the input data of the decoding module. The transformer layer can be placed before the multi-layer perceptron to implement the integration operation of the vector information associated with the predicted text generated by the previous decoding operation and the first hidden state. This facilitates the multi-head model to more accurately predict the text.

[0145] Therefore, after obtaining the vector information associated with the predicted text generated by the previous decoding operation and the first hidden state, the context information of the input data of the decoding module can be integrated through the context information integration layer. The input data is input into the multilayer perceptron, which performs a weighted linear combination of the input data and then performs a nonlinear transformation using a preset activation function to obtain output data. The output data is input into the language model header, which generates an unnormalized prediction value based on the output data.

[0146] Figure 6 Another model architecture diagram provided in the embodiment of the present disclosure is as follows: Figure 6 As shown, the model architecture includes at least a base model 61 associated with a large language model and a multi-head model 62. The multi-head model 62 includes multiple decoding modules, each of which includes a multi-layer perceptron 63, a language model head 64, and a context information integration layer 65. The multiple decoding modules reuse the language model head in the large language model.

[0147] Based on the above model architecture, the vector information associated with the predicted text generated by the previous decoding operation and the context information of the first hidden state can be integrated based on the context information integration layer 65, thereby improving the reception rate of the multi-head model 62.

[0148] The decoding acceleration method provided in this embodiment can improve the reception rate of the multi-head model by setting a transformer layer and integrating context information based on the transformer layer.

[0149] Figure 7 A flowchart of a decoding acceleration method provided by another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 7 As shown, step 204 includes:

[0150] Step 701: perform a concatenation operation on the multiple predicted texts and the text to be predicted to obtain a text to be verified, where each minimum text unit in the text to be verified is a text unit.

[0151] Step 702: Input the text to be verified into the large language model, and perform encoding operation on the text to be verified through the large language model to obtain a second hidden state corresponding to the text to be verified, wherein the second hidden state includes vocabulary parameters corresponding to each text unit in the text to be verified.

[0152] Step 703: For each text unit corresponding to the predicted text, determine a target candidate word corresponding to the vocabulary parameter corresponding to the text unit in a preset vocabulary.

[0153] Step 704: Determine whether the target candidate word is consistent with the predicted text predicted by the multi-head model.

[0154] Step 705: If yes, determine that the predicted text passes verification, and determine the predicted text as the target predicted text.

[0155] Step 706: If not, it is determined that the predicted text verification has failed, and the verification operation on other predicted texts after the predicted text is stopped.

[0156] In this embodiment, after a plurality of predicted texts are generated by the language model head and the multi-head model, a verification operation may be performed on the predicted texts generated by the multi-head model by the large language model.

[0157] Optionally, a concatenation operation can be performed on multiple predicted texts and the text to be predicted to obtain the text to be verified, where each minimum text unit in the text to be verified is a text unit. The text to be verified is input into the large language model, and the large language model is used to encode the text to be verified during the encoding phase to obtain a second hidden state corresponding to the text to be verified, wherein the second hidden state includes the vocabulary parameters (vocabulary size) corresponding to each text unit in the text to be verified.

[0158] Furthermore, the text units corresponding to the currently to-be-verified predicted text are sequentially determined according to the order of the decoding modules, and the target candidate words corresponding to the word list parameters corresponding to the text units are determined in the preset word list. It is determined whether the target candidate words are consistent with the predicted text predicted by the multi-head model. If they are consistent, it is determined that the predicted text verification has passed, and the predicted text is determined as the target predicted text. Otherwise, it is determined that the predicted text verification has failed. In order to further improve the verification efficiency during the verification process, after the current predicted text verification fails, the verification operation for other predicted texts after the predicted text can be stopped.

[0159] The decoding acceleration method provided in this embodiment inputs the spliced ​​text to be verified into a large language model, performs encoding operations on the text to be verified through the large language model, and for each text unit corresponding to the predicted text, determines the target candidate word corresponding to the vocabulary parameter corresponding to the text unit in the preset vocabulary, and determines whether the target candidate word is consistent with the predicted text predicted by the multi-head model, thereby being able to implement verification operations on multiple predicted texts at one time, further improving the decoding speed.

[0160] Figure 8 This is a schematic diagram of the structure of the decoding acceleration device provided by the embodiment of the present disclosure, such as Figure 8As shown, the device includes: an acquisition module 81, an encoding module 82, a decoding module 83 and a verification module 84. Among them, the acquisition module 81 is used to obtain the text to be predicted sent by the terminal device. The encoding module 82 is used to input the text to be predicted into the encoding layer preset in the large language model, and obtain the first hidden state corresponding to the text to be predicted output by the encoding layer. The decoding module 83 is used to perform a decoding operation based on the first hidden state through the language model head in the large language model and the preset multi-head model to generate multiple predicted texts, and the multi-head model includes multiple decoding modules, and the decoding module includes a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head. The verification module 84 is used to perform a verification operation on the multiple predicted texts based on the large language model, and send the multiple target predicted texts that have passed the verification to the terminal device.

[0161] Further, based on any of the above embodiments, the decoding module includes: a first decoding unit configured to perform a decoding operation on the first hidden state using the language model head to obtain a first predicted text obtained in a first round of decoding; a second decoding unit configured to sequentially generate a plurality of second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state using multiple decoding modules in the multi-head model; and a determination unit configured to determine the first predicted text and the plurality of second predicted texts as the plurality of predicted texts.

[0162] Furthermore, based on any of the above embodiments, the first decoding unit includes: a prediction subunit, configured to input the first hidden state into the language model head to obtain an unnormalized prediction value output by the language model head; and a mapping subunit, configured to determine a first predicted text predicted by the large language model based on the unnormalized prediction value and a preset vocabulary.

[0163] Further, based on any of the above embodiments, the second decoding unit includes: an acquisition subunit for acquiring the historical prediction text generated by the previous round of decoding operation, wherein the historical prediction text includes any one of the first prediction text and the second prediction text generated by the previous decoding module. A vectorization subunit for performing a vectorization operation on the historical prediction text to obtain the vector information corresponding to the historical prediction text. An input subunit for inputting the vector information and the first hidden state into the next decoding module in the multi-head model to obtain the unnormalized prediction value output by the next decoding module. A determination subunit for determining the second prediction text predicted by the next decoding module based on the unnormalized prediction value output by the next decoding module and a preset vocabulary. A termination subunit for determining the confidence associated with the unnormalized prediction value output by the next decoding module, and determining whether the preset termination condition is currently met based on the confidence. A first processing subunit for determining the second prediction text generated by each decoding module in the multi-head model as the multiple second prediction texts if met. The second processing sub-unit is used to determine the second predicted text predicted by the next decoding module as the current historical predicted text if the condition is not met, return to execute the step of performing vectorization operation on the historical predicted text to obtain the vector information corresponding to the historical predicted text, until the termination condition is met, and determine the second predicted text generated by each decoding module in the multi-head model as the multiple second predicted texts.

[0164] Further, based on any of the above embodiments, the termination subunit is used to: determine the probabilities of multiple candidate words in the preset word list that match the unnormalized prediction value output by the next decoding module. Determine the probability of the candidate words that meet the preset conditions as the confidence level. Determine whether the confidence level is greater than a preset threshold associated with the next decoding module. If the confidence level is greater than or equal to the preset threshold, determine that the termination condition is not currently met. If the confidence level is less than the preset threshold, determine that the termination condition is currently met.

[0165] Further, based on any of the above embodiments, multiple decoding modules in the multi-head model are connected sequentially, and the size of the preset threshold associated with each decoding module is proportional to the size of the sequence number of the decoding module.

[0166] Furthermore, based on any of the above embodiments, the apparatus further includes: a determination module configured to determine a first number of the plurality of target predicted texts that have passed verification; and an adjustment module configured to dynamically adjust a preset threshold associated with each decoding module based on a ratio of the first number to a second number of the plurality of predicted texts. The preset threshold is inversely proportional to the ratio.

[0167] Furthermore, on the basis of any of the above embodiments, the decoding module further includes a context information integration layer, and the context information integration layer is used to integrate the context information of the input data of the decoding module. The device includes: an acquisition module, which is used to acquire input data and integrate the context information of the input data of the decoding module through the context information integration layer, wherein the input data includes the vector information associated with the predicted text generated by the last decoding operation and the first hidden state. An input module is used to input the input data into the multilayer perceptron, so that the multilayer perceptron performs a weighted linear combination on the input data and then performs a nonlinear transformation through a preset activation function to obtain output data. A prediction module is used to input the output data into a language model head, so that the language model head generates an unnormalized prediction value based on the output data.

[0168] Further, based on any of the above embodiments, the verification module includes: a splicing unit for splicing the multiple predicted texts and the text to be predicted to obtain the text to be verified, wherein each minimum text unit in the text to be verified is a text unit. An input unit for inputting the text to be verified into the large language model, encoding the text to be verified through the large language model, and obtaining a second hidden state corresponding to the text to be verified, wherein the second hidden state includes the vocabulary parameters corresponding to each text unit in the text to be verified. A determination unit for determining, for each text unit corresponding to the predicted text, a target candidate word corresponding to the vocabulary parameter corresponding to the text unit in a preset vocabulary. A comparison unit for determining whether the target candidate word is consistent with the predicted text predicted by the multi-head model. A first verification unit for determining that the predicted text has passed verification if yes, and determining the predicted text as the target predicted text. A second verification unit for determining that the predicted text has failed verification if no, and stopping verification operations on other predicted texts after the predicted text.

[0169] Furthermore, based on any of the above embodiments, the device further includes: an acquisition module for acquiring a preset training data set, wherein the training data set includes multiple training data, the training data including original text and annotation information, and the annotation information is multiple texts generated by the large language model through autoregressive decoding. An input module for inputting the training data into the multi-head model to obtain a prediction result output by the multi-head model. A calculation module for calculating a loss value corresponding to the multi-head model based on the prediction result and annotation information. A judgment module for determining whether the multi-head model meets a preset convergence condition based on the loss value. A first training module for determining whether the multi-head model has converged if so, completing the training operation of the multi-head model. A second training module for performing reverse gradient adjustment on the parameters associated with the multi-head model based on the loss value if not, and returning to the step of inputting the training data into the multi-head model to obtain the prediction result output by the multi-head model until the multi-head model meets the preset convergence condition, completing the training operation of the multi-head model.

[0170] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0171] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, including:

[0172] at least one processor; and

[0173] a memory communicatively connected to the at least one processor; wherein,

[0174] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any one of the above embodiments.

[0175] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method described in any one of the above embodiments.

[0176] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0177] Figure 9Schematic diagram of an electronic device provided for an embodiment of the present disclosure. Electronic device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0178] like Figure 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0179] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0180] The computing unit 901 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the decoding acceleration method. For example, in some embodiments, the decoding acceleration method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the decoding acceleration method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the decoding acceleration method by any other appropriate means (e.g., by means of firmware).

[0181] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0182] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0183] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0184] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0185] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0186] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0187] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0188] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A decoding acceleration method, comprising: Obtain the text to be predicted sent by the terminal device; Inputting the text to be predicted into a coding layer preset in a large language model, and obtaining a first hidden state corresponding to the text to be predicted output by the coding layer; Performing a decoding operation on the first hidden state through a language model head in the large language model to obtain a first predicted text obtained by a first round of decoding; Generate multiple second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state through multiple decoding modules in the preset multi-head model; Determining the first predicted text and the plurality of second predicted texts as a plurality of predicted texts, wherein the multi-head model includes a plurality of decoding modules, each of the decoding modules includes a multi-layer perceptron and a language model head, and the plurality of decoding modules reuses the language model head; Performing a verification operation on the multiple predicted texts based on the large language model, and sending the multiple target predicted texts that have passed the verification to the terminal device; The step of sequentially generating a plurality of second predicted texts based on the predicted text of the previous decoding operation and the first hidden state by the plurality of decoding modules in the multi-head model includes: Obtaining a historical prediction text generated by a previous round of decoding operation, wherein the historical prediction text includes any one of the first prediction text and a second prediction text generated by a previous decoding module; Performing a vectorization operation on the historical prediction text to obtain vector information corresponding to the historical prediction text; Inputting the vector information and the first hidden state into a next decoding module in the multi-head model to obtain an unnormalized prediction value output by the next decoding module; The second predicted text predicted by the next decoding module is determined based on the unnormalized prediction value output by the next decoding module and a preset vocabulary.

2. The method according to claim 1, wherein The decoding operation on the first hidden state by the language model head to obtain a first predicted text obtained by a first round of decoding includes: Inputting the first hidden state into the language model head to obtain an unnormalized prediction value output by the language model head; A first predicted text predicted by the large language model is determined based on the unnormalized predicted value and a preset vocabulary.

3. The method according to claim 1, wherein After determining the second predicted text predicted by the next decoding module based on the unnormalized predicted value output by the next decoding module and the preset vocabulary, the method further includes: Determining a confidence level associated with the unnormalized prediction value output by the next decoding module, and determining whether a preset termination condition is currently satisfied based on the confidence level; If the conditions are met, determining the second predicted texts generated by each decoding module in the multi-head model as the multiple second predicted texts; If it is not satisfied, the second predicted text predicted by the next decoding module is determined as the current historical predicted text, and the process returns to the step of performing the vectorization operation on the historical predicted text to obtain the vector information corresponding to the historical predicted text until the termination condition is met, and the second predicted text generated by each decoding module in the multi-head model is determined as the multiple second predicted texts.

4. The method according to claim 3, wherein: Determining a confidence level associated with the unnormalized prediction value output by the next decoding module, and determining whether a preset termination condition is currently satisfied based on the confidence level, includes: Determining, in the preset vocabulary, the probabilities of a plurality of candidate words that match the unnormalized prediction values ​​output by the next decoding module; Determining the probability of the candidate word meeting the preset condition as the confidence level; determining whether the confidence level is greater than a preset threshold associated with the next decoding module; If the confidence level is greater than or equal to the preset threshold, it is determined that the termination condition is not currently met; If the confidence level is less than the preset threshold, it is determined that the termination condition is currently met.

5. The method according to claim 4, wherein In the multi-head model, multiple decoding modules are connected sequentially, and the size of the preset threshold associated with each decoding module is proportional to the size of the sequence number of the decoding module.

6. The method according to claim 4, further comprising: verifying the plurality of predicted texts based on the large language model and sending the plurality of target predicted texts that have passed the verification to the terminal device; determining a first number of the plurality of target predicted texts that pass the verification; dynamically adjusting a preset threshold associated with each decoding module based on a ratio of the first number to a second number of the plurality of predicted texts; The size of the preset threshold is inversely proportional to the size of the ratio.

7. The method according to any one of claims 1 to 6, wherein: The decoding module also includes a context information integration layer, which is used to integrate context information of input data of the decoding module; The method comprises: Obtaining input data, and integrating context information of the input data of the decoding module through the context information integration layer, wherein the input data includes vector information associated with the predicted text generated by the previous decoding operation and the first hidden state; Inputting the input data into a multilayer perceptron, so that the multilayer perceptron performs weighted linear combination on the input data and then performs nonlinear transformation through a preset activation function to obtain output data; The output data is input into a language model head, so that the language model head generates an unnormalized prediction value based on the output data.

8. The method according to any one of claims 1 to 6, wherein: The verifying operation on the plurality of predicted texts based on the large language model includes: performing a splicing operation on the multiple predicted texts and the text to be predicted to obtain a text to be verified, wherein each minimum text unit in the text to be verified is a text unit; Inputting the text to be verified into the large language model, encoding the text to be verified using the large language model to obtain a second hidden state corresponding to the text to be verified, wherein the second hidden state includes vocabulary parameters corresponding to each text unit in the text to be verified; For each text unit corresponding to the predicted text, determining a target candidate word corresponding to the vocabulary parameter corresponding to the text unit in a preset vocabulary; Determining whether the target candidate word is consistent with the predicted text predicted by the multi-head model; If yes, determining that the predicted text verification has passed, and determining the predicted text as the target predicted text; If not, it is determined that the prediction text verification fails, and the verification operation on other prediction texts after the prediction text is stopped.

9. The method according to any one of claims 1 to 6, further comprising: Obtain a preset training data set, wherein the training data set includes multiple training data, the training data includes original text and annotation information, and the annotation information is multiple texts generated by the large language model through autoregressive decoding; Inputting the training data into the multi-head model to obtain a prediction result output by the multi-head model; Calculate the loss value corresponding to the multi-head model based on the prediction result and the annotation information; Determining whether the multi-head model meets a preset convergence condition according to the loss value; If so, it is determined that the multi-head model has converged, and the training operation of the multi-head model is completed; If not, reverse gradient adjustment is performed on the parameters associated with the multi-head model based on the loss value, and the step of inputting the training data into the multi-head model to obtain the prediction result output by the multi-head model is returned to, until the multi-head model meets the preset convergence conditions, and the training operation of the multi-head model is completed.

10. A decoding acceleration device, comprising: An acquisition module, used to acquire the text to be predicted sent by the terminal device; an encoding module, configured to input the text to be predicted into an encoding layer preset in a large language model, and obtain a first hidden state corresponding to the text to be predicted output by the encoding layer; a decoding module, configured to generate a plurality of predicted texts by performing a decoding operation based on the first hidden state using a language model head in a large language model and a preset multi-head model, wherein the multi-head model includes a plurality of decoding modules, each of which includes a multi-layer perceptron and a language model head, and the plurality of decoding modules reuse the language model head; a verification module, configured to perform a verification operation on the plurality of predicted texts based on the large language model, and send the plurality of target predicted texts that pass the verification to the terminal device; The decoding module includes: a first decoding unit, configured to perform a decoding operation on the first hidden state through the language model head to obtain a first predicted text obtained by a first round of decoding; A second decoding unit, configured to generate a plurality of second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state in sequence through a plurality of decoding modules in the multi-head model; a determining unit, configured to determine the first predicted text and the plurality of second predicted texts as the plurality of predicted texts; Wherein, the second decoding unit includes: an acquisition subunit, configured to acquire a historical prediction text generated by a previous round of decoding operation, wherein the historical prediction text includes any one of the first prediction text and the second prediction text generated by a previous decoding module; A vectorization subunit, configured to perform a vectorization operation on the historical prediction text to obtain vector information corresponding to the historical prediction text; An input subunit, configured to input the vector information and the first hidden state into a next decoding module in the multi-head model to obtain an unnormalized prediction value output by the next decoding module; A determination subunit is configured to determine a second predicted text predicted by the next decoding module based on the unnormalized prediction value output by the next decoding module and a preset vocabulary.

11. The device according to claim 10, wherein The first decoding unit includes: a prediction subunit, configured to input the first hidden state into the language model head to obtain an unnormalized prediction value output by the language model head; A mapping subunit is configured to determine a first predicted text predicted by the large language model based on the unnormalized predicted value and a preset vocabulary.

12. The device according to claim 10, wherein The second decoding unit further includes: a termination subunit, configured to determine a confidence level associated with the unnormalized prediction value output by the next decoding module, and determine whether a preset termination condition is currently satisfied based on the confidence level; A first processing subunit is configured to determine the second predicted texts generated by each decoding module in the multi-head model as the plurality of second predicted texts if the conditions are met; The second processing sub-unit is used to determine the second predicted text predicted by the next decoding module as the current historical predicted text if the condition is not met, return to execute the step of performing vectorization operation on the historical predicted text to obtain the vector information corresponding to the historical predicted text, until the termination condition is met, and determine the second predicted text generated by each decoding module in the multi-head model as the multiple second predicted texts.

13. The device according to claim 12, wherein The terminator unit is used for: Determining, in the preset vocabulary, the probabilities of a plurality of candidate words that match the unnormalized prediction values ​​output by the next decoding module; Determining the probability of the candidate word meeting the preset condition as the confidence level; determining whether the confidence level is greater than a preset threshold associated with the next decoding module; If the confidence level is greater than or equal to the preset threshold, it is determined that the termination condition is not currently met; If the confidence level is less than the preset threshold, it is determined that the termination condition is currently met.

14. The device according to claim 13, wherein In the multi-head model, multiple decoding modules are connected sequentially, and the size of the preset threshold associated with each decoding module is proportional to the size of the sequence number of the decoding module.

15. The apparatus according to claim 13, further comprising: a determination module, configured to determine a first number of the plurality of target predicted texts that have passed the verification; an adjusting module, configured to dynamically adjust a preset threshold associated with each decoding module based on a ratio of the first number to a second number of the plurality of predicted texts; The size of the preset threshold is inversely proportional to the size of the ratio.

16. The device according to any one of claims 10 to 15, wherein: The decoding module also includes a context information integration layer, which is used to integrate context information of input data of the decoding module; The device comprises: an acquisition module, configured to acquire input data and integrate context information of the input data of the decoding module through the context information integration layer, wherein the input data includes vector information associated with the predicted text generated by the previous decoding operation and the first hidden state; An input module, configured to input the input data into a multilayer perceptron, so that the multilayer perceptron performs a weighted linear combination on the input data and then performs a nonlinear transformation using a preset activation function to obtain output data; A prediction module is configured to input the output data into a language model head, so that the language model head generates an unnormalized prediction value based on the output data.

17. The device according to any one of claims 10 to 15, wherein: The verification module includes: a concatenation unit configured to concatenate the plurality of predicted texts and the text to be predicted to obtain a text to be verified, wherein each minimum text unit in the text to be verified is a text unit; an input unit, configured to input the text to be verified into the large language model, and perform an encoding operation on the text to be verified using the large language model to obtain a second hidden state corresponding to the text to be verified, wherein the second hidden state includes vocabulary parameters corresponding to each text unit in the text to be verified; a determination unit configured to determine, for each text unit corresponding to the predicted text, a target candidate word corresponding to a vocabulary parameter corresponding to the text unit in a preset vocabulary; A comparison unit, configured to determine whether the target candidate word is consistent with the predicted text predicted by the multi-head model; a first verification unit, configured to determine that if the predicted text passes verification, and determine the predicted text as the target predicted text; The second verification unit is configured to, if not, determine that the verification of the predicted text fails, and stop verifying other predicted texts following the predicted text.

18. The device according to any one of claims 10 to 15, further comprising: An acquisition module is configured to acquire a preset training data set, wherein the training data set includes a plurality of training data, the training data including original text and annotation information, and the annotation information is a plurality of texts generated by the large language model through autoregressive decoding; An input module, configured to input the training data into the multi-head model to obtain a prediction result output by the multi-head model; A calculation module, configured to calculate a loss value corresponding to the multi-head model based on the prediction result and the annotation information; A judgment module, configured to determine whether the multi-head model satisfies a preset convergence condition according to the loss value; A first training module is configured to determine that the multi-head model has converged and complete the training operation of the multi-head model if yes; The second training module is used to perform reverse gradient adjustment on the parameters associated with the multi-head model based on the loss value if not, and return to execute the step of inputting the training data into the multi-head model to obtain the prediction result output by the multi-head model until the multi-head model meets the preset convergence conditions, thereby completing the training operation of the multi-head model.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9.

Images