Decoding acceleration method, device and equipment, computer readable storage medium and product

Abstract

The invention provides a decoding acceleration method, device and equipment, a computer readable storage medium and a product, and relates to the field of artificial intelligence, in particular to the field of deep learning. According to the specific implementation scheme, a to-be-predicted text sent by terminal equipment is obtained; inputting the to-be-predicted text into a preset coding layer in the large language model, and obtaining a first hidden state corresponding to the to-be-predicted text output by the coding layer; decoding operation is carried out on the basis of the first hidden state through a language model header in the large language model and a preset multi-head model, multiple prediction texts are generated, the multi-head model comprises multiple decoding modules, each decoding module comprises a multi-layer perceptron and the language model header, and the multiple decoding modules multiplex the language model header; and performing verification operation on the plurality of prediction texts based on the large language model, and sending the plurality of target prediction texts passing verification to the terminal equipment. Therefore, the speed of the large language model in the decoding stage is increased, and the data processing efficiency of the large language model is improved.

Description

Technical Field

[0001] The present disclosure relates to deep learning in artificial intelligence, and more particularly to a decoding acceleration method, apparatus, device, computer-readable storage medium, and product. Background Art

[0002] The current mainstream large language model prediction method is the autoregressive decoding scheme. When predicting a text, this scheme mainly includes two stages. The first stage is the prefill stage (encoding stage), and the second stage is the decode stage (decoding stage).

[0003] After obtaining the hidden state generated in the encoding phase, the large language model performs a forward calculation during the decoding phase to output only one predicted text unit (token). This process is repeated until all tokens are output. However, the decoding operation using the above method is often inefficient, resulting in a long prediction operation time. Summary of the invention

[0004] The present disclosure provides a decoding acceleration method, apparatus, device, computer-readable storage medium, and product for accelerating the decoding stage of a large language model.

[0005] According to a first aspect of the present disclosure, a decoding acceleration method is provided, comprising:

[0006] Obtain the text to be predicted sent by the terminal device;

[0007] Inputting the text to be predicted into a coding layer preset in a large language model, and obtaining a first hidden state corresponding to the text to be predicted output by the coding layer;

[0008] Generate multiple predicted texts by performing a decoding operation based on the first hidden state through a language model head in a large language model and a preset multi-head model, wherein the multi-head model includes multiple decoding modules, the decoding modules include a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head;

[0009] A verification operation is performed on the multiple predicted texts based on the large language model, and multiple target predicted texts that pass the verification are sent to the terminal device.

[0010] According to a second aspect of the present disclosure, a decoding acceleration device is provided, comprising:

[0011] An acquisition module, used for acquiring the text to be predicted sent by the terminal device;

[0012] An encoding module, used for inputting the text to be predicted into an encoding layer preset in a large language model, and obtaining a first hidden state corresponding to the text to be predicted output by the encoding layer;

[0013] A decoding module, used to perform a decoding operation based on the first hidden state through a language model head in a large language model and a preset multi-head model to generate multiple predicted texts, wherein the multi-head model includes multiple decoding modules, and the decoding modules include a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head;

[0014] A verification module is used to perform a verification operation on the multiple predicted texts based on the large language model, and send the multiple target predicted texts that pass the verification to the terminal device.

[0015] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0021] The technology disclosed in the present invention improves the speed of a large language model in the decoding stage and improves the efficiency of data processing by the large language model.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 A diagram of the system architecture on which the embodiments of the present disclosure are based;

[0025] Figure 2 A flowchart of a decoding acceleration method provided by an embodiment of the present disclosure;

[0026] Figure 3 A flowchart of a decoding acceleration method provided by another embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of a model architecture provided for an embodiment of the present disclosure;

[0028] Figure 5 A flowchart of a decoding acceleration method provided by another embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of another model architecture provided for an embodiment of the present disclosure;

[0030] Figure 7 A flowchart of a decoding acceleration method provided by another embodiment of the present disclosure;

[0031] Figure 8 A schematic diagram of the structure of a decoding acceleration device provided in an embodiment of the present disclosure;

[0032] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] Glossary:

[0035] Large Language Model (LLM): refers to a large-scale neural network model based on deep learning technology, which is specially used to process and generate natural language text. Large language models usually have a very large parameter scale, usually containing hundreds of millions to hundreds of billions of parameters. This scale enables them to capture complex language patterns and semantic relationships.

[0036] Token: During the decoding process of a large language model (such as GPT), the generated "token" usually refers to the basic unit of text generated by the model. In the Chinese context, a token can be a Chinese character, a word, or a subword (i.e., part of a word), depending on the segmentation or encoding method used by the model.

[0037] Multi-Layer Perceptron (MLP): A classic feed-forward artificial neural network structure. It consists of at least three layers: an input layer, one or more hidden layers, and an output layer. Each layer consists of several neurons (or nodes), and the neurons between layers are fully connected. It can be used for a variety of tasks, including classification, regression, and pattern recognition.

[0038] Language Model Head (LM Head): Usually refers to the output layer of the model, which can map the hidden state of the model to the probability distribution of each word in the vocabulary.

[0039] Hidden State refers to the intermediate representation generated by the model through its internal layers (such as hidden layers) when processing input data. The hidden state is passed between the layers of the model to capture the characteristics and contextual information of the input data.

[0040] The present disclosure provides a decoding acceleration method, device, equipment, computer-readable storage medium and product, which are applied to the deep learning field in the field of artificial intelligence to achieve the acceleration effect of large language models in the decoding stage.

[0041] The current mainstream scenario of large language model prediction is the autoregressive decoding solution. This solution mainly includes two stages when predicting a text. The first stage is the pre-fil stage (encoding stage), and the second stage is the decoding stage (decoding stage). In the decoding stage, the large language model performs a forward calculation when inferring, outputs only one token, and continues to execute until all tokens are output. This decoding method is relatively inefficient.

[0042] In the related art, in order to improve the decoding efficiency of the large language model during the prediction process, a parallel decoding solution is provided. For example, a draft token can be generated by using a small model with the same vocabulary or the same series. When a large model is inferred, a larger model often takes longer to decode a token (one decoding step), while a small model can decode a token much faster.

[0043] However, when using the above method for parallel decoding, a small model needs to be loaded additionally when the large language model is actually deployed for inference, which increases the memory overhead. In addition, when the batch size is large, it is easy to trigger the computing bottleneck, and there is no acceleration compared to the original method.

[0044] In the process of solving the above technical problems, the inventors found through research that in order to improve the decoding speed, a multi-head model can be designed based on the original large language model. Thus, in a forward data processing process, multiple tokens can be quickly generated at one time based on the multi-head model.

[0045] In addition, the multi-head model can include multiple decoding modules, and the multiple decoding modules reuse the language model head in the large language model, thereby saving a lot of additional video memory overhead of the multi-head model and further broadening the application scope of accelerated decoding based on the multi-head model.

[0046] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0047] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 1-Figure 9 The embodiments of the present disclosure are further refined.

[0048] Figure 1 This is a system architecture diagram based on the embodiments of the present disclosure, such as Figure 1 As shown, the system architecture based on the present disclosure includes at least a terminal device 11 and a server 12. The server 12 is provided with a decoding acceleration device, which can be written in languages ​​such as C / C++, Java, Shell or Python.

[0049] Based on the above system architecture, the user can input the text to be predicted through the terminal device 11. After obtaining the text to be predicted, the server 12 can extract the hidden state associated with the text to be predicted through the large language model. And generate multiple predicted texts at one time based on the hidden state through the preset language model head and the multi-head model. Accelerate the decoding stage. In addition, the large language model can realize the verification operation of multiple predicted texts at one time, further improving the data processing speed of the large language model.

[0050] Figure 2 A flowchart of a decoding acceleration method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the method includes:

[0051] Step 201: Obtain the text to be predicted sent by the terminal device.

[0052] The execution subject of this embodiment is a decoding acceleration device. The decoding acceleration device can be coupled to a server. The server can be connected to a terminal device for communication, so it can exchange information with the terminal device.

[0053] The Big Language Model is a large-scale neural network model based on deep learning technology, which is specially used to process and generate natural language text. Users can input the text to be predicted into the Big Language Model according to actual needs, so that the Big Language Model can process the data of the text to be predicted and generate prediction results.

[0054] Optionally, the user may input the text to be predicted on the terminal device, so the decoding acceleration device may obtain the text to be predicted sent by the terminal device.

[0055] For example, the user can input a question text into the large language model according to actual needs, and the large language model can extract the hidden state based on the question text, and generate a predicted text based on the decoding operation of the hidden state, so as to implement the answer operation to the question text based on the predicted text.

[0056] Step 202: input the text to be predicted into a coding layer preset in a large language model, and obtain a first hidden state corresponding to the text to be predicted output by the coding layer.

[0057] In this embodiment, the large language model includes a preset encoding layer, which can extract hidden features of the text to be predicted input by the terminal device to generate a first hidden state corresponding to the text to be predicted.

[0058] Optionally, the encoding layer can perform data processing on the text to be predicted through self-attention mechanism, multi-head attention mechanism, feedforward neural network, layer normalization, residual connection and position encoding, so that the encoding layer can effectively process and convert input data, capture complex semantic and grammatical relationships, and generate a first hidden state corresponding to the text to be predicted.

[0059] Step 203: Perform decoding operations based on the first hidden state through the language model head in the large language model and the preset multi-head model to generate multiple predicted texts, wherein the multi-head model includes multiple decoding modules, and the decoding module includes a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head.

[0060] The current prediction method of large language models is an autoregressive decoding scheme. Generally, only one predicted text is generated in one decoding process, and the process is continued until all predicted texts are generated. However, the autoregressive decoding scheme is often inefficient.

[0061] In this embodiment, in order to realize the decoding acceleration operation, a multi-head model capable of performing decoding operation can be pre-trained. The multi-head model can include multiple decoding modules, each of which can realize the decoding operation and generate a predicted text. Therefore, multiple predicted texts can be generated in one decoding process.

[0062] The decoding module may include a multi-layer perceptron and a language model head. Multiple decoding modules in the multi-head model may reuse the language model head in the large language model. Therefore, the video memory overhead of the multi-head model can be saved. In addition, the multi-layer perceptron may be composed of a residual network. The additional multi-layer perceptron uses the unnormalized prediction values ​​(logits) output by the large model for self-distillation training, so that each decoding module learns the ability to predict the next prediction text.

[0063] Therefore, after the hidden state associated with the text to be predicted is obtained, a decoding operation can be performed based on the first hidden state by using the language model head in the large language model and the preset multi-head model to generate multiple predicted texts.

[0064] Among them, the first predicted text can be generated by the language model head in the large language model, and then each decoding module in the multi-head model can generate multiple predicted texts based on the previous predicted text and the hidden state.

[0065] Step 204: Verify the multiple predicted texts based on the large language model, and send the multiple target predicted texts that pass the verification to the terminal device.

[0066] In this embodiment, after a plurality of predicted texts are generated by using the language model head and the multi-head model, further verification operations may be performed on the predicted texts.

[0067] Therefore, the multiple predicted texts and the text to be predicted may be concatenated and input into the large language model, so that the large language model performs a verification operation on the multiple predicted texts to determine whether the predicted text is available.

[0068] Furthermore, multiple target predicted texts that have passed verification may be determined from the multiple predicted texts, and the multiple target predicted texts may be sent to the terminal device, so that the user can view the multiple target predicted texts on the terminal device.

[0069] For example, after obtaining the question text sent by the user through the terminal device, the answer text of the question text can be predicted by the large language model. In the prediction process, in order to improve the prediction speed, the hidden state associated with the question text can be decoded in parallel by the language model head and the multi-head model to generate multiple predicted texts in one forward data processing process. The predicted text verified by the large language model is determined as the answer text among the multiple predicted texts. The answer text is sent to the terminal device for the user to view.

[0070] The decoding acceleration method provided in this embodiment can generate multiple predicted texts at one time based on multiple decoding modules in the multi-head model by pre-training the multi-head model in the decoding stage. In addition, by making multiple decoding modules reuse the language model head in the large language model, the large amount of additional video memory overhead caused by the multi-head model can be effectively reduced.

[0071] Optionally, based on any of the above embodiments, the method further includes:

[0072] A preset training data set is obtained, wherein the training data set includes multiple training data, the training data includes original text and annotation information, and the annotation information is multiple texts generated by the large language model through autoregressive decoding.

[0073] The training data is input into the multi-head model to obtain the prediction result output by the multi-head model.

[0074] The loss value corresponding to the multi-head model is calculated based on the prediction results and the annotation information.

[0075] Determine whether the multi-head model meets a preset convergence condition based on the loss value.

[0076] If so, it is determined that the multi-head model has converged, and the training operation of the multi-head model is completed.

[0077] If not, reverse gradient adjustment is performed on the parameters associated with the multi-head model based on the loss value, and the step of inputting the training data into the multi-head model to obtain the prediction result output by the multi-head model is returned to execute until the multi-head model meets the preset convergence conditions, thereby completing the training operation of the multi-head model.

[0078] In this embodiment, before performing a decoding operation based on a multi-head model, it is first necessary to perform a training operation on the multi-head model so that the multi-head model has decoding capabilities.

[0079] Optionally, a preset training data set may be obtained, wherein the training data set includes a plurality of training data, each training data includes original text and annotation information, and the annotation information is a plurality of texts generated by a large language model through autoregressive decoding.

[0080] After obtaining the training data set, the multi-head model can be iteratively trained using the training data set.

[0081] Optionally, for each training data, the original text in the training data can be input into the multi-head model to obtain the prediction result output by the multi-head model. The prediction result is obtained by decoding the hidden state associated with the original text by the multi-head model.

[0082] The loss value corresponding to the multi-head model is calculated based on the prediction results and the annotation information. Among them, any loss value algorithm can be used to calculate the loss value corresponding to the multi-head model, and the present disclosure does not limit this.

[0083] Furthermore, it is possible to determine whether the multi-head model meets a preset convergence condition based on the loss value. The convergence condition includes, but is not limited to, the loss value being less than a preset loss value threshold, the difference between the loss values ​​of two rounds of training being less than a preset difference threshold, etc. Alternatively, the convergence condition may also be that the training duration reaches a preset duration threshold, the number of training rounds reaches a preset number threshold, etc. The present disclosure does not limit this.

[0084] If so, it is determined that the multi-head model has converged, and it is determined that the multi-head model can accurately perform the decoding operation, so the training operation of the multi-head model can be completed. Otherwise, it indicates that the multi-head model does not currently have the ability to accurately decode. Then, the parameters associated with the multi-head model are reversely adjusted based on the loss value, and the step of inputting the training data into the multi-head model to obtain the prediction result output by the multi-head model is returned until the multi-head model meets the preset convergence conditions and the training operation of the multi-head model is completed.

[0085] The decoding acceleration method provided in this embodiment obtains a training data set including original text and annotation information in advance, so that the multi-head model can be iteratively trained based on the training data set. This enables the multi-head model to have a decoding capability similar to that of a large language model. When the preset convergence condition is met, the iterative training of the multi-head model is stopped. This enables the multi-head model to accurately generate a predicted text based on the input data that is similar to the predicted text generated by the large language model using an autoregressive decoding method. The data processing accuracy of the multi-head model is improved.

[0086] Further, based on any of the above embodiments, step 203 includes:

[0087] The first hidden state is decoded by the language model head to obtain a first predicted text obtained by a first round of decoding.

[0088] Multiple second predicted texts are generated in sequence based on the predicted text of the previous round of decoding operation and the first hidden state through multiple decoding modules in the multi-head model.

[0089] The first predicted text and the plurality of second predicted texts are determined as the plurality of predicted texts.

[0090] In this embodiment, the first predicted text can be generated by the language model head in the large language model, and subsequently, multiple predicted texts can be generated based on the previous predicted text and the hidden state by each decoding module in the multi-head model.

[0091] Therefore, after the first hidden state is acquired, the first hidden state may be decoded by the language model head using an autoregressive decoding method to obtain a first predicted text obtained by the first round of decoding.

[0092] The first predicted text is used as the predicted text of the last round of decoding operation. Therefore, each decoding module in the multi-head model can generate multiple second predicted texts based on the predicted text of the last round of decoding operation and the first hidden state in turn. The first predicted text and the multiple second predicted texts are determined as multiple predicted texts.

[0093] The decoding acceleration method provided in this embodiment can improve the accuracy of the generated predicted text by performing the next round of decoding operation based on the predicted text obtained in the previous round of decoding operation and the first hidden state. In addition, multiple predicted texts can be quickly generated in one forward data processing process to improve the decoding speed.

[0094] Further, based on any of the above embodiments, the decoding operation on the first hidden state through the language model head to obtain the first predicted text obtained by the first round of decoding includes:

[0095] The first hidden state is input into the language model head to obtain an unnormalized prediction value output by the language model head.

[0096] A first predicted text predicted by the large language model is determined based on the unnormalized predicted value and a preset vocabulary.

[0097] In this embodiment, after the first hidden state is input into the language model head, the language model head can perform a linear transformation operation on the first hidden state to generate an unnormalized prediction value. The unnormalized prediction value (logits) is a vector with the same size as the vocabulary in the preset vocabulary, where each element corresponds to the score of a word in the vocabulary. Logits can be positive, negative or zero, representing the relative importance of each word in a given context.

[0098] Therefore, after obtaining the unnormalized prediction value, the first predicted text predicted by the large language model can be determined based on the unnormalized prediction value and the preset word list. Among them, multiple candidate words matching the unnormalized prediction value can be determined in the preset word list, and the candidate word with the highest probability is determined as the first predicted text.

[0099] The decoding acceleration method provided in this embodiment pre-builds a preset word list, so that after obtaining an unnormalized prediction value based on the language model head prediction, the first predicted text can be accurately determined based on the unnormalized prediction value and the preset word list, thereby improving the accuracy and efficiency of text prediction.

[0100] Figure 3 A flowchart of a decoding acceleration method provided by another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 3 As shown, the multiple decoding modules in the multi-head model sequentially generate multiple second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state, including:

[0101] Step 301: Obtain a historical prediction text generated by a previous round of decoding operation, wherein the historical prediction text includes any one of the first prediction text and a second prediction text generated by a previous decoding module.

[0102] Step 302: perform vectorization operation on the historical prediction text to obtain vector information corresponding to the historical prediction text.

[0103] Step 303: input the vector information and the first hidden state into the next decoding module in the multi-head model to obtain an unnormalized prediction value output by the next decoding module.

[0104] Step 304: Determine a second predicted text predicted by the next decoding module based on the unnormalized prediction value output by the next decoding module and a preset vocabulary.

[0105] Step 305: Determine the confidence associated with the unnormalized prediction value output by the next decoding module, and determine whether a preset termination condition is currently met based on the confidence.

[0106] Step 306: If satisfied, the second predicted texts generated by each decoding module in the multi-head model are determined as the multiple second predicted texts.

[0107] Step 307, if it is not satisfied, the second predicted text predicted by the next decoding module is determined as the current historical predicted text, and the step of performing the vectorization operation on the historical predicted text to obtain the vector information corresponding to the historical predicted text is returned to, until the termination condition is met, and the second predicted text generated by each decoding module in the multi-head model is determined as the multiple second predicted texts.

[0108] In this embodiment, each decoding module in the multi-head model can generate multiple second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state in sequence.

[0109] Optionally, the historical prediction text generated by the previous round of decoding operation can be obtained, wherein the historical prediction text can be the first prediction text generated by an autoregressive method, or the historical prediction text can also be the second prediction text generated by the previous decoding module, and the present disclosure does not impose any restrictions on this.

[0110] Furthermore, the decoding module may also include an embedding layer, so the historical prediction text may be vectorized through the embedding layer to obtain the vector information corresponding to the historical prediction text. The vector information and the first hidden state are input into the next decoding module in the multi-head model to obtain the unnormalized prediction value output by the next decoding module.

[0111] Based on the unnormalized prediction value output by the next decoding module, multiple matching candidate words are determined in the preset word list, and the candidate words whose probabilities meet the preset conditions are determined as the second predicted text predicted by the next decoding module.

[0112] It should be noted that, since the multi-head model includes multiple decoding modules, each decoding module performs decoding operations based on the predicted text generated by the previous round of decoding operations and the first hidden state. Therefore, the difficulty of generating predicted texts for decoding modules with a later ranking is higher, and the accuracy of the generated predicted texts is lower than that of the predicted texts generated by the first N decoding modules. Therefore, in order to improve the content quality of the generated predicted texts and avoid the problems of long prediction time and consumption of computing resources caused by the continuous generation of low-quality predicted texts, termination conditions can be set in advance.

[0113] Therefore, after the next decoding module completes decoding, the confidence associated with the next decoding module can be determined based on the unnormalized prediction value output by the next decoding module, and whether the preset termination condition is currently met can be determined based on the confidence.

[0114] If it is satisfied, it indicates that the content quality of the currently generated prediction text is not high, so the subsequent decoding operation can be stopped, and the second prediction text generated by each decoding module in the multi-head model is determined as multiple second prediction texts.

[0115] On the contrary, it indicates that the quality of the currently generated prediction text is good, and the next round of decoding operation can be continued. Therefore, the second prediction text predicted by the next decoding module can be determined as the current historical prediction text, and the step of performing vectorization operation on the historical prediction text to obtain the vector information corresponding to the historical prediction text is returned until the termination condition is met, and the second prediction text generated by each decoding module in the multi-head model is determined as multiple second prediction texts.

[0116] Figure 4 A schematic diagram of a model architecture provided in an embodiment of the present disclosure, such as Figure 4 As shown, the model architecture includes at least a basic model 41 associated with a large language model and a multi-head model 42, wherein the multi-head model 42 includes multiple decoding modules, wherein the decoding module includes a multi-layer perceptron 43 and a language model head 44. Among them, multiple decoding modules reuse the language model head in the large language model.

[0117] Based on the above model architecture, after obtaining the text to be predicted, the basic model 41 can be used to perform encoding operations to obtain the hidden state 45. The hidden state 45 is input into the language model head in the large language model to obtain the first predicted text 46 predicted by the large language model. The first predicted text 46 is determined as the current historical predicted text, and the historical predicted text is vectorized. The vector information and the hidden state 45 are input into the first decoding module in the multi-head model 42, so that the multi-layer perceptron 43 and the language model head 44 in the decoding module process the input data to obtain the second predicted text 47. After each decoding module outputs the predicted text, it can be determined whether the termination condition is met. When the termination condition is met, the subsequent decoding operation is stopped.

[0118] The decoding acceleration method provided in this embodiment can improve the accuracy of the generated predicted text by performing the next round of decoding operation based on the predicted text obtained in the previous round of decoding operation and the first hidden state. In addition, a termination condition is pre-set, so that the decoding operation can be terminated accurately based on the termination condition, avoiding the continuous generation of low-quality predicted text, thereby improving decoding efficiency and decoding accuracy.

[0119] Further, based on any of the above embodiments, step 305 includes:

[0120] Determine the probabilities of multiple candidate words that match the unnormalized prediction values ​​output by the next decoding module in the preset word list.

[0121] The probability of the candidate word satisfying the preset condition is determined as the confidence.

[0122] Determine whether the confidence level is greater than a preset threshold associated with the next decoding module.

[0123] If the confidence level is greater than or equal to the preset threshold, it is determined that the termination condition is not currently met.

[0124] If the confidence level is less than the preset threshold, it is determined that the termination condition is currently met.

[0125] In this embodiment, since each decoding module performs the decoding operation of this round based on the predicted text outputted in the previous round and the first hidden state, a preset threshold associated with each decoding module can be set. The termination condition can be that when the confidence associated with the decoding module is less than the preset threshold, the subsequent decoding operation is stopped.

[0126] Optionally, the probabilities of multiple candidate words that match the unnormalized prediction value output by the next decoding module can be determined in the preset word list. The probabilities of candidate words that meet the preset conditions are determined as confidence. Among them, the candidate word with the largest probability can be determined, and the largest probability is determined as the confidence.

[0127] Determine whether the confidence is greater than a preset threshold associated with the next decoding module. If greater, determine that the termination condition is not currently met. If less, determine that the termination condition is currently met.

[0128] Among them, multiple decoding modules in the multi-head model are connected in sequence, and the size of the preset threshold associated with each decoding module is proportional to the size of the sequence number of the decoding module.

[0129] It should be noted that, since each decoding module performs decoding operation based on the predicted text generated by the previous round of decoding operation and the first hidden state, the difficulty of generating predicted text and the lower accuracy of the decoding module with a later ranking are higher.

[0130] Therefore, in the multi-head model, multiple decoding modules are connected in sequence, and the size of the preset threshold associated with each decoding module is proportional to the size of the sequence number of the decoding module. That is, the preset threshold associated with the later-ranked decoding module is larger. Taking the three-head model as an example, preset thresholds of 0.4, 0.45, and 0.5 can be set for the three decoding modules respectively.

[0131] The decoding acceleration method provided in this embodiment can determine the content quality of the predicted text generated by the decoding module based on the confidence corresponding to the decoding module and the preset threshold by setting a corresponding preset threshold for each decoding module in advance. In addition, since each decoding module performs the decoding operation of this round based on the predicted text output in the previous round and the first hidden state, by setting the preset threshold, when the content quality of the predicted text generated by the decoding module is low, the subsequent decoding operation can be stopped to avoid the continuous generation of low-quality predicted text.

[0132] Figure 5 A flowchart of a decoding acceleration method provided by another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 5 As shown, after step 204, the following steps are also included:

[0133] Step 501: Determine a first quantity of the plurality of target predicted texts that have passed the verification.

[0134] Step 502: dynamically adjust a preset threshold associated with each decoding module based on a ratio of the first number to a second number of the plurality of predicted texts, wherein the value of the preset threshold is inversely proportional to the value of the ratio.

[0135] In this embodiment, the preset threshold value can be dynamically adjusted based on the computational efficiency associated with the previous round of forward data processing operations, wherein the computational efficiency can be obtained by calculating the ratio of the first number of the plurality of target prediction texts that have passed verification to the second number of the plurality of prediction texts.

[0136] Therefore, after obtaining a plurality of preset texts in each round of forward data processing operation, a verification operation can be performed on the predicted texts through the large language model to obtain a plurality of target predicted texts that have passed the verification. A first number of the plurality of target predicted texts that have passed the verification is determined. A ratio of the first number to a second number of the plurality of predicted texts is calculated to obtain a calculation efficiency.

[0137] Furthermore, the preset threshold can be dynamically adjusted based on the computational efficiency. The size of the preset threshold is inversely proportional to the size of the ratio. Since the higher the computational efficiency, the more accurate the prediction results generated by the multi-head module. Therefore, a lower preset threshold can be set for it, so that the multi-head model with higher accuracy can generate more predicted texts in one forward data processing process.

[0138] As an implementable method, in the prediction text verification stage, another set of thresholds can be set for each decoding module. According to the order of the decoding modules, the confidence associated with each decoding module can be compared with the threshold in turn. If it is greater than the threshold, it is sent to the large language model for verification. Otherwise, the language text is discarded and no verification operation is performed.

[0139] The decoding acceleration method provided in this embodiment can determine the current prediction accuracy of the multi-head model by determining the computational efficiency corresponding to the multi-head model. And by determining the preset threshold associated with each decoding module in the next round of decoding operation based on the computational efficiency, it can ensure a higher computational efficiency. The computational efficiency is improved in the two stages of generating and verifying the preset text, and the decoding speed is further improved.

[0140] Further, based on any of the above embodiments, the decoding module further includes a context information integration layer, and the context information integration layer is used to integrate context information of input data of the decoding module. The method includes:

[0141] Input data is obtained, and context information of the input data of the decoding module is integrated through the context information integration layer, wherein the input data includes vector information associated with the predicted text generated by the last decoding operation and the first hidden state.

[0142] The input data is input to the multilayer perceptron, so that the multilayer perceptron performs weighted linear combination on the input data, and then performs nonlinear transformation through a preset activation function to obtain output data.

[0143] The output data is input into a language model head so that the language model head generates an unnormalized prediction value based on the output data.

[0144] In this embodiment, the decoding module also includes a context information integration layer (transformer layer), which is used to integrate the context information of the input data of the decoding module. The transformer layer can be set before the multi-layer perceptron to implement the integration operation of the vector information associated with the predicted text generated by the last decoding operation and the first hidden state. This facilitates the multi-head model to more accurately perform the prediction operation of the predicted text.

[0145] Therefore, after obtaining the vector information associated with the predicted text generated by the last decoding operation and the first hidden state, the context information of the input data of the decoding module can be integrated through the context information integration layer. The input data is input to the multilayer perceptron, so that the multilayer perceptron performs weighted linear combination on the input data, and then performs nonlinear transformation through a preset activation function to obtain output data. The output data is input to the language model head, so that the language model head generates an unnormalized prediction value based on the output data.

[0146] Figure 6 Another model architecture diagram provided for the embodiment of the present disclosure is as follows: Figure 6 As shown, the model architecture at least includes a basic model 61 associated with a large language model and a multi-head model 62, wherein the multi-head model 62 includes multiple decoding modules, wherein the decoding module includes a multi-layer perceptron 63, a language model head 64 and a context information integration layer 65. Among them, multiple decoding modules reuse the language model head in the large language model.

[0147] Based on the above model architecture, the vector information associated with the predicted text generated by the last decoding operation and the context information of the first hidden state can be integrated based on the context information integration layer 65, so as to improve the reception rate of the multi-head model 62.

[0148] The decoding acceleration method provided in this embodiment can improve the reception rate of the multi-head model by setting a transformer layer and integrating context information based on the transformer layer.

[0149] Figure 7 A flowchart of a decoding acceleration method provided by another embodiment of the present disclosure is provided. Based on any of the above embodiments, Figure 7 As shown, step 204 includes:

[0150] Step 701: perform a concatenation operation on the multiple predicted texts and the text to be predicted to obtain a text to be verified, wherein each minimum text unit in the text to be verified is a text unit.

[0151] Step 702: input the text to be verified into the large language model, and perform encoding operation on the text to be verified through the large language model to obtain a second hidden state corresponding to the text to be verified, wherein the second hidden state includes vocabulary parameters corresponding to each text unit in the text to be verified.

[0152] Step 703: for each text unit corresponding to the predicted text, determine in a preset vocabulary a target candidate word corresponding to the vocabulary parameter corresponding to the text unit.

[0153] Step 704: Determine whether the target candidate word is consistent with the predicted text predicted by the multi-head model.

[0154] Step 705: If yes, determine that the predicted text has passed the verification, and determine the predicted text as the target predicted text.

[0155] Step 706: If not, it is determined that the predicted text verification has failed, and the verification operation on other predicted texts after the predicted text is stopped.

[0156] In this embodiment, after a plurality of predicted texts are generated by using a language model head and a multi-head model, a verification operation may be performed on the predicted texts generated by the multi-head model using a large language model.

[0157] Optionally, a plurality of predicted texts and the text to be predicted may be concatenated to obtain the text to be verified, wherein each minimum text unit in the text to be verified is a text unit. The text to be verified is input into the large language model, and the large language model is used to encode the text to be verified in the encoding stage to obtain the second hidden state corresponding to the text to be verified, wherein the second hidden state includes the vocabulary parameter (vocabulary size) corresponding to each text unit in the text to be verified.

[0158] Furthermore, the text units corresponding to the current predicted text to be verified are determined in sequence according to the order of the decoding modules, and the target candidate words corresponding to the word list parameters corresponding to the text units are determined in the preset word list. It is determined whether the target candidate word is consistent with the predicted text predicted by the multi-head model. If they are consistent, it is determined that the predicted text verification has passed, and the predicted text is determined as the target predicted text. Otherwise, it is determined that the predicted text verification has failed. In order to further improve the verification efficiency in the verification process, after the current predicted text verification fails, the verification operation of other predicted texts after the predicted text can be stopped.

[0159] The decoding acceleration method provided in this embodiment, by inputting the spliced ​​text to be verified into a large language model, encoding the text to be verified through the large language model, determining the target candidate word corresponding to the vocabulary parameter corresponding to the text unit in a preset vocabulary for each text unit corresponding to the predicted text, and determining whether the target candidate word is consistent with the predicted text predicted by the multi-head model, thereby being able to implement verification operations on multiple predicted texts at one time, further improving the decoding speed.

[0160] Figure 8 A schematic diagram of the structure of a decoding acceleration device provided in an embodiment of the present disclosure is shown in FIG. Figure 8As shown, the device includes: an acquisition module 81, an encoding module 82, a decoding module 83 and a verification module 84. Among them, the acquisition module 81 is used to obtain the text to be predicted sent by the terminal device. The encoding module 82 is used to input the text to be predicted into the encoding layer preset in the large language model, and obtain the first hidden state corresponding to the text to be predicted output by the encoding layer. The decoding module 83 is used to perform a decoding operation based on the first hidden state through the language model head in the large language model and the preset multi-head model to generate multiple predicted texts, and the multi-head model includes multiple decoding modules, and the decoding module includes a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head. The verification module 84 is used to verify the multiple predicted texts based on the large language model, and send the multiple target predicted texts that have passed the verification to the terminal device.

[0161] Further, based on any of the above embodiments, the decoding module includes: a first decoding unit, configured to perform a decoding operation on the first hidden state through the language model head to obtain a first predicted text obtained by the first round of decoding. A second decoding unit, configured to generate a plurality of second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state in sequence through a plurality of decoding modules in the multi-head model. A determination unit, configured to determine the first predicted text and the plurality of second predicted texts as the plurality of predicted texts.

[0162] Further, based on any of the above embodiments, the first decoding unit includes: a prediction subunit, used to input the first hidden state into the language model head to obtain an unnormalized prediction value output by the language model head; a mapping subunit, used to determine a first predicted text predicted by the large language model based on the unnormalized prediction value and a preset vocabulary.

[0163] Further, on the basis of any of the above embodiments, the second decoding unit includes: an acquisition subunit, which is used to acquire the historical prediction text generated by the previous round of decoding operation, wherein the historical prediction text includes any one of the first prediction text and the second prediction text generated by the previous decoding module. A vectorization subunit, which is used to perform vectorization operation on the historical prediction text to obtain the vector information corresponding to the historical prediction text. An input subunit, which is used to input the vector information and the first hidden state into the next decoding module in the multi-head model to obtain the unnormalized prediction value output by the next decoding module. A determination subunit, which is used to determine the second prediction text predicted by the next decoding module based on the unnormalized prediction value output by the next decoding module and a preset vocabulary. A termination subunit, which is used to determine the confidence associated with the unnormalized prediction value output by the next decoding module, and determine whether the preset termination condition is currently met based on the confidence. A first processing subunit, which is used to determine the second prediction text generated by each decoding module in the multi-head model as the multiple second prediction texts if it is met. The second processing sub-unit is used to determine the second predicted text predicted by the next decoding module as the current historical predicted text if it is not satisfied, and return to execute the step of performing vectorization operation on the historical predicted text to obtain the vector information corresponding to the historical predicted text until the termination condition is met, and determine the second predicted text generated by each decoding module in the multi-head model as the multiple second predicted texts.

[0164] Further, based on any of the above embodiments, the termination subunit is used to: determine the probability of multiple candidate words matching the unnormalized prediction value output by the next decoding module in the preset word list. Determine the probability of the candidate words that meet the preset conditions as the confidence. Determine whether the confidence is greater than a preset threshold associated with the next decoding module. If the confidence is greater than or equal to the preset threshold, it is determined that the termination condition is not currently met. If the confidence is less than the preset threshold, it is determined that the termination condition is currently met.

[0165] Further, based on any of the above embodiments, multiple decoding modules in the multi-head model are connected sequentially, and the size of the preset threshold associated with each decoding module is proportional to the size of the serial number of the decoding module.

[0166] Further, based on any of the above embodiments, the device further comprises: a determination module, configured to determine a first number of the plurality of target prediction texts that have passed the verification. An adjustment module, configured to dynamically adjust a preset threshold associated with each decoding module based on a ratio of the first number to a second number of the plurality of prediction texts. The size of the preset threshold is inversely proportional to the size of the ratio.

[0167] Further, on the basis of any of the above embodiments, the decoding module also includes a context information integration layer, and the context information integration layer is used to integrate the context information of the input data of the decoding module. The device includes: an acquisition module, which is used to acquire input data and integrate the context information of the input data of the decoding module through the context information integration layer, wherein the input data includes the vector information associated with the predicted text generated by the last decoding operation and the first hidden state. An input module is used to input the input data into the multilayer perceptron, so that the multilayer perceptron performs a weighted linear combination on the input data, and then performs a nonlinear transformation through a preset activation function to obtain output data. A prediction module is used to input the output data into a language model head, so that the language model head generates an unnormalized prediction value based on the output data.

[0168] Further, on the basis of any of the above embodiments, the verification module includes: a splicing unit, which is used to splice the multiple predicted texts and the text to be predicted to obtain the text to be verified, and each minimum text unit in the text to be verified is a text unit. An input unit, which is used to input the text to be verified into the large language model, and encode the text to be verified through the large language model to obtain the second hidden state corresponding to the text to be verified, wherein the second hidden state includes the vocabulary parameters corresponding to each text unit in the text to be verified. A determination unit, which is used to determine the target candidate word corresponding to the vocabulary parameter corresponding to the text unit in the preset vocabulary for each text unit corresponding to the predicted text. A comparison unit, which is used to determine whether the target candidate word is consistent with the predicted text predicted by the multi-head model. A first verification unit, which is used to determine if the predicted text is verified, and determine the predicted text as the target predicted text. A second verification unit, which is used to determine if the predicted text is verified, and stop the verification operation on other predicted texts after the predicted text.

[0169] Further, on the basis of any of the above embodiments, the device further includes: an acquisition module, which is used to acquire a preset training data set, wherein the training data set includes multiple training data, the training data includes original text and annotation information, and the annotation information is multiple texts generated by the large language model through autoregressive decoding. An input module, which is used to input the training data into the multi-head model to obtain the prediction result output by the multi-head model. A calculation module, which is used to calculate the loss value corresponding to the multi-head model based on the prediction result and the annotation information. A judgment module, which is used to determine whether the multi-head model meets the preset convergence condition according to the loss value. A first training module, which is used to determine if the multi-head model converges and complete the training operation of the multi-head model. A second training module, which is used to perform reverse gradient adjustment on the parameters associated with the multi-head model based on the loss value if not, and return to execute the step of inputting the training data into the multi-head model to obtain the prediction result output by the multi-head model until the multi-head model meets the preset convergence condition and completes the training operation of the multi-head model.

[0170] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0171] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, including:

[0172] at least one processor; and

[0173] a memory communicatively connected to the at least one processor; wherein,

[0174] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of the above embodiments.

[0175] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method described in any one of the above embodiments.

[0176] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0177] Fig. 9A schematic diagram of an electronic device provided for an embodiment of the present disclosure. The electronic device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0178] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0179] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0180] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a decoding acceleration method. For example, in some embodiments, the decoding acceleration method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the decoding acceleration method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the decoding acceleration method in any other appropriate manner (e.g., by means of firmware).

[0181] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0182] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0183] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0184] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0185] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0186] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0187] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0188] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A decoding acceleration method, comprising: Obtain the text to be predicted sent by the terminal device; Inputting the text to be predicted into a coding layer preset in a large language model, and obtaining a first hidden state corresponding to the text to be predicted output by the coding layer; Generate multiple predicted texts by performing a decoding operation based on the first hidden state through a language model head in a large language model and a preset multi-head model, wherein the multi-head model includes multiple decoding modules, the decoding modules include a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head; A verification operation is performed on the multiple predicted texts based on the large language model, and multiple target predicted texts that pass the verification are sent to the terminal device.

2. The method according to claim 1, wherein: The step of performing a decoding operation based on the first hidden state by using a language model head in the large language model and a preset multi-head model to generate a plurality of predicted texts includes: Performing a decoding operation on the first hidden state through the language model head to obtain a first predicted text obtained by a first round of decoding; Generate multiple second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state in sequence through multiple decoding modules in the multi-head model; The first predicted text and the plurality of second predicted texts are determined as the plurality of predicted texts.

3. The method according to claim 2, wherein: The decoding operation on the first hidden state through the language model head to obtain a first predicted text obtained by a first round of decoding includes: Inputting the first hidden state into the language model head to obtain an unnormalized prediction value output by the language model head; A first predicted text predicted by the large language model is determined based on the unnormalized predicted value and a preset vocabulary.

4. The method according to claim 2, wherein: The method of generating a plurality of second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state by using a plurality of decoding modules in the multi-head model in sequence includes: Acquire a historical prediction text generated by a previous round of decoding operation, wherein the historical prediction text includes any one of the first prediction text and a second prediction text generated by a previous decoding module; Performing a vectorization operation on the historical prediction text to obtain vector information corresponding to the historical prediction text; Inputting the vector information and the first hidden state into a next decoding module in the multi-head model to obtain an unnormalized prediction value output by the next decoding module; Determine a second predicted text predicted by the next decoding module based on the unnormalized predicted value output by the next decoding module and a preset vocabulary; Determining a confidence level associated with the unnormalized prediction value output by the next decoding module, and determining whether a preset termination condition is currently met based on the confidence level; If satisfied, determining the second predicted texts generated by each decoding module in the multi-head model as the multiple second predicted texts; If it is not satisfied, the second predicted text predicted by the next decoding module is determined as the current historical predicted text, and the process returns to the step of performing the vectorization operation on the historical predicted text to obtain the vector information corresponding to the historical predicted text until the termination condition is met, and the second predicted text generated by each decoding module in the multi-head model is determined as the multiple second predicted texts.

5. The method according to claim 4, wherein: Determining the confidence associated with the unnormalized prediction value output by the next decoding module, and determining whether a preset termination condition is currently met based on the confidence, includes: Determining, in the preset word list, the probabilities of a plurality of candidate words that match the unnormalized prediction values ​​output by the next decoding module; Determining the probability of the candidate word satisfying the preset condition as the confidence level; Determining whether the confidence level is greater than a preset threshold associated with the next decoding module; If the confidence level is greater than or equal to the preset threshold, it is determined that the termination condition is not currently met; If the confidence level is less than the preset threshold, it is determined that the termination condition is currently met.

6. The method according to claim 5, wherein: In the multi-head model, multiple decoding modules are connected in sequence, and the size of the preset threshold associated with each decoding module is proportional to the size of the sequence number of the decoding module.

7. The method according to claim 5, further comprising: verifying the plurality of predicted texts based on the large language model and sending the plurality of target predicted texts that have passed the verification to the terminal device; Determining a first number of the plurality of target predicted texts that have passed the verification; dynamically adjusting a preset threshold associated with each decoding module based on a ratio of the first number to a second number of the plurality of predicted texts; The size of the preset threshold is inversely proportional to the size of the ratio.

8. The method according to any one of claims 1 to 7, wherein: The decoding module also includes a context information integration layer, which is used to integrate the context information of the input data of the decoding module; The method comprises: Acquire input data, and integrate the context information of the input data of the decoding module through the context information integration layer, wherein the input data includes vector information associated with the predicted text generated by the last decoding operation and the first hidden state; Inputting the input data into the multilayer perceptron, so that the multilayer perceptron performs weighted linear combination on the input data and then performs nonlinear transformation through a preset activation function to obtain output data; The output data is input into a language model head so that the language model head generates an unnormalized prediction value based on the output data.

9. The method according to any one of claims 1 to 7, wherein: The verifying operation on the plurality of predicted texts based on the large language model includes: Performing a splicing operation on the multiple predicted texts and the text to be predicted to obtain a text to be verified, wherein each minimum text unit in the text to be verified is a text unit; Inputting the text to be verified into the large language model, encoding the text to be verified through the large language model, and obtaining a second hidden state corresponding to the text to be verified, wherein the second hidden state includes a vocabulary parameter corresponding to each text unit in the text to be verified; For each text unit corresponding to the predicted text, determining a target candidate word corresponding to the word list parameter corresponding to the text unit in a preset word list; Determining whether the target candidate word is consistent with the predicted text predicted by the multi-head model; If yes, it is determined that the predicted text verification is passed, and the predicted text is determined as the target predicted text; If not, it is determined that the prediction text verification fails, and the verification operation on other prediction texts after the prediction text is stopped.

10. The method according to any one of claims 1 to 7, further comprising: Obtain a preset training data set, wherein the training data set includes a plurality of training data, the training data includes original text and annotation information, and the annotation information is a plurality of texts generated by the large language model through autoregressive decoding; Inputting the training data into the multi-head model to obtain a prediction result output by the multi-head model; Calculate the loss value corresponding to the multi-head model based on the prediction result and the annotation information; Determining whether the multi-head model satisfies a preset convergence condition according to the loss value; If yes, it is determined that the multi-head model has converged, and the training operation of the multi-head model is completed; If not, reverse gradient adjustment is performed on the parameters associated with the multi-head model based on the loss value, and the step of inputting the training data into the multi-head model to obtain the prediction result output by the multi-head model is returned to execute until the multi-head model meets the preset convergence conditions, thereby completing the training operation of the multi-head model.

11. A decoding acceleration device, comprising: An acquisition module, used for acquiring the text to be predicted sent by the terminal device; An encoding module, used for inputting the text to be predicted into an encoding layer preset in a large language model, and obtaining a first hidden state corresponding to the text to be predicted output by the encoding layer; A decoding module, used to perform a decoding operation based on the first hidden state through a language model head in a large language model and a preset multi-head model to generate multiple predicted texts, wherein the multi-head model includes multiple decoding modules, and the decoding modules include a multi-layer perceptron and a language model head, and the multiple decoding modules reuse the language model head; A verification module is used to perform a verification operation on the multiple predicted texts based on the large language model, and send the multiple target predicted texts that pass the verification to the terminal device.

12. The device according to claim 11, wherein The decoding module comprises: A first decoding unit, configured to perform a decoding operation on the first hidden state through the language model head to obtain a first predicted text obtained by a first round of decoding; A second decoding unit, configured to generate a plurality of second predicted texts based on the predicted text of the previous round of decoding operation and the first hidden state in sequence through a plurality of decoding modules in the multi-head model; A determining unit is configured to determine the first predicted text and the plurality of second predicted texts as the plurality of predicted texts.

13. The device according to claim 12, wherein: The first decoding unit comprises: A prediction subunit, configured to input the first hidden state into the language model head to obtain an unnormalized prediction value output by the language model head; A mapping subunit is used to determine a first predicted text predicted by the large language model based on the unnormalized prediction value and a preset vocabulary.

14. The device according to claim 12, wherein: The second decoding unit comprises: An acquisition subunit, used to acquire a historical prediction text generated by a previous round of decoding operation, wherein the historical prediction text includes any one of the first prediction text and a second prediction text generated by a previous decoding module; A vectorization subunit, used for performing a vectorization operation on the historical prediction text to obtain vector information corresponding to the historical prediction text; An input subunit, used to input the vector information and the first hidden state into a next decoding module in the multi-head model to obtain an unnormalized prediction value output by the next decoding module; A determination subunit, configured to determine a second predicted text predicted by the next decoding module based on the unnormalized prediction value output by the next decoding module and a preset vocabulary; A termination subunit, configured to determine a confidence level associated with the unnormalized prediction value output by the next decoding module, and determine whether a preset termination condition is currently satisfied based on the confidence level; A first processing subunit is configured to determine the second predicted texts generated by each decoding module in the multi-head model as the plurality of second predicted texts if the condition is satisfied; The second processing sub-unit is used to determine the second predicted text predicted by the next decoding module as the current historical predicted text if it is not satisfied, and return to execute the step of performing vectorization operation on the historical predicted text to obtain the vector information corresponding to the historical predicted text until the termination condition is met, and determine the second predicted text generated by each decoding module in the multi-head model as the multiple second predicted texts.

15. The device according to claim 14, wherein: The terminator unit is used for: Determining, in the preset word list, the probabilities of a plurality of candidate words that match the unnormalized prediction values ​​output by the next decoding module; Determining the probability of the candidate word satisfying the preset condition as the confidence level; Determining whether the confidence level is greater than a preset threshold associated with the next decoding module; If the confidence level is greater than or equal to the preset threshold, it is determined that the termination condition is not currently met; If the confidence level is less than the preset threshold, it is determined that the termination condition is currently met.

16. The device according to claim 15, wherein: In the multi-head model, multiple decoding modules are connected in sequence, and the size of the preset threshold associated with each decoding module is proportional to the size of the sequence number of the decoding module.

17. The apparatus according to claim 15, further comprising: A determination module, used to determine a first number of the plurality of target predicted texts that have passed the verification; an adjustment module, configured to dynamically adjust a preset threshold associated with each decoding module based on a ratio of the first number to a second number of the plurality of predicted texts; The size of the preset threshold is inversely proportional to the size of the ratio.

18. The device according to any one of claims 11 to 17, wherein: The decoding module also includes a context information integration layer, which is used to integrate the context information of the input data of the decoding module; The device comprises: An acquisition module, used for acquiring input data, integrating the context information of the input data of the decoding module through the context information integration layer, wherein the input data includes the vector information associated with the predicted text generated by the last decoding operation and the first hidden state; An input module, used for inputting the input data into the multilayer perceptron, so that the multilayer perceptron performs weighted linear combination on the input data and then performs nonlinear transformation through a preset activation function to obtain output data; A prediction module is used to input the output data into a language model head so that the language model head generates an unnormalized prediction value based on the output data.

19. The device according to any one of claims 11 to 17, wherein: The verification module comprises: A concatenation unit, configured to concatenate the plurality of predicted texts and the text to be predicted to obtain a text to be verified, wherein each minimum text unit in the text to be verified is a text unit; An input unit, used for inputting the text to be verified into the large language model, encoding the text to be verified through the large language model, and obtaining a second hidden state corresponding to the text to be verified, wherein the second hidden state includes a vocabulary parameter corresponding to each text unit in the text to be verified; A determination unit, configured to determine, for each text unit corresponding to the predicted text, a target candidate word corresponding to a word list parameter corresponding to the text unit in a preset word list; A comparison unit, used to determine whether the target candidate word is consistent with the predicted text predicted by the multi-head model; a first verification unit, configured to determine that if yes, the predicted text verification is passed, and determine the predicted text as the target predicted text; The second verification unit is used to determine that the prediction text verification fails, and stop the verification operation on other prediction texts after the prediction text.

20. The device according to any one of claims 11 to 17, further comprising: An acquisition module, used to acquire a preset training data set, wherein the training data set includes a plurality of training data, the training data includes original text and annotation information, and the annotation information is a plurality of texts generated by the large language model through autoregressive decoding; An input module, used to input the training data into the multi-head model to obtain a prediction result output by the multi-head model; A calculation module, used to calculate the loss value corresponding to the multi-head model based on the prediction result and the annotation information; A judgment module, used to determine whether the multi-head model meets a preset convergence condition according to the loss value; A first training module is used to determine if the multi-head model converges and complete the training operation of the multi-head model; The second training module is used to, if not, perform reverse gradient adjustment on the parameters associated with the multi-head model based on the loss value, and return to execute the step of inputting the training data into the multi-head model to obtain the prediction result output by the multi-head model, until the multi-head model meets the preset convergence conditions, thereby completing the training operation of the multi-head model.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images