Decoding method based on attention mechanism and text generation method of large language model

Abstract

Embodiments of the present application provide a decoding method based on an attention mechanism and a text generation method of a large language model. The decoding method based on the attention mechanism is applied to a first decoding layer of a model based on the attention mechanism. The model includes a plurality of decoding layers in cascade. The first decoding layer is any decoding layer except the first decoding layer among the plurality of decoding layers. The decoding method based on the attention mechanism includes: performing linear transformation on an input vector of the first decoding layer to obtain a first value vector; obtaining an attention matrix of a second decoding layer in the large language model, the second decoding layer being before the first decoding layer; and performing weighted summation on the first value vector according to the attention matrix of the second decoding layer to obtain an attention output of the first decoding layer. The method decodes by obtaining the attention matrix of the second decoding layer, saves the calculation process of obtaining the attention matrix based on the input vector of the first decoding layer, and improves the decoding speed based on the attention mechanism.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and more specifically, to an attention-based decoding method and a text generation method for large language models. Background Technology

[0002] In the inference process of attention-based models, decoding requires calculating an attention matrix to capture the relationships and dependencies between different elements in the input sequence. Attention mechanisms can dynamically adjust the importance of each lexical term to the current processing task, thereby improving the model's inference performance.

[0003] In the inference process of attention-based models, the computational complexity of the decoding process is high, which results in a significant amount of time being spent on decoding. Therefore, the decoding speed of existing attention-based models is slow. Summary of the Invention

[0004] This application provides an attention-based decoding method and a text generation method for large language models to solve the problem of slow decoding speed in existing attention-based models.

[0005] Firstly, this application proposes an attention-based decoding method applied to the first decoding layer of an attention-based model, wherein the model includes multiple cascaded decoding layers, and the first decoding layer is any decoding layer other than the first decoding layer among the multiple decoding layers, comprising:

[0006] The input vector of the first decoding layer is linearly transformed to obtain the first value vector;

[0007] Obtain the attention matrix of the second decoding layer in the model, wherein the second decoding layer is preceding the first decoding layer;

[0008] Based on the attention matrix of the second decoding layer, the first value vector is weighted and summed to obtain the attention output of the first decoding layer.

[0009] Secondly, this application proposes a text generation method using a large language model, wherein the large language model includes an embedding layer, an encoder, and a decoder, and the decoder includes multiple cascaded decoding layers, including:

[0010] The text to be processed is input into the embedding layer, which converts the text into a text vector and outputs it to the encoder.

[0011] The encoder generates a contextual representation of the text vector and outputs it to the decoder.

[0012] The decoder generates an output sequence based on the context and processes the output sequence into corresponding text, wherein a portion of the decoding layer in the decoder performs an attention-based decoding method as described in any of the first aspects.

[0013] Thirdly, this application proposes a decoding device based on an attention mechanism, comprising:

[0014] The transformation module is used to linearly transform the input vector of the first decoding layer of the attention mechanism-based model to obtain a first value vector; wherein the model includes multiple cascaded decoding layers, and the first decoding layer is any decoding layer other than the first decoding layer among the multiple decoding layers;

[0015] The acquisition module is used to acquire the attention matrix of the second decoding layer in the model, wherein the second decoding layer is before the first decoding layer;

[0016] The output module is used to perform a weighted summation of the first value vector based on the attention matrix of the second decoding layer to obtain the attention output of the first decoding layer.

[0017] Fourthly, this application proposes an electronic device, including: a memory and a processor;

[0018] The memory is used to store computer programs;

[0019] The processor is configured to execute a computer program stored in the memory, wherein the computer program, when executed, causes the processor to perform the method as described in either the first or second aspect.

[0020] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method described in either the first or second aspect.

[0021] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in either the first or second aspect.

[0022] The attention-based decoding method and text generation method for large language models provided in this application are applied to the first decoding layer of an attention-based model. The attention-based model includes multiple cascaded decoding layers, with the first decoding layer being any one of these layers. The method includes linearly transforming the input vector of the first decoding layer to obtain a first value vector; obtaining the attention matrix of the second decoding layer in the attention-based model, which precedes the first decoding layer; and performing a weighted summation of the first value vector based on the attention matrix of the second decoding layer to obtain the attention output of the first decoding layer. In this method, during the decoding process of the first decoding layer, the attention matrix of the second decoding layer preceding the first decoding layer is obtained, and the attention output of the first decoding layer is further obtained based on the attention matrix of the second decoding layer; that is, the attention matrix of the second decoding layer is used as the attention matrix of the first decoding layer. The first decoding layer performs decoding by obtaining the attention matrix of the second decoding layer, saving the computation process of obtaining the attention matrix based on the input vector in the first decoding layer and improving the decoding speed based on the attention mechanism. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating an attention-based decoding method provided in this application embodiment. Figure 1 ;

[0025] Figure 2 This application provides a schematic diagram of a process for determining a second decoding layer according to an embodiment of the present application.

[0026] Figure 3 A flowchart illustrating an attention-based decoding method provided in this application embodiment. Figure 2 ;

[0027] Figure 4 A flowchart illustrating a text generation method for a large language model provided in an embodiment of this application;

[0028] Figure 5 A schematic diagram of the structure of an attention-based decoding device provided in an embodiment of this application;

[0029] Figure 6 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] The attention-based model includes multiple cascaded decoding layers. During the inference process, each decoding layer further processes the input features and gradually optimizes the generated output.

[0032] In cascaded decoding layers, each layer first performs a linear transformation on the input vector to obtain a key vector and a query vector, and then further calculates the attention matrix based on the key and query vectors. Therefore, the computational complexity of attention-based decoding is high, resulting in a significant time consumption. Consequently, existing attention-based models exhibit slow decoding speeds.

[0033] In view of this, this application proposes an attention-based decoding method applied to the first decoding layer of an attention-based model. The first decoding layer can be any decoding layer other than the first decoding layer among multiple decoding layers. During the decoding process of the first decoding layer, the attention matrix of the second decoding layer preceding the first decoding layer is obtained. Furthermore, the attention output of the first decoding layer is obtained based on the attention matrix of the second decoding layer; that is, the attention matrix of the second decoding layer is used as the attention matrix of the first decoding layer. In this method, the first decoding layer performs decoding by obtaining the attention matrix of the second decoding layer, saving the computation process of obtaining the attention matrix based on the input vector in the first decoding layer and improving the decoding speed of the attention-based mechanism.

[0034] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0035] Figure 1 A flowchart illustrating an attention-based decoding method provided in this application embodiment. Figure 1The attention-based model includes multiple cascaded decoding layers. The attention-based decoding method provided in this application is applied to the first decoding layer of the attention-based model. The execution entity of this method can be any device implementing the attention-based model, which can be implemented in software and / or hardware. The method may include, for example, the following steps:

[0036] S101. Perform a linear transformation on the input vector of the first decoding layer to obtain the first value vector.

[0037] The first decoding layer is any decoding layer in the attention-based model other than the first decoding layer. The input vector of the first decoding layer is the output vector of the previous decoding layer.

[0038] A value vector is the result of a linear transformation of an input vector, and it includes the feature information of the input vector. The first value vector is the vector obtained by linearly transforming the input vector of the first decoding layer.

[0039] A linear transformation can be performed on the input vector of the first decoding layer, based on the weight matrix of the value vectors in the decoding layer. The decoding layer contains pre-trained weight matrices of the value vectors; for example, multiplying the input vector of the first decoding layer by the weight matrix of the value vectors yields the first value vector.

[0040] Alternatively, the first value vector can be obtained based on the weight matrix and the bias matrix of the value vector. The bias matrix of the value vector is a matrix pre-trained in the decoding layer. In this way, the input vector of the first decoding layer is multiplied by the weight matrix of the value vector, and the matrix obtained by multiplication is added to the bias matrix of the value vector to obtain the first value vector.

[0041] S102. Obtain the attention matrix of the second decoding layer in the model. The second decoding layer is before the first decoding layer.

[0042] The attention matrix is ​​a matrix composed of attention weights. Attention weights represent the importance of each element in the input vector to other elements in the input vector. The attention matrix can represent the dependencies and correlations between the elements in the input vector.

[0043] The second decoding layer is the decoding layer in the model that precedes the first decoding layer. The model includes multiple cascaded decoding layers, with the output of each decoding layer serving as the input to the next decoding layer; that is, the earlier decoding layer decodes before the later decoding layer. Therefore, the second decoding layer has already completed decoding and obtained its attention matrix before the first decoding layer decodes.

[0044] In this approach, after the second decoding layer obtains the attention matrix, it can store the attention matrix of the second decoding layer. During the decoding process of the first decoding layer, the attention matrix of the second decoding layer in the model can be obtained, and the stored attention matrix of the second decoding layer can be read.

[0045] Obtain the attention matrix of the second decoding layer in the model. Optionally, the attention matrix of the second decoding layer that is close to the first decoding layer can be selected. For example, the attention matrix of the second decoding layer that is adjacent to the first decoding layer can be selected in the model. For example, the attention matrix of the second decoding layer that is two layers ahead of the first decoding layer can be selected in the model.

[0046] S103. Based on the attention matrix of the second decoding layer, the first value vector is weighted and summed to obtain the attention output of the first decoding layer.

[0047] Attention output is the result of aggregating input vectors based on an attention mechanism. It is obtained by weighted summation of the value vectors mapped from the input vectors, with the weights in the summation being the attention weights in the attention matrix. Attention output represents the context-aware representation of each element in the input sequence.

[0048] The attention output of the first decoding layer is obtained by weighted summation of the first value vector based on the attention matrix of the second decoding layer. The attention matrix of the second decoding layer consists of attention weights. In this method, the attention output of the first decoding layer is obtained by weighted summation of the first value vector based on the attention weights in the attention matrix of the second decoding layer.

[0049] The attention-based decoding method of this application is applied to the first decoding layer of an attention-based model. The model includes multiple cascaded decoding layers, with the first decoding layer being any one of the multiple decoding layers except for the first decoding layer itself. The method includes: linearly transforming the input vector of the first decoding layer to obtain a first value vector; obtaining the attention matrix of a second decoding layer in the model, where the second decoding layer precedes the first decoding layer; and performing a weighted summation of the first value vector based on the attention matrix of the second decoding layer to obtain the attention output of the first decoding layer. In this method, the first decoding layer performs decoding by obtaining the attention matrix of the second decoding layer, saving the computational process of obtaining the attention matrix based on the input vector and improving the decoding speed of the attention-based mechanism.

[0050] In the above embodiments, the attention matrix of the second decoding layer can be calculated from the input vector of the second decoding layer. Specifically, the input vector of the second decoding layer is first linearly transformed to obtain the first query vector and the first key vector. Then, the square root of the dimension of the first key vector is determined, and the ratio of the dot product of the first query vector and the first key vector to the square root of the dimension of the first key vector is normalized to obtain the attention matrix of the second decoding layer.

[0051] In this method, the input vector of the second decoding layer is linearly transformed to obtain the first query vector. This transformation can be based on the weight matrix of the query vector in the decoding layer. The decoding layer contains a pre-trained weight matrix of the query vector. For example, the first query vector is obtained by multiplying the input vector of the second decoding layer with the weight matrix of the query vector. Optionally, the first query vector can also be obtained based on the weight matrix and the bias matrix of the query vector, where the bias matrix is ​​a pre-trained matrix in the decoding layer. In this method, the input vector of the second decoding layer is multiplied with the weight matrix of the query vector, and the resulting matrix is ​​then added to the bias matrix of the query vector to obtain the first query vector.

[0052] The first key vector is obtained by linearly transforming the input vector of the second decoding layer. This transformation can be based on the weight matrix of the key vector in the decoding layer. The decoding layer contains a pre-trained weight matrix of the key vector. For example, the first key vector is obtained by multiplying the input vector of the second decoding layer with the weight matrix of the key vector. Alternatively, the first key vector can also be obtained based on the weight matrix and the bias matrix of the key vector, where the bias matrix is ​​a pre-trained matrix in the decoding layer. In this method, the input vector of the second decoding layer is multiplied with the weight matrix of the key vector, and the resulting matrix is ​​then added to the bias matrix of the key vector to obtain the first key vector, and the square root of the dimension of the first key vector is determined.

[0053] After obtaining the first query vector and the first key vector through linear transformation, the ratio of the dot product of the first query vector and the first key vector to the square root of the dimension of the first key vector is normalized to obtain the attention matrix of the second decoding layer, as shown in equation (1):

[0054] (Equation 1)

[0055] Where Q is the first query vector, K is the first key vector, the softmax function is the activation function of the attention mechanism, outputs is the attention matrix, and d is the dimension of the first key vector.

[0056] During the decoding process of the second decoding layer, the attention matrix of the second decoding layer can be obtained according to the method in the above embodiments.

[0057] Optionally, in the attention mechanism-based decoding method provided in the embodiments of this application, before linearly transforming the input vector of the first decoding layer to obtain the first value vector, any input sequence can be input into the attention mechanism-based model, and the attention matrix of each decoding layer can be obtained during the inference process of the attention mechanism-based model. Based on the attention matrix of each decoding layer, the second decoding layer is determined for the first decoding layer. The specific steps for determining the second decoding layer are illustrated below.

[0058] Figure 2 This is a schematic diagram illustrating a process for determining the second decoding layer, provided as an embodiment of this application. Figure 2 As shown, the method may include, for example, the following steps:

[0059] S201. Determine the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer.

[0060] The third decoding layer is any decoding layer in the attention-based model that precedes the first decoding layer.

[0061] To determine the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer, a similarity index can be calculated between them. The similarity index can be any index that measures the similarity of matrices, such as cosine similarity, Euclidean distance, or Manhattan distance.

[0062] One possible approach is to calculate the cosine similarity between the attention matrices of the first and third decoding layers. A higher cosine similarity indicates a higher similarity, meaning the closer the similarity is to 1, the higher the similarity. Therefore, a higher cosine similarity between the attention matrices of the first and third decoding layers indicates a higher degree of similarity between them.

[0063] Optionally, the similarity between the attention matrices of the first and third decoding layers can be determined based on the similarity between the last row of the attention matrices of the first and third decoding layers. Since the last row of the attention matrix typically corresponds to the attention distribution of the query vector at the last time step, which tends to be a summary processing step of the entire input sequence, the calculation of the similarity between the attention matrices of the first and third decoding layers can be simplified by calculating the similarity between the last rows of the attention matrices of the first and third decoding layers.

[0064] In this approach, the last row of the attention matrix of the first decoding layer and the last row of the attention matrix of the third decoding layer can be extracted, and the similarity between the last rows of the attention matrices of the first and third decoding layers can be determined as the similarity between their respective attention matrices. For example, the cosine similarity between the last rows of the attention matrices of the first and third decoding layers can be determined as the similarity between their respective attention matrices.

[0065] S202. Determine the second decoding layer based on the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer.

[0066] One possible approach is to determine any third decoding layer as a candidate second decoding layer if the similarity between the attention matrix of the first decoding layer and the attention matrix of any third decoding layer is greater than a preset threshold, thereby obtaining one or more candidate second decoding layers. Alternatively, a candidate second decoding layer that is closest to the first decoding layer among the one or more candidate second decoding layers can be selected as the second decoding layer.

[0067] For example, if the cosine similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer is determined, it is then determined whether the cosine similarity is greater than a preset threshold. The preset threshold can be set to, for example, 0.95. If the cosine similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer is greater than the preset threshold, it indicates a high similarity between the two, and the third decoding layer is identified as a candidate second decoding layer. By determining the cosine similarity between the attention matrix of the first decoding layer and the attention matrix of any third decoding layer, one or more candidate second decoding layers are obtained. Optionally, in the state of obtaining one or more candidate second decoding layers, a candidate second decoding layer can be randomly selected as the second decoding layer, or the candidate second decoding layer closest to the first decoding layer among the one or more candidate second decoding layers can be selected as the second decoding layer.

[0068] One possible approach is to determine the second decoding layer based on the similarity between the attention matrix of the first decoding layer and the attention matrix of any third decoding layer, selecting the third decoding layer with the highest similarity to the attention matrix of the first decoding layer. For example, if the cosine similarity between the attention matrix of the first decoding layer and the attention matrix of any third decoding layer is determined, then the third decoding layer with the highest cosine similarity to the attention matrix of the first decoding layer is selected as the second decoding layer. Optionally, multiple first decoding layers can all use the same third decoding layer as the second decoding layer. For example, for the nth, (n+1), and (n+2)th decoding layers, the (n-1)th decoding layer can be designated as the second decoding layer.

[0069] The above embodiment determines the second decoding layer based on the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer. This method ensures a high similarity between the determined attention matrix of the second decoding layer and the attention matrix of the first decoding layer. In the decoding method of the attention mechanism in this application, after linearly transforming the input vector of the first decoding layer to obtain a first value vector, the attention matrix of the second decoding layer determined according to the method of the above embodiment can be obtained. Then, based on the determined attention matrix of the second decoding layer, a weighted sum of the first value vector is performed to obtain the attention output of the first decoding layer. This can improve the decoding speed based on the attention mechanism while reducing the error caused by the attention of the first decoding layer obtained from the attention matrix of the second decoding layer.

[0070] Optionally, the second decoding layer can be determined based on the position of the first decoding layer. In this approach, the second decoding layer is adjacent to the first decoding layer. Because the attention matrices of adjacent decoding layers have high similarity during inference in the attention mechanism-based model, the previous decoding layer adjacent to the first decoding layer can be determined as the second decoding layer. In this approach, in the decoding method of the attention mechanism in this application, the attention matrix of the second decoding layer determined according to the method of the above embodiments is obtained, and the first value vector is weighted and summed according to the determined attention matrix of the second decoding layer to obtain the attention output of the first decoding layer. This can improve the decoding speed of the attention mechanism-based model while reducing the error caused by the attention of the first decoding layer obtained from the attention matrix of the second decoding layer.

[0071] The above is an embodiment of a method for determining a second decoding layer before linearly transforming the input vector of the first decoding layer to obtain a first value vector. In the decoding method of the attention mechanism of this application, after linearly transforming the input vector of the first decoding layer to obtain a first value vector, a determined attention matrix of the second decoding layer can be obtained, and the first value vector is weighted and summed according to the determined attention matrix of the second decoding layer to obtain the attention output of the first decoding layer.

[0072] Figure 3 A flowchart illustrating an attention-based decoding method provided in this application embodiment. Figure 2 .like Figure 3 As shown, the l-th decoding layer is the first decoding layer, and the lm-th decoding layer is the second decoding layer. The lm-th decoding layer first performs a linear transformation on the input vector of the second decoding layer to obtain the query vector of the lm-th decoding layer. Key vector Sum value vector Confirm key vector The dimension d, and based on the query vector Key vector The attention matrix is ​​obtained by taking the square root of dimension d, and then applied to the value vector. Weighted summation yields the attention output of the lm-th decoding layer. The l-th decoding layer performs a linear transformation on the input vector of the l-th decoding layer to obtain the value vector of the l-th decoding layer. And based on the attention matrix of the lm-th decoding layer, the value vector of the l-th decoding layer is... Weighted summation yields the attention output of the l-th decoding layer.

[0073] Compared to the lm-th decoding layer, the l-th decoding layer saves the steps of linearly transforming the input vector of the l-th decoding layer to obtain the key vector and query vector of the l-th decoding layer, and calculating the attention matrix of the l-th decoding layer based on the key vector, query vector and the dimension of the key vector of the l-th decoding layer, thus improving the decoding speed of the l-th decoding layer.

[0074] The attention-based decoding method of this application is applied to the first decoding layer of an attention-based model. The model includes multiple cascaded decoding layers, and the first decoding layer is any decoding layer other than the first decoding layer. The method includes: linearly transforming the input vector of the first decoding layer to obtain a first value vector; obtaining the attention matrix of a second decoding layer in the model, where the second decoding layer precedes the first decoding layer; and performing a weighted summation of the first value vector based on the attention matrix of the second decoding layer to obtain the attention output of the first decoding layer. In this method, the first decoding layer performs decoding by obtaining the attention matrix of the second decoding layer, saving the computation process of obtaining the attention matrix based on the input vector in the first decoding layer and improving the decoding speed of the attention-based mechanism.

[0075] Figure 4 This is a flowchart illustrating a text generation method based on a large language model, as provided in an embodiment of this application. The large language model includes an embedding layer, an encoder, and a decoder, with the decoder comprising multiple cascaded decoding layers. Figure 3As shown, the specific steps may include:

[0076] S401. Input the text to be processed into the embedding layer, convert the text to be processed into a text vector through the embedding layer, and output it to the encoder.

[0077] Embedding layers can convert discrete text data into continuous vector representations.

[0078] The text to be processed is determined based on the task being handled by the large language model. For example, in dialogue generation or question-answering tasks, the text to be processed could be previous dialogue turns or the input question text. For example, in a text continuation task, the text to be processed could be the input starting text.

[0079] The text to be processed is input into the embedding layer, which converts the text into a vector representation, resulting in a text vector, which is then output to the encoder.

[0080] S402. Generate a contextual representation of the text vector through the encoder and output it to the decoder.

[0081] The encoder consists of multiple cascaded coding layers, each of which further processes and refines the features of the input vector. The decoder is used to capture the semantic and contextual information of the text vector, extract the semantic features of the text vector, and generate the contextual representation of the text vector.

[0082] The text vector input encoder captures the semantic relationships of the text vectors through the decoder, generates a contextual representation of the text vectors, and outputs it to the decoder.

[0083] S403. Generate an output sequence through the decoder based on the context representation, and process the output sequence into the corresponding text, wherein a portion of the decoding layer in the decoder performs a decoding method based on an attention mechanism.

[0084] The decoder comprises multiple cascaded decoding layers. The context representation of the text vector is input into the decoder to generate an output sequence. Some decoding layers execute the attention-based decoding method described above, yielding the attention output of that layer. Furthermore, the attention output of this layer can be input into the feedback-forward neural network within the decoding layer for nonlinear transformation, further capturing the features of the text vector.

[0085] After the decoder generates the output sequence, the output sequence corresponds to the text output by the model. Therefore, the output sequence can be processed into the corresponding text. For example, the format of the output sequence can be corrected to make the output sequence conform to the text format and syntax, so as to obtain the corresponding text, which is the text output by the large language model.

[0086] The text generation method of the large language model in this application embodiment uses the attention mechanism-based decoding method in the decoder to perform decoding in a part of the decoding layer, which improves the decoding speed of the decoder in the large language model and further improves the inference speed of the large language model.

[0087] Figure 5 This is a schematic diagram of the structure of an attention-based decoding device provided in an embodiment of this application. Figure 5 As shown, the attention-based decoding device 500 includes a transformation module 501, an acquisition module 502, and an output module 503. Optionally, it may also include a first determination module and / or a second determination module.

[0088] Transformation module 501 is used to linearly transform the input vector of the first decoding layer of the attention mechanism-based model to obtain a first value vector; wherein, the attention mechanism-based model includes multiple cascaded decoding layers, and the first decoding layer is any decoding layer other than the first decoding layer among the multiple decoding layers.

[0089] The acquisition module 502 is used to acquire the attention matrix of the second decoding layer in the model, which is before the first decoding layer.

[0090] The output module 503 is used to perform a weighted summation of the first value vector based on the attention matrix of the second decoding layer to obtain the attention output of the first decoding layer.

[0091] In one implementation, the first determining module is specifically used to perform a linear transformation on the input vector of the second decoding layer to obtain a first query vector and a first key vector; determine the square root of the dimension of the first key vector; and normalize the ratio of the dot product of the first query vector and the first key vector to the square root of the dimension of the first key vector to obtain the attention matrix of the second decoding layer.

[0092] In one implementation, before performing a linear transformation on the input vector of the first decoding layer to obtain the first value vector, the second determining module is specifically used to determine the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer, where the third decoding layer is any decoding layer in the model before the first decoding layer; and to determine the second decoding layer based on the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer.

[0093] For example, the second determining module is specifically used to extract the last row of the attention matrix of the first decoding layer and the last row of the attention matrix of the third decoding layer; and to determine the similarity between the last row of the attention matrix of the first decoding layer and the last row of the attention matrix of the third decoding layer as the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer.

[0094] In one implementation, the second determining module is specifically used to: if the similarity between the attention matrix of the first decoding layer and the attention matrix of any third decoding layer is greater than a preset threshold, determine any third decoding layer as a candidate second decoding layer, and obtain one or more candidate second decoding layers; randomly select one candidate second decoding layer from the one or more candidate second decoding layers as the second decoding layer, or select the candidate second decoding layer that is closest to the first decoding layer from the one or more candidate second decoding layers as the second decoding layer.

[0095] In one implementation, the second determining module is specifically used to determine the third decoding layer with the highest similarity to the attention matrix of the first decoding layer as the second decoding layer based on the similarity between the attention matrix of the first decoding layer and the attention matrix of any third decoding layer.

[0096] In one implementation, the second decoding layer is adjacent to the first decoding layer.

[0097] The apparatus provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0098] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0099] Figure 6 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 may include at least one processor 601 for implementing the attention-based decoding method provided in the embodiments of this application.

[0100] Optionally, the electronic device 600 further includes at least one memory 602 for storing program instructions and / or data. The memory 602 is coupled to the processor 601. The coupling in this embodiment is an indirect coupling or communication connection between devices, units, or modules, and can be electrical, mechanical, or other forms, used for information exchange between devices, units, or modules. The processor 601 may operate in conjunction with the memory 602. The processor 601 may execute program instructions stored in the memory 602. At least one of the at least one memory may be included in the processor.

[0101] Optionally, the electronic device 600 further includes a communication interface 603 for communicating with other devices via a transmission medium, thereby enabling the electronic device 600 to communicate with other devices. The communication interface 603 may be, for example, a transceiver, interface, bus, circuit, or a device capable of transmitting and receiving functions. The processor 601 can utilize the communication interface 603 to transmit and receive data and / or information, and to implement the methods provided in the embodiments of this application. For details, please refer to the detailed descriptions in the preceding embodiments; further elaboration is not repeated here.

[0102] This application embodiment does not limit the specific connection medium between the processor 601, memory 602, and communication interface 603. This application embodiment... Figure 6 The processor 601, memory 602, and communication interface 603 are connected via bus 604. Bus 604 is... Figure 6 The connections between other components are shown in thick lines only and are not intended to be limiting. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0103] It should be understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0104] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0105] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run by a processor, it causes the computer to perform the methods as described in any of the foregoing embodiments.

[0106] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the foregoing embodiments.

[0107] The terms “unit”, “module”, etc., used in this specification may be used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution.

[0108] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed apparatus, devices, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0109] The unit described as a separate component may or may not be physically separate. The component shown as a unit may or may not be a physical unit; that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. This computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., digital video disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0112] If this function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0113] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0114] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A decoding method based on an attention mechanism, characterized in that, The method includes: applying a first decoding layer to an attention-based model, wherein the model includes multiple cascaded decoding layers, and the first decoding layer is any decoding layer other than the first decoding layer among the multiple decoding layers; the method includes: The input vector of the first decoding layer is linearly transformed to obtain the first value vector; Obtain the attention matrix of the second decoding layer in the model, wherein the second decoding layer is preceding the first decoding layer; Based on the attention matrix of the second decoding layer, the first value vector is weighted and summed to obtain the attention output of the first decoding layer.

2. The method according to claim 1, characterized in that, The method further includes: The input vector of the second decoding layer is linearly transformed to obtain the first query vector and the first key vector; Determine the square root of the dimension of the first key vector, and normalize the ratio of the dot product of the first query vector and the first key vector to the square root of the dimension of the first key vector to obtain the attention matrix of the second decoding layer.

3. The method according to claim 1, characterized in that, Before performing a linear transformation on the input vector of the first decoding layer to obtain the first value vector, the method further includes: Determine the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer, wherein the third decoding layer is any decoding layer in the model preceding the first decoding layer; The second decoding layer is determined based on the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer.

4. The method according to claim 3, characterized in that, Determining the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer includes: Extract the last row of the attention matrix of the first decoding layer, and the last row of the attention matrix of the third decoding layer; The similarity between the last row of the attention matrix of the first decoding layer and the last row of the attention matrix of the third decoding layer is determined as the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer.

5. The method according to claim 3, characterized in that, Determining the second decoding layer based on the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer includes: If the similarity between the attention matrix of the first decoding layer and the attention matrix of any of the third decoding layers is greater than a preset threshold, then any of the third decoding layers is determined as a candidate second decoding layer, and one or more candidate second decoding layers are obtained. Alternatively, one of the candidate second decoding layers may be randomly selected as the second decoding layer, or the candidate second decoding layer that is closest to the first decoding layer among the one or more candidate second decoding layers may be selected as the second decoding layer.

6. The method according to claim 3, characterized in that, Determining the second decoding layer based on the similarity between the attention matrix of the first decoding layer and the attention matrix of the third decoding layer includes: Based on the similarity between the attention matrix of the first decoding layer and the attention matrix of any of the third decoding layers, the third decoding layer with the highest similarity to the attention matrix of the first decoding layer is determined as the second decoding layer.

7. The method according to claim 1, characterized in that, The second decoding layer is adjacent to the first decoding layer.

8. A text generation method for a large language model, characterized in that, The large language model includes an embedding layer, an encoder, and a decoder, wherein the decoder includes multiple cascaded decoding layers, and the method includes: The text to be processed is input into the embedding layer, which converts the text into a text vector and outputs it to the encoder. The encoder generates a contextual representation of the text vector and outputs it to the decoder. The decoder generates an output sequence based on the context and processes the output sequence into corresponding text, wherein a portion of the decoding layer in the decoder performs the attention-based decoding method as described in any one of claims 1 to 7.

9. A decoding device based on an attention mechanism, characterized in that, include: The transformation module is used to linearly transform the input vector of the first decoding layer of the attention mechanism-based model to obtain a first value vector; wherein, the model includes multiple cascaded decoding layers, and the first decoding layer is any decoding layer other than the first decoding layer among the multiple decoding layers; The acquisition module is used to acquire the attention matrix of the second decoding layer in the model, wherein the second decoding layer is before the first decoding layer; The output module is used to perform a weighted summation of the first value vector based on the attention matrix of the second decoding layer to obtain the attention output of the first decoding layer.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory, wherein the computer program, when executed, causes the processor to perform the method of any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images