Neural network model processing method, device, equipment, medium and program product

Abstract

The present application relates to a processing method, device, computer equipment, computer-readable storage medium and computer program product for a neural network model. The method comprises: obtaining an embedding matrix corresponding to an input sequence through an embedding representation layer of a target neural network model; obtaining the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer when obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model; when the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix satisfy a preset relationship, reading a pre-calculated factor matrix from a local cache, and calculating the attention weight corresponding to the embedding matrix according to the embedding matrix and the pre-calculated factor matrix; the weight matrix includes a query matrix and a key matrix, and the pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, which can take into account both the amount of calculation and the amount of parameters in the model calculation process.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium and computer program product for processing a neural network model. Background Art

[0002] With the continuous development of artificial intelligence and natural language processing technology, the Transformer model based on neural networks has become one of the most important models in deep learning. The Transformer model is used to process sequence data and is implemented based on the attention mechanism. It is widely used in different fields, mainly including natural language processing, computer vision, speech recognition, reinforcement learning and other fields.

[0003] The components of the Transformer model include self-attention mechanism, multi-head attention mechanism, etc. The self-attention mechanism can calculate the similarity of elements at any two positions in the sequence. The multi-head attention mechanism is an extension of the self-attention mechanism. By introducing multiple attention heads, the multi-head attention mechanism enables the model to learn richer feature representations from different subspaces.

[0004] In the Transformer model, attention weight is the part with the largest amount of calculation. In related technologies, weight fusion technology is proposed to reduce the amount of calculation, but it leads to an increase in the number of parameters. How to balance the amount of calculation and the number of parameters is an urgent problem to be solved. Summary of the invention

[0005] Based on this, it is necessary to provide a processing method, device, computer equipment, computer-readable storage medium and computer program product for a neural network model that can take into account both the amount of calculation and the amount of parameters in order to address the above technical problems.

[0006] In a first aspect, the present application provides a method for processing a neural network model, comprising:

[0007] Obtain the embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model;

[0008] When obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, obtaining the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer;

[0009] When the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, a pre-computed factor matrix is ​​read from the local cache, and the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix and the pre-computed factor matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-computed factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution degree of the elements in the input sequence.

[0010] In a second aspect, the present application also provides a processing device for a neural network model, comprising:

[0011] An embedding representation module is used to obtain an embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model;

[0012] A model parameter acquisition module, configured to obtain a feature dimension of the embedding matrix and a feature dimension of a weight matrix of the attention layer when obtaining an attention weight corresponding to the embedding matrix through the attention layer of the target neural network model;

[0013] An attention weight calculation module is used to read a pre-calculated factor matrix from a local cache when the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, and calculate the attention weight corresponding to the embedding matrix according to the embedding matrix and the pre-calculated factor matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution of the elements in the input sequence.

[0014] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0015] Obtain the embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model;

[0016] When obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, obtaining the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer;

[0017] When the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, a pre-computed factor matrix is ​​read from the local cache, and the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix and the pre-computed factor matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-computed factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution degree of the elements in the input sequence.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0019] Obtain the embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model;

[0020] When obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, obtaining the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer;

[0021] When the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, a pre-computed factor matrix is ​​read from the local cache, and the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix and the pre-computed factor matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-computed factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution degree of the elements in the input sequence.

[0022] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0023] Obtain the embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model;

[0024] When obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, obtaining the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer;

[0025] When the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, a pre-computed factor matrix is ​​read from the local cache, and the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix and the pre-computed factor matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-computed factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution degree of the elements in the input sequence.

[0026] The processing method, apparatus, computer device, computer-readable storage medium and computer program product of the above-mentioned neural network model obtain the embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model, and when obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, obtain the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer, and when the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix satisfy the preset relationship, read the pre-calculated factor matrix cached locally, and calculate the attention weight corresponding to the embedding matrix according to the embedding matrix and the pre-calculated factor matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution of elements in the input sequence.

[0027] Since the cached pre-computed factor matrix is ​​used to calculate the attention weights corresponding to the embedding matrix, the pre-computed factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix. This can avoid the problem of large amount of calculation caused by directly calculating the attention weights corresponding to the embedding matrix according to the embedding matrix, the query matrix and the key matrix during the model inference process. At the same time, it is also set that the cached pre-computed factor matrix is ​​used to calculate the attention weights corresponding to the embedding matrix only when the feature dimension of the embedding matrix and the feature dimension of the weight matrix meet the preset relationship. This can avoid the situation where the number of parameters of the pre-computed factor matrix is ​​greater than the original number of parameters of the query matrix and the key matrix while reducing the amount of calculation. Therefore, the number of parameters and the amount of data reading and writing of the target neural network model can be saved, and the model calculation efficiency can be further improved, so as to achieve a balance between the amount of calculation and the amount of parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1An application environment diagram of a method for processing a neural network model in one embodiment;

[0030] Figure 2 A schematic diagram of a process flow of a neural network model processing method in one embodiment;

[0031] Figure 3 A schematic diagram of a flow chart for calculating attention weights in one embodiment;

[0032] Figure 4 is a schematic diagram of a calculation process of an attention score in one embodiment;

[0033] Figure 5 A schematic diagram of a process flow of a neural network model in another embodiment;

[0034] Figure 6 A schematic diagram of adaptively selecting whether to perform calculation amount optimization in one embodiment;

[0035] Figure 7 is a structural block diagram of a processing device for a neural network model in one embodiment;

[0036] Figure 8 is an internal structure diagram of a computer device in one embodiment;

[0037] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] The processing method of the neural network model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. In one embodiment, the computer device can obtain the embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model; when the attention weight corresponding to the embedding matrix is ​​obtained through the attention layer of the target neural network model, the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer are obtained; when the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix meet the preset relationship, the pre-calculated factor matrix cached is read from the local, and the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix and the pre-calculated factor matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution of the elements in the input sequence.

[0040] The computer device may be the terminal 102 or the server 104 mentioned above. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0041] In a specific application scenario, the target neural network model is a language model based on Transformer. The terminal 102 can obtain a source sentence input by a user, the language of the source sentence is a first language, the terminal 102 can generate a translation request based on the input source sentence, for translating the source sentence into a target sentence, the language of the target sentence is a second language, the terminal 102 sends the translation request to the server 104, the server 104 uses the target neural network model to translate the source sentence carried in the translation request, and outputs the target sentence, wherein, in the process of the server 104 using the target neural network model to translate the source sentence, the embedding matrix corresponding to the source sentence as the input sequence can be obtained through the embedding representation layer of the target neural network model; when the attention weight corresponding to the embedding matrix is ​​obtained through the attention layer of the target neural network model, the server 104 can read the feature dimension of the embedding matrix and the feature dimension of the weight matrix of the attention layer from the local; when the feature dimension of the embedding matrix and the feature dimension of the weight matrix meet the preset relationship, the server 104 can read the pre-calculated factor matrix obtained by performing a dot product operation on the query matrix and the key matrix from the local, calculate the attention weight corresponding to the embedding matrix according to the embedding matrix and the pre-calculated factor matrix, and the server 104 can output the target sentence corresponding to the source sentence according to the attention weight.

[0042] In another specific application scenario, the target neural network model is a Transformer-based image recognition model, which is used to identify the category of objects in an input image. The terminal 102 can obtain an input image, and when image recognition is required for the input image, the input image can be sent to the server 104. Before the server 104 executes the method provided in the embodiment of the present application, the visual tokens of the input image can be obtained. These visual tokens are input as an input sequence into the target neural network model, and the embedding matrix corresponding to the input sequence is obtained through the embedding representation layer of the target neural network model. Similarly, the server 104 can obtain the attention weight corresponding to the embedding matrix according to the method provided in the embodiment of the present application, and obtain the image recognition result about the input image based on the attention weight, and the image recognition result indicates the category to which the object in the input image belongs.

[0043] Of course, in some embodiments, the processing method of the neural network model applied to the above application scenario can also be executed separately by the terminal 102. In some embodiments, the target neural network model can also be other models based on Transformer, for example, a large cross-modal model of pictures and texts, an image positioning model, a text generation model, and so on.

[0044] In an exemplary embodiment, Figure 2 As shown, a processing method for a neural network model is provided, and the method is applied to Figure 1 The computer device (terminal 102 or server 104) in the example is used for description, and the steps include the following steps 202 to 206. Among them:

[0045] Step 202, obtaining an embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model.

[0046] Among them, the target neural network model is a neural network model based on Transformer. For example, the target neural network model can be a translation model, image recognition model, text generation model, etc. built based on Transformer. There are several different attention mechanisms in Transformer. The first is the self-attention mechanism in the encoder Encoder. In the attention layer of the encoder, the attention weight of the source sequence and itself needs to be calculated. The second is the self-attention mechanism in the decoder Decoder. In the attention layer of the decoder, the attention weight of the target sequence and itself needs to be calculated. The third is in the end-to-end structure Encoder-Decoder from encoder to decoder, which needs to calculate the attention weight of the target sequence to the source sequence. In the embodiment of the present application, the input sequence can be the above-mentioned source sequence, and the attention layer of the target neural network model is used to calculate the attention weight of the source sequence and itself. In the embodiment of the present application, the input sequence can also be the above-mentioned target sequence, and the attention layer of the target neural network model is used to calculate the attention weight of the target sequence and itself. In the embodiment of the present application, the input sequence can also include the target sequence and the source sequence, and the attention layer of the target neural network model is used to calculate the attention weight of the target sequence to the source sequence. In different application scenarios, the source sequence can be objects of different forms. For example, the source sequence can be an input sentence, an input image, an input text, an input video, etc., and the target sequence can be a partial sentence currently output when decoding is not completed.

[0047] In an embodiment of the present application, the target neural network model includes at least an embedding representation layer and an attention layer. The embedding representation layer is used to generate a corresponding embedding representation for the input sequence. The embedding representation is a high-dimensional vector representation. Each element in the input sequence will be mapped to a point in a high-dimensional vector space. This mapping process is called embedding. That is, the computer device inputs the input sequence into the embedding representation layer of the target neural network model, and the embedding representation layer can generate a corresponding embedding vector for each element in the input sequence, thereby obtaining an embedding matrix corresponding to the entire input sequence. The embedding matrix is ​​then passed by the computer device to the attention layer of the target neural network model, and the weight matrix in the attention layer is used to calculate the attention weight (Attention Weight) corresponding to the embedding matrix.

[0048] Step 204, when obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, obtain the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer.

[0049] Generally, when a computer device obtains the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, it calculates the attention weight corresponding to the embedding matrix based on the embedding matrix, the query matrix, and the key matrix. Taking the self-attention mechanism as an example, Figure 3 As shown, a schematic diagram of a process of calculating the attention weight corresponding to the embedding matrix according to the embedding matrix, the query matrix and the key matrix is ​​shown, including steps 302 to 310:

[0050] Step 302, calculate the query vector matrix corresponding to the embedding matrix according to the query matrix in the attention layer;

[0051] Step 304, calculate the key vector matrix corresponding to the embedding matrix according to the key matrix in the attention layer;

[0052] Step 306, calculating the value vector matrix corresponding to the embedding matrix according to the value matrix in the attention layer;

[0053] Step 308, performing a matrix dot product operation on the transpose of the query vector matrix and the key vector matrix to obtain an attention weight;

[0054] Step 310, perform matrix multiplication operation on the attention weight and the value vector matrix to obtain the attention score.

[0055] The attention score is obtained by weighting each value vector in the value vector matrix according to the attention weight, which is the encoded representation assigned by the attention layer to each element in the input sequence.

[0056] Figure 4 FIG. 1 is a schematic diagram of a calculation process of an attention score in one embodiment. Figure 4, taking the input sequence S={S1,S2,S3,S4} as an example, the input sequence includes 4 elements. The embedding representation layer can generate a corresponding embedding vector for each element in the input sequence, thereby obtaining the embedding matrix X={X1,X2,X3,X4} corresponding to the entire input sequence. Subsequently, the embedding matrix {X1, X2, X3, X4} is passed to the attention layer of the target neural network model by the computer device, and the weight matrices WQ, WK and WV in the attention layer are used to obtain the query vector matrix Q={Q1, Q2, Q3, Q4}, key vector matrix K={K1, K2, K3, K4} and value vector matrix V={V1, V2, V3, V4} corresponding to the embedding matrix respectively. According to the query vector matrix {Q1, Q2, Q3, Q4} and the key vector matrix {K1, K2, K3, K4}, the attention weight QK={QK1, QK2, QK3, QK4} corresponding to the embedding matrix can be calculated. According to the attention weight {QK1, QK2, QK3, QK4} and the value vector matrix {V1, V2, V3, V4}, the attention score Z={Z1, Z2, Z3, Z4} can be calculated.

[0057] It can be seen from this process that in order to obtain the attention weight corresponding to the embedding matrix, the computer device executes at least steps 302 to 306, that is, it needs to read the query matrix and key matrix of the target neural network model and then perform three calculation processes.

[0058] In order to reduce the amount of calculation when calculating the attention weight, in the embodiment of the present application, when it is necessary to obtain the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, the computer device can read the pre-calculated factor matrix that has been calculated and cached locally, and calculate the attention weight corresponding to the embedding matrix according to the embedding matrix and the pre-calculated factor matrix. The weight matrix includes a query matrix and a key matrix. The pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix. The attention weight is used to characterize the contribution of the elements in the input sequence.

[0059] Taking the input sequence including S1 and S2 as an example, the derivation process of the pre-calculated factor matrix is ​​as follows:

[0060] Attention Weight = (S1 * WQ + b_Q) * (S2*WK + b_K)_T

[0061] = (S1 * WQ + b_Q ) * (WK _T * S2_T + b_K.view(1,k)_T)

[0062] =S1 * WQ * WK _T *S2 + S1 * WQ * b_K.view(1,k)_T + b_Q.view(1,q)* WK_T * S2_T + b_Q.view(1,q)* b_K.view(1,k)_T

[0063] = S1 * (WQ * WK _T) *S2 + S1 * (WQ * b_K.view(1,k)_T) + (b_Q.view(1,q)* WK _T) * S2_T + (b_Q.view(1,q)* b_K.view(1,k)_T).

[0064] Among them, b_Q represents the bias term of the weight matrix WQ, b_K represents the bias term of the weight matrix WK, "_T" represents the transposed matrix of the matrix, b_K.view(1,k) represents the conversion of the bias term b_K of the weight matrix WK into a matrix of size (1, k), k is the dimension of the weight matrix WK, b_Q.view(1,q) represents the conversion of the bias term of the weight matrix WQ into a matrix of size (1, q), q is the dimension of the weight matrix WQ. It can be understood that the values ​​of k and q are usually equal.

[0065] Based on the last line of the above formula, it can be known that the attention weight is finally expressed as the sum of four items. The pre-calculated factor in the embodiment of the present application at least includes the calculation result M1 corresponding to WQ * WK _T in the first item S1 * (WQ * WK _T) * S2. In this way, when the first item S1 * (WQ * WK _T) * S2 is calculated by the attention layer, the calculation result of the first item can be directly obtained according to the embedding matrix and the pre-calculated factor matrix, that is, the first item is S1 * M1 * S2, and the calculation result of the first item can be used to obtain the attention weight corresponding to the embedding matrix. Since in the model training or reasoning process, only the embedding matrix and the pre-calculated factor matrix need to be calculated twice to obtain the calculation result of the first item S1 * M1 * S2, compared to the calculation of S1 * (WQ * WK _T) * S2 in the model training or reasoning process, which requires 3 calculation processes, it is obvious that the amount of calculation can be reduced and the model operation efficiency can be improved. In some embodiments, based on the above derivation results, the bias item matrix can also be calculated offline, and the bias item matrix includes: the calculation result M2 of WQ * b_K.view(1,k)_T in the second item, the calculation result M3 of b_Q.view(1,q)* WK _T in the third item, and the calculation result M4 of b_Q.view(1,q)* b_K.view(1,k)_T in the fourth item to further reduce the amount of calculation.

[0066] It can be understood that if it is a self-attention mechanism, S1 and S2 in the above formula have the same value.

[0067] It can be understood that if there is no bias term in the weight matrix, that is, the bias term is 0, the above formula can be expressed as: Attention Weight = (S1 * WQ) * (S2*WK)_T

[0068] = S1 * (WQ * WK _T) *S2.

[0069] Furthermore, this method of storing pre-calculated factors and calculating attention weights by reading the locally stored pre-calculated factors to reduce the amount of calculation brings about a change in the number of parameters, because the weight matrices WQ and WK were originally stored, and now the pre-calculated factors M1 need to be stored.

[0070] In order to avoid the situation where the number of parameters of the pre-calculated factors increases in magnitude relative to the original number of parameters of the weight matrices WQ and WK, resulting in an increase in the amount of data read, insignificant optimization of computational efficiency, and a large amount of storage space of the computer device, when the computer device needs to obtain the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, it obtains the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer, wherein the characteristic dimension of the embedding matrix is ​​the vector dimension of the input sequence embedded in the high-dimensional vector space, also known as the Embedding Dimension, which can be recorded as d_model, and the characteristic dimension of the weight matrix of the attention layer is the hidden layer dimension in the target neural network model, which is the dimension of the feature representation that the model can learn, and it directly affects the model's processing ability for the input data and the fineness of feature extraction, which can be recorded as d_hidden. The computer device can obtain these two values ​​locally, and these two values ​​can be equal. It should be noted that for a built neural network model, these two values ​​are fixed, and the computer device can execute the steps of obtaining the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer before performing model inference or model training.

[0071] The computer device may measure whether to calculate the attention weight corresponding to the embedding matrix according to the embedding matrix, the query matrix and the key matrix, or to calculate the attention weight corresponding to the embedding matrix according to the embedding matrix and the pre-calculated factor matrix during the model reasoning or training process, based on the relationship between the feature dimension of the embedding matrix and the feature dimension of the weight matrix of the attention layer. The computer device may perform the step of determining whether the feature dimension of the embedding matrix and the weight matrix of the attention layer satisfy a preset relationship before performing model reasoning or model training.

[0072] Step 206, when the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, read the pre-computed factor matrix cached locally, and calculate the attention weight corresponding to the embedding matrix according to the embedding matrix and the pre-computed factor matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-computed factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution of the elements in the input sequence.

[0073] That is to say, the computer device reads the pre-calculated factor matrix from the local cache only when the feature dimension d_model of the embedding matrix and the feature dimension d_hidden of the weight matrix satisfy the preset relationship, and calculates the attention weights corresponding to the embedding matrix according to the embedding matrix and the pre-calculated factor matrix, so as to avoid the problem that the number of parameters of the pre-calculated factors increases by an order of magnitude relative to the original number of parameters of the weight matrices WQ and WK, resulting in an increase in the amount of data read, unclear optimization of computing efficiency, and a large amount of storage space occupied by the computer device.

[0074] The principle of setting the condition that the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy the preset relationship is derived as follows: the length of the input sequence is denoted as N, the feature dimension of the embedding vector is denoted as d_model, then the size of the embedding matrix is ​​N*d_model.

[0075] For the single-head attention mechanism, the size of its weight matrices WQ and WK is d_model*d_hidden, then the total parameter amount of the two weight matrices is recorded as Num1, Num1=2*d_model*d_hidden; the pre-calculated factor M1=WQ * WK _T, the parameter amount of the pre-calculated factor is recorded as Num2, Num2=d_model* d_model. In other words, when Num2 is less than or equal to Num1, using the embedding matrix and the pre-calculated factor matrix to calculate the attention weight corresponding to the embedding matrix will not lead to an increase in the number of parameters. Num2 is less than or equal to Num1, that is, d_model≤2*d_hidden. Therefore, based on this conclusion, a preset relationship can be set, that is, the preset relationship satisfied by the feature dimension of the embedding matrix and the feature dimension of the weight matrix can be d_model≤2*d_hidden, or d_model≤d_hidden, d_model≤2*d_hidden±a, a can be a smaller value, such as 20, 10, and so on.

[0076] For the multi-head attention mechanism, the number of heads is recorded as hc, and the size of the weight matrices WQ and WK in each head is d_model*d_hidden. The total number of parameters of the two weight matrices is recorded as Num3, Num3= 2*d_model*d_hidden*hc, the pre-calculated factor M1=WQ * WK _T, and the number of parameters of the pre-calculated factor is recorded as Num4, Num4=d_model* d_model*hc. In other words, when Num3 is less than or equal to Num3, using the embedding matrix and the pre-calculated factor matrix to calculate the attention weights corresponding to the embedding matrix will not lead to an increase in the number of parameters. Num3 is less than or equal to Num4, that is, d_model≤2*d_hidden. This conclusion is consistent with the above conclusion and has nothing to do with the number of heads hc of the multi-head attention.

[0077] It can be understood that when d_model and d_hidden are the same, this conclusion always holds true, and the computer device can use the embedding matrix and the pre-calculated factor matrix to calculate the attention weight corresponding to the embedding matrix.

[0078] It can be understood that the calculation result M2 of WQ * b_K.view(1,k)_T in the second item, the calculation result M3 of b_Q.view(1,q)* WK_T in the third item, and the calculation result M4 of b_Q.view(1,q)* b_K.view(1,k)_T in the fourth item, since the number of parameters in the bias term is very small, the number of parameters in the calculation results M2, M3 and M4 is not in the same order of magnitude as the number of parameters in M1 and can be ignored.

[0079] In the processing method of the above neural network model, since the cached pre-calculated factor matrix is ​​used to calculate the attention weights corresponding to the embedding matrix, the pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix. This can avoid the problem of large amount of calculation caused by directly calculating the attention weights corresponding to the embedding matrix according to the embedding matrix, the query matrix and the key matrix during model inference or training. At the same time, it is also provided that the locally cached pre-calculated factor matrix is ​​used to calculate the attention weights corresponding to the embedding matrix only when the feature dimension of the embedding matrix and the feature dimension of the weight matrix meet the preset relationship. This can avoid the situation where the number of parameters of the pre-calculated factor matrix is ​​far greater than the number of parameters of the query matrix and the key matrix while reducing the amount of calculation. Therefore, the number of parameters and the amount of data reading and writing of the target neural network model can be saved, and the model calculation efficiency can be further improved. The number of parameters will not increase significantly, and the storage space of the computer device can be saved, thereby taking into account both the amount of calculation and the amount of parameters.

[0080] In an exemplary embodiment, Figure 5FIG. 1 is a flowchart of processing a neural network model in another embodiment, referring to FIG. Figure 5 In addition to the above steps 202 to 206, step 208 is also included. When the feature dimension of the embedding matrix and the feature dimension of the weight matrix do not satisfy the preset relationship, the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix, the query matrix and the key matrix.

[0081] In this embodiment, the decision on whether to optimize the amount of calculation is made based on whether the feature dimension of the embedding matrix and the feature dimension of the weight matrix meet the specified preset relationship. That is, when the feature dimension of the embedding matrix and the feature dimension of the weight matrix meet the preset relationship, the cached pre-calculated factor matrix is ​​used to calculate the attention weight corresponding to the embedding matrix. When the feature dimension of the embedding matrix and the feature dimension of the weight matrix do not meet the preset relationship, the attention weight corresponding to the embedding matrix is ​​directly calculated according to the embedding matrix, the query matrix and the key matrix. In this way, the parameter amount and data reading and writing amount of the target neural network model can be saved, the model calculation efficiency can be further improved, and both the amount of calculation and the amount of parameters can be taken into account.

[0082] In an exemplary embodiment, before obtaining the attention weights corresponding to the embedding matrix through the attention layer of the target neural network model, the method also includes: obtaining a query matrix and a key matrix in the attention layer of the target neural network model; performing a dot product operation based on the query matrix and the key matrix to obtain a pre-calculated factor matrix; and storing the pre-calculated factor matrix.

[0083] In this embodiment, before performing model inference, the computer device can obtain the query matrix and key matrix in the attention layer of the target neural network model; perform a dot product operation based on the query matrix and the key matrix to obtain a pre-calculated factor matrix; store the pre-calculated factor matrix in a local storage space, and the pre-calculated factor can be read from the storage space when the attention weight corresponding to the embedding matrix is ​​obtained through the attention layer of the target neural network model.

[0084] In an exemplary embodiment, performing a dot product operation on the query matrix and the key matrix to obtain a pre-calculated factor matrix includes:

[0085] ;

[0086] in, represents the precomputed factor matrix, Represents the query matrix in the attention layer. The query matrix is ​​used to perform matrix multiplication with the embedding matrix to obtain the query vector matrix corresponding to the embedding matrix. Represents the key matrix in the attention layer. The key matrix is ​​used to perform matrix multiplication with the embedding matrix to obtain the key vector matrix corresponding to the embedding matrix.

[0087] In an exemplary embodiment, the query matrix has a first bias term, and the key matrix has a second bias term; when the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, a pre-calculated factor matrix cached locally is read, and the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix and the pre-calculated factor matrix, including:

[0088] When the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy the preset relationship, the pre-calculated factor matrix and the bias matrix are read from the local cache, and the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix, the pre-calculated factor matrix and the bias matrix;

[0089] Among them, the bias item matrix includes a first bias item matrix, a second bias item matrix and a third bias item matrix. The first bias item matrix is ​​obtained by performing a dot product operation on the query matrix and the second bias item, the second bias item matrix is ​​obtained by performing a dot product operation on the key matrix and the first bias item, and the third bias item matrix is ​​obtained by performing a dot product operation on the first bias item and the second bias item.

[0090] Based on the previous introduction, the bias item matrix is ​​read from the local cache. The bias item matrix includes: the calculation result M2 of WQ *b_K.view(1,k)_T in the second item is the first bias item matrix, the calculation result M3 of b_Q.view(1,q)* WK _T in the third item is the second bias item matrix, and the calculation result M4 of b_Q.view(1,q)* b_K.view(1,k)_T in the fourth item is the third bias item matrix.

[0091] In this embodiment, the amount of calculation can be further reduced by reading the pre-calculated factor matrix and bias matrix from the local cache when the feature dimension of the embedding matrix and the feature dimension of the weight matrix meet the preset relationship, and calculating the attention weights corresponding to the embedding matrix according to the embedding matrix, the pre-calculated factor matrix and the bias matrix. At the same time, the number of parameters of the bias matrix is ​​very small, and the number of parameters of the calculation results M2, M3 and M4 is not in the same order of magnitude as the number of parameters of M1, which can be ignored. Therefore, while further reducing the amount of calculation, it will not cause a large increase in the number of parameters, and both the amount of calculation and the amount of parameters can be taken into account.

[0092] In an exemplary embodiment, the method also includes: when the feature dimension of the embedding matrix and the feature dimension of the weight matrix do not satisfy a preset relationship, calculating the attention weight corresponding to the embedding matrix based on the embedding matrix, the query matrix, the key matrix, the first bias item of the query matrix and the second bias item of the key matrix.

[0093] In this embodiment, whether to optimize the amount of calculation is determined based on whether the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix meet the preset relationship. That is, when the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix meet the preset relationship, the cached pre-calculated factor matrix and the bias matrix are used to calculate the attention weight corresponding to the embedding matrix. When the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix do not meet the preset relationship, the attention weight corresponding to the embedding matrix is ​​directly calculated based on the embedding matrix, the query matrix, the key matrix, the first bias item of the query matrix and the second bias item of the key matrix. In this way, the parameter amount and data reading and writing amount of the target neural network model can be saved, and the model calculation efficiency can be further improved, so as to achieve a balance between the amount of calculation and the amount of parameters.

[0094] In an exemplary embodiment, the target neural network model is implemented based on a multi-head attention mechanism, and the attention layer of the target neural network model is any attention layer in the multi-head attention. The method also includes: after calculating the attention weight corresponding to the embedding matrix in each attention layer, obtaining the attention score in each attention layer based on the attention weight, concatenating and fusing the attention scores corresponding to the embedding matrix calculated in each attention layer to obtain a multi-head attention score, which is used to characterize the attention coding representation of the elements in the input sequence.

[0095] like Figure 6 FIG. 1 is a schematic diagram of an embodiment of adaptively selecting whether to optimize the amount of calculation. Figure 6 , obtain the hyperparameters of the target neural network model, including the feature dimension d_model of the embedding matrix and the feature dimension d_hidden of the weight matrix of the attention layer, obtain the weight matrix in the attention layer of the target neural network model, including the query matrix and the key matrix, perform a dot product operation on the query matrix and the key matrix to obtain the pre-calculated factor matrix, and store the pre-calculated factor matrix locally. If d_model and d_hidden meet the specified preset relationship, the calculation amount is optimized. If d_model and d_hidden do not meet the specified preset relationship, the calculation amount is not optimized.

[0096] In a specific embodiment, the processing method of the neural network model executed by a computer device includes the following steps:

[0097] 1. Obtain the hyperparameters of the target neural network model, including the feature dimensions of the embedding matrix and the feature dimensions of the weight matrix of the attention layer;

[0098] 2. Obtain the weight matrix of the target neural network model. The weight matrix includes the query matrix and the key matrix. Perform a dot product operation on the query matrix and the key matrix to obtain a pre-calculated factor matrix, and store the pre-calculated factor matrix locally.

[0099] 3. Get the bias term of the weight matrix, the query matrix has the first bias term, and the key matrix has the second bias term;

[0100] 4. Perform a dot product operation on the query matrix and the second bias item to obtain a first bias item matrix, perform a dot product operation on the key matrix and the first bias item to obtain a second bias item matrix, perform a dot product operation on the first bias item and the second bias item to obtain a third bias item matrix, and store the first bias item matrix, the second bias item matrix, and the third bias item matrix locally;

[0101] 5. Obtain the input sequence and obtain the embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model;

[0102] 6. When the attention weights corresponding to the embedding matrix are obtained through the attention layer of the target neural network model, when the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy the preset relationship, the pre-calculated factor matrix and the bias item matrix are read from the local cache, and the attention weights corresponding to the embedding matrix are calculated according to the embedding matrix, the pre-calculated factor matrix and the bias item matrix; when the feature dimension of the embedding matrix and the feature dimension of the weight matrix do not satisfy the preset relationship, the attention weights corresponding to the embedding matrix are calculated according to the embedding matrix, the query matrix and the key matrix, as well as the first bias item of the query matrix and the second bias item of the key matrix. The attention weights are used to characterize the contribution of the elements in the input sequence.

[0103] 7. Obtain the attention score corresponding to the input sequence according to the attention weight;

[0104] 8. Obtain the output result corresponding to the input sequence based on the attention score.

[0105] The method provided in the embodiment of the present application can adaptively choose whether to optimize the amount of computation to ensure that the model obtains latency and power consumption benefits under completely equivalent logic; by offline calculation and saving the matrix multiplication results, it can ensure that the amount of model computation is reduced, which is applicable to both single-head attention mechanism and multi-head attention mechanism.

[0106] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiment of the present application also provides a processing device for a neural network model for implementing the processing method of the neural network model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the processing device for one or more neural network models provided below can refer to the limitations of the processing method for the neural network model above, and will not be repeated here.

[0108] In an exemplary embodiment, Figure 7 As shown, a processing device 700 for a neural network model is provided, comprising: an embedding representation module 702, a model parameter acquisition module 704 and an attention weight calculation module 706, wherein:

[0109] An embedding representation module 702, used to obtain an embedding matrix corresponding to the input sequence through an embedding representation layer of a target neural network model;

[0110] A model parameter acquisition module 704 is used to obtain the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer when obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model;

[0111] The attention weight calculation module 706 is used to read the pre-calculated factor matrix from the local cache when the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, and calculate the attention weight corresponding to the embedding matrix according to the embedding matrix and the pre-calculated factor matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution of the elements in the input sequence.

[0112] In an exemplary embodiment, the attention weight calculation module 706 is also used to calculate the attention weight corresponding to the embedding matrix based on the embedding matrix, the query matrix and the key matrix when the feature dimension of the embedding matrix and the feature dimension of the weight matrix do not satisfy a preset relationship.

[0113] In an exemplary embodiment, the processing device 700 of the neural network model further includes:

[0114] The pre-calculation module is used to obtain the query matrix and the key matrix in the attention layer of the target neural network model; perform a dot product operation on the query matrix and the key matrix to obtain a pre-calculated factor matrix; and store the pre-calculated factor matrix.

[0115] In an exemplary embodiment, the pre-calculation module is further configured to Compute the precomputed factor matrix; where, represents the precomputed factor matrix, Represents the query matrix in the attention layer. The query matrix is ​​used to perform matrix multiplication with the embedding matrix to obtain the query vector matrix corresponding to the embedding matrix. Represents the key matrix in the attention layer. The key matrix is ​​used to perform matrix multiplication with the embedding matrix to obtain the key vector matrix corresponding to the embedding matrix.

[0116] In an exemplary embodiment, the query matrix has a first bias item, and the key matrix has a second bias item; the attention weight calculation module is also used to read the pre-calculated factor matrix and the bias item matrix from the local cache when the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, and calculate the attention weight corresponding to the embedding matrix according to the embedding matrix, the pre-calculated factor matrix and the bias item matrix; wherein the bias item matrix includes a first bias item matrix, a second bias item matrix and a third bias item matrix, the first bias item matrix is ​​obtained by performing a dot product operation on the query matrix and the second bias item, the second bias item matrix is ​​obtained by performing a dot product operation on the key matrix and the first bias item, and the third bias item matrix is ​​obtained by performing a dot product operation on the first bias item and the second bias item.

[0117] In an exemplary embodiment, the attention weight calculation module is also used to calculate the attention weight corresponding to the embedding matrix based on the embedding matrix, the query matrix, the key matrix, the first bias item of the query matrix and the second bias item of the key matrix when the feature dimension of the embedding matrix and the feature dimension of the weight matrix do not satisfy a preset relationship.

[0118] In an exemplary embodiment, the target neural network model is implemented based on a multi-head attention mechanism, the attention layer of the target neural network model is any attention layer of the multi-head attention, and the processing device 700 of the neural network model further includes:

[0119] The attention score calculation module is used to calculate the attention weight corresponding to the embedding matrix in each attention layer, obtain the attention score in each attention layer based on the attention weight, concatenate and fuse the attention scores corresponding to the embedding matrix calculated in each attention layer to obtain the multi-head attention score, which is used to characterize the attention coding representation of the elements in the input sequence.

[0120] The processing device 700 of the above-mentioned neural network model uses a calculated pre-calculated factor matrix to calculate the attention weights corresponding to the embedding matrix. The pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix. This can avoid the problem of large amount of calculation caused by directly calculating the attention weights corresponding to the embedding matrix according to the embedding matrix, the query matrix and the key matrix. At the same time, it is also provided that the calculated pre-calculated factor matrix is ​​used to calculate the attention weights corresponding to the embedding matrix only when the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix satisfy a preset relationship. This can avoid the situation where the parameter amount of the pre-calculated factor matrix is ​​greater than the parameter amount of the query matrix and the key matrix while reducing the amount of calculation. Therefore, the parameter amount and data reading and writing amount of the target neural network model can be saved, and the model calculation efficiency can be further improved, so as to achieve a balance between the amount of calculation and the amount of parameters.

[0121] Each module in the processing device 700 of the neural network model can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0122] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a processing method for a neural network model is implemented.

[0123] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a processing method for a neural network model is implemented.

[0124] Those skilled in the art will understand that Figure 8 , Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the processing method of the neural network model provided in the embodiment of the present application are implemented.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the processing method of the neural network model provided in the embodiment of the present application are implemented.

[0127] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the processing method of the neural network model provided in the embodiment of the present application.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0129] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0131] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A processing method for a neural network model, characterized in that: The method comprises: Obtain the embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model; When obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, obtaining the characteristic dimension of the embedding matrix and the characteristic dimension of the weight matrix of the attention layer; When the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, the pre-calculated factor matrix and the bias item matrix are read from the local cache, and the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix, the pre-calculated factor matrix and the bias item matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution degree of the elements in the input sequence; the query matrix has a first bias item, and the key matrix has a second bias item; the bias item matrix includes a first bias item matrix, a second bias item matrix and a third bias item matrix, the first bias item matrix is ​​obtained by performing a dot product operation on the query matrix and the second bias item, the second bias item matrix is ​​obtained by performing a dot product operation on the key matrix and the first bias item, and the third bias item matrix is ​​obtained by performing a dot product operation on the first bias item and the second bias item; The target neural network model is used to translate the source sentence carried in the translation request as the input sequence to obtain the attention weight, and output the target sentence corresponding to the source sentence according to the attention weight; or The target neural network model is used to perform image recognition processing on the input image as the input sequence to obtain the attention weight, and obtain an image recognition result about the input image according to the attention weight, wherein the image recognition result indicates the category to which the object in the input image belongs.

2. The method according to claim 1, characterized in that The method further comprises: When the feature dimension of the embedding matrix and the feature dimension of the weight matrix do not satisfy the preset relationship, the attention weight corresponding to the embedding matrix is ​​calculated according to the embedding matrix, the query matrix, the key matrix, the first bias term of the query matrix and the second bias term of the key matrix.

3. The method according to claim 1, characterized in that Before obtaining the attention weight corresponding to the embedding matrix through the attention layer of the target neural network model, the method further includes: Obtaining a query matrix and a key matrix in an attention layer of the target neural network model; Performing a dot product operation on the query matrix and the key matrix to obtain a pre-calculated factor matrix; The pre-computed factor matrix is ​​stored.

4. The method according to claim 3, characterized in that The performing a dot product operation on the query matrix and the key matrix to obtain a pre-calculated factor matrix includes: ； in, represents the precomputed factor matrix, represents a query matrix in the attention layer, and the query matrix is ​​used to perform a matrix multiplication operation with the embedding matrix to obtain a query vector matrix corresponding to the embedding matrix, Represents a key matrix in the attention layer, and the key matrix is ​​used to perform a matrix multiplication operation with the embedding matrix to obtain a key vector matrix corresponding to the embedding matrix.

5. The method according to any one of claims 1 to 4, characterized in that: The target neural network model is implemented based on a multi-head attention mechanism, the attention layer of the target neural network model is any attention layer of the multi-head attention, and the method further includes: After calculating the attention weight corresponding to the embedding matrix in each attention layer, the attention score in each attention layer is obtained based on the attention weight, and the attention scores corresponding to the embedding matrix calculated in each attention layer are concatenated and then fused to obtain a multi-head attention score, which is used to characterize the attention encoding representation of the elements in the input sequence.

6. A processing device for a neural network model, characterized in that: The device comprises: An embedding representation module is used to obtain an embedding matrix corresponding to the input sequence through the embedding representation layer of the target neural network model; A model parameter acquisition module, configured to obtain a feature dimension of the embedding matrix and a feature dimension of a weight matrix of the attention layer when obtaining an attention weight corresponding to the embedding matrix through the attention layer of the target neural network model; An attention weight calculation module is used to read the pre-calculated factor matrix and the bias item matrix from the local cache when the feature dimension of the embedding matrix and the feature dimension of the weight matrix satisfy a preset relationship, and calculate the attention weight corresponding to the embedding matrix according to the embedding matrix, the pre-calculated factor matrix and the bias item matrix; wherein the weight matrix includes a query matrix and a key matrix, the pre-calculated factor matrix is ​​a matrix obtained by performing a dot product operation on the query matrix and the key matrix, and the attention weight is used to characterize the contribution degree of the elements in the input sequence; the query matrix has a first bias item, and the key matrix has a second bias item; the bias item matrix includes a first bias item matrix, a second bias item matrix and a third bias item matrix, the first bias item matrix is ​​obtained by performing a dot product operation on the query matrix and the second bias item, the second bias item matrix is ​​obtained by performing a dot product operation on the key matrix and the first bias item, and the third bias item matrix is ​​obtained by performing a dot product operation on the first bias item and the second bias item; The target neural network model is used to translate the source sentence carried in the translation request as the input sequence to obtain the attention weight, and output the target sentence corresponding to the source sentence according to the attention weight; or The target neural network model is used to perform image recognition processing on the input image as the input sequence to obtain the attention weight, and obtain an image recognition result about the input image according to the attention weight, wherein the image recognition result indicates the category to which the object in the input image belongs.

7. The device according to claim 6, characterized in that The attention weight calculation module is also used to calculate the attention weight corresponding to the embedding matrix according to the embedding matrix, the query matrix, the key matrix, the first bias item of the query matrix and the second bias item of the key matrix when the feature dimension of the embedding matrix and the feature dimension of the weight matrix do not satisfy the preset relationship.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images