Method and electronic apparatus for computation on transformer-based neural network

Abstract

This invention proposes a computational method and electronic device for bridging an attention layer and a multilayer perceptron layer in a neural network based on a transformation model. The method includes receiving the residuals and attention matrix of the attention layer; obtaining a first low-rank weight matrix and a second low-rank weight matrix of the output score generated by low-rank decomposition of the weight matrix of the output score; obtaining a third merging weight; and performing calculations based on the residuals, attention matrix, the first low-rank weight matrix of the output score, and the third merging weight to generate the output of a specified path in the multilayer perceptron layer, wherein the third merging weight is pre-calculated based on the second low-rank weight matrix of the output score and the weight matrix of the multilayer perceptron layer.

Description

[Technical Field]

[0001] This invention relates to a computational method and electronic device for neural networks based on a transformation model. [Previous Technology]

[0002] Transitional models are deep learning architectures that have revolutionized natural language processing across various tasks and achieved state-of-the-art results. Attention mechanisms are a core component of transitional models, allowing deep learning models to focus on relevant parts of the input data. In typical transitional models, attention weights are calculated by multiplying the weighted query input by the weighted key input (dot product), followed by scaling and a softmax function. The attention score is obtained by multiplying the attention weights by the weighted numerical input matrix, followed by weight calculation. However, the computational burden and memory access requirements of this architecture make it difficult to achieve immediate application on resource-constrained devices. [Summary of the Invention]

[0003] This invention proposes a computational method and its electronic device for neural networks based on a transformation model.

[0004] In one embodiment of the present invention, the above method includes receiving query input, key input and numerical input from the layer preceding the attention layer, obtaining a first merge weight pre-calculated based on a weight matrix for the query and a weight matrix for the key, obtaining a second merge weight pre-calculated based on a weight matrix for the numerical value and a weight matrix for the output score, and performing calculations based on the query input, key input, numerical input, first merge weight and second merge weight to generate an attention score for the attention layer.

[0005] In one embodiment of the present invention, the electronic device includes a processor configured to receive query input, key input and numerical input from the layer preceding the attention layer, obtain a first merge weight pre-calculated based on a weight matrix for the query and a weight matrix for the key, obtain a second merge weight pre-calculated based on a weight matrix for the numerical value and a weight matrix for the output score, and perform calculations based on the query input, key input, numerical input, first merge weight and second merge weight to generate an attention score for the attention layer.

[0006] This invention proposes a computational method and electronic device for bridging the attention layer and multilayer perceptron layer of a neural network based on a conversion model.

[0007] In one embodiment of the present invention, the above method includes receiving the residual of the attention layer and the attention matrix, obtaining the low-rank weight matrix of the first output score and the low-rank weight matrix of the second output score generated by the low-rank decomposition of the weight matrix of the output score, and obtaining the third merging weight, and performing calculations based on the residual, the attention matrix, the low-rank weight matrix of the first output score and the third merging weight to generate the output of the specified path of the multilayer perceptron layer, wherein the third merging weight is pre-calculated based on the low-rank weight matrix of the second output score and the weight matrix of the multilayer perceptron layer.

[0008] In one embodiment of the present invention, the electronic device includes a processor for receiving the residuals and attention matrix of the attention layer, obtaining a first low-rank weight matrix of the output score and a second low-rank weight matrix of the output score generated by the low-rank decomposition of the weight matrix of the output score, obtaining a third merging weight, and performing calculations based on the residuals, attention matrix, first low-rank weight matrix of the output score and third merging weight to generate the output of a specified path of the multilayer perceptron layer, wherein the third merging weight is pre-calculated based on the second low-rank weight matrix of the output score and the weight matrix of the multilayer perceptron layer.

Implementation Method

[0010] Some exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The number of elements shown in the following drawings is for illustrative purposes only, and the present invention is not intended to be limited thereto. Furthermore, these embodiments are only a part of the present invention and do not disclose all possible implementations of the present invention. More precisely, these embodiments are merely examples of methods and electronic devices within the scope of the present invention's patent application.

[0011] FIG1 is a schematic diagram of an electronic device according to an embodiment of the present invention. FIG1 first introduces the various components and their configuration relationships in the electronic device, and the detailed functions will be disclosed together with the flowcharts of the subsequent embodiments.

[0012] Referring to FIG1, the electronic device 100 of this embodiment includes at least a processor 110 and a memory 120. The electronic device 100 may be an electronic system or a computer system. The processor 110 is used to perform calculations on the neural network based on the transformation model, and may be a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a programmable general-purpose or special-purpose microprocessor, a digital signal processor (DSP), a field programmable array (FPGA), an application-specific integrated circuit (ASIC), other similar devices, integrated circuits, or combinations thereof. The memory 120 is used to store data, and may be various forms of random access memory (RAM), such as dynamic random-access memory (DRAM), or other similar devices, integrated circuits, or combinations thereof.

[0013] FIG2 is a flowchart illustrating a computation method for a neural network based on a conversion model according to an embodiment of the present invention, wherein the steps of FIG2 can be implemented by the electronic device 100 of FIG1.

[0014] Referring to both Figure 2 and Figure 1, the processor 110 receives query input, key input, and numerical input from the layer preceding the attention layer (step S202). Here, the query input, key input, and numerical input can be feature maps output from the convolutional layer of the neural network.

[0015] Next, the processor 110 obtains a first merge weight pre-calculated based on the query weight matrix and the key weight matrix (step S204), and a second merge weight pre-calculated based on the numerical weight matrix and the output score weight matrix (step S206). In this embodiment, the first merge weight and the second merge weight can be pre-calculated offline and pre-stored in memory 120 to reduce the number of parameters and operations, thereby reducing online computation time, memory access, quantization error, and hardware resources. Afterwards, the processor 110 calculates based on the query input, key input, numerical input, first merge weight, and second merge weight to generate the attention score of the attention layer (step S208), thereby solving the problems in the prior art. More details will be fully described later.

[0016] Figure 3 is a schematic diagram of the calculation method of attention layer for general conversion-based neural networks in the prior art.

[0017] Referring to Figure 3, the attention weight XS is calculated by the inner product of the weighted query input Q (which is given a linear weight WQ) and the weighted key input K (which is given a linear weight WK), followed by scaling and the application of the softmax function. The attention score XO is calculated by the matrix multiplication of the attention matrix Xh and the weighted numerical input V (which is given a linear weight WV), followed by a linear weight WO.

[0018] The goal of the method proposed in this embodiment is to improve the calculation of 310 and 320 in Figure 3 by using weight merging techniques (hereinafter referred to as "QK merging" and "VO merging" respectively). The inner product calculation in

[0019] 310 can be rewritten as, where and. The QK merging here will involve substitution, expansion and merging, where the inner product can be further rewritten as the following equation (1): It should be noted that WQK represents the aforementioned first merging weight, which is the product of the transpose of the weight matrix WQ used for query and the weight matrix WK T used for key. The multiplication calculation in

[0020] 320 can be rewritten as, where. The proposed VO merging will involve substitution and merging, where the multiplication calculation can be further rewritten as the following equation (2): It should be noted that WVO represents the aforementioned second merging weight, which is the weight matrix WV used for numerical values ​​and the weight matrix WO used for output scores.

[0021] Figure 4 is a schematic diagram of a computation method for a neural network based on a transformation model according to an embodiment of the present invention, wherein the steps of Figure 4 can be implemented by the electronic device 100 of Figure 1.

[0022] Referring to both Figure 4 and Figure 1, in this embodiment, the processor 110 will obtain the first merging weight WQK and the second merging weight WVO, which are pre-calculated and pre-stored in memory 120. It should be noted that the calculation in 410 corresponds to the QK merging derived in equation (1). The processor 110 will multiply the query input XQ, the first merging weight WVO, and the transpose of the key input XK T to generate a first multiplication result. Next, the processor 110 will calculate the first multiplication result to generate the attention weight XS. For example, the processor 110 may perform a scaling calculation on the first multiplication result to generate a scaled multiplication result, and may further apply the softmax function to the scaled multiplication result to generate the attention weight XS. In addition, the calculation in 420 corresponds to the VO merging derived in equation (2). The processor 110 will multiply the attention weight XS, the numerical input XV, and the second merging weight WVO to generate the attention score XO of the attention layer.

[0023] In terms of performance, assume that the dimensions of XQ and XK are both N×D, the dimensions of WQ and WK are both D×d, and the dimensions of Q and K are both N×d. Compared with the prior art, the technique proposed in this embodiment requires fewer operations (i.e., fewer instructions). Intuitively, assume N=1024, D=384, and d=384. Compared with the prior art, the technique proposed in this embodiment can reduce the number of parameters by 62% (i.e., reduce latency). Furthermore, since the technique proposed in this embodiment does not use approximations, accuracy can be guaranteed.

[0024] In another embodiment, to further reduce the number of parameters, the first merge weight can be pre-calculated based on the low-rank decomposition of the weight matrix used for querying and the low-rank decomposition of the weight matrix used for the key. Specifically, equation (1) can be further decomposed into low-rank weights as shown in equation (3) below: that is, the low-rank decomposition of the weight matrix WQ used for querying will produce a first low-rank weight matrix UQ and a second low-rank weight matrix SQ used for querying, wherein the low-rank decomposition of the weight matrix WK used for the key will produce a first low-rank weight matrix UK and a second low-rank weight matrix SK used for the key. In this case, the first merge weight will be the product of the first merged low-rank weight UQ' and the transpose of the first low-rank weight matrix SKT used for the key, wherein the first low-rank merge weight UQ' is the product of the first low-rank weight matrix UQ used for querying, the second low-rank weight matrix SQ used for querying, and the transpose of the first low-rank weight matrix UKT used for the key.

[0025] The weight merging mechanism can also be applied to other layers besides the attention layer in a neural network based on a transformation model. For example, Figure 5 is a flowchart illustrating a computation method for bridging the attention layer and multilayer perceptron layers of a neural network based on a transformation model according to an embodiment of the present invention, wherein the steps in Figure 5 can also be implemented by the electronic device 100 of Figure 1.

[0026] Referring simultaneously to Figures 5 and 1, the processor 120 receives the residual and attention matrix of the attention layer (step S502). The processor 120 obtains a first low-rank weight matrix and a second low-rank weight matrix for the output score, generated by the low-rank decomposition of the weight matrix for the output score, and obtains a third merge weight, which is pre-calculated based on the second low-rank weight matrix for the output score and the weight matrix of the multilayer perceptron layer (step S504). In this embodiment, the third merge weight can be pre-calculated offline and pre-stored in memory 120 to reduce the number of parameters and operations. The processor 120 calculates based on the residual, the attention matrix, the first low-rank weight matrix for the output score, and the third merge weight to generate the output of the specified path of the multilayer perceptron layer (step S506). Here, the specified path is a path that spans the attention layer and the multilayer perceptron layer. More details will be fully described later.

[0027] Figure 6 is a schematic diagram of the method for calculating the attention layer and multilayer perceptron layer in a general conversion-based neural network according to the prior art.

[0028] Referring to Figure 6, the residual Xr will be added to the attention score of the attention layer 601 to produce an addition result, where the attention score is the product of the attention weight Xh and the weight matrix WO used to output the score. The addition result will be split into two paths: the high-frequency signal of the addition result will be input to the multilayer perceptron layer 602 for layer normalization calculation and given linear weight WG, while the low-frequency signal of the addition result (i.e., Xm) will be directly output.

[0029] The goal of the method proposed in this embodiment is to improve the operation across two-layer paths by using a weight merging technique (referred to as "SOG merging"). Furthermore, the layer normalization calculation is a linear transformation, and therefore can be deferred (i.e., calculated after the weight calculation).

[0030] The SOG merging proposed in this embodiment will involve inverse matrix, low-rank decomposition and merging, wherein the calculation across two layers can be written as the following equation (4): Here, Xg represents the output of the specified path of the multilayer perceptron layer, and WSOG represents the aforementioned third merging weight matrix, which is the product of the second low-rank weight matrix SO used for output scores and the weight matrix WG of the multilayer perceptron layer.

[0031] Figure 7 is a schematic diagram of a computation method for crossing the attention layer and multilayer perceptron layer of a neural network based on a conversion model according to an embodiment of the present invention, wherein the method of Figure 7 can be implemented by the electronic device 100 of Figure 1.

[0032] Referring to Figure 7 and Figure 1, the processor 110 obtains from memory 120 the first low-rank weight matrix UO for outputting the score, the second low-rank weight matrix SO for outputting the score, the inverse matrix (UOSO)-1 ​​of the product of the first low-rank weight matrix UO and the second low-rank weight matrix SO for outputting the score, and the third merge weight WSOG. Alternatively, based on double-precision calculation, (UOSO)-1 ​​can also be replaced by (WO)-1.

[0033] In the path 710 that does not pass through the multilayer perceptron layer, the processor 110 sums the products of the residual Xr with the attention matrix Xh, the first low-rank weight matrix UO of the output score, and the second low-rank weight matrix SO of the output score to generate the output Xm of the path 710 of the multilayer perceptron layer. These operations can also be represented by the following equation (5):

[0034] In the second path 720 spanning the attention layer and the multilayer perceptron layer, the processor 110 sums the attention matrix Xh and the residual Xr with the inverse matrix (UOSO)⁻¹ of the product of the first low-rank weight matrix used for the output score and the second low-rank weight matrix used for the output score to generate a first mediation result. Next, the processor 110 multiplies the first mediation result, the first low-rank weight matrix UO used for the output score, and the third merging weight WSOG to generate a second mediation result. Then, the processor 110 performs layer normalization on the second mediation result to generate the output Xg of the path 720 of the multilayer perceptron layer.

[0035] In terms of performance, assuming the dimensions of UO and SO are D×r and r×D respectively, the dimension of WO-1 is 2(D×D), and the dimension of WSOG is r×8D. The method proposed in this embodiment requires fewer operations compared to existing technologies. Intuitively, assuming the rank r = D / 4, the method proposed in this embodiment can reduce the number of parameters by 62% compared to existing technologies.

[0036] As another more aggressive merging method, equation (4) can be used for additional merging, as presented in the following equation (6): Here, WOU is the fourth merging weight, which is the product of the inverse of the product of the first low-rank weight matrix used for the output score and the second low-rank weight matrix used for the output score (UOSO)-1 ​​and the first low-rank weight matrix UO of the output score.

[0037] Figure 8 is a schematic diagram of a computation method for crossing the attention layer and multilayer perceptron layer of a neural network based on a conversion model, according to another embodiment of the present invention, wherein the method of Figure 8 can be implemented by the electronic device 100 of Figure 1.

[0038] Refer to Figure 8 and Figure 1 at the same time. The processor 110 will obtain from the memory 120 the first low-rank weight matrix UO for outputting the score, the second low-rank weight matrix SO for outputting the score, the third merging weight WSOG and the fourth merging weight WOU.

[0039] In the path 810 that does not pass through the multilayer perceptron layer, the processor 110 sums up the residual Xr and the product of the attention matrix Xh, the first low-rank weight matrix UO used for output score and the second low-rank weight matrix SO used for output score to generate the output Xm of the path 810 of the multilayer perceptron layer.

[0040] In the path 820 that crosses the attention layer and the multilayer perceptron layer, the processor 110 sums the product of the residual Xr and the fourth merging weight WOU and the product of the attention matrix Xh and the first low-rank weight matrix UO of the output score to generate the first mediation result.

[0041] Next, the processor 110 multiplies the first mediation result and the third merging weight WSOG to generate a second mediation result. The processor 110 performs layer normalization calculation on the second mediation result to generate the output of the first path of the multilayer perceptron layer, and then generates the output Xg of the path 820 of the multilayer perceptron layer.

[0042] In terms of performance, assuming the dimensions of UO and SO are D×r and r×D respectively, the dimension of WOU is 2(D×r), and the dimension of WSOG is r×8D. The method proposed in this embodiment requires fewer operations compared to existing technologies. Intuitively, assuming the rank r = D / 4, the method proposed in this embodiment can reduce the number of parameters by 62% compared to existing technologies.

[0043] In summary, the present invention proposes various effective methods for computation on neural networks based on conversion models, so as to reduce online computation time, memory access, quantization error and resource consumption while ensuring accuracy.

[0044] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims and their equivalents. [Simplified Explanation of the Diagram]

[0009] Figure 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 2 is a flowchart of a calculation method for a neural network based on a conversion model according to an embodiment of the present invention. Figure 3 is a schematic diagram of a calculation method for an attention layer in a general conversion model-based neural network according to the prior art. Figure 4 is a schematic diagram of a calculation method for a neural network based on a conversion model according to an embodiment of the present invention. Figure 5 is a flowchart of a calculation method for crossing an attention layer and a multilayer perceptron layer in a neural network based on a conversion model according to an embodiment of the present invention. Figure 6 is a schematic diagram of a calculation method for an attention layer and a multilayer perceptron layer in a general conversion model-based neural network according to the prior art. Figure 7 is a schematic diagram of a calculation method for crossing an attention layer and a multilayer perceptron layer in a neural network based on a conversion model according to an embodiment of the present invention. Figure 8 is a schematic diagram of a calculation method for crossing an attention layer and a multilayer perceptron layer in a neural network based on a conversion model according to another embodiment of the present invention.

Claims

1. A computational method for bridging attention layers and multilayer perceptron layers in a neural network based on a transformation model, comprising: Receive the residuals and attention matrix of the attention layer; A first low-rank weight matrix for output score and a second low-rank weight matrix for output score are obtained from the low-rank decomposition of the weight matrix for output score, and a third merge weight is obtained, wherein the third merge weight is pre-calculated based on the second low-rank weight matrix for output score and the weight matrix of the multilayer perceptron layer; and the output of the specified path of the multilayer perceptron layer is generated based on the residual, the attention matrix, the first low-rank weight matrix for output score and the third merge weight.

2. The method of claim 1, wherein the step of calculating the output of the specified path of the multilayer perceptron layer based on the residual, the attention matrix, the first low-rank weight matrix for output score, and the third merging weight includes: Obtain the inverse matrix of the product of the first low-rank weight matrix used for output score and the second low-rank weight matrix used for output score; sum the products of the attention matrix and the residual, the inverse matrix of the product of the first low-rank weight matrix used for output score and the low-rank weight matrix used for the second output score to generate a first mediation result; multiply the first mediation result, the first low-rank weight matrix used for output score and the third merging weight to generate a second mediation result; and perform layer normalization on the second mediation result to generate the output of the specified path of the multilayer perceptron layer.

3. The method of claim 1, wherein the step of calculating the output of the specified path of the multilayer perceptron layer based on the residual, the attention matrix, the first low-rank weight matrix for output score, and the third merging weight includes: A fourth merging weight is obtained, wherein the fourth merging weight is pre-computed based on the first low-rank weight matrix for output score and the inverse matrix of the product of the first low-rank weight matrix for output score and the second low-rank weight matrix for output score; the product of the residual and the fourth merging weight and the product of the attention matrix and the first low-rank weight matrix for output score are summed to generate a first mediation result; the first mediation result and the third merging weight are multiplied to generate a second mediation result; and the second mediation result is layer normalized to generate the output of the specified path of the multilayer perceptron layer.

4. The method as described in claim 1, further comprising: The residuals, the attention matrix, the product of the first low-rank weight matrix used for output scores and the second low-rank weight matrix used for output scores are summed to generate the output of another specified path of the multilayer perceptron layer.

5. An electronic device for bridging an attention layer and multiple perceptron layers in a neural network based on a conversion model, comprising: The processor is configured to: receive the residuals and attention matrix of the attention layer; A first low-rank weight matrix for the output score and a second low-rank weight matrix for the output score are obtained from the low-rank decomposition of the weight matrix of the output score; a third merge weight is obtained, wherein the third merge weight is pre-calculated based on the second low-rank weight matrix for the output score and the weight matrix of the multilayer perceptron layer; and the output of the specified path of the multilayer perceptron layer is generated based on the residual, the attention matrix, the first low-rank weight matrix for the output score and the third merge weight.

6. The electronic device as claimed in claim 5, further comprising: The memory is used to pre-store the third merging weights and the first and second low-rank matrices for outputting the scores, which are generated by the low-rank decomposition of the weight matrix for outputting the scores.

Images