Hardware Accelerator Applied to Transformer Neural Network and Its Computing Method

Abstract

The present application discloses a hardware accelerator applied to a Transformer neural network and its calculation method. The hardware accelerator includes: a data storage module, a systolic array module, a bias addition module, a dot product attention function operation module, an activation function operation module, a residual addition module, a layer normalization function operation module, and a control module. The systolic array module includes a plurality of basic operation units for performing matrix calculations, and the control module is used to control the calculation process of the hardware accelerator. Based on the systolic array module capable of performing all matrix calculations in the Transformer neural network, the above-mentioned hardware accelerator can effectively improve the operation speed and efficiency of the Transformer neural network.

Description

Technical Field

[0001] This application relates to the technical field of neural networks, and particularly to a hardware accelerator applied to a Transformer neural network and its calculation method. Background Art

[0002] The Transformer network is a neural network model for solving natural language processing problems. Its model architecture is as Figure 1 shown, mainly including an encoder stack and a decoder stack. The encoder stack and the decoder stack each contain N encoder layers and multiple decoder layers. During the calculation process of the Transformer neural network, the input sequence first undergoes word vector embedding layer processing and position encoding superposition processing to obtain an input matrix. This input matrix is input to the encoder stack and undergoes operations of multiple encoder layers in sequence to obtain the output matrix of the encoder stack. After the encoding stage is the decoding stage. In each step of the decoding stage, an element of the target sentence is output to achieve natural language processing.

[0003] Each encoder layer and decoder layer is composed of a multi-head attention layer and a feed-forward layer. The multi-head attention layer includes three input matrices of the same size, namely the first input matrix, the second input matrix, and the third input matrix. The feed-forward layer only includes one input matrix. Refer to Figure 2 , the multi-head attention layer includes multiple attention heads. Each attention head contains three linear layers, namely the first linear layer, the second linear layer, and the third linear layer. Among them, the first linear layer contains the first weight matrix and the first bias matrix, the second linear layer contains the second weight matrix and the second bias matrix, and the third linear layer contains the third weight matrix and the third bias matrix. After the input matrix undergoes linear processing by these three linear layers, then the scaled dot product attention function operation is performed, and then the obtained multiple matrices are concatenated to obtain a concatenated matrix. This concatenated matrix undergoes linear processing by the fourth linear layer (this linear layer contains the fourth weight matrix and the fourth bias matrix) and subsequent residual addition operation and layer normalization function operation to obtain the final output matrix of the multi-head attention layer. This output matrix will be used as the input of the feed-forward layer. The feed-forward layer includes the fifth linear layer and the sixth linear layer. The fifth linear layer contains the fifth weight matrix and the fifth bias matrix, and the sixth linear layer contains the sixth weight matrix and the sixth bias matrix. There is an activation function between these two linear layers. After the input matrix undergoes linear processing by these two linear layers and the activation function operation between them, through subsequent residual addition calculation and layer normalization function operation, the output matrix of the feed-forward layer can be obtained.

[0004] Currently, the above-mentioned computing process is run on general computing platforms such as CPUs or GPUs, and no dedicated hardware accelerator has been developed for Transformer neural networks. To improve the operation speed and efficiency of this neural network, it is urgent to design a dedicated hardware accelerator. Summary of the Invention

[0005] In order to improve the operation speed and efficiency of Transformer neural networks, this application discloses a hardware accelerator applied to Transformer neural networks and its calculation method through the following embodiments.

[0006] The first aspect of this application discloses a hardware accelerator applied to Transformer neural networks, including:

[0007] A data storage module, including an input matrix storage unit, a weight matrix storage unit, a bias matrix storage unit, and a temporary matrix storage unit. The input end of the input matrix storage unit serves as the input interface of the entire hardware accelerator. The output ends of the input matrix storage unit, the weight matrix storage unit, and the temporary matrix storage unit are all connected to the input end of the systolic array module. The output end of the bias matrix storage unit is connected to the input end of the bias addition module. The weight matrix storage unit is used to store the fourth weight matrix, the fifth weight matrix, and the sixth weight matrix. These three weight matrices are all longitudinally divided into multiple sub-matrices, and the number of columns of each sub-matrix is the same as the number of columns of the first weight matrix;

[0008] A systolic array module, which includes multiple basic arithmetic units for performing matrix calculations. The number of rows of the basic arithmetic unit is the same as the number of rows of any input matrix in the multi-head attention layer. The number of columns of the basic arithmetic unit is the same as the number of columns of the first weight matrix. The output end of the systolic array module is connected to the input end of the bias addition module, the input end of the dot product attention function operation module, and the input end of the temporary matrix storage unit;

[0009] A bias addition module, including multiple adders. The number of adders is the same as the number of rows of any input matrix in the multi-head attention layer. The output end of the bias addition module is connected to the input end of the temporary matrix storage unit, the input end of the activation function operation module, and the input end of the residual addition module;

[0010] A dot product attention function operation module, whose output end is connected to the input end of the temporary matrix storage unit;

[0011] An activation function operation module, whose output end is connected to the input end of the temporary matrix storage unit;

[0012] The residual addition module includes multiple adders. The number of adders is the same as the number of rows of any input matrix in the multi-head attention layer. The input end of the residual addition module is connected to the output end of the input matrix storage unit, and the output end is connected to the input end of the layer normalization function operation module;

[0013] The layer normalization function operation module, and the output end of the layer normalization function operation module serves as the output interface of the entire hardware accelerator;

[0014] The control module is respectively connected to the data storage module, the systolic array module, the bias addition module, the dot product attention function operation module, the activation function operation module, the residual addition module, and the layer normalization function operation module.

[0015] Optionally, the following control process is configured in the control module:

[0016] Determine whether the operation executed in the current hardware accelerator is the operation of the multi-head attention layer or the operation of the feed-forward layer;

[0017] If the operation executed is the operation of the multi-head attention layer, then perform the following operations:

[0018] Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the first linear layer for the first input matrix, the operation of the second linear layer for the second input matrix, and the operation of the third linear layer for the third input matrix, and control the dot product attention function operation module to perform a function operation on the operation results of the first linear layer, the second linear layer, and the third linear layer to obtain a concatenated matrix;

[0019] Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fourth linear layer for the concatenated matrix; and control the residual addition module to perform a residual calculation on the operation result of the fourth linear layer to obtain a first intermediate matrix;

[0020] Control the layer normalization function operation module to perform a function operation on the first intermediate matrix to obtain the output matrix of the multi-head attention layer;

[0021] If the operation executed is the operation of the feed-forward layer, then perform the following operations:

[0022] Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fifth linear layer for the input matrix of the feed-forward layer; and control the activation function operation module to perform an activation function operation on the operation result of the fifth linear layer to obtain an activation matrix;

[0023] Control the data storage module, the systolic array module, and the bias addition module to perform the arithmetic operations of the sixth linear layer on the activation matrix; and, control the residual addition module to perform residual calculation on the arithmetic result of the sixth linear layer to obtain the second intermediate matrix.

[0024] Control the layer normalization function arithmetic module to perform function operations on the second intermediate matrix to obtain the output matrix of the feedforward layer.

[0025] Optionally, the bias matrix storage unit is used to store the first bias matrix, the second bias matrix, the third bias matrix, the fourth bias matrix, the fifth bias matrix, and the sixth bias matrix, wherein the fourth bias matrix, the fifth bias matrix, and the sixth bias matrix are all divided into multiple sub-matrices, and the number of columns of each sub-matrix is the same as the number of columns of the first weight matrix.

[0026] Optionally, the input matrix storage unit includes a first matrix storage subunit and a second matrix storage subunit;

[0027] The first matrix storage subunit is used to store the first input matrix of the multi-head attention layer and the input matrix of the feedforward layer;

[0028] The second matrix storage subunit is used to store the second input matrix and the third input matrix of the multi-head attention layer.

[0029] Optionally, the temporary matrix storage unit includes a first temporary matrix storage subunit, a second temporary matrix storage subunit, and a third temporary matrix storage subunit.

[0030] Optionally, the output end of the systolic array module is connected to the input end of the third temporary matrix storage subunit;

[0031] The output end of the bias addition module is connected to the input end of the first temporary matrix storage subunit and the input end of the second temporary matrix storage subunit;

[0032] The output end of the dot product attention function arithmetic module is connected to the input end of the first temporary matrix storage subunit;

[0033] The output end of the activation function arithmetic module is connected to the input end of the third temporary matrix storage subunit.

[0034] Optionally, the dot product attention function arithmetic module includes a filter matrix storage unit and multiple function arithmetic units, and the number of function arithmetic units is the same as the number of rows of any input matrix in the multi-head attention layer;

[0035] The filter matrix storage unit is respectively connected to each function arithmetic unit;

[0036] The function operation unit includes a maximum value acquisition subunit, a summation subunit, a logarithmic operation subunit, and an exponential operation subunit.

[0037] Optionally, the layer normalization function operation module includes an intermediate matrix storage unit, a first mean calculation unit, a second mean calculation unit, a first square calculation unit, a second square calculation unit, a reciprocal square root calculation unit, and an output matrix calculation unit;

[0038] The output end of the intermediate matrix storage unit is connected to the output matrix calculation unit. The intermediate matrix storage unit is used to acquire and store the intermediate matrix, where the intermediate matrix is the first intermediate matrix in the processing of the multi-head attention layer or the second intermediate matrix in the processing of the feed-forward layer;

[0039] The output end of the first mean calculation unit is respectively connected to the first square calculation unit and the output matrix calculation unit. The first mean calculation unit is used to calculate the mean of each row of elements in the intermediate matrix;

[0040] The output end of the first square calculation unit is connected to the reciprocal square root calculation unit. The first square calculation unit is used to perform a square operation on the value input by the first mean calculation unit to obtain the square of the mean of each row of elements in the intermediate matrix;

[0041] The output end of the second square calculation unit is connected to the second mean calculation unit. The second square calculation unit is used to perform a square operation on each element in the intermediate matrix to obtain a square matrix;

[0042] The output end of the second mean calculation unit is connected to the reciprocal square root calculation unit. The second mean calculation unit is used to calculate the mean of each row of elements in the square matrix;

[0043] The output end of the reciprocal square root calculation unit is connected to the output matrix calculation unit. The reciprocal square root calculation unit is used to obtain the reciprocal of the square root of the variance of each row of elements in the intermediate matrix according to the square of the mean of each row of elements in the intermediate matrix and the mean of each row of elements in the square matrix;

[0044] The output matrix calculation unit is used to perform layer normalization processing on each element of the intermediate matrix, the mean of each row of elements in the intermediate matrix, and the reciprocal of the square root of the variance of each row of elements in the intermediate matrix to obtain the final output matrix of the multi-head attention layer or the feed-forward layer.

[0045] Optionally, the first mean calculation unit includes a plurality of first mean calculation sub-units, the second mean calculation unit includes a plurality of second mean calculation sub-units, the first square calculation unit includes a plurality of first square calculation sub-units, the second square calculation unit includes a plurality of second square calculation sub-units, the reciprocal square root calculation unit includes a plurality of reciprocal square root calculation sub-units, and the output matrix calculation unit includes a plurality of output matrix calculation sub-units;

[0046] The number of the first mean calculation sub-units, the second mean calculation sub-units, the first square calculation sub-units, the second square calculation sub-units, the reciprocal square root calculation sub-units, and the output matrix calculation sub-units is the same as the number of rows of any input matrix in the multi-head attention layer.

[0047] A second aspect of the present application discloses a calculation method for a hardware accelerator applied to a Transformer neural network. The calculation method is applied to a control module, and the control module is located in the hardware accelerator applied to the Transformer neural network described in the first aspect of the present application. The calculation method includes:

[0048] Determine whether the operation executed in the current hardware accelerator is an operation of the multi-head attention layer or an operation of the feed-forward layer;

[0049] If the operation executed is an operation of the multi-head attention layer, the following steps include:

[0050] Step S101: Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the first linear layer on the first input matrix, the operation of the second linear layer on the second input matrix, and the operation of the third linear layer on the third input matrix, and control the dot product attention function operation module to perform a function operation on the operation results of the first linear layer, the second linear layer, and the third linear layer to obtain a concatenated matrix;

[0051] Step S102: Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fourth linear layer on the concatenated matrix; and control the residual addition module to perform a residual calculation on the operation result of the fourth linear layer to obtain a first intermediate matrix;

[0052] Step S103: Control the layer normalization function operation module to perform a function operation on the first intermediate matrix to obtain the output matrix of the multi-head attention layer;

[0053] If the operation executed is an operation of the feed-forward layer, then execute the following steps:

[0054] Step S201, control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fifth linear layer on the input matrix of the feedforward layer; and, control the activation function operation module to perform the activation function operation on the operation result of the fifth linear layer to obtain the activation matrix;

[0055] Step S202, control the data storage module, the systolic array module, and the bias addition module to perform the operation of the sixth linear layer on the activation matrix; and, control the residual addition module to perform residual calculation on the operation result of the sixth linear layer to obtain the second intermediate matrix;

[0056] Step S203, control the layer normalization function operation module to perform the function operation on the second intermediate matrix to obtain the output matrix of the feedforward layer.

[0057] Optionally, step S101 includes:

[0058] Step S1011, obtain the first input matrix from the first matrix storage subunit, obtain the first weight matrix from the weight matrix storage unit, input the first input matrix and the first weight matrix into the systolic array module to perform matrix calculation, then input the result of the matrix calculation into the bias addition module to add with the first bias matrix to obtain the first temporary matrix, and finally input the first temporary matrix into the first temporary matrix storage subunit;

[0059] Step S1012, obtain the second input matrix from the second matrix storage subunit, obtain the second weight matrix from the weight matrix storage unit, input the second input matrix and the second weight matrix into the systolic array module to perform matrix calculation, then input the result of the matrix calculation into the bias addition module to add with the second bias matrix to obtain the second temporary matrix, and finally input the second temporary matrix into the second temporary matrix storage subunit;

[0060] Step S1013, take out the first temporary matrix from the first temporary matrix storage subunit, take out the transposed matrix of the second temporary matrix from the second temporary matrix storage subunit, input the first temporary matrix and the transposed matrix of the second temporary matrix into the systolic array module to perform matrix calculation, and then input the result of the matrix calculation into the dot product attention function operation module;

[0061] Step S1014: Control the dot product attention function operation module to perform a function operation on the received matrix data to obtain a third temporary matrix, and store the third temporary matrix in the first temporary matrix storage subunit; meanwhile, obtain the third input matrix from the second matrix storage subunit, obtain the third weight matrix from the weight matrix storage unit, input the third input matrix and the third weight matrix into the systolic array module to perform matrix calculation, then input the result of the matrix calculation into the bias addition module to add it to the third bias matrix to obtain a fourth temporary matrix, and finally input the fourth temporary matrix into the second temporary matrix storage subunit;

[0062] Step S1015: Take out the third temporary matrix from the first temporary matrix storage subunit, take out the fourth temporary matrix from the second temporary matrix storage subunit, input these two matrices into the systolic array module to perform matrix calculation, and then input the obtained matrix to be spliced into the third temporary matrix storage subunit;

[0063] Step S1016: Repeat steps S1011 - S1015 until the preset number of times is satisfied. The preset number of times is the same as the number of attention heads included in the multi - head attention layer. After execution, the multiple matrices to be spliced stored in the third temporary matrix storage subunit form a spliced matrix.

[0064] Optionally, step S102 includes:

[0065] Step S1021: Take out the spliced matrix from the third temporary matrix storage subunit, take out a sub - matrix of the fourth weight matrix from the weight matrix storage unit, input the spliced matrix and the sub - matrix of the fourth weight matrix into the systolic array module to perform matrix calculation, then input the result of the matrix calculation into the bias addition module to add it to a sub - matrix of the fourth bias matrix, and input the addition result into the residual addition module to add it to a sub - matrix of the first input matrix to obtain a first intermediate sub - matrix;

[0066] Step S1022: Repeat step S1021 until the preset number of times is satisfied. After execution, the obtained multiple first intermediate sub - matrices form a first intermediate matrix. Among them, in different execution processes, different sub - matrices of the fourth weight matrix, different sub - matrices of the fourth bias matrix, and different sub - matrices of the first input matrix are taken out.

[0067] Optionally, step S201 includes:

[0068] Step S2011: Take out the input matrix of the feedforward layer from the first matrix storage subunit, take out a sub-matrix of the fifth weight matrix from the weight matrix storage unit, input the input matrix of the feedforward layer and the sub-matrix of the fifth weight matrix into the systolic array module to perform matrix calculation, and input the result of the matrix calculation into the bias addition module to add it with a sub-matrix of the fifth bias matrix. Input the addition result into the activation function operation module to obtain an activation sub-matrix, and input the activation sub-matrix into the third temporary matrix storage subunit;

[0069] Step S2012: Repeat Step S2011 until the preset number of times is met. After execution, the multiple activation sub-matrices stored in the third temporary matrix storage subunit form an activation matrix. Among them, in different execution processes, different sub-matrices of the fifth weight matrix are taken out, and different sub-matrices of the fifth bias matrix are taken out.

[0070] Optionally, Step S202 includes:

[0071] Step S2021: Take out the activation matrix from the third temporary matrix storage subunit, take out a sub-matrix of the sixth weight matrix from the weight matrix storage unit, input the activation matrix and the sub-matrix of the sixth weight matrix into the systolic array module to perform matrix calculation, and input the result of the matrix calculation into the bias addition module to add it with a sub-matrix of the sixth bias matrix. Input the addition result into the residual addition module to add it with a sub-matrix of the feedforward layer input matrix to obtain a second intermediate sub-matrix;

[0072] Step S2022: Repeat Step S2021 until the preset number of times is met. After execution, the multiple second intermediate sub-matrices obtained form a second intermediate matrix. Among them, in different execution processes, different sub-matrices of the sixth weight matrix are taken out, different sub-matrices of the sixth bias matrix are taken out, and different sub-matrices of the feedforward layer input matrix are taken out.

[0073] Optionally, Step S103 or Step S203 includes:

[0074] Input all elements of the intermediate matrix into the intermediate matrix storage unit in column order. Among them, if Step S103 is executed, the intermediate matrix is the first intermediate matrix; if Step S203 is executed, the intermediate matrix is the second intermediate matrix;

[0075] Input each element of each row of the intermediate matrix into multiple first mean calculation sub-units respectively to calculate the mean value of each element in each row of the intermediate matrix; and input each element of each row of the intermediate matrix into multiple second square calculation sub-units respectively to obtain a square matrix;

[0076] Input the mean value of each element in each row of the intermediate matrix into multiple first square calculation sub-units respectively to obtain the square of the mean value of each element in each row of the intermediate matrix;

[0077] Input each element of each row of the square matrix into multiple second mean calculation sub-units respectively to calculate the mean value of each element in each row of the square matrix;

[0078] Input the square of the mean value of each element in each row of the intermediate matrix and the mean value of each element in each row of the square matrix into multiple square root reciprocal calculation sub-units respectively to obtain the reciprocal of the square root of the variance of each element in each row of the intermediate matrix;

[0079] Input each element of the intermediate matrix, the mean value of each element in each row of the intermediate matrix, and the reciprocal of the square root of the variance of each element in each row of the intermediate matrix into multiple output matrix calculation sub-units respectively to obtain the final output matrix of the multi-head attention layer or the feed-forward layer.

[0080] A third aspect of the present application discloses a computer device, including:

[0081] A memory for storing a computer program;

[0082] A processor for implementing the steps of the calculation method of the hardware accelerator applied to the Transformer neural network as described in the second aspect of the present application when executing the computer program.

[0083] A fourth aspect of the present application discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is processed and executed, it implements the steps of the calculation method of the hardware accelerator applied to the Transformer neural network as described in the second aspect of the present application.

[0084] This application discloses a hardware accelerator applied to a Transformer neural network and its calculation method. The hardware accelerator includes: a data storage module, a systolic array module, a bias addition module, a dot product attention function operation module, an activation function operation module, a residual addition module, a layer normalization function operation module, and a control module. Among them, the data storage module includes a weight matrix storage unit for storing a fourth weight matrix, a fifth weight matrix, and a sixth weight matrix. These three weight matrices are all longitudinally divided into multiple sub-matrices, and the number of columns of each sub-matrix is the same as the number of columns of the first weight matrix. The systolic array module includes multiple basic operation units for performing matrix calculations, and the control module is used to control the calculation process of the hardware accelerator. Based on the systolic array module capable of performing all matrix calculations in the Transformer neural network, the above hardware accelerator can effectively improve the operation speed and efficiency of the Transformer neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the technical solutions of this application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0086] Figure 1 It is a schematic diagram of the model architecture of the Transformer neural network;

[0087] Figure 2 It is an architecture diagram of the multi-head attention layer in the Transformer neural network;

[0088] Figure 3 It is a schematic diagram of the matrix operation process of the multi-head attention layer in the Transformer neural network;

[0089] Figure 4 It is a schematic diagram of the matrix operation process of the feed-forward layer in the Transformer neural network;

[0090] Figure 5 It is a schematic diagram of the hardware architecture of a hardware accelerator applied to a Transformer neural network disclosed in the embodiments of this application;

[0091] Figure 6 It is a schematic diagram of the structure of the temporary matrix storage unit in a hardware accelerator applied to a Transformer neural network disclosed in the embodiments of this application;

[0092] Figure 7Schematic diagram of the division of the fourth weight matrix, fifth weight matrix, and sixth weight matrix stored in the weight matrix storage unit in a hardware accelerator applied to a Transformer neural network disclosed in an embodiment of the present application;

[0093] Figure 8 Schematic diagram of the hardware architecture of the dot product attention function operation module in a hardware accelerator applied to a Transformer neural network disclosed in an embodiment of the present application;

[0094] Figure 9 Schematic diagram of the structure of a function operation unit of the dot product attention function operation module in a hardware accelerator applied to a Transformer neural network disclosed in an embodiment of the present application;

[0095] Figure 10 Schematic diagram of the hardware architecture of the layer normalization function operation module in a hardware accelerator applied to a Transformer neural network disclosed in an embodiment of the present application. Detailed implementation manners

[0096] In order to improve the operation speed and efficiency of the Transformer neural network, the present application discloses a hardware accelerator applied to the Transformer neural network and its calculation method through the following embodiments.

[0097] In the present application, the three inputs of the multi-head attention layer are respectively defined as Q, K, and V, and the input of the feed-forward layer is defined as X. The size of the input tensor X is the same as the sizes of the input tensors Q, K, and V, and they are all equal to [batch_size, s, dmodel] (batch_size represents how many input sequences there are at a time, s represents how many words there are in an input sequence, and the size of dmodel represents the size of the neural network model). Considering the case where batch_size is 1, the input tensor can be considered to degenerate into a matrix, so all operations can be considered as operations on the input matrix (even if batch_size is greater than 1, it can be considered as multiple input matrices with the same size but different elements, and the same and non-interfering operations are performed on them and then merged together). Figure 3 Matrix operation process for the multi-head attention layer, Figure 4The matrix operation process for the feed-forward layer. Among them, the sizes of the first weight matrix WQ1, the second weight matrix WK1, and the third weight matrix WV1 are all dmodel×dk, the size of the fourth weight matrix WG is dmodel×dmodel, the size of the fifth weight matrix W1 is dmodel×dff, and the size of the sixth weight matrix W2 is dff×dmodel. In actual operation, dmodel = h×dk, and dff = 4dmodel, where h represents the number of attention heads in the multi-head attention layer.

[0098] Combined with Figure 4 , the matrix operation performed by the feed-forward layer can be expressed by the following formula:

[0099] Output = layerNorm(x + ReLU(xW1 + b1)W2 + b2).

[0100] Among them, LayerNorm represents the layer normalization function operation, x is the input of the feed-forward layer, b1 represents the fifth bias matrix, and b2 represents the sixth bias matrix.

[0101] The first embodiment of this application discloses a hardware accelerator applied to a Transformer neural network. Refer to Figure 5 and Figure 6 shown in the structural diagram. This hardware accelerator includes: a control module, a data storage module, a systolic array module, a bias addition module, a dot product attention function operation module, an activation function operation module, a residual addition module, and a layer normalization function operation module.

[0102] The data storage module includes an input matrix storage unit, a weight matrix storage unit, a bias matrix storage unit, and a temporary matrix storage unit.

[0103] The output ends of the input matrix storage unit, the weight matrix storage unit, and the temporary matrix storage unit are all connected to the input end of the systolic array module, and the output end of the bias matrix storage unit is connected to the input end of the bias addition module.

[0104] The input end of the input matrix storage unit serves as the input interface of the entire hardware accelerator. In one implementation, the input matrix storage unit includes a first matrix storage subunit and a second matrix storage subunit. The first matrix storage subunit is used to store the first input matrix of the multi-head attention layer and the input matrix of the feed-forward layer. The second matrix storage subunit is used to store the second input matrix and the third input matrix of the multi-head attention layer.

[0105] The weight matrix storage unit is used to store a fourth weight matrix, a fifth weight matrix, and a sixth weight matrix. These three weight matrices are each vertically divided into multiple sub-matrices, and the number of columns of each sub-matrix is the same as the number of columns of the first weight matrix.

[0106] The bias matrix storage unit is used to store a first bias matrix, a second bias matrix, a third bias matrix, a fourth bias matrix, a fifth bias matrix, and a sixth bias matrix. Among them, the fourth bias matrix, the fifth bias matrix, and the sixth bias matrix are each divided into multiple sub-matrices, and the number of columns of each sub-matrix is the same as the number of columns of the first weight matrix.

[0107] The temporary matrix storage unit includes a first temporary matrix storage subunit, a second temporary matrix storage subunit, and a third temporary matrix storage subunit.

[0108] Among them, the output end of the systolic array module is connected to the input end of the third temporary matrix storage subunit.

[0109] The output end of the bias addition module is connected to the input end of the first temporary matrix storage subunit and the input end of the second temporary matrix storage subunit.

[0110] The output end of the dot product attention function operation module is connected to the input end of the first temporary matrix storage subunit.

[0111] The output end of the activation function operation module is connected to the input end of the third temporary matrix storage subunit.

[0112] In some embodiments, the first matrix storage subunit, the second matrix storage subunit, the first temporary matrix storage subunit, and the third temporary matrix storage subunit are connected to the systolic array module through a four-way selector, and the second temporary matrix storage subunit and the weight matrix storage unit are connected to the systolic array module through a two-way selector.

[0113] The systolic array module includes multiple basic operation units for performing matrix calculations. The number of rows of the basic operation units is the same as the number of rows of any input matrix in the multi-head attention layer, the number of columns of the basic operation units is the same as the number of columns of the first weight matrix, and the output end of the systolic array module is connected to the input end of the bias addition module, the input end of the dot product attention function operation module, and the input end of the temporary matrix storage unit.

[0114] The number of rows of any input matrix in the multi-head attention layer is denoted as s, the number of columns of the first weight matrix is denoted as dk, and the size of the systolic array module is s×dk, that is, it contains s rows and dk columns of basic arithmetic units. In a typical Transformer neural network, the size of dk is 64, and in different processing flows, s takes different values, and typical values are 32, 64, 128, etc.

[0115] The matrix calculations involved in the calculation process of the Transformer neural network include: multiplying the first input matrix Q by the first weight matrix WQ1, multiplying the second input matrix K by the second weight matrix WK1, multiplying the third input matrix V by the third weight matrix WV1, multiplying the concatenated matrix P by the fourth weight matrix WG, multiplying the input matrix X of the feed-forward layer by the fifth weight matrix W1, and multiplying the activation matrix ReLU(XW1 + b1) by the sixth weight matrix W2.

[0116] The first weight matrix, the second weight matrix, and the third weight matrix can directly use the s×dk systolic array module for operations, and the size of the matrix obtained by their multiplication is the same as the size of the systolic array module.

[0117] In order to enable the matrix calculations involving the fourth weight matrix WG, the fifth weight matrix W1, and the sixth weight matrix W2 to directly use the systolic array module for execution, these three large matrices are partitioned and respectively partitioned into sub-matrices with dk columns. For the partitioning results, see Figure 7 . Among them, the fourth weight matrix WG is partitioned into h sub-matrices, namely WG1, WG2, ……, WGh; the fifth weight matrix W1 is partitioned into 4h sub-matrices, namely W11, W12, ……, W14h; the sixth weight matrix W2 is partitioned into h sub-matrices, namely W21, W22, ……, W2h.

[0118] The bias addition module includes multiple adders, the number of the adders is the same as the number of rows of any input matrix in the multi-head attention layer, and the output end of the bias addition module is connected to the input end of the temporary matrix storage unit, the input end of the activation function operation module, and the input end of the residual addition module.

[0119] The output end of the dot product attention function operation module is connected to the input end of the temporary matrix storage unit.

[0120] The dot product attention function operation module (Mask-Softmax function operation module) is used to perform a scaled dot product attention function operation on the input data, and this function operation can be represented by the following formula:

[0121]

[0122] Among them, Attention(Qi, Ki, Vi) represents the operation result of the dot product attention function operation module for the i-th attention head, and Attention(Qh, Kh, Vh) represents the operation result of the dot product attention function operation module for the h-th attention head. Qi, Ki, and Vi respectively represent the outputs of the three linear layers in the i-th attention head, and Qh, Kh, and Vh respectively represent the outputs of the three linear layers in the h-th attention head. The multi-head attention layer includes a total of h attention heads. The Mask operation is an operation on the input of the dot product attention function (softmax function). During this operation, a Mask matrix is used to block those unnecessary softmax function inputs so that they do not participate in subsequent operations (for unnecessary inputs, change their values to negative infinity).

[0123] According to the formula disclosed above, a matrix calculation is involved in the operation process of the scaled dot product attention function, that is, the matrix Qi is multiplied by the matrix Ki T (the transposed matrix of the matrix Ki). When s is less than or equal to dk, the systolic array module can directly execute this matrix calculation; when s is greater than dk, then Qi needs to be split. Generally speaking, however, the amount of computation in this part is very small compared to the matrix multiplication operation of the entire multi-head attention layer. Even if Qi is split, it has little impact on the hardware utilization rate of the entire design. Therefore, it can be not split.

[0124] In addition, another matrix calculation is involved in the operation process of the above scaled dot product attention function. The output result of the softmax function will be multiplied by Vi, and this matrix multiplication calculation can also be directly implemented using a systolic array module with a size of s×dk.

[0125] The output end of the activation function operation module is connected to the input end of the temporary matrix storage unit. Among them, the activation function operation module uses the ReLU activation function.

[0126] The residual addition module includes multiple adders. The number of adders is the same as the number of rows of any input matrix in the multi-head attention layer. The input end of the residual addition module is connected to the output end of the input matrix storage unit, and the output end is connected to the input end of the layer normalization function operation module.

[0127] The output end of the layer normalization function operation module serves as the output interface of the entire hardware accelerator.

[0128] The control module is respectively connected to the data storage module, the systolic array module, the bias addition module, the dot product attention function operation module, the activation function operation module, the residual addition module, and the layer normalization function operation module.

[0129] Further, the following control processes are configured in the control module:

[0130] Determine whether the operation being executed in the current hardware accelerator is for a multi-head attention layer or a feed-forward layer.

[0131] If the operation being executed is for a multi-head attention layer, then perform the following operations:

[0132] Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the first linear layer on the first input matrix, the operation of the second linear layer on the second input matrix, and the operation of the third linear layer on the third input matrix, and control the dot product attention function operation module to perform a function operation on the operation results of the first linear layer, the second linear layer, and the third linear layer to obtain a concatenated matrix.

[0133] Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fourth linear layer on the concatenated matrix. And control the residual addition module to perform a residual calculation on the operation result of the fourth linear layer to obtain a first intermediate matrix.

[0134] Control the layer normalization function operation module to perform a function operation on the first intermediate matrix to obtain the output matrix of the multi-head attention layer.

[0135] If the operation being executed is for a feed-forward layer, then perform the following operations:

[0136] Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fifth linear layer on the input matrix of the feed-forward layer. And control the activation function operation module to perform an activation function operation on the operation result of the fifth linear layer to obtain an activation matrix.

[0137] Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the sixth linear layer on the activation matrix. And control the residual addition module to perform a residual calculation on the operation result of the sixth linear layer to obtain a second intermediate matrix.

[0138] Control the layer normalization function operation module to perform a function operation on the second intermediate matrix to obtain the output matrix of the feed-forward layer.

[0139] In actual operation, for the dot product attention function operation module, name its input matrix as D (with a size of s×s), and define the output matrix as Y (with a size of s×s). The Mask matrix M used for the Mask operation is a matrix with the same size as D, but all its internal elements are 0 or 1. The operation performed by this module is expressed by the following formula:

[0140]

[0141] Among them, Y(i, j) represents the element in the i-th row and j-th column of the output matrix Y, D(i, j) represents the element in the i-th row and j-th column of the input matrix D, M(i, j) represents the element in the i-th row and j-th column of the Mask matrix M, and exp() represents the exponential operation.

[0142] To avoid complex division operations and the overflow problem that may occur due to the excessively large output data caused by the e exponential operation, this embodiment adopts the method of converting the softmax function to logarithmic domain operation and subtracting the maximum value from all inputs simultaneously. This method can be expressed by the following formula:

[0143]

[0144] Among them,

[0145] Based on the above formula, the dot product attention function operation module can be divided into four stages, namely the first stage: finding the maximum value, the second stage: calculating The third stage: calculating ln(SUM), and the fourth stage: calculating exp(x i -x max -ln(sum)).

[0146] See Figure 8 and Figure 9 As shown in the structural diagram, the dot product attention function operation module (Mask-Softmax function operation module) includes a filter matrix storage unit and multiple function operation units. The number of function operation units is the same as the number of rows of any input matrix in the multi-head attention layer.

[0147] The filter matrix storage unit is respectively connected to each function operation unit. The filter matrix storage unit is used to store the Mask matrix M.

[0148] The function operation unit includes a maximum value acquisition subunit, a summation subunit, a logarithmic operation subunit, and an exponential operation subunit. Among them, the specific implementation of the exponential operation subunit and the logarithmic operation subunit can use methods such as lookup tables.

[0149] These four subunits correspond to the four stages in the function operation process. Among them, the maximum value acquisition subunit is used to execute the operation in the first stage, the summation subunit is used to execute the operation in the second stage, the logarithmic operation subunit is used to execute the operation in the third stage, and the exponential operation subunit is used to execute the operation in the fourth stage.

[0150] Each function operation unit is responsible for performing the mask-softmax operation on a corresponding row of the input matrix D. The following describes the four stages of the mask-softmax operation for the i-th function operation unit:

[0151] The first stage: Find the maximum value D(i,max) among the s matrix elements from D(i,1) to D(i,s). The specific operation process includes: First, set D(i,max) to negative infinity, and then in s cycles, input D(i,1), D(i,2), …, D(i,s) in sequence. In the t-th cycle, if M(i,t) equals 0 and D(i,t) is greater than D(i,max), then change the value of D(i,max) to D(i,t).

[0152] The second stage: Calculate and store it in the variable SUM. The method is to first initialize SUM to 0, and each time calculate Then if M(i,j) equals 0, then If M(i,j) equals 1, the value of SUM remains unchanged.

[0153] The third stage: For the SUM value obtained from the operation in the second stage, in this stage, it is used as the input of the logarithmic operation unit to calculate ln(SUM).

[0154] The fourth stage: Input D(i,1), D(i,2), …, D(i,s) in sequence as the input of the exponential (EXP) operation unit to calculate Y(i,1), Y(i,2), …, Y(i,s). If M(i,t) equals 1, then Y(i,t) equals 0; if M(i,t) equals 0, then Y(i,1), Y(i,2), …, Y(i,s) are the final output results of the i-th function operation unit.

[0155] In the embodiment of the present application, the input of the layer normalization function operation module is the intermediate matrix G of size s×dmodel, and the output is also a matrix of the same size (named Output), and its calculation can be expressed as:

[0156]

[0157] where, Output(i,j) represents the element in the i-th row and j-th column of the output matrix, var(G,i) represents the variance of the elements in the i-th row of the intermediate matrix G, G(i,j) represents the element in the i-th row and j-th column of the intermediate matrix G, E(G,i) represents the mean of the elements in the i-th row of the intermediate matrix, γ j represents the second parameter, β jRepresents the third parameter. ε is the first parameter, which is used to prevent the denominator from being zero so that the operation result becomes infinite, and its value is 10 -8 . The second parameter includes d model ones (γ1, γ2, …, γ j , …, γ dmodel ), which are respectively used to calculate the elements of different columns of the output matrix. The third parameter includes d model ones (β1, β2, …, β j , …, β dmodel ), which are respectively used to calculate the elements of different columns of the output matrix. Both the second parameter and the third parameter are preset values.

[0158] Generally, the following formula is used to calculate the mean value of the elements in the i-th row of the intermediate matrix:

[0159]

[0160] Generally, the following formula is used to calculate the variance of the elements in the i-th row of the intermediate matrix:

[0161]

[0162] In the process of performing layer normalization processing through the above formula, in order to obtain the variance value of each row of elements in the intermediate matrix, it is necessary to first calculate the average value of each row of elements in the intermediate matrix, then respectively obtain the difference between each element and the average value, square the difference and then perform an accumulation operation. Such a calculation process has relatively cumbersome steps. In the actual processing process, it will generate a large delay, increase the operation time of the Transformer neural network, and reduce the operation efficiency of the Transformer neural network.

[0163] In the embodiments of the present application, in order to reduce the delay, two optimized methods are provided. The first optimized method is to pre-calculate E(G, i) and E(G.*G, i) (G.*G represents the matrix obtained by squaring each element of the intermediate matrix G) while the systolic array module is still operating, and only s accumulators are needed for calculation. These s accumulators should be directly connected to the input end of the layer normalization function module. The key of the second optimized method lies in using another method to calculate the variance of the elements in the i-th row of the intermediate matrix, and the calculation formula is as follows:

[0164]

[0165] Based on the above disclosed optimized method, the structural diagram of the layer normalization function operation module disclosed in the embodiments of the present application is shown in Figure 10 and includes an intermediate matrix storage unit, a first mean calculation unit, a second mean calculation unit, a first square calculation unit, a second square calculation unit, a reciprocal square root calculation unit and an output matrix calculation unit.

[0166] The output end of the intermediate matrix storage unit is connected to the output matrix calculation unit. The intermediate matrix storage unit is used to obtain and store the intermediate matrix, where the intermediate matrix is the first intermediate matrix in the processing of the multi-head attention layer or the second intermediate matrix in the processing of the feed-forward layer.

[0167] The output end of the first mean calculation unit is respectively connected to the first square calculation unit and the output matrix calculation unit. The first mean calculation unit is used to calculate the mean of each row of elements in the intermediate matrix.

[0168] The output end of the first square calculation unit is connected to the reciprocal of square root calculation unit. The first square calculation unit is used to perform a square operation on the value input by the first mean calculation unit to obtain the square of the mean of each row of elements in the intermediate matrix.

[0169] The output end of the second square calculation unit is connected to the second mean calculation unit. The second square calculation unit is used to perform a square operation on each element in the intermediate matrix to obtain a square matrix.

[0170] The output end of the second mean calculation unit is connected to the reciprocal of square root calculation unit. The second mean calculation unit is used to calculate the mean of each row of elements in the square matrix.

[0171] The output end of the reciprocal of square root calculation unit is connected to the output matrix calculation unit. The reciprocal of square root calculation unit is used to obtain the reciprocal of the square root of the variance of each row of elements in the intermediate matrix according to the square of the mean of each row of elements in the intermediate matrix and the mean of each row of elements in the square matrix.

[0172] The output matrix calculation unit is used to perform layer normalization processing on each element of the intermediate matrix, the mean of each row of elements in the intermediate matrix, and the reciprocal of the square root of the variance of each row of elements in the intermediate matrix to obtain the final output matrix of the multi-head attention layer or the feed-forward layer.

[0173] Furthermore, the first mean calculation unit includes a plurality of first mean calculation sub-units, the second mean calculation unit includes a plurality of second mean calculation sub-units, the first square calculation unit includes a plurality of first square calculation sub-units, the second square calculation unit includes a plurality of second square calculation sub-units, the reciprocal of square root calculation unit includes a plurality of reciprocal of square root calculation sub-units, and the output matrix calculation unit includes a plurality of output matrix calculation sub-units.

[0174] The number of the first mean calculation sub-unit, the second mean calculation sub-unit, the first square calculation sub-unit, the second square calculation sub-unit, the square root reciprocal calculation sub-unit, and the output matrix calculation sub-unit is the same as the number of rows of any input matrix in the multi-head attention layer.

[0175] Combined with Figure 10 the disclosed structural diagram, the specific implementation process of the layer normalization function operation module is as follows:

[0176] Input the intermediate matrix G into the layer normalization function operation module, and input a column of elements of this matrix each time. That is, input G(1,1)-G(s,1) at the first moment, input G(1,j)-G(s,j) at the j-th moment, and so on until input G(1,dmodel)-G(s,dmodel) at the last moment. At the same time, the intermediate matrix storage unit, the first mean calculation unit, the second square calculation unit, and the second mean calculation unit in the layer normalization function operation module perform the following operations: store the intermediate matrix G in the "intermediate matrix storage unit"; accumulate and calculate Accumulate and calculate After the input of the intermediate matrix G is completed, and Use the first square calculation unit to calculate E(G,1) 2 , E(G,2) 2 and E(G,s) 2 .

[0177] According to the operation results of the first mean calculation unit, the second mean calculation unit, and the first square calculation unit, use the adder in the square root reciprocal calculation unit to calculate var(G,1) = E(G,1) 2 -E(G.*G,1), var(G,2) = E(G,2) 2 -E(G.*G,2), ……, var(G,s) = E(G,s) 2 -E(G.*G,s). Then use the "x^(-0.5)" operation unit to find r1 = (var(G,1)+ε)^(-0.5), r2 = (var(G,2)+ε)^(-0.5), ……, r s = (var(G,s)+ε)^(-0.5).

[0178] According to the operation results of the intermediate matrix storage unit, the first mean calculation unit, and the square root reciprocal calculation unit, the output matrix calculation unit outputs according to the formula Calculate the final output matrix. The outputs at the first moment are Output(1,1), Output(2,1), ……, Output(s,1), the outputs at the second moment are Output(1,2), Output(2,2), ……, Output(s,2), until the dmodel-th moment, the outputs Output(1,dmodel), Output(2,dmodel), ……, Output(s,dmodel), and obtain the final output matrix of the layer normalization function operation module.

[0179] The second embodiment of the present application discloses a calculation method applied to a hardware accelerator for a Transformer neural network. The calculation method is applied to a control module, and the control module is located in the hardware accelerator for a Transformer neural network described in the first embodiment of the present application. The calculation method includes:

[0180] Determine whether the operation being executed in the current hardware accelerator is the operation of the multi-head attention layer or the operation of the feed-forward layer.

[0181] If the operation being executed is the operation of the multi-head attention layer, the following steps are included:

[0182] Step S101, control the data storage module, the systolic array module, and the bias addition module to perform the operation of the first linear layer on the first input matrix, the operation of the second linear layer on the second input matrix, and the operation of the third linear layer on the third input matrix, and control the dot product attention function operation module to perform a function operation on the operation results of the first linear layer, the second linear layer, and the third linear layer to obtain a concatenated matrix.

[0183] Step S102, control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fourth linear layer on the concatenated matrix. And control the residual addition module to perform a residual calculation on the operation result of the fourth linear layer to obtain a first intermediate matrix.

[0184] Step S103, control the layer normalization function operation module to perform a function operation on the first intermediate matrix to obtain the output matrix of the multi-head attention layer.

[0185] If the operation being executed is the operation of the feed-forward layer, then the following steps are executed:

[0186] Step S201, control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fifth linear layer on the input matrix of the feed-forward layer. And control the activation function operation module to perform an activation function operation on the operation result of the fifth linear layer to obtain an activation matrix.

[0187] Step S202: Control the data storage module, the systolic array module, and the bias addition module to perform the arithmetic operations of the sixth linear layer on the activation matrix. Further, control the residual addition module to perform residual calculation on the arithmetic result of the sixth linear layer to obtain a second intermediate matrix.

[0188] Step S203: Control the layer normalization function arithmetic module to perform function operations on the second intermediate matrix to obtain the output matrix of the feedforward layer.

[0189] Furthermore, step S101 includes:

[0190] Step S1011: Obtain a first input matrix from the first matrix storage subunit, obtain a first weight matrix from the weight matrix storage unit, input the first input matrix and the first weight matrix into the systolic array module to perform matrix calculation, and then input the result of the matrix calculation into the bias addition module to add it to the first bias matrix Bias Q1 to obtain a first temporary matrix, and finally input the first temporary matrix into the first temporary matrix storage subunit.

[0191] Step S1012: Obtain a second input matrix from the second matrix storage subunit, obtain a second weight matrix from the weight matrix storage unit, input the second input matrix and the second weight matrix into the systolic array module to perform matrix calculation, and then input the result of the matrix calculation into the bias addition module to add it to the second bias matrix Bias K1 to obtain a second temporary matrix, and finally input the second temporary matrix into the second temporary matrix storage subunit.

[0192] Step S1013: Take out the first temporary matrix from the first temporary matrix storage subunit, take out the transposed matrix of the second temporary matrix from the second temporary matrix storage subunit, input the first temporary matrix and the transposed matrix of the second temporary matrix into the systolic array module to perform matrix calculation, and then input the result of the matrix calculation into the dot product attention function arithmetic module.

[0193] Step S1014: Control the dot product attention function arithmetic module to perform function operations on the received matrix data to obtain a third temporary matrix, and store the third temporary matrix in the first temporary matrix storage subunit. Meanwhile, obtain a third input matrix from the second matrix storage subunit, obtain a third weight matrix from the weight matrix storage unit, input the third input matrix and the third weight matrix into the systolic array module to perform matrix calculation, and then input the result of the matrix calculation into the bias addition module to add it to the third bias matrix BiasV1 Perform addition to obtain a fourth temporary matrix, and finally input the fourth temporary matrix into the second temporary matrix storage subunit.

[0194] Step S1015: Take out the third temporary matrix from the first temporary matrix storage subunit, take out the fourth temporary matrix from the second temporary matrix storage subunit, input these two matrices into the systolic array module to perform matrix calculations, and then input the obtained matrix to be stitched into the third temporary matrix storage subunit.

[0195] Step S1016: Repeat steps S1011 - S1015 until a preset number of times is met. The preset number of times is the same as the number of attention heads included in the multi - head attention layer. After execution, the multiple matrices to be stitched stored in the third temporary matrix storage subunit form a stitched matrix.

[0196] Further, step S102 includes:

[0197] Step S1021: Take out the stitched matrix from the third temporary matrix storage subunit, take out a sub - matrix of the fourth weight matrix from the weight matrix storage unit, input the stitched matrix and the sub - matrix of the fourth weight matrix into the systolic array module to perform matrix calculations, then input the result of the matrix calculation into the bias addition module to add with a sub - matrix of the fourth bias matrix, and input the addition result into the residual addition module to add with a sub - matrix of the first input matrix to obtain a first intermediate sub - matrix.

[0198] Step S1022: Repeat step S1021 until the preset number of times is met. After execution, the obtained multiple first intermediate sub - matrices form a first intermediate matrix. Among them, in different execution processes, different sub - matrices of the fourth weight matrix, different sub - matrices of the fourth bias matrix, and different sub - matrices of the first input matrix are taken out.

[0199] Further, step S201 includes:

[0200] Step S2011: Take out the input matrix of the feed - forward layer from the first matrix storage subunit, take out a sub - matrix of the fifth weight matrix from the weight matrix storage unit, input the input matrix of the feed - forward layer and the sub - matrix of the fifth weight matrix into the systolic array module to perform matrix calculations, and input the result of the matrix calculation into the bias addition module to add with a sub - matrix of the fifth bias matrix, then input the addition result into the activation function operation module to obtain an activation sub - matrix, and input the activation sub - matrix into the third temporary matrix storage subunit.

[0201] In step S2012, step S2011 is repeated until the preset number of times is satisfied. After execution, the multiple activation sub-matrices stored in the third temporary matrix storage sub-unit form an activation matrix. Among them, in different execution processes, different sub-matrices in the fifth weight matrix are taken out, and different sub-matrices in the fifth bias matrix are taken out.

[0202] Further, step S202 includes:

[0203] In step S2021, the activation matrix is taken out from the third temporary matrix storage sub-unit, and a sub-matrix of the sixth weight matrix is taken out from the weight matrix storage unit. The activation matrix and the sub-matrix of the sixth weight matrix are input into the systolic array module to perform matrix calculation, and the result of the matrix calculation is input into the bias addition module to be added to a sub-matrix of the sixth bias matrix. The addition result is input into the residual addition module and added to a sub-matrix of the feed-forward layer input matrix to obtain a second intermediate sub-matrix.

[0204] In step S2022, step S2021 is repeatedly executed until the preset number of times is satisfied. After execution, the multiple second intermediate sub-matrices obtained form a second intermediate matrix. Among them, in different execution processes, different sub-matrices in the sixth weight matrix are taken out, different sub-matrices in the sixth bias matrix are taken out, and different sub-matrices in the feed-forward layer input matrix are taken out.

[0205] Further, step S103 or step S203 includes:

[0206] All elements of the intermediate matrix are sequentially input into the intermediate matrix storage unit according to the column order. Among them, if step S103 is executed, the intermediate matrix is the first intermediate matrix; if step S203 is executed, the intermediate matrix is the second intermediate matrix.

[0207] Each row of elements of the intermediate matrix is respectively input into multiple first mean calculation sub-units to calculate the mean of each row of elements in the intermediate matrix. And each row of elements of the intermediate matrix is respectively input into multiple second square calculation sub-units to obtain a square matrix.

[0208] The mean of each row of elements in the intermediate matrix is respectively input into multiple first square calculation sub-units to obtain the square of the mean of each row of elements in the intermediate matrix.

[0209] Each row of elements in the square matrix is respectively input into multiple second mean calculation sub-units to calculate the mean of each row of elements in the square matrix.

[0210] The square of the mean value of each row element in the intermediate matrix and the mean value of each row element in the square matrix are respectively input into a plurality of square root reciprocal calculation sub-units to obtain the square root reciprocal of the variance of each row element in the intermediate matrix.

[0211] Each element of the intermediate matrix, the mean value of each row element in the intermediate matrix, and the square root reciprocal of the variance of each row element in the intermediate matrix are respectively input into a plurality of output matrix calculation sub-units to obtain the final output matrix of the multi-head attention layer or the feed-forward layer.

[0212] The third embodiment of the present application discloses a computer device, including:

[0213] A memory for storing a computer program;

[0214] A processor for implementing the steps of the calculation method of the hardware accelerator applied to the Transformer neural network as described in the second embodiment of the present application when executing the computer program.

[0215] The fourth embodiment of the present application discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is processed and executed, it implements the steps of the calculation method of the hardware accelerator applied to the Transformer neural network as described in the second embodiment of the present application.

[0216] The present application discloses a hardware accelerator applied to the Transformer neural network and its calculation method. The hardware accelerator includes: a data storage module, a systolic array module, a bias addition module, a dot product attention function operation module, an activation function operation module, a residual addition module, a layer normalization function operation module, and a control module. Among them, the control module is used to control the calculation process of the hardware accelerator. The systolic array module includes a plurality of basic operation units for performing matrix calculations and can execute all matrix calculations in the operation process of the Transformer neural network. This hardware accelerator belongs to a dedicated accelerator for the Transformer neural network and can effectively improve the operation speed and efficiency of the Transformer neural network.

[0217] The above has described the present application in detail in combination with specific implementation manners and exemplary examples, but these descriptions should not be construed as limitations on the present application. Those skilled in the art understand that without departing from the spirit and scope of the present application, various equivalent replacements, modifications, or improvements can be made to the technical solutions and their implementation manners of the present application, and these all fall within the scope of the present application. The protection scope of the present application is subject to the appended claims.

Claims

1. A hardware accelerator applied to a Transformer neural network, characterized in that, Comprising: A data storage module, including an input matrix storage unit, a weight matrix storage unit, a bias matrix storage unit, and a temporary matrix storage unit. The input end of the input matrix storage unit serves as the input interface of the entire hardware accelerator. The output ends of the input matrix storage unit, the weight matrix storage unit, and the temporary matrix storage unit are all connected to the input end of the systolic array module. The output end of the bias matrix storage unit is connected to the input end of the bias addition module. The weight matrix storage unit is used to store a first weight matrix, a second weight matrix, a third weight matrix, a fourth weight matrix, a fifth weight matrix, and a sixth weight matrix. The fourth weight matrix, the fifth weight matrix, and the sixth weight matrix are all longitudinally divided into multiple sub-matrices, and the number of columns of each sub-matrix is the same as the number of columns of the first weight matrix. The first weight matrix, the second weight matrix, and the third weight matrix are the basic operation units of the multi-layer attention layer; A systolic array module, which includes multiple basic operation units for performing matrix calculations. The number of rows of the basic operation unit is the same as the number of rows of any input matrix in the multi-head attention layer, and the number of columns of the basic operation unit is the same as the number of columns of the first weight matrix. The output end of the systolic array module is connected to the input end of the bias addition module, the input end of the dot product attention function operation module, and the input end of the temporary matrix storage unit; A bias addition module, including multiple adders. The number of adders is the same as the number of rows of any input matrix in the multi-head attention layer. The output end of the bias addition module is connected to the input end of the temporary matrix storage unit, the input end of the activation function operation module, and the input end of the residual addition module; A dot product attention function operation module, and the output end of the dot product attention function operation module is connected to the input end of the temporary matrix storage unit; An activation function operation module, and the output end of the activation function operation module is connected to the input end of the temporary matrix storage unit; A residual addition module, including multiple adders. The number of adders is the same as the number of rows of any input matrix in the multi-head attention layer. The input end of the residual addition module is connected to the output end of the input matrix storage unit, and the output end is connected to the input end of the layer normalization function operation module; A layer normalization function operation module, and the output end of the layer normalization function operation module serves as the output interface of the entire hardware accelerator; A control module, which is respectively connected to the data storage module, the systolic array module, the bias addition module, the dot product attention function operation module, the activation function operation module, the residual addition module, and the layer normalization function operation module.

2. The hardware accelerator applied to the Transformer neural network according to claim 1, characterized in that The following control process is configured in the control module: Judge whether the operation executed in the current hardware accelerator is the operation of the multi-head attention layer or the operation of the feed-forward layer; If the operation executed is the operation of the multi-head attention layer, then perform the following operations: Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the first linear layer for the first input matrix, the operation of the second linear layer for the second input matrix, and the operation of the third linear layer for the third input matrix, and control the dot product attention function operation module to perform a function operation on the operation results of the first linear layer, the second linear layer, and the third linear layer to obtain a concatenated matrix; Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fourth linear layer for the concatenated matrix; and control the residual addition module to perform a residual calculation on the operation result of the fourth linear layer to obtain a first intermediate matrix; Control the layer normalization function operation module to perform a function operation on the first intermediate matrix to obtain the output matrix of the multi-head attention layer; If the operation of the feed-forward layer is being performed, perform the following operations: Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fifth linear layer for the input matrix of the feed-forward layer; and control the activation function operation module to perform an activation function operation on the operation result of the fifth linear layer to obtain an activation matrix; Control the data storage module, the systolic array module, and the bias addition module to perform the operation of the sixth linear layer for the activation matrix; and control the residual addition module to perform a residual calculation on the operation result of the sixth linear layer to obtain a second intermediate matrix; Control the layer normalization function operation module to perform a function operation on the second intermediate matrix to obtain the output matrix of the feed-forward layer.

3. The hardware accelerator applied to the Transformer neural network according to claim 1, wherein The bias matrix storage unit is used to store a first bias matrix, a second bias matrix, a third bias matrix, a fourth bias matrix, a fifth bias matrix, and a sixth bias matrix, wherein the fourth bias matrix, the fifth bias matrix, and the sixth bias matrix are each divided into a plurality of sub-matrices, and the number of columns of each sub-matrix is the same as the number of columns of the first weight matrix.

4. The hardware accelerator applied to the Transformer neural network according to claim 1, characterized in that The input matrix storage unit includes a first matrix storage sub-unit and a second matrix storage sub-unit; The first matrix storage sub-unit is used to store the first input matrix of the multi-head attention layer and the input matrix of the feed-forward layer; The second matrix storage sub-unit is used to store the second input matrix and the third input matrix of the multi-head attention layer.

5. The hardware accelerator applied to the Transformer neural network according to claim 1, characterized in that, The temporary matrix storage unit includes a first temporary matrix storage sub-unit, a second temporary matrix storage sub-unit, and a third temporary matrix storage sub-unit.

6. The hardware accelerator applied to a Transformer neural network according to claim 5, wherein The output end of the systolic array module is connected to the input end of the third temporary matrix storage sub-unit; The output end of the bias addition module is connected to the input end of the first temporary matrix storage sub-unit and the input end of the second temporary matrix storage sub-unit; The output end of the dot product attention function operation module is connected to the input end of the first temporary matrix storage sub-unit; The output end of the activation function operation module is connected to the input end of the third temporary matrix storage sub-unit.

7. The hardware accelerator applied to the Transformer neural network according to claim 1, wherein The dot product attention function operation module includes a filter matrix storage unit and a plurality of function operation units, and the number of the function operation units is consistent with the number of rows of any input matrix in the multi-head attention layer; The filter matrix storage unit is respectively connected to each of the function operation units; The function operation unit includes a maximum value acquisition subunit, a summation subunit, a logarithm operation subunit, and an exponential operation subunit.

8. The hardware accelerator applied to the Transformer neural network according to claim 1, characterized in that, The layer normalization function operation module includes an intermediate matrix storage unit, a first mean calculation unit, a second mean calculation unit, a first square calculation unit, a second square calculation unit, a square root reciprocal calculation unit, and an output matrix calculation unit; The output end of the intermediate matrix storage unit is connected to the output matrix calculation unit. The intermediate matrix storage unit is used to acquire and store the intermediate matrix, and the intermediate matrix is the first intermediate matrix in the processing process of the multi-head attention layer or the second intermediate matrix in the processing process of the feed-forward layer; The output end of the first mean calculation unit is respectively connected to the first square calculation unit and the output matrix calculation unit. The first mean calculation unit is used to calculate the mean value of each row of elements in the intermediate matrix; The output end of the first square calculation unit is connected to the square root reciprocal calculation unit. The first square calculation unit is used to perform a square operation on the value input by the first mean calculation unit to obtain the square of the mean value of each row of elements in the intermediate matrix; The output end of the second square calculation unit is connected to the second mean calculation unit. The second square calculation unit is used to perform a square operation on each element in the intermediate matrix to obtain a square matrix; The output end of the second mean calculation unit is connected to the square root reciprocal calculation unit. The second mean calculation unit is used to calculate the mean value of each row of elements in the square matrix; The output end of the square root reciprocal calculation unit is connected to the output matrix calculation unit. The square root reciprocal calculation unit is used to obtain the reciprocal of the square root of the variance of each row of elements in the intermediate matrix according to the square of the mean value of each row of elements in the intermediate matrix and the mean value of each row of elements in the square matrix; The output matrix calculation unit is used to perform layer normalization processing on each element of the intermediate matrix, the mean value of each row of elements in the intermediate matrix, and the reciprocal of the square root of the variance of each row of elements in the intermediate matrix to obtain the final output matrix of the multi-head attention layer or the feed-forward layer.

9. The hardware accelerator applied to the Transformer neural network according to claim 8, wherein The first mean calculation unit includes a plurality of first mean calculation subunits, the second mean calculation unit includes a plurality of second mean calculation subunits, the first square calculation unit includes a plurality of first square calculation subunits, the second square calculation unit includes a plurality of second square calculation subunits, the square root reciprocal calculation unit includes a plurality of square root reciprocal calculation subunits, and the output matrix calculation unit includes a plurality of output matrix calculation subunits; The numbers of the first mean calculation sub-unit, the second mean calculation sub-unit, the first square calculation sub-unit, the second square calculation sub-unit, the reciprocal square root calculation sub-unit, and the output matrix calculation sub-unit are all consistent with the number of rows of any input matrix in the multi-head attention layer.

10. A computing method for a hardware accelerator applied to a Transformer neural network, characterized in that, The calculation method is applied to a control module, which is located in the hardware accelerator for Transformer neural network according to any one of claims 1-9. The calculation method includes: Determine whether the operation executed in the current hardware accelerator is the operation of the multi-head attention layer or the operation of the feed-forward layer; If the operation executed is the operation of the multi-head attention layer, the following steps include: Step S101, control the data storage module, the systolic array module, and the bias addition module to perform the operation of the first linear layer on the first input matrix, the operation of the second linear layer on the second input matrix, and the operation of the third linear layer on the third input matrix, and control the dot product attention function operation module to perform a function operation on the operation results of the first linear layer, the second linear layer, and the third linear layer to obtain a concatenated matrix; Step S102, control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fourth linear layer on the concatenated matrix; and control the residual addition module to perform a residual calculation on the operation result of the fourth linear layer to obtain a first intermediate matrix; Step S103, control the layer normalization function operation module to perform a function operation on the first intermediate matrix to obtain the output matrix of the multi-head attention layer; If the operation executed is the operation of the feed-forward layer, then perform the following steps: Step S201, control the data storage module, the systolic array module, and the bias addition module to perform the operation of the fifth linear layer on the input matrix of the feed-forward layer; and control the activation function operation module to perform an activation function operation on the operation result of the fifth linear layer to obtain an activation matrix; Step S202, control the data storage module, the systolic array module, and the bias addition module to perform the operation of the sixth linear layer on the activation matrix; and control the residual addition module to perform a residual calculation on the operation result of the sixth linear layer to obtain a second intermediate matrix; Step S203, control the layer normalization function operation module to perform a function operation on the second intermediate matrix to obtain the output matrix of the feed-forward layer.

11. The calculation method of the hardware accelerator applied to the Transformer neural network according to claim 10, characterized in that, The step S101 includes: Step S1011, obtain the first input matrix from the first matrix storage sub-unit, obtain the first weight matrix from the weight matrix storage unit, input the first input matrix and the first weight matrix into the systolic array module to perform matrix calculation, then input the result of the matrix calculation into the bias addition module to add with the first bias matrix to obtain a first temporary matrix, and finally input the first temporary matrix into the first temporary matrix storage sub-unit; Step S1012: Obtain the second input matrix from the second matrix storage subunit, obtain the second weight matrix from the weight matrix storage unit, input the second input matrix and the second weight matrix into the systolic array module to perform matrix calculations, then input the result of the matrix calculations into the bias addition module to add it to the second bias matrix to obtain a second temporary matrix, and finally input the second temporary matrix into the second temporary matrix storage subunit; Step S1013: Take out the first temporary matrix from the first temporary matrix storage subunit, take out the transposed matrix of the second temporary matrix from the second temporary matrix storage subunit, input the first temporary matrix and the transposed matrix of the second temporary matrix into the systolic array module to perform matrix calculations, and then input the result of the matrix calculations into the dot product attention function operation module; Step S1014: Control the dot product attention function operation module to perform a function operation on the received matrix data to obtain a third temporary matrix, and store the third temporary matrix in the first temporary matrix storage subunit; meanwhile, obtain the third input matrix from the second matrix storage subunit, obtain the third weight matrix from the weight matrix storage unit, input the third input matrix and the third weight matrix into the systolic array module to perform matrix calculations, then input the result of the matrix calculations into the bias addition module to add it to the third bias matrix to obtain a fourth temporary matrix, and finally input the fourth temporary matrix into the second temporary matrix storage subunit; Step S1015: Take out the third temporary matrix from the first temporary matrix storage subunit, take out the fourth temporary matrix from the second temporary matrix storage subunit, input these two matrices into the systolic array module to perform matrix calculations, and then input the obtained matrix to be concatenated into the third temporary matrix storage subunit; Step S1016: Repeat steps S1011 - S1015 until the preset number of times is met. The preset number of times is the same as the number of attention heads included in the multi - head attention layer. After execution, the multiple matrices to be concatenated stored in the third temporary matrix storage subunit form a concatenated matrix.

12. The calculation method of the hardware accelerator applied to the Transformer neural network according to claim 11, characterized in that, The step S102 includes: Step S1021: Take out the concatenated matrix from the third temporary matrix storage subunit, take out a sub - matrix of the fourth weight matrix from the weight matrix storage unit, input the concatenated matrix and the sub - matrix of the fourth weight matrix into the systolic array module to perform matrix calculations, then input the result of the matrix calculations into the bias addition module to add it to a sub - matrix of the fourth bias matrix, and input the addition result into the residual addition module to add it to a sub - matrix of the first input matrix to obtain a first intermediate sub - matrix; Step S1022: Repeat step S1021 until the preset number of times is satisfied. After completion, the multiple first intermediate sub-matrices obtained form a first intermediate matrix. Among them, in different execution processes, different sub-matrices are taken out from the fourth weight matrix, different sub-matrices are taken out from the fourth bias matrix, and different sub-matrices are taken out from the first input matrix.

13. The calculation method of the hardware accelerator applied to the Transformer neural network according to claim 12, wherein, The step S201 includes: Step S2011: Take out the input matrix of the feedforward layer from the first matrix storage sub-unit, take out a sub-matrix of the fifth weight matrix from the weight matrix storage unit, input the input matrix of the feedforward layer and the sub-matrix of the fifth weight matrix into the systolic array module to perform matrix calculation, and input the result of the matrix calculation into the bias addition module to add it with a sub-matrix of the fifth bias matrix, input the addition result into the activation function operation module to obtain an activation sub-matrix, and input the activation sub-matrix into the third temporary matrix storage sub-unit; Step S2012: Repeat step S2011 until the preset number of times is satisfied. After completion, the multiple activation sub-matrices stored in the third temporary matrix storage sub-unit form an activation matrix. Among them, in different execution processes, different sub-matrices are taken out from the fifth weight matrix and different sub-matrices are taken out from the fifth bias matrix.

14. The calculation method of the hardware accelerator applied to the Transformer neural network according to claim 13, wherein The step S202 includes: Step S2021: Take out the activation matrix from the third temporary matrix storage sub-unit, take out a sub-matrix of the sixth weight matrix from the weight matrix storage unit, input the activation matrix and the sub-matrix of the sixth weight matrix into the systolic array module to perform matrix calculation, and input the result of the matrix calculation into the bias addition module to add it with a sub-matrix of the sixth bias matrix, and input the addition result into the residual addition module to add it with a sub-matrix of the feedforward layer input matrix to obtain a second intermediate sub-matrix; Step S2022: Repeat step S2021 until the preset number of times is satisfied. After completion, the multiple second intermediate sub-matrices obtained form a second intermediate matrix. Among them, in different execution processes, different sub-matrices are taken out from the sixth weight matrix, different sub-matrices are taken out from the sixth bias matrix, and different sub-matrices are taken out from the feedforward layer input matrix.

15. The calculation method of the hardware accelerator applied to the Transformer neural network according to claim 14, wherein The step S103 or the step S203 includes: Input all elements of the intermediate matrix into the intermediate matrix storage unit in column order. Among them, if step S103 is executed, the intermediate matrix is the first intermediate matrix; if step S203 is executed, the intermediate matrix is the second intermediate matrix; Input each row of elements of the intermediate matrix into multiple first mean calculation sub-units to calculate the mean of each row of elements in the intermediate matrix; and input each row of elements of the intermediate matrix into multiple second square calculation sub-units to obtain a square matrix; The mean values of each row of elements in the intermediate matrix are respectively input into a plurality of first square calculation sub-units to obtain the squares of the mean values of each row of elements in the intermediate matrix; Each row of elements in the square matrix is respectively input into a plurality of second mean value calculation sub-units to calculate the mean value of each row of elements in the square matrix; The squares of the mean values of each row of elements in the intermediate matrix and the mean values of each row of elements in the square matrix are respectively input into a plurality of square root reciprocal calculation sub-units to obtain the square root reciprocals of the variances of each row of elements in the intermediate matrix; Each element of the intermediate matrix, the mean value of each row of elements in the intermediate matrix, and the square root reciprocal of the variance of each row of elements in the intermediate matrix are respectively input into a plurality of output matrix calculation sub-units to obtain the final output matrix of the multi-head attention layer or the feed-forward layer.

Images