Hardware architecture and computational methods for Fastformer neural networks

Abstract

The present application discloses a hardware architecture and a calculation method for Fastformer neural network. The hardware architecture includes a data storage module, a read buffer module, a write buffer module, a calculation module and a control module; wherein the control module is used to control the hardware architecture to perform calculation operations, and the calculation module includes a linear calculation submodule, a nonlinear calculation submodule and an intermediate storage submodule; the linear calculation module includes a plurality of vector calculation units for performing linear operations including vector matrix multiplication, matrix addition and element-by-element product, and the nonlinear submodule is used to perform normalized exponential function operations. The hardware architecture and its calculation method can effectively improve the calculation speed and efficiency of Fastformer neural network.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a hardware architecture and a calculation method thereof applied to a Fastformer neural network. Background Art

[0002] The Fastformer network is a neural network model designed for natural language processing tasks. Its model architecture is as follows: Figure 1 As shown in Figure 2, the Fastformer network uses an additive attention mechanism that can build contextual information with linear complexity.

[0003] The number of rows of the input matrix Input of the Fastforemer network represents the hidden dimension of the network, and the number of columns represents the length of the input sequence. The hidden dimension of the network is generally 256, and the length of the input sequence is 128-65536, which is determined by the input length of the text processed by different tasks; in general, the length of the input sequence is greater than the hidden dimension of the network. First, the input matrix is ​​transformed into a query matrix, a key matrix and a value matrix through three parallel linear layers (query linear layer, key linear layer and value linear layer, each of which includes a pre-trained weight matrix and bias vector); then the query matrix is ​​subjected to normalized exponential processing to obtain a global query vector; then the global query vector and the key matrix are subjected to element-by-element multiplication operation, so that the global query vector learns the context-related key matrix to obtain an intermediate key matrix, and then the intermediate key matrix is ​​subjected to normalized exponential processing to obtain a global key vector; then the global key vector and the value matrix are subjected to element-by-element multiplication operation, so that the global key vector learns the context-related value matrix to obtain an intermediate value matrix; the intermediate value matrix is ​​transformed into a global value matrix through a pre-trained linear layer (similar to the above three parallel linear layers, also including pre-trained weight matrix and bias vector); finally, the global value matrix is ​​added to the query matrix to obtain an output matrix.

[0004] Currently, the Fastformer network model is operated on general computing platforms such as CPU or GPU, and no dedicated hardware architecture has been developed for Fastformer neural networks. If the operation speed and efficiency of the neural network are to be improved, a dedicated hardware architecture needs to be designed urgently. Summary of the invention

[0005] In order to improve the computing speed and efficiency of the Fastformer neural network, this application discloses a hardware architecture and a computing method thereof applied to the Fastformer neural network through the following aspects.

[0006] In a first aspect, the present application discloses a hardware architecture applied to a Fastformer neural network, including: a data storage module, a read buffer module, a write buffer module, a calculation module, and a control module;

[0007] The data storage module is used to receive and store the input matrix to be calculated, and to store pre-trained parameters, and to store the intermediate data generated by the calculation module during the calculation process, and to store the calculation results of the hardware architecture; the input end of the data storage module is the input interface of the entire hardware architecture, and is connected to the output end of the write buffer module; the output end of the data storage module is connected to the input end of the read buffer module, and serves as the output interface of the hardware architecture;

[0008] The input end of the read buffer module is also connected to the output end of the calculation module, and the output end of the read buffer module is connected to the input end of the calculation module;

[0009] The input end of the write buffer module is connected to the output end of the calculation module;

[0010] The calculation module includes a linear calculation submodule, a nonlinear calculation submodule and an intermediate storage submodule; the linear calculation submodule is used to perform linear operations including vector matrix multiplication, matrix addition, and element-by-element product; the nonlinear submodule is used to perform normalized exponential function operations; and the intermediate storage submodule is used to receive the calculation results of the nonlinear submodule and transmit them to the linear calculation submodule; wherein the input end of the linear calculation submodule is respectively connected to the output end of the read buffer and the output end of the intermediate storage submodule, and the output end is respectively connected to the input end of the read buffer, the input end of the write buffer module and the input end of the nonlinear calculation submodule; the output end of the nonlinear calculation submodule is connected to the input end of the intermediate storage submodule;

[0011] The control module is respectively connected to the data storage module, the read buffer module, the write buffer module, and the linear calculation submodule and the nonlinear calculation submodule in the calculation module.

[0012] Optionally, the control module is used to perform the following operations:

[0013] Control the data storage module, the read buffer module and the linear calculation submodule to perform a linear layer query operation on the input matrix, obtain a query matrix and output it to the read buffer module and the write buffer module respectively; the write buffer module transmits the query matrix to the data storage module;

[0014] Control the data storage module, the read buffer module, and the linear calculation submodule to perform vector-matrix multiplication on the query matrix, obtain an intermediate query vector, and transmit it to the nonlinear calculation submodule;

[0015] The nonlinear calculation submodule is controlled to perform a normalized exponential function operation on the intermediate query vector to obtain a normalized query vector and output it to the intermediate storage submodule; at the same time, the data storage module, the read buffer module and the linear calculation submodule are controlled to perform an attention key linear layer operation on the input matrix to obtain a key matrix and output it to the write buffer module; the write buffer module transmits the key matrix to the data storage module;

[0016] Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication operations on the normalized query vector and the query matrix to obtain a global query vector, and output it to the read buffer module;

[0017] Control the read buffer module, the linear calculation submodule and the data storage module to perform element-by-element product of the global query vector and the key matrix to obtain an intermediate key matrix, perform vector-matrix multiplication operation on the intermediate key matrix to obtain an intermediate key vector and output it to the nonlinear calculation submodule, and simultaneously control the linear calculation submodule to output the intermediate key matrix to the write buffer module; the write buffer module stores the intermediate key matrix to the data storage module;

[0018] The nonlinear calculation submodule is controlled to perform a normalized exponential function operation on the intermediate key vector to obtain a normalized key vector and output it to the intermediate storage submodule; at the same time, the data storage module, the read buffer module and the linear calculation submodule are controlled to perform an attention value linear layer operation on the input matrix to obtain a value matrix and output it to the write buffer module; the write buffer module transmits the value matrix to the data storage module;

[0019] Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication operations on the normalized key vector and the key matrix, obtain a global key vector, and output it to the read buffer module;

[0020] The linear calculation submodule, the read buffer module, and the data storage module are controlled to perform element-by-element product operations on the value matrix and the global key vector, and the operation results are subjected to linear layer operations to obtain a global value matrix; the global value matrix and the query matrix are added to obtain an output matrix of the neural network and output to the write buffer module; the write buffer module transmits the output matrix to the data storage module.

[0021] Optionally, the weight matrix storage unit is used to store the query weight matrix, the key weight matrix, the value weight matrix and the transformation weight matrix; wherein the number of rows and columns of the query weight matrix, the key weight matrix, the value weight matrix and the transformation weight matrix are equal to the number of rows of the input matrix.

[0022] Optionally, the bias vector storage unit is used to store a query bias vector, a key bias vector, a value bias vector and a transformation bias vector; wherein the query bias vector, the key bias vector, the value bias vector and the transformation bias vector are all column vectors whose number of rows is equal to the number of rows of the input matrix.

[0023] Optionally, the parameter vector unit is used to store a first parameter vector and a second parameter vector; the first parameter vector and the second parameter vector are both column vectors whose number of columns is equal to the number of rows of the input matrix.

[0024] Optionally, the linear calculation submodule includes a plurality of vector calculation units, wherein each vector calculation unit includes an element-by-element product operation unit, an inner product unit, a temporary vector storage unit and an addition unit;

[0025] The element-by-element product operation unit includes a plurality of multipliers; wherein the number of the multipliers is equal to the number of rows of any input matrix; the input end of the element-by-element product operation unit is respectively connected to the output end of the read buffer module and the output end of the temporary vector storage unit, and the output end of the element-by-element product operation unit is respectively connected to the inner product unit and the temporary vector storage unit; the element-by-element product operation unit is used to perform element-by-element multiplication operation on two input vectors;

[0026] The input end of the inner product unit is connected to the output end of the element-by-element product operation unit, and the output end of the inner product unit is connected to the addition unit;

[0027] The input end of the temporary vector storage unit is connected to the output end of the element-by-element product operation unit, and the output end of the temporary vector storage unit is respectively connected to the input end of the element-by-element product operation unit and the input end of the write buffer module;

[0028] The input end of the adding unit is respectively connected to the output end of the inner product unit and the output end of the read buffer module, and the output end of the adding unit is connected to the input end of the read buffer module and the input end of the write buffer module.

[0029] Optionally, when a linear layer operation is performed in the linear calculation submodule, the control module is further configured to perform the following operations:

[0030] Step 101, control the read buffer module to input the target column of the matrix to be calculated and the target row of the weight matrix to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, and the target accumulation result is obtained through accumulation operation and output to the addition unit, and at the same time control the read buffer module to input the corresponding element value of the bias matrix to the addition unit; the addition unit adds the target accumulation result and the corresponding element value in the bias matrix to obtain the target value;

[0031] Step 102, repeatedly executing step 101 to obtain multiple target values, until the target column is the last column in the matrix to be calculated;

[0032] Step 103, concatenate multiple target values ​​to obtain a result matrix; wherein the size of the result matrix is ​​consistent with the size of the matrix to be calculated.

[0033] Optionally, when the vector-matrix multiplication operation is performed in the linear calculation submodule, the control module is further configured to perform the following operations:

[0034] Step 200, determining whether the vector to be calculated by the current linear calculation unit is a normalized query vector or a normalized key vector;

[0035] If the vector to be calculated by the current linear calculation unit is not a normalized query vector or a normalized key vector, the following operations are performed:

[0036] Step 211, control the read buffer module to input the target column of the vector to be calculated and the matrix to be calculated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, and the target accumulation result is obtained through accumulation operation and output to the addition unit, and the addition unit directly outputs the target accumulation result;

[0037] Step 212, repeating step 211 to obtain multiple target accumulation results until the target column is the last column of the matrix to be calculated;

[0038] Step 213, concatenate multiple target accumulation results to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of columns of the matrix to be operated;

[0039] If the input vector of the current linear computation unit is a normalized query vector or a normalized key vector, the following operations are performed:

[0040] Step 221, control the read buffer module to split the vector to be operated into a first number of sub-row vectors, where the dimension of the sub-row vectors is equal to the number of rows of the input matrix of the neural network; and control the read buffer module to split each column vector of the matrix to be operated into a first number of sub-column vectors, where the dimension of the sub-column vectors is equal to the number of rows of the input matrix of the neural network;

[0041] Step 222, control the read buffer module to input the target sub-row vector of the vector to be calculated and the corresponding sub-column vector of the target column in the matrix to be calculated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, and the target cumulative result obtained by the accumulation operation is output to the adding unit; the adding unit adds the target cumulative result and the target accumulated result obtained by the last execution of step 222, and outputs the target value;

[0042] Step 223, repeatedly executing steps 221 to 222 until the target sub-row vector is the last sub-vector in the vector to be calculated;

[0043] Step 224, repeatedly executing steps 221 to 223 to obtain multiple target values, until the target column is the last column in the matrix to be calculated;

[0044] Step 225, concatenate multiple target values ​​to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be operated.

[0045] Optionally, when the linear calculation submodule performs the vector-matrix element-by-element product and multiplication mixed operation, the control module is further configured to perform the following operations:

[0046] Step 301, control the read buffer module to input the first vector to be operated and the target column of the matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the write buffer module and the element-by-element product operation unit respectively;

[0047] Step 302, control the read buffer module to input the second vector to be operated to the element-by-element product operation unit; the element-by-element product operation unit performs element-by-element product operation on the second vector to be operated and the intermediate operation vector, and outputs the corresponding result to the inner product unit, obtains the target value through accumulation operation, and outputs it to the addition unit, and the addition unit directly outputs the target value;

[0048] Step 303: repeating steps 301 and 302 to obtain multiple target values ​​and multiple intermediate operation vectors until the target column is the last column of the matrix to be operated;

[0049] Step 304: concatenate multiple target values ​​to a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be operated; and concatenate multiple intermediate operation vectors to obtain an intermediate operation matrix; wherein the size of the intermediate operation matrix is ​​consistent with the size of the matrix to be operated.

[0050] Optionally, when the linear calculation submodule performs vector-matrix element-by-element product, linear layer, and additive mixing operations, the control module is further used to perform the following operations:

[0051] Step 401, control the read buffer module to input the first vector to be operated and the target column of the first matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the element-by-element product operation unit;

[0052] Step 402, control the target row of the weight matrix input by the read buffer module to the element-by-element product operation unit; the element-by-element product operation unit performs element-by-element product operation on the target row and the intermediate operation vector, and outputs the corresponding result to the inner product unit, obtains the target accumulation result through accumulation operation and outputs it to the addition unit; at the same time, control the read buffer module input offset vector corresponding to the element value corresponding to the second matrix to be operated to the addition unit;

[0053] Step 403, the adding unit adds the target accumulation result, the element value corresponding to the bias vector, and the element value corresponding to the second matrix to be operated to obtain the target value;

[0054] Step 404, repeatedly executing steps 402 to 403 to obtain multiple target values ​​until the last row in the target behavior weight matrix of the weight matrix;

[0055] Step 405, repeatedly executing steps 401 to 404 to obtain multiple target values, until the target column of the first matrix to be calculated is the last column of the first matrix to be calculated;

[0056] Step 406, concatenate the multiple target values ​​to obtain a result matrix; wherein the size of the result matrix is ​​consistent with the size of the first matrix to be calculated.

[0057] The second aspect of the present application discloses a calculation method for a hardware architecture applied to a Fastformer neural network. The calculation method is applied to a control module, and the control module is located in the hardware architecture applied to a Fastformer neural network according to any one of claims 1 to 10. The calculation method includes:

[0058] Step 001, control the data storage module, the read buffer module and the linear calculation submodule to perform a linear layer query operation on the input matrix, obtain a query matrix and output it to the read buffer module and the write buffer module respectively; the write buffer module transmits the query matrix to the data storage module;

[0059] Step 002: Control the data storage module, the read buffer module, and the linear calculation submodule to perform vector-matrix multiplication on the query matrix, obtain an intermediate query vector, and transmit it to the nonlinear calculation submodule;

[0060] Step 003: Control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate query vector, obtain a normalized query vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention key linear layer operation on the input matrix, obtain a key matrix and output it to the write buffer module; the write buffer module transmits the key matrix to the data storage module;

[0061] Step 004: Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication on the normalized query vector and the query matrix to obtain a global query vector, and output it to the read buffer module;

[0062] Step 005: Control the read buffer module, the linear calculation submodule and the data storage module to perform element-by-element product on the global query vector and the key matrix to obtain an intermediate key matrix, perform vector-matrix multiplication on the intermediate key matrix to obtain an intermediate key vector and output it to the nonlinear calculation submodule, and simultaneously control the linear calculation submodule to output the intermediate key matrix to the write buffer module; the write buffer module stores the intermediate key matrix to the data storage module;

[0063] Step 006: Control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate key vector to obtain a normalized key vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention value linear layer operation on the input matrix to obtain a value matrix and output it to the write buffer module; the write buffer module transmits the value matrix to the data storage module;

[0064] Step 007: Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication on the normalized key vector and the key matrix to obtain a global key vector and output it to the read buffer module;

[0065] Step 008: Control the linear calculation submodule, the read buffer module, and the data storage module to perform element-by-element product operations on the value matrix and the global key vector, and perform linear layer operations on the operation results to obtain a global value matrix; perform addition operations on the global value matrix and the query matrix to obtain the output matrix of the neural network and output it to the write buffer module; the write buffer module transfers the output matrix to the data storage module.

[0066] Optionally, when the control module controls the linear calculation submodule to perform a linear layer operation, the calculation method further includes:

[0067] Step 101, control the read buffer module to input the target column of the matrix to be calculated and the target row of the weight matrix to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, and the target accumulation result is obtained through accumulation operation and output to the addition unit, and at the same time control the read buffer module to input the corresponding element value of the bias matrix to the addition unit; the addition unit adds the target accumulation result and the corresponding element value in the bias matrix to obtain the target value;

[0068] Step 102, repeatedly executing step 101 to obtain multiple target values, until the target column is the last column in the matrix to be calculated;

[0069] Step 103, concatenate multiple target values ​​to obtain a result matrix; wherein the size of the result matrix is ​​consistent with the size of the matrix to be calculated.

[0070] Optionally, when the control module controls the linear calculation submodule to perform a vector-matrix multiplication operation, the calculation method further includes:

[0071] Step 200, determining whether the vector to be calculated by the current linear calculation unit is a normalized query vector or a normalized key vector;

[0072] If the vector to be calculated by the current linear calculation unit is not a normalized query vector or a normalized key vector, the following operations are performed:

[0073] Step 211, control the read buffer module to input the target column of the vector to be calculated and the matrix to be calculated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, and the target accumulation result is obtained through accumulation operation and output to the addition unit, and the addition unit directly outputs the target accumulation result;

[0074] Step 212, repeating step 211 to obtain multiple target accumulation results until the target column is the last column of the matrix to be calculated;

[0075] Step 213, concatenate multiple target accumulation results to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of columns of the matrix to be operated;

[0076] If the input vector of the current linear computation unit is a normalized query vector or a normalized key vector, the following operations are performed:

[0077] Step 221, control the read buffer module to split the vector to be operated into a first number of sub-row vectors, where the dimension of the sub-row vectors is equal to the number of rows of the input matrix of the neural network; and control the read buffer module to split each column vector of the matrix to be operated into a first number of sub-column vectors, where the dimension of the sub-column vectors is equal to the number of rows of the input matrix of the neural network;

[0078] Step 222, control the read buffer module to input the target sub-row vector of the vector to be calculated and the corresponding sub-column vector of the target column in the matrix to be calculated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, and the target cumulative result obtained by the accumulation operation is output to the adding unit; the adding unit adds the target cumulative result and the target accumulated result obtained by the last execution of step 222, and outputs the target value;

[0079] Step 223, repeatedly executing steps 221 to 222 until the target sub-row vector is the last sub-vector in the vector to be calculated;

[0080] Step 224, repeatedly executing steps 221 to 223 to obtain multiple target values, until the target column is the last column in the matrix to be calculated;

[0081] Step 225, concatenate multiple target values ​​to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be operated.

[0082] Optionally, when the control module controls the linear calculation submodule to perform a vector-matrix element-by-element active multiplication mixed operation, the calculation method further includes:

[0083] Step 301, control the read buffer module to input the first vector to be operated and the target column of the matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the write buffer module and the element-by-element product operation unit respectively;

[0084] Step 302, control the read buffer module to input the second vector to be operated to the element-by-element product operation unit; the element-by-element product operation unit performs element-by-element product operation on the second vector to be operated and the intermediate operation vector, and outputs the corresponding result to the inner product unit, obtains the target value through accumulation operation, and outputs it to the addition unit, and the addition unit directly outputs the target value;

[0085] Step 303: repeating steps 301 and 302 to obtain multiple target values ​​and multiple intermediate operation vectors until the target column is the last column of the matrix to be operated;

[0086] Step 304: concatenate multiple target values ​​to a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be operated; and concatenate multiple intermediate operation vectors to obtain an intermediate operation matrix; wherein the size of the intermediate operation matrix is ​​consistent with the size of the matrix to be operated.

[0087] Optionally, when the control module controls the linear calculation submodule to perform vector-matrix element-by-element product, linear layer, and additive mixed operations, the calculation method further includes:

[0088] Step 401, control the read buffer module to input the first vector to be operated and the target column of the first matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the element-by-element product operation unit;

[0089] Step 402, control the target row of the weight matrix input by the read buffer module to the element-by-element product operation unit; the element-by-element product operation unit performs element-by-element product operation on the target row and the intermediate operation vector, and outputs the corresponding result to the inner product unit, obtains the target accumulation result through accumulation operation and outputs it to the addition unit; at the same time, control the read buffer module input offset vector corresponding to the element value corresponding to the second matrix to be operated to the addition unit;

[0090] Step 403, the adding unit adds the target accumulation result, the element value corresponding to the bias vector, and the element value corresponding to the second matrix to be operated to obtain the target value;

[0091] Step 404, repeatedly executing steps 402 to 403 to obtain multiple target values ​​until the last row in the target behavior weight matrix of the weight matrix;

[0092] Step 405, repeatedly executing steps 401 to 404 to obtain multiple target values, until the target column of the first matrix to be calculated is the last column of the first matrix to be calculated;

[0093] Step 406, concatenate the multiple target values ​​to obtain a result matrix; wherein the size of the result matrix is ​​consistent with the size of the first matrix to be calculated.

[0094] The present application discloses a hardware architecture and a calculation method thereof applied to a Fastformer neural network through the above aspects. The hardware architecture includes a data storage module, a read buffer module, a write buffer module, a calculation module and a control module; wherein the control module is used to control the hardware architecture to perform calculation operations, and the calculation module includes a linear calculation submodule, a nonlinear calculation submodule and an intermediate storage submodule; the linear calculation module includes a plurality of vector calculation units for performing linear operations including vector matrix multiplication, matrix addition and element-by-element product, and the nonlinear submodule is used to perform normalized exponential function operations. The hardware architecture and its calculation method can effectively improve the calculation speed and efficiency of the Fastformer neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 This is a schematic diagram of the model architecture of the Fastformer neural network;

[0096] Figure 2 A schematic diagram of the model architecture that re-describes the Fastformer neural network model based on the six basic functions summarized in this application;

[0097] Figure 3 A schematic diagram of the structure of a hardware architecture applied to a Fastformer neural network disclosed in an embodiment of the present application;

[0098] Figure 4 A schematic diagram of the hardware architecture of a data storage module used in a hardware architecture of a Fastformer neural network disclosed in an embodiment of the present application;

[0099] Figure 5 A schematic diagram of the hardware architecture of a vector computing unit in a hardware architecture applied to a Fastformer neural network disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0100] To facilitate the description of the technical solution of the application, some concepts involved in the application are first described below. In the application, the operation of multiplying corresponding elements in a vector or matrix one by one is called an element-by-element product operation. As an example, the element-by-element product operation process of two 2-dimensional row vectors can be expressed by the following formula:

[0101] (A1 A2)*(B1 B2)=(A1×B1 A2×B2)

[0102] First, the reasoning process of the Fastformer neural network is further explained. Figure 1 Figure 1 is a schematic diagram of the Fastformer neural network model architecture. Figure 1 As shown in the figure, the input of the Fastformer model is a matrix E with d rows and N columns, which can be regarded as a d-dimensional column vector e1, e2, ..., e N-1 , e N The output of the entire model is also a matrix, named Output, and the shape of the output matrix is ​​the same as E. The reasoning calculation process of the Fsatformer neural network is as follows:

[0103] Step 01: The input matrix E is first processed by the query linear layer (Query Transformation in the figure), the key linear layer (Key Transformation in the figure), and the value linear layer (Value Transformation in the figure) to obtain the query matrix Q, key matrix K, and value matrix V of the same size as the input matrix E. The calculation of the linear layer here can be regarded as a d-row and d-column weight matrix (Weight) multiplied by the d-row and N-column input matrix. The intermediate result is then added to each column with a bias column vector (Bias) of dimension d, and finally a d-row and N-column query matrix Q, key matrix K, and value matrix V are obtained. Among them, the weight matrix Weight and the bias column vector Bias are intermediate storage parameters of the linear layer. Combined with Figure 1 , the process of three linear layers can be expressed by the following formula:

[0104]

[0105]

[0106]

[0107] Step 02: Based on the trained intermediate parameter vector (W q is a d-dimensional column vector, is a d-dimensional row vector), perform normalized exponential calculation on the query matrix Q, and obtain a global query vector denoted as q, where the dimension of the global query vector q is d. The calculation process can be expressed by the following formula:

[0108]

[0109] q=(α1,α2,……,α N )×Q T

[0110] Step 03: The global query vector q is sequentially compared with the N column vectors k1, k2, ..., k in the key matrix K. N Multiply the corresponding elements one by one to obtain N column vectors p1, p2, ..., p N , combined into the intermediate key matrix P, the calculation process can be expressed by the following formula:

[0111] P=(p1,p2,......,p N )=q T *(k1,k2,......,k N )

[0112] In this application, the operation of multiplying the corresponding elements in a vector one by one is called an element-by-element product operation.

[0113] Step 04: Based on the trained intermediate parameter vector (With W q similar, is a d-dimensional column vector), perform normalized exponential calculation on the intermediate key matrix P, and obtain a global key vector denoted as k; where the dimension of the global key vector k is also d. The calculation process can be expressed by the following formula:

[0114]

[0115] k=(β1,β2,...,β N )×P T

[0116] Step 05: The global query vector k is sequentially compared with all N column vectors v1, v2, ..., v in the value matrix V. N Multiply the corresponding elements one by one to obtain N column vectors u1, u2, ..., u N , combined into the intermediate value matrix U, the calculation process can be expressed by the following formula:

[0117] U=(u1,u2,......,u N ) = k T *(v1, v2, ..., v N )

[0118] Step 06: The intermediate value matrix U is then processed by a pre-trained linear layer (represented as Transformation in the figure) to obtain the global value matrix R (r1, r2, ..., u N ), the calculation process can be expressed by the following formula:

[0119] R=Weight R ×U+(Bias R ,......,Bias R )

[0120] Step 07: Output matrix Output = Q + R.

[0121] The above is the specific calculation process of the Fastformer neural network. In order to improve the operation speed and efficiency of the Fastformer neural network, this application discloses a hardware architecture applied to the Fastformer neural network through the following embodiments.

[0122] First, this application summarizes the operations involved in the Fastformer network into six basic functions, where the input and output of each function are expressed in matrix form, such as a matrix M with d rows and N columns represented as M (d，N) , the d-dimensional row vector is represented by M (1，d) , the d-dimensional column vector is represented by M (d，1) .

[0123] The first function F1 is used to represent the calculation process of each linear processing layer. The general calculation formula is:

[0124]

[0125] The input and output of the function can be expressed as: (d，N) =F1(W (d，d) , X (d，N) , B (d，1) ).

[0126] The second function F2 is used to represent the multiplication operation of a d-dimensional row vector and d rows and N columns, that is, it is used to process the d-dimensional intermediate parameter vector to obtain the intermediate operation result for normalized exponential processing. The general operation formula is: Y = X1 × X2, which can be expressed as follows according to the input and output of the function:

[0127] The third function F3 is used to represent the multiplication operation of an N-dimensional row vector and an N-row d-column matrix. It is used to process the N-dimensional normalized query vector or normalized key vector to obtain the corresponding global query matrix and global key matrix. The general operation formula is: Y = X1 × X2. According to the input and output of the function, it can be expressed as

[0128] The fourth function F4 is used to represent the element-by-element product operation of a d-dimensional column vector and a d-row N-column matrix in sequence. The general operation formula is:

[0129]

[0130] According to the input and output of the function, it can be expressed as

[0131] The fifth function F5 is used to represent the normalized exponential processing of N-dimensional row vectors. The general calculation formula is: in According to the function input and output, it can be expressed as Y (1，N) =F5(X (1，N) ).

[0132] The sixth function F6 is used to represent the addition operation of two d-row and N-column matrices. The general operation formula is: Y = R + Q. According to the function input and output, it can be expressed as Y (d，N) =F6(R (d，N) , Q (d，N) ).

[0133] Among them, functions F1, F2, F3, F4 and F6 are all linear calculations, specifically involving vector matrix multiplication, addition and element-by-element product operations; function F5 is a nonlinear calculation, specifically involving the Softmax function. Based on the above six basic functions, this application re-describes the Fastformer neural network architecture, such as Figure 2 shown.

[0134] Based on the redescription of the Fastformer neural network architecture, this application discloses a hardware architecture applied to the Fastformer neural network, such as Figure 3 The hardware architecture includes a data storage module, a read buffer module, a write buffer module, a calculation module and a control module.

[0135] The data storage module is used to receive external input data to be calculated and store them in parallel so that they can be called in subsequent calculation processes. The data storage module is also used to store the pre-trained parameters involved in the Fastformer neural network calculation process, and temporarily store the intermediate data generated in the calculation process so that they can be called in subsequent calculation processes. It also outputs the calculation results of the entire Fastformer neural network.

[0136] In some embodiments, the data storage module includes an input matrix storage unit, a weight matrix storage unit, a bias vector storage unit, a parameter vector storage unit, a temporary matrix storage unit, and an output matrix storage unit. Figure 4 The input end of the input matrix storage unit is the input end of the hardware architecture, and the output end of the input matrix storage unit is connected to the input end of the read buffer module, which is used to receive and store an input matrix with d rows and N columns, so that the input matrix can be called multiple times through the read buffer module in the subsequent calculation process.

[0137] The output ends of the weight matrix storage unit, the bias vector storage unit, the parameter vector storage unit, and the temporary matrix storage unit are connected to the input end of the read buffer module; wherein the weight matrix storage unit is used to store the pre-trained query weight matrix Weight with d rows and columns. Q , key weight matrix Weight K , value weight matrix Weight y And the transformation weight matrix Weight R; The bias vector storage unit is used to store the pre-trained query bias vector Bias with d rows o , key bias vector Bias K , value bias vector Bias V and transformation bias vector Bias R , where the bias vectors are all column vectors. The parameter vector unit is used to store the first parameter vector W with d rows that has been pre-trained. q and the second parameter vector W k , where the parameter vectors are all column vectors.

[0138] The input end of the temporary matrix storage unit is connected to the output end of the write buffer module, and is used to store the intermediate data generated in the calculation process so as to be called in the subsequent calculation process.

[0139] The input end of the output matrix storage unit is connected to the output end of the write buffer module, and the output end of the output matrix storage unit is the output end of the hardware architecture, which is used to output the calculation result of the entire Fastformer neural network.

[0140] The input end of the read buffer module is also connected to the output end of the calculation module, and the output end of the read buffer module is connected to the input end of the calculation module. The read buffer module is used to call the required data from the data storage module according to the specific calculation process, output it according to the format requirements of the calculation module, and receive the data output by the calculation module for the next step.

[0141] The input end of the write buffer module is connected to the output end of the calculation module. When the intermediate calculation result output by the calculation module is needed in the subsequent calculation process, the write buffer module receives such calculation result and transmits it to the temporary matrix storage unit in the data storage module for temporary storage; when the settlement result output by the calculation module is the calculation result of the entire Fastformer, the write buffer module receives such calculation result and transmits it to the output matrix storage unit in the data storage module for storage and output.

[0142] The calculation module includes a linear calculation submodule, a nonlinear calculation submodule and an intermediate storage submodule. The linear calculation submodule is used to perform linear operations including vector matrix multiplication, addition, and element-by-element product. The input end of the linear calculation submodule is respectively connected to the output end of the read buffer and the output end of the intermediate storage submodule, and the output end is respectively connected to the input end of the read buffer, the input end of the write buffer module and the input end of the nonlinear calculation submodule. The nonlinear submodule is used to perform a normalized exponential function (Softmax function) operation, and its output end is connected to the input end of the intermediate storage submodule. The intermediate storage submodule is used to receive the calculation results of the nonlinear submodule and transmit them to the linear calculation submodule.

[0143] The control module is respectively connected to the data storage module, the read buffer module, the write buffer module, and the linear calculation submodule and the nonlinear calculation submodule in the calculation module. The control module is used to perform the following operations:

[0144] Step 001, control the data storage module, the read buffer module and the linear calculation submodule to perform a linear layer query operation on the input matrix, obtain a query matrix and output it to the read buffer module and the write buffer module respectively; wherein the write buffer module transmits the query matrix to the temporary matrix storage unit in the data storage module.

[0145] Step 001 is equivalent to the linear calculation submodule calculating F1 (Weight Q ,E,Bias Q ), the calculation result query matrix Q is returned to the read buffer module (as the input of step 002) and the write buffer module (and then returned to the data storage area as the input of steps 004 and 008).

[0146] Step 002, controlling the data storage module, the read buffer module, and the linear calculation submodule to perform a vector-matrix multiplication operation on the query matrix Q, obtaining an intermediate query vector and transmitting it to the nonlinear calculation submodule;

[0147] Step 002 is equivalent to the linear calculation submodule calculating separately And output the calculation result intermediate query vector to the nonlinear calculation submodule.

[0148] Step 003, control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate query vector, obtain a normalized query vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention key linear layer operation on the input matrix, obtain a key matrix and output it to the write buffer module; the write buffer module transfers the key matrix to the temporary matrix storage unit.

[0149] Step 003 is equivalent to the nonlinear calculation submodule calculating the result (α1, α2, ..., α N ) and temporarily stored in the intermediate storage submodule of the calculation module as the input of step 004. At the same time, the linear calculation submodule calculates F1 (Weight K ,E,Bias K ), the calculation result key matrix K is first output to the write buffer module, and then returned to the data storage module, ready to be used as the input of step 005.

[0150] Step 004, control the linear calculation submodule, intermediate storage submodule, data storage module, and read buffer module to perform vector-matrix multiplication operation on the normalized query vector and the query matrix to obtain a global query vector, and output it to the read buffer module.

[0151] Step 004 is equivalent to the linear operation module calculating F3((α1, α2, ..., α N ), Q T ), returns the calculated global query vector q to the read buffer, ready to be used as input for step 005.

[0152] Step 005, control the read buffer module, linear calculation submodule and data storage module to perform element-by-element product of the global query vector and the key matrix to obtain an intermediate key matrix, perform vector-matrix multiplication operation on the intermediate key matrix to obtain an intermediate key vector and output it to the nonlinear calculation submodule, and simultaneously control the linear calculation submodule to output the intermediate key matrix to the write buffer module; the write buffer module stores the intermediate key matrix to a temporary matrix storage unit.

[0153] Step 005 is equivalent to the mixed calculation F4(q T , K) and The final result of the hybrid calculation is output to the nonlinear calculation submodule, and the intermediate key matrix P is output to the write buffer module as the input of step 007.

[0154] Step 006, control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate key vector, obtain a normalized key vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention value linear layer operation on the input matrix, obtain a value matrix and output it to the write buffer module; the write buffer module transfers the value matrix to the temporary matrix storage unit in the data storage module.

[0155] Step 006 is equivalent to the nonlinear calculation submodule function module calculating the result (β1, β2, ..., β N ) and temporarily stored in the intermediate storage submodule of the calculation module as the input of step 007. At the same time, the linear calculation submodule calculates F1 (Weight V ,E,Bias V ), the calculation result value matrix V is first output to the write buffer module, and then returned to the data storage module, ready to be used as the input of step 008.

[0156] Step 007, control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication operations on the normalized key vector and the key matrix, obtain a global key vector and output it to the read buffer module.

[0157] Step 007 is equivalent to the linear operation module calculating F3((β1, β2, ..., β N ), PT), and returns the calculated global key vector k to the read buffer module, ready to be used as the input of step 008

[0158] Step 008, control the linear calculation submodule, the read buffer module, and the data storage module to perform element-by-element product operations on the value matrix and the global key vector, and perform linear layer operations on the operation results to obtain a global value matrix; perform addition operations on the global value matrix and the query matrix to obtain the output matrix of the neural network and output it to the write buffer module; the write buffer module transfers the output matrix to the output matrix storage unit in the data storage module.

[0159] Step 008 is equivalent to the mixed calculation F4(k T , V), F1(Weigh t R ,U,Bias R ), F6(Q, R), get the final result matrix Output, return it to the write buffer module, and output it to the data storage module as the output matrix of the entire Fastformer neural network.

[0160] The operations performed in the above control module not only fully consider the maximization of the hardware utilization of the computing module (the linear computing submodule is not idle in each step), but also try to reduce the amount of intermediate data generated, thereby reducing storage overhead and further improving the efficiency of the entire hardware architecture.

[0161] In some embodiments, the linear calculation submodule includes a plurality of vector calculation units. The vector calculation unit is used to implement vector element-by-element product operations, multiplication operations, and addition operations. By performing parallel calculations on a plurality of vector units, element-by-element product, multiplication, and addition operations between a vector and a matrix, or between matrices, can be implemented. It is also possible to implement element-by-element product, multiplication, and addition operations between a vector and a matrix, or between matrices, by performing multiple calculations on a single vector unit.

[0162] See also Figure 5 The vector operation unit of the hardware architecture applied to the Fastformer neural network disclosed in the embodiment of the present application includes an element-by-element product operation unit, an inner product unit, a temporary vector storage unit and an addition unit. The input vectors of the vector calculation unit are respectively recorded as input vectors x(x1, x2, ..., x d ) and the input vector y(y1, y2, ..., y d ).

[0163] The element-by-element product operation unit includes a plurality of multipliers, wherein the number of multipliers is equal to the number of rows d of any input matrix, that is, the hidden layer dimension d of the neural network; the input end of the element-by-element product operation unit is respectively connected to the output end of the read buffer module and the output end of the temporary vector storage unit, and the output end thereof is respectively connected to the inner product unit and the temporary vector storage unit. The element-by-element product operation unit is used to perform element-by-element multiplication operation on two input vectors of length d.

[0164] The input end of the inner product unit is connected to the output end of the element-by-element product unit, and the output end is connected to the addition unit. The inner product unit is used to accumulate the element-by-element product operation results of two vectors of length d, and the result obtained is the inner product result of the two vectors of length d.

[0165] The input end of the temporary vector storage unit is connected to the output end of the element-by-element product operation unit, and the output end thereof is respectively connected to the input end of the element-by-element product operation unit and the input end of the write buffer module. The temporary vector storage unit is used to store the element-by-element product operation result of two vectors of length d in the form of a vector.

[0166] The input end of the adding unit is respectively connected to the output end of the inner product unit and the output end of the read buffer module, and the output end thereof is connected to the input end of the read buffer module and the input end of the write buffer module.

[0167] When the linear layer operation is performed in the linear calculation submodule, that is, the separate operation of the F1 function is performed, the control module is used to perform the following operations:

[0168] Step 101: Control the read buffer module to input the matrix to be operated X (d，N) The target column and the corresponding weight matrix W (d，d) The target row of the element-by-element product operation unit is output to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, and the target accumulation result is output to the addition unit after the accumulation operation, and at the same time controls the read buffer module to input the bias vector B (d，1) The corresponding element value is sent to the addition unit; the addition unit adds the target accumulation result and the bias vector B (d，1) Add the corresponding element values ​​in to get the target value.

[0169] It should be noted that, in this embodiment, the target columns are the first column, the second column, ..., the Nth column in the matrix X to be calculated, and the target rows are the first row, the second row, ..., the dth row in the corresponding weight matrix W.

[0170] Step 102, repeating step 101 to obtain multiple target values, until the target column is the last column in the matrix to be calculated and the target behavior corresponds to the last row in the weight matrix.

[0171] Step 103, concatenate the multiple target values ​​to obtain a result matrix; wherein the size of the result matrix is ​​consistent with the size of the matrix to be calculated.

[0172] In this embodiment, the first execution of step 101 obtains the value of the first row and first column element in the result matrix, and the second execution of step 101 obtains the value of the first row and second column element in the result matrix. After executing step 101 d·N times, the complete output result of the F1 function is obtained. In some other embodiments, the linear submodule configures multiple vector operation units to treat the operation matrix X (d，N) Each column vector of is input and calculated in parallel, and the d·N execution steps 101 are distributed to multiple vector calculation units to improve the calculation efficiency.

[0173] When the vector-matrix multiplication operation is performed in the linear calculation submodule, that is, the single operation of function F2 or the single operation of function F3 is performed, the control module is used to perform the following operations:

[0174] Step 200 , determining whether the vector to be calculated by the current linear calculation unit is a normalized query vector or a normalized key vector.

[0175] Because function F2 is used to process the d-dimensional intermediate parameter vector to obtain the intermediate operation result for normalized exponential processing, its vector to be operated is a pre-trained d-dimensional parameter vector, which can be directly calculated using the vector operation unit; while function F3 is used to process the N-dimensional normalized query vector or normalized key vector to obtain the corresponding global query matrix and global key matrix, and its vector to be operated is an N-dimensional vector, which needs to be split into multiple d-dimensional vectors and then calculated using the vector operation unit. Therefore, when performing vector-matrix multiplication in the linear submodule, it is necessary to first determine the type of the input vector to be operated.

[0176] If the vector to be calculated in the current linear calculation unit is not a normalized query vector or a normalized key vector, that is, the function F2 is executed, the following operations are performed:

[0177] Step 211, control the read buffer module to input the vector to be operated and the matrix to be calculated The target column is sent to the element-by-element product unit; the element-by-element product unit outputs the corresponding element-by-element product result to the inner product unit, and the target accumulation result is obtained through accumulation operation and output to the addition unit, and the addition unit directly outputs the target accumulation result.

[0178] It should be noted that in step 211, the target columns of the matrix to be operated are the first column, the second column, ..., the Nth column in sequence.

[0179] Step 212, repeating step 211 to obtain multiple target accumulation results, until the target column is the last column of the matrix to be calculated.

[0180] Step 213, concatenate multiple target accumulation results to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of columns of the matrix to be calculated.

[0181] In this embodiment, the first execution of step 211 obtains the value of the first element in the result column vector, and the second execution of step 211 obtains the value of the second element in the result column vector. After N executions of step 211, the complete output result of function F2 is obtained. In some other embodiments, the linear submodule configures multiple vector operation units to treat the operation matrix Each column vector of is input and calculated in parallel, and the N execution steps 211 are distributed to multiple vector calculation units to improve the calculation efficiency.

[0182] If the input vector of the current linear calculation unit is a normalized query vector or a normalized key vector, that is, the function F3 is executed, the following operations are performed:

[0183] Step 221, control the read buffer module to read the vector to be operated split into a first number of sub-row vectors, the dimension of which is equal to the number of rows d of the input matrix of the neural network; and at the same time control the read buffer module to convert the matrix to be operated Each column vector of is split into a first number of sub-column vectors, and the dimension of the sub-column vectors is equal to the number of rows d of the input matrix of the neural network.

[0184] Step 222: Control the read buffer module to input the vector to be operated The target sub-row vector and the matrix to be operated The corresponding sub-column vector of the target column in the inner product is sent to the element-by-element product unit; the element-by-element product unit outputs the corresponding element-by-element product result to the inner product unit, and the target accumulation result is obtained through accumulation operation and output to the addition unit; the addition unit adds the target accumulation result and the target cumulative result obtained by the last execution of step 222, and outputs the target value.

[0185] Step 223, repeating step 222 until the target sub-row vector is the last sub-vector in the vector to be calculated;

[0186] As an example, the vector to be operated Split into T sub-row vectors X 11 , X 12 ,......,X 1T, the matrix to be operated Each column vector in is split into T sub-column vectors, where the first column vector is split into T sub-column vectors X 211 , X 212 ,......,X 21T , the d-th sub-column vector is split into T sub-column vectors X 2d1 , X 2d2 ,......,X 2dT .

[0187] In this embodiment, the result of executing step 222 for the first time is the first sub-row vector X 11 and the first subcolumn vector X in the first column of the matrix to be operated 211 The inner product result of is output as the partial sum of the first element value in the result column vector to the write buffer module for temporary storage; the result obtained by executing step 222 for the second time is the second sub-row vector X 12 and the second subcolumn vector X in the first column of the matrix to be operated 212 The inner product result of the first execution of step 222 is added to the partial sum, and the new partial sum is temporarily stored by writing into the buffer module; until the T-th execution of step 222, the T-th sub-row vector X is obtained. 1T and the Tth subcolumn vector X in the first column of the matrix to be operated 21T The inner product result plus the partial sum accumulated previously is used as the vector to be calculated and the matrix to be calculated The value corresponding to the first element in the column vector of the multiplication result.

[0188] Step 224, repeatedly executing steps 222 to 223 to obtain a plurality of the target values, until the target column is the last column in the matrix to be calculated.

[0189] In this embodiment, by repeatedly executing step 222 to step 223, the target values ​​obtained in sequence are the values ​​corresponding to the first, second, ... dth elements in the result column vector.

[0190] Step 225, concatenate the multiple target values ​​to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be calculated.

[0191] According to the above steps 221-225, in this embodiment, it is necessary to perform d·T operations to obtain the operation result of function F3. In some other embodiments, the linear submodule configures multiple vector operation units to treat the operation matrix Each column vector of is input and calculated in parallel, and d·T operations are distributed to multiple vector calculation units to improve the calculation efficiency.

[0192] The Fastformer neural network also involves the mixed operation of vector-matrix element-by-element product and multiplication, that is, the mixed operation of function F4 and function F2. The calculation result of function F4 is used as the matrix to be operated in function F2.

[0193] When the linear calculation submodule performs the mixed operation of vector-matrix element-by-element product and multiplication, that is, the mixed operation of function F4 and function F2 is performed, the control module is also used to perform the following operations:

[0194] It should be noted that in this operation, the input of function F4 is referred to as the first vector to be operated. and the matrix to be calculated The input of function F2 is called the second vector to be operated

[0195] Step 301: Control the read buffer module to input the first vector to be operated and the matrix to be calculated The target column is sent to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the write buffer module and the element-by-element product operation unit respectively.

[0196] Step 302: Control the read buffer module to input the second vector to be operated To the element-by-element product unit; the element-by-element product unit performs element-by-element product operation on the corresponding rows in the intermediate operation vector and the second vector to be operated, and outputs the corresponding result to the inner product unit, obtains the target value through accumulation operation and outputs it to the addition unit, and the addition unit directly outputs the target value.

[0197] Step 303, repeating steps 301 to 302 to obtain multiple target values ​​and multiple intermediate operation vectors, until the target column is the last column of the matrix to be operated.

[0198] In this embodiment, step 301 and step 302 are executed for the first time to obtain the value corresponding to the first element in the result column vector and to obtain the vector corresponding to the first column in the intermediate operation matrix.

[0199] Step 304: concatenate multiple target values ​​to a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be calculated; and concatenate multiple intermediate calculation vectors to obtain an intermediate calculation matrix having a size consistent with the size of the matrix to be calculated.

[0200] In this embodiment, through N operations, the complete output result of the mixed operation of function F4 and function F2 and the complete result of the operation of function F4 are obtained. In some other embodiments, the linear submodule is configured with multiple vector operation units to treat the operation matrix Each column vector of is input and calculated in parallel, N operations are distributed to multiple vector calculation units to improve the calculation efficiency.

[0201] The Fastformer neural network also involves mixed operations of vector-matrix element-by-element product, linear layer, and matrix addition, that is, mixed operations of function F4, function F1, and function F6. The calculation result of function F4 is used as the matrix to be calculated in function F1, and the calculation result of function F1 is used as the matrix to be calculated in function F6.

[0202] When the linear calculation submodule performs the vector-matrix element-by-element product, the linear layer, and the additive mixing operation, the control module is also used to perform the following operations:

[0203] It should be noted that in this operation, the input of function F4 is referred to as the first vector to be operated. and the matrix to be calculated The input of function F6 is called the second matrix to be operated

[0204] Step 401: Control the read buffer module to input the first vector to be operated and the first matrix to be calculated The target column is sent to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the element-by-element product operation unit.

[0205] Among them, the first matrix to be calculated The target columns are the first column, the second column, ..., the Nth column of the first matrix to be operated.

[0206] Step 402, control the target row of the weight matrix input by the buffer module to the element-by-element product operation unit; the element-by-element product operation unit performs element-by-element product operation on the target row and the intermediate operation vector, and outputs the corresponding result to the inner product unit, obtains the target accumulation result through accumulation operation and outputs it to the addition unit; at the same time, control the element value corresponding to the input bias vector of the buffer module and the second matrix to be operated The corresponding element value goes to the addition unit.

[0207] Among them, the target rows of the weight matrix are the first row, the second row, ..., the dth row of the weight matrix.

[0208] Step 403: The adding unit adds the target accumulation result, the element value corresponding to the bias matrix, and the element value corresponding to the second matrix to be calculated to obtain a target value.

[0209] Step 404, repeating steps 402 to 403 to obtain multiple target values ​​until the target behavior of the weight matrix reaches the last row in the weight matrix.

[0210] In this embodiment, by executing step 402 to step 402 N times, the values ​​corresponding to the N elements of the target column vector in the result matrix are obtained.

[0211] Step 405, repeating steps 401 to 404 to obtain multiple target values ​​until the target column is the first matrix to be calculated The last column of .

[0212] Step 406, concatenate multiple target values ​​to obtain a result matrix; wherein the size of the result matrix is ​​consistent with the size of the first matrix to be calculated.

[0213] In this embodiment, by executing steps 401 to 405 d times, the values ​​corresponding to all elements in the result matrix are obtained. In general, in this embodiment, the complete result of the mixed operation of function F4, function F1 and function F6 is obtained through the calculation of dN vector calculation units. These dN calculations can be processed in parallel by multiple vector calculation units.

[0214] It can be seen from the operation modes supported by the above-mentioned vector computing units that the hardware architecture applied to the Fastformer neural network disclosed in this embodiment determines the parallelism of the entire hardware architecture by flexibly configuring the number of vector computing units to adapt to hardware computing platforms with different resources. If the currently used hardware platform has fewer resources and the upper limit of the communication bandwidth between the computing module and the storage module is small, fewer vector computing units can be used; otherwise, the number of vector computing units can be increased.

[0215] The second embodiment of the present application discloses a calculation method for a hardware architecture applied to a Fastformer neural network. The calculation method is applied to a control module, and the control module is located in the hardware architecture applied to the Fastformer neural network described in the first embodiment of the present application. The calculation method includes:

[0216] Step 001, controlling the data storage module, the read buffer module and the linear calculation submodule to perform a linear layer query operation on the input matrix, obtaining a query matrix and outputting it to the read buffer module and the write buffer module respectively; the write buffer module transmits the query matrix to the temporary matrix storage unit;

[0217] Step 002: Control the data storage module, the read buffer module, and the linear calculation submodule to perform vector-matrix multiplication on the query matrix to obtain an intermediate query vector and transmit it to the nonlinear calculation submodule;

[0218] Step 003: Control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate query vector to obtain a normalized query vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention key linear layer operation on the input matrix to obtain a key matrix and output it to the write buffer module; the write buffer module transmits the key matrix to the temporary matrix storage unit;

[0219] Step 004: Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication on the normalized query vector and the query matrix to obtain a global query vector, and output it to the read buffer module;

[0220] Step 005: Control the read buffer module, the linear calculation submodule and the data storage module to perform element-by-element product on the global query vector and the key matrix to obtain an intermediate key matrix, perform vector-matrix multiplication on the intermediate key matrix to obtain an intermediate key vector and output it to the nonlinear calculation submodule, and simultaneously control the linear calculation submodule to output the intermediate key matrix to the write buffer module; the write buffer module stores the intermediate key matrix to the temporary matrix storage unit;

[0221] Step 006: Control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate key vector to obtain a normalized key vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention value linear layer operation on the input matrix to obtain a value matrix and output it to the write buffer module; the write buffer module transmits the value matrix to the temporary matrix storage unit;

[0222] Step 007: Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication on the normalized key vector and the key matrix to obtain a global key vector and output it to the read buffer module;

[0223] Step 008: Control the linear calculation submodule, the read buffer module, and the data storage module to perform element-by-element product operations on the value matrix and the global key vector, and perform linear layer operations on the operation results to obtain a global value matrix; perform addition operations on the global value matrix and the query matrix to obtain an output matrix of the neural network and output it to the write buffer module; the write buffer module transmits the output matrix to the output matrix storage unit in the data storage module.

[0224] When the control module controls the linear calculation submodule to perform linear layer operations, the calculation method further includes:

[0225] Step 101, control the read buffer module to input the target column of the matrix to be calculated and the target row of the weight matrix to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, obtains the target accumulation result through accumulation operation and outputs it to the addition unit, and at the same time controls the read buffer module to input the corresponding element value of the bias matrix to the addition unit; the addition unit adds the target accumulation result and the corresponding element value in the bias matrix to obtain the target value;

[0226] Step 102, repeatedly executing step 101 to obtain multiple target values, until the target column is the last column in the matrix to be calculated;

[0227] Step 103, concatenate the multiple target values ​​to obtain a result matrix; wherein the size of the result matrix is ​​consistent with the size of the matrix to be calculated.

[0228] When the control unit controls the linear calculation submodule to perform a vector-matrix multiplication operation, the calculation method further includes:

[0229] Step 200, determining whether the vector to be calculated by the current linear calculation unit is a normalized query vector or a normalized key vector;

[0230] If the vector to be calculated by the current linear calculation unit is not a normalized query vector or a normalized key vector, the following operations are performed:

[0231] Step 211, control the read buffer module to input the target column of the vector to be operated and the matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, obtains the target accumulation result through accumulation operation and outputs it to the addition unit, and the addition unit directly outputs the target accumulation result;

[0232] Step 212, repeating step 211 to obtain multiple target accumulation results, until the target column is the last column of the matrix to be calculated;

[0233] Step 213, concatenating multiple target accumulation results to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of columns of the matrix to be calculated;

[0234] If the input vector of the current linear computation unit is a normalized query vector or a normalized key vector, the following operations are performed:

[0235] Step 221, control the read buffer module to split the vector to be operated into a first number of sub-row vectors, the dimension of the sub-row vectors is equal to the number of rows of the input matrix of the neural network; at the same time, control the read buffer module to split each column vector of the matrix to be operated into a first number of sub-column vectors, the dimension of the sub-column vectors is equal to the number of rows of the input matrix of the neural network;

[0236] Step 222, control the read buffer module to input the target sub-row vector of the vector to be calculated and the corresponding sub-column vector of the target column in the matrix to be calculated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, and obtains the target accumulation result through accumulation operation and outputs it to the adding unit; the adding unit adds the target accumulation result and the target cumulative result obtained by the last execution of step 222, and outputs the target value;

[0237] Step 223, repeatedly executing steps 221 to 222 until the target sub-row vector is the last sub-vector in the vector to be calculated;

[0238] Step 224, repeatedly executing steps 221 to 223 to obtain a plurality of target values, until the target column is the last column in the matrix to be calculated;

[0239] Step 225, concatenate the multiple target values ​​to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be calculated.

[0240] When the control unit controls the linear calculation submodule to perform the vector-matrix element-by-element active multiplication mixed operation, the calculation method further includes:

[0241] Step 301, control the read buffer module to input the first vector to be operated and the target column of the matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the write buffer module and the element-by-element product operation unit respectively;

[0242] Step 302, control the read buffer module to input the second vector to be operated to the element-by-element product operation unit; the element-by-element product operation unit performs an element-by-element product operation on the second vector to be operated and the intermediate operation vector, and outputs the corresponding result to the inner product unit, obtains a target value through accumulation operation, and outputs it to the addition unit, and the addition unit directly outputs the target value;

[0243] Step 303: repeating steps 301 and 302 to obtain a plurality of target values ​​and a plurality of intermediate operation vectors, until the target column is the last column of the matrix to be operated;

[0244] Step 304: concatenate multiple target values ​​to a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be operated; and concatenate multiple intermediate operation vectors to obtain an intermediate operation matrix; wherein the size of the intermediate operation matrix is ​​consistent with the size of the matrix to be operated. When the control unit controls the linear calculation submodule to perform vector-matrix element-by-element product, linear layer and additive mixing operations, the calculation method further includes:

[0245] Step 401, control the read buffer module to input the first vector to be operated and the target column of the first matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the element-by-element product operation unit;

[0246] Step 402, control the target row of the weight matrix input by the read buffer module to the element-by-element product operation unit; the element-by-element product operation unit performs element-by-element product operation on the target row and the intermediate operation vector, and outputs the corresponding result to the inner product unit, obtains the target accumulation result through accumulation operation and outputs it to the addition unit; at the same time, control the element value corresponding to the bias vector input by the read buffer module and the element value corresponding to the second matrix to be operated to the addition unit;

[0247] Step 403, the adding unit adds the target accumulation result, the element value corresponding to the bias vector and the element value corresponding to the second matrix to be operated to obtain a target value;

[0248] Step 404, repeatedly executing steps 402 to 403 to obtain a plurality of target values, until the target row of the weight matrix is ​​the last row in the weight matrix;

[0249] Step 405, repeatedly executing steps 401 to 404 to obtain a plurality of target values, until the target column of the first matrix to be calculated is the last column of the first matrix to be calculated;

[0250] Step 406, concatenate the multiple target values ​​to obtain a result matrix; wherein the size of the result matrix is ​​consistent with the size of the first matrix to be calculated.

[0251] The third embodiment of the present application discloses a computer device, including:

[0252] Memory for storing computer programs;

[0253] A processor is used to implement the steps of the calculation method applied to the hardware architecture of the Fastformer neural network as described in the second embodiment of the present application when executing the computer program.

[0254] The fourth embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is processed and executed, the steps of the calculation method applied to the hardware architecture of the Fastformer neural network as described in the second embodiment of the present application are implemented.

[0255] The present application discloses a hardware architecture and a calculation method thereof applied to a Fastformer neural network through the above embodiments, wherein the hardware architecture includes a data storage module, a read buffer module, a write buffer module, a calculation module and a control module; wherein the control module is used to control the hardware architecture to perform calculation operations, and the calculation module includes a linear calculation submodule, a nonlinear calculation submodule and an intermediate storage submodule; the linear calculation module includes a plurality of vector calculation units for performing linear operations including vector matrix multiplication, matrix addition and element-by-element product, and the nonlinear submodule is used for performing normalized exponential function operations. The hardware architecture can effectively improve the calculation speed and efficiency of the Fastformer neural network.

[0256] Furthermore, the hardware architecture applied to the Fastformer neural network disclosed in the present application can support input data of different lengths when implementing the Fastformer function, and improve the utilization of the hardware through the designed control process. Furthermore, the vector calculation unit can support all linear calculation parts in the Fastformer neural network, and by designing a mixed operation process of different functions, the larger intermediate key matrix and intermediate value matrix U in the Fastformer neural network model are processed in the vector calculation unit in the last step, reducing storage overhead. The hardware architecture of the basic vector calculation unit has a scalable feature, and the degree of parallelism can be flexibly adjusted according to the hardware resources and the bandwidth size of the memory access; according to the calculation method disclosed in the present application, all calculations in the Fastformer network model can be fully realized, and at the same time, the generation of intermediate data can be minimized to ensure the efficiency of the overall system.

[0257] The present application is described in detail above in conjunction with specific implementation methods and exemplary examples, but these descriptions cannot be understood as limiting the present application. Those skilled in the art understand that, without departing from the spirit and scope of the present application, a variety of equivalent replacements, modifications or improvements can be made to the technical solution of the present application and its implementation methods, all of which fall within the scope of the present application. The scope of protection of the present application shall be subject to the attached claims.

Claims

1. A hardware architecture for Fastformer neural network, characterized in that: include: Data storage module, read buffer module, write buffer module, calculation module and control module; The data storage module is used to receive and store the input matrix to be calculated, and to store pre-trained parameters, and to store the intermediate data generated by the calculation module during the calculation process, and to store the calculation results of the hardware architecture; the input end of the data storage module is the input interface of the entire hardware architecture, and is connected to the output end of the write buffer module; the output end of the data storage module is connected to the input end of the read buffer module, and serves as the output interface of the hardware architecture; The input end of the read buffer module is also connected to the output end of the calculation module, and the output end of the read buffer module is connected to the input end of the calculation module; The input end of the write buffer module is connected to the output end of the calculation module; The calculation module includes a linear calculation submodule, a nonlinear calculation submodule and an intermediate storage submodule; the linear calculation submodule is used to perform linear operations including vector matrix multiplication, matrix addition, and element-by-element product, the nonlinear calculation submodule is used to perform normalized exponential function operations, and the intermediate storage submodule is used to receive the calculation results of the nonlinear calculation submodule and transmit them to the linear calculation submodule; wherein the input end of the linear calculation submodule is respectively connected to the output end of the read buffer module and the output end of the intermediate storage submodule, and the output end is respectively connected to the input end of the read buffer, the input end of the write buffer module and the input end of the nonlinear calculation submodule; the output end of the nonlinear calculation submodule is connected to the input end of the intermediate storage submodule; The control module is respectively connected to the data storage module, the read buffer module, the write buffer module, and the linear calculation submodule and the nonlinear calculation submodule in the calculation module; The control module is used to perform the following operations: Control the data storage module, the read buffer module and the linear calculation submodule to perform a linear layer query operation on the input matrix to obtain a query matrix and output it to the read buffer module and the write buffer module respectively; the write buffer module transmits the query matrix to the data storage module; Control the data storage module, the read buffer module, and the linear calculation submodule to perform a vector-matrix multiplication operation on the query matrix to obtain an intermediate query vector and transmit it to the nonlinear calculation submodule; Control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate query vector to obtain a normalized query vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention key linear layer operation on the input matrix to obtain a key matrix and output it to the write buffer module; the write buffer module transmits the key matrix to the data storage module; Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication on the normalized query vector and the query matrix to obtain a global query vector, and output it to the read buffer module; Control the read buffer module, the linear calculation submodule and the data storage module to perform element-by-element product of the global query vector and the key matrix to obtain an intermediate key matrix, perform vector-matrix multiplication operation on the intermediate key matrix to obtain an intermediate key vector and output it to the nonlinear calculation submodule, and simultaneously control the linear calculation submodule to output the intermediate key matrix to the write buffer module; the write buffer module stores the intermediate key matrix to the data storage module; Control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate key vector to obtain a normalized key vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention value linear layer operation on the input matrix to obtain a value matrix and output it to the write buffer module; the write buffer module transmits the value matrix to the data storage module; Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication on the normalized key vector and the key matrix to obtain a global key vector and output it to the read buffer module; Control the linear calculation submodule, the read buffer module, and the data storage module to perform element-by-element product operations on the value matrix and the global key vector, and perform linear layer operations on the operation results to obtain a global value matrix; perform addition operations on the global value matrix and the query matrix to obtain an output matrix of the neural network and output it to the write buffer module; the write buffer module transmits the output matrix to the data storage module.

2. The hardware architecture for Fastformer neural network according to claim 1, characterized in that: The data storage module includes an input matrix storage unit, a weight matrix storage unit, a bias vector storage unit, a parameter vector storage unit, a temporary matrix storage unit and an output matrix storage unit; in The input end of the input matrix storage unit is the input end of the hardware architecture, and the output end of the input matrix storage unit is connected to the input end of the read buffer module; The output ends of the weight matrix storage unit, the bias vector storage unit, the parameter vector storage unit, and the temporary matrix storage unit are connected to the input end of the read buffer module; The input end of the temporary matrix storage unit is connected to the output end of the write buffer module; The input end of the output matrix storage unit is connected to the output end of the write buffer module, and the output end of the output matrix storage unit is the output end of the hardware architecture.

3. The hardware architecture for Fastformer neural network according to claim 1, characterized in that: The linear calculation submodule includes a plurality of vector calculation units, wherein each of the vector calculation units includes an element-by-element product operation unit, an inner product unit, a temporary vector storage unit and an addition unit; The element-by-element product operation unit includes a plurality of multipliers; wherein the number of the multipliers is equal to the number of rows of any input matrix; the input end of the element-by-element product operation unit is respectively connected to the output end of the read buffer module and the output end of the temporary vector storage unit, and the output end of the element-by-element product operation unit is respectively connected to the inner product unit and the temporary vector storage unit; the element-by-element product operation unit is used to perform element-by-element multiplication operation on two input vectors; The input end of the inner product unit is connected to the output end of the element-by-element product operation unit, and the output end of the inner product unit is connected to the addition unit; The input end of the temporary vector storage unit is connected to the output end of the element-by-element product operation unit, and the output end of the temporary vector storage unit is respectively connected to the input end of the element-by-element product operation unit and the input end of the write buffer module; The input end of the adding unit is respectively connected to the output end of the inner product unit and the output end of the read buffer module, and the output end of the adding unit is connected to the input end of the read buffer module and the input end of the write buffer module.

4. The hardware architecture for Fastformer neural network according to claim 3, characterized in that: When the linear layer operation is performed in the linear calculation submodule, the control module is also used to perform the following operations: Step 101, control the read buffer module to input the target column of the matrix to be calculated and the target row of the weight matrix to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, obtains the target accumulation result through accumulation operation and outputs it to the addition unit, and at the same time controls the read buffer module to input the corresponding element value of the bias matrix to the addition unit; The adding unit adds the target accumulation result to the corresponding element value in the bias matrix to obtain the target value; Step 102, repeatedly executing step 101 to obtain multiple target values, until the target column is the last column in the matrix to be calculated; Step 103, concatenating the multiple target values ​​to obtain a result matrix; The size of the result matrix is ​​consistent with the size of the matrix to be operated.

5. The hardware architecture for Fastformer neural network according to claim 3, characterized in that: When the vector-matrix multiplication operation is performed in the linear calculation submodule, the control module is also used to perform the following operations: Step 200, determining whether the vector to be calculated by the current linear calculation unit is a normalized query vector or a normalized key vector; If the vector to be calculated by the current linear calculation unit is not a normalized query vector or a normalized key vector, the following operations are performed: Step 211, control the read buffer module to input the target column of the vector to be operated and the matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, obtains the target accumulation result through accumulation operation and outputs it to the addition unit, and the addition unit directly outputs the target accumulation result; Step 212, repeating step 211 to obtain multiple target accumulation results, until the target column is the last column of the matrix to be calculated; Step 213, concatenating multiple target accumulation results to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of columns of the matrix to be calculated; If the input vector of the current linear computation unit is a normalized query vector or a normalized key vector, the following operations are performed: Step 221, control the read buffer module to split the vector to be operated into a first number of sub-row vectors, the dimension of the sub-row vectors is equal to the number of rows of the input matrix of the neural network; at the same time, control the read buffer module to split each column vector of the matrix to be operated into a first number of sub-column vectors, the dimension of the sub-column vectors is equal to the number of rows of the input matrix of the neural network; Step 222, control the read buffer module to input the target sub-row vector of the vector to be operated and the corresponding sub-column vector of the target column in the matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the inner product unit, and obtains the target accumulation result through accumulation operation and outputs it to the addition unit; The adding unit adds the target accumulated result and the target accumulated result obtained by the last execution of step 222, and outputs the target value; Step 223, repeatedly executing steps 221 to 222 until the target sub-row vector is the last sub-vector in the vector to be calculated; Step 224, repeatedly executing steps 221 to 223 to obtain a plurality of target values, until the target column is the last column in the matrix to be calculated; Step 225, concatenate the multiple target values ​​to obtain a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be calculated.

6. The hardware architecture for Fastformer neural network according to claim 3, characterized in that: When the linear calculation submodule performs the vector-matrix element-by-element product and the multiplication mixed operation, the control module is also used to perform the following operations: Step 301, control the read buffer module to input the first vector to be operated and the target column of the matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the write buffer module and the element-by-element product operation unit respectively; Step 302, control the read buffer module to input the second vector to be operated to the element-by-element product operation unit; the element-by-element product operation unit performs an element-by-element product operation on the second vector to be operated and the intermediate operation vector, and outputs the corresponding result to the inner product unit, obtains a target value through accumulation operation, and outputs it to the addition unit, and the addition unit directly outputs the target value; Step 303: repeating steps 301 and 302 to obtain a plurality of target values ​​and a plurality of intermediate operation vectors, until the target column is the last column of the matrix to be operated; Step 304: concatenate multiple target values ​​to a result column vector; wherein the dimension of the result column vector is equal to the number of rows of the matrix to be calculated; and concatenate multiple intermediate calculation vectors to obtain an intermediate calculation matrix; wherein the size of the intermediate calculation matrix is ​​consistent with the size of the matrix to be calculated.

7. The hardware architecture for Fastformer neural network according to claim 3, characterized in that: When the linear calculation submodule performs the vector-matrix element-by-element product, the linear layer, and the additive mixing operation, the control module is also used to perform the following operations: Step 401, control the read buffer module to input the first vector to be operated and the target column of the first matrix to be operated to the element-by-element product operation unit; the element-by-element product operation unit outputs the corresponding element-by-element product operation result to the temporary vector storage unit for splicing to obtain an intermediate operation vector; the temporary vector storage unit transmits the intermediate operation vector to the element-by-element product operation unit; Step 402, control the target row of the weight matrix input by the read buffer module to the element-by-element product operation unit; the element-by-element product operation unit performs element-by-element product operation on the target row and the intermediate operation vector, and outputs the corresponding result to the inner product unit, obtains the target accumulation result through accumulation operation and outputs it to the addition unit; at the same time, control the read buffer module input bias vector corresponding to the element value corresponding to the second matrix to be operated to the addition unit; Step 403, the adding unit adds the target accumulation result, the element value corresponding to the bias vector and the element value corresponding to the second matrix to be operated to obtain a target value; Step 404, repeatedly executing steps 402 to 403 to obtain a plurality of target values, until the target row of the weight matrix is ​​the last row in the weight matrix; Step 405, repeatedly executing steps 401 to 404 to obtain a plurality of target values, until the target column of the first matrix to be calculated is the last column of the first matrix to be calculated; Step 406, concatenate the multiple target values ​​to obtain a result matrix; wherein the size of the result matrix is ​​consistent with the size of the first matrix to be calculated.

8. A method for calculating the hardware architecture of a Fastformer neural network, characterized in that: The calculation method is applied to a control module, the control module is located in the hardware architecture applied to the Fastformer neural network according to any one of claims 1 to 7, and the calculation method includes: Step 001, controlling the data storage module, the read buffer module and the linear calculation submodule to perform a query linear layer operation on the input matrix, obtaining a query matrix and outputting it to the read buffer module and the write buffer module respectively; the write buffer module transmits the query matrix to the data storage module; Step 002: Control the data storage module, the read buffer module, and the linear calculation submodule to perform vector-matrix multiplication on the query matrix to obtain an intermediate query vector and transmit it to the nonlinear calculation submodule; Step 003: Control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate query vector to obtain a normalized query vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention key linear layer operation on the input matrix to obtain a key matrix and output it to the write buffer module; the write buffer module transmits the key matrix to the data storage module; Step 004: Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication on the normalized query vector and the query matrix to obtain a global query vector, and output it to the read buffer module; Step 005: Control the read buffer module, the linear calculation submodule and the data storage module to perform element-by-element product on the global query vector and the key matrix to obtain an intermediate key matrix, perform vector-matrix multiplication on the intermediate key matrix to obtain an intermediate key vector and output it to the nonlinear calculation submodule, and simultaneously control the linear calculation submodule to output the intermediate key matrix to the write buffer module; the write buffer module stores the intermediate key matrix to the data storage module; Step 006: Control the nonlinear calculation submodule to perform a normalized exponential function operation on the intermediate key vector to obtain a normalized key vector and output it to the intermediate storage submodule; at the same time, control the data storage module, the read buffer module and the linear calculation submodule to perform an attention value linear layer operation on the input matrix to obtain a value matrix and output it to the write buffer module; the write buffer module transmits the value matrix to the data storage module; Step 007: Control the linear calculation submodule, the intermediate storage submodule, the data storage module, and the read buffer module to perform vector-matrix multiplication on the normalized key vector and the key matrix to obtain a global key vector and output it to the read buffer module; Step 008: Control the linear calculation submodule, the read buffer module, and the data storage module to perform element-by-element product operations on the value matrix and the global key vector, and perform linear layer operations on the operation results to obtain a global value matrix; perform addition operations on the global value matrix and the query matrix to obtain an output matrix of the neural network and output it to the write buffer module; the write buffer module transmits the output matrix to the data storage module.

Images