An FPGA-based Transformer encoder hardware accelerator and its computation method

Abstract

This invention relates to an FPGA-based Vision Transformer encoder hardware accelerator and its computation method, comprising: a processor module, a DDR memory module, a data scheduling module, a matrix storage module, a residual addition module, a first matrix multiplication module, a GeLU operation module, a second matrix multiplication module, a third matrix multiplication module, and a softmax calculation module. The processor module is used for dynamically scheduling the accelerator and performing auxiliary calculations. Simultaneously, the computation array adopts a data-parallel computation strategy with fixed outputs, exhibiting strong network scale adaptability, high input / output data reuse, and efficient on-chip and off-chip data transfer. This hardware accelerator is a dedicated accelerator for Vision Transformer neural networks, effectively improving the computation speed and efficiency of Vision Transformer neural networks while significantly reducing computational power consumption.

Description

Technical Field

[0001] This invention relates to an FPGA-based Vision Transformer encoder hardware accelerator and its computation method, primarily targeting computer vision problems in edge devices, such as classification and detection tasks, and belongs to the field of deep learning technology. Background Technology

[0002] Recently, the attention-based Transformer architecture has achieved excellent performance in natural language processing tasks and has been widely applied. Inspired by this, the Vision Transformer network extends it to the field of machine vision. Research shows that reliance on CNNs is not necessary in machine vision; the Vision Transformer (ViT) model can achieve results comparable to the current state-of-the-art convolutional networks, while significantly reducing the computational resources required for its training.

[0003] The architecture of its encoder section is as follows Figure 1 As shown, it mainly consists of a layer normalization function operation layer, a multi-head attention layer, and a feedforward layer. First, the input matrix undergoes layer normalization function and multi-head attention layer operations to obtain the output of the multi-head attention layer. Then, the output of the multi-head attention layer is added to the input with residuals, followed by layer normalization and feedforward layer calculations. Finally, the output of the feedforward layer is added to the output of the multi-head attention layer with residuals to obtain the encoder layer's result. The multi-head attention layer contains three input matrices and multiple attention heads. Each attention head contains q, K, V weight matrices and q, K, V bias matrices, which can perform linear mappings on each input. The mapping results are processed by attention operations to obtain multiple matrices. The new matrix obtained by concatenating these matrices undergoes another linear transformation using the linear mapping weight matrix and linear mapping bias matrix to obtain the multi-head attention layer's output. The feedforward layer mainly consists of a first feedforward weight matrix, a second feedforward weight matrix, a first feedforward bias matrix, and a second feedforward bias matrix. It obtains the feedforward calculation result by performing linear transformations, function activation, and linear transformations on the input.

[0004] Currently, the aforementioned computational processes all run on general-purpose computing platforms such as CPUs or GPUs. In actual computation, significant latency occurs due to the complex processing steps. Existing Transformer or VisionTransformer accelerators are adapted to relatively fixed network sizes, offering limited configurable performance, and their data scheduling schemes have considerable room for optimization. To improve the computational speed and efficiency of Vision Transformer neural networks and reduce computational power consumption, there is an urgent need to design dedicated hardware accelerators. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a hardware accelerator and its computational method for the Vision Transformer encoder. This accelerator features separate hardware circuit designs for each part of the Vision Transformer encoder. It can work collaboratively with a processor according to the proposed computational method, thereby improving computational speed and efficiency, enhancing data reuse, reducing redundant data transfer to improve efficiency, and saving computational power consumption.

[0006] The present invention also provides a quantization scheme for deploying the above-described accelerator and computing method.

[0007] The technical solution of the present invention is as follows:

[0008] A hardware accelerator for the encoder portion of a Vision Transformer includes:

[0009] Processor module, DDR memory module, data scheduling module, matrix storage module, residual addition module, first matrix multiplication module, GeLU operation module, second matrix multiplication module, third matrix multiplication module, softmax calculation module;

[0010] The processor module is connected to the DDR memory module, the data scheduling module, and the residual addition module. The data scheduling module is connected to the matrix storage module, the first matrix multiplication module, the GeLU operation module, the second matrix multiplication module, and the third matrix multiplication module.

[0011] The second matrix multiplication module is connected to the softmax calculation module;

[0012] The processor module is used to read parameters and temporary data stored in the DDR memory module, perform layer normalization function calculations with high precision within the processor module, provide dynamic control and data to the data scheduling module when performing other calculations besides the layer normalization function, receive data returned by the data scheduling module or the residual addition module, and write temporary data and calculation results to the DDR memory module; the DDR memory module is used to store parameters, input data, temporary data generated by calculation, and final results; the data scheduling module is used to control the module's operation and schedule data when performing other calculations besides the layer normalization function; the residual addition module is used to perform residual addition operations; the matrix storage module is used to perform... The system stores data and parameters to be processed during operations other than the layer normalization function; the first matrix multiplication module is used to perform linear layer operations, multiplying two matrices simultaneously input in a specific manner and adding them to a third matrix input in a specific manner; the GeLU operation module is used to perform operations on the data in the feedforward layer using the GeLU activation function; the second matrix multiplication module is used to calculate the first matrix multiplication in the dot product attention function; the softmax calculation module is used to perform the softmax operation; the third matrix multiplication module is used to calculate the second matrix multiplication in the dot product attention function, multiplying two matrices simultaneously input in a specific manner and adding them to a third input matrix.

[0013] According to a preferred embodiment of the present invention, the data scheduling module includes a controller unit, an address generation unit, and a data gating unit; the controller unit is connected to the address generation unit and the data gating unit; the processor module is connected to the controller unit and the data gating unit; the controller unit, the address generation unit, and the data gating unit are all connected to the matrix storage module.

[0014] The controller unit is used to: control the matrix storage module, the address generation unit, and the data gating unit; the address generation unit is used to: generate the addresses of each data storage unit in the matrix storage module; the data gating unit is used to: select the data path between the processor module and each data storage unit.

[0015] According to a preferred embodiment of the present invention, the matrix storage module includes a data signal gating unit, a first data storage unit, a second data storage unit, and a third data storage unit; the data signal gating unit is connected to the first data storage unit, the second data storage unit, and the third data storage unit; the controller unit, the address generation unit, and the data gating unit are all connected to the data signal gating unit.

[0016] The data signal gating unit is used to: select the addresses and data of the first data storage unit, the second data storage unit, and the third data storage unit; the first data storage unit, the second data storage unit, and the third data storage unit are all used for: buffering input, output, and intermediate data.

[0017] According to a preferred embodiment of the present invention, the first matrix multiplication module includes multiple parallel multiply-accumulate units;

[0018] The multiply-accumulate unit is used for matrix multiplication operations.

[0019] According to a preferred embodiment of the present invention, the second matrix multiplication module includes a multiplication unit and an addition tree unit;

[0020] The multiplication unit is used to multiply the input data; the addition tree unit is used to add the results of the multiplication unit in parallel.

[0021] According to a preferred embodiment of the present invention, the softmax calculation module performs the softmax operation using equation (I):

[0022]

[0023] In equation (I), z i Let z be the output value of the i-th node, C be the number of output nodes (i.e., the number of categories), and z be the number of categories. max The maximum value of all node outputs is obtained by using the softmax function to convert the multi-class output values ​​into a probability distribution ranging from [0,1] to 1; softmax(z i ) indicates that for z i Perform softmax calculation, z c This is the output value of the c-th node.

[0024] According to a preferred embodiment of the present invention, the third matrix multiplication module includes multiple parallel multiply-accumulate units.

[0025] According to a preferred embodiment of the present invention, the softmax calculation module includes a maximum value unit, a first cache unit, an adder unit, an exponentiation unit, an accumulator unit, a second cache unit, and a division unit; as follows: Figure 3 As shown;

[0026] First, the input data is fed into the maximum value unit and the first cache unit respectively. Then, the results of the maximum value unit and the first cache unit are fed into the adder unit. The result of the adder unit is fed into the exponentiation unit. Next, the result of the exponentiation unit is fed into the accumulator unit and the second cache unit. Finally, the outputs of the accumulator unit and the second cache unit are both fed into the division unit to obtain the final softmax result.

[0027] The maximum value unit is used to: traverse the input data of softmax and select the largest value as the output; the first cache unit is used to: cache the input data; the adder unit is used to: subtract the result of the first cache unit from the result of the maximum value unit; the exponentiation unit is used to: perform exponentiation on the adder result; the accumulator unit is used to: accumulate the result of the exponentiation operation; the second cache unit is used to: cache the result of the exponentiation unit; the division unit is used to: divide the result of the accumulator unit by the output of the second cache unit.

[0028] According to a preferred embodiment of the present invention, the GeLU arithmetic unit utilizes a lookup table to perform approximate calculations using the following formula:

[0029]

[0030] The above-mentioned hardware accelerator operation method for the Vision Transformer encoder part includes:

[0031] The input data is processed sequentially through the first layer normalization function operation, multi-head attention layer calculation, first residual summation, second layer normalization function operation, feedforward layer calculation and second residual summation, and finally the final calculation result is obtained.

[0032] According to a preferred embodiment of the present invention, the following operations are performed before the above-described method for operating the hardware accelerator for the Vision Transformer encoder section is run: the input matrix X, weight matrix W, and weight matrix W are... Q W K W V W O W1, W2, bias matrices B, B Q B K B V B O B1 and B2 are all stored in the DDR memory module, and there is still extra space inside the DDR memory module for storing intermediate temporary data;

[0033] W, W Q W K W V W O W 1 W 2 These are, respectively, the layer normalized weights, Q-weight matrix, K-weight matrix, V-weight matrix, linear mapping weight matrix, first feedforward weight matrix, and second feedforward weight matrix; B, B Q B K B V B O B 1B 2 These are, respectively, the layer normalized bias, Q bias matrix, K bias matrix, V bias matrix, linear mapping bias matrix, first feedforward bias matrix, and second feedforward bias matrix;

[0034] W, W Q W K W V W O W1, W2, B, B Q B K B V B O B1 and B2 are parameters of the Vision Transformer, obtained by training the dataset through a neural network.

[0035] According to a preferred embodiment of the present invention, the first layer normalization function operation includes:

[0036] The processor module calculates the mean and variance of the input matrix X in the DDR memory module using equations (II) and (III), respectively:

[0037]

[0038]

[0039] In equations (II) and (III), i = 1, 2, ..., d k j = 1, 2, ..., d model Let X(i,j) represent the element in the i-th row and j-th column of the input matrix X, μ(i) represent the mean of the elements in the i-th row of the input matrix X, and σ(i) represent the mean of the elements in the input matrix X. The variance of the elements in the i-th row; d k The number of image markers in the input image; d model This refers to the size of the neural network model;

[0040] The processor module performs layer normalization processing on each element of the input matrix X, the mean of each row of elements in the input matrix X, the reciprocal of the square root of the variance of each row of elements in the input matrix X, the weight matrix W and the bias matrix B in the DDR memory module according to equation (IV), and obtains the final output matrix X of the layer normalization function operation. norm1 :

[0041]

[0042] In equation (IV), ε is an infinitesimal, W(i,j) represents the j-th element in the i-th row of the learned weight matrix W, B(j) represents the j-th element of the bias matrix B, and X norm1 (i,j) represents the element in the i-th row and j-th column of the resulting matrix, X.norm1 This is the output matrix of the first layer normalization function operation, which is stored in the DDR memory module.

[0043] According to a preferred embodiment of the present invention, multi-head attention layer calculation includes:

[0044] The processor module configures the data scheduling module to allocate the data matrix X in the DDR memory module respectively. norm1 Weight matrix W Q With bias matrix B Q The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively.

[0045] X norm1 The processor module, acting as matrix Q, configures the data scheduling module to store matrix Q and weight matrix W in the matrix storage module. Q With bias matrix B Q The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to perform matrix multiplication on matrix Q and weight matrix W. Q Matrix multiplication, the result of matrix multiplication and the bias matrix B Q Perform matrix addition to obtain the result matrix X. Q The processor module will perform matrix addition on the resulting matrix X. Q The data is transmitted back through the data scheduling module and stored in the DDR memory module;

[0046] The processor module configures the data scheduling module to allocate the weight matrix W in the DDR memory module respectively. K With bias matrix B K The data is transmitted to the matrix storage module and stored in the second and third data storage units, respectively. Matrix Q serves as matrix K. The processor module, through the configuration data scheduling module, processes matrix K and weight matrix W in the matrix storage module. K With bias matrix B K The data is fed into the first matrix multiplication module, enabling the first matrix multiplication module to perform matrix multiplication on matrix K and weight matrix W. K Matrix multiplication, the result of matrix multiplication and the bias matrix B K Perform matrix addition to obtain the result matrix X. K The processor module will perform matrix addition on the resulting matrix X. K The data is transmitted back through the data scheduling module and stored in the DDR memory module;

[0047] The processor module configures the data scheduling module to allocate the weight matrix W in the DDR memory module respectively. V With bias matrix B VThe data is transmitted to the matrix storage module and stored in the second and third data storage units, respectively. Matrix K acts as matrix V. The processor module configures the data scheduling module to process matrix V and weight matrix W in the matrix storage module. V With bias matrix B V The data is fed into the first matrix multiplication module, enabling the first matrix multiplication module to perform matrix multiplication on matrix V and weight matrix W. V Matrix multiplication, the result of matrix multiplication and the bias matrix B V Perform matrix addition to obtain the result matrix X. V The processor module will perform matrix addition on the resulting matrix X. V The data is transmitted back through the data scheduling module and stored in the DDR memory module;

[0048] The processor module configures the data scheduling module to allocate the temporary data matrix X in the DDR memory module. Q Temporary data matrix X K With temporary data matrix X V The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively.

[0049] The processor module configures the data scheduling module to store the temporary data matrix X in the matrix storage module. Q With temporary data matrix X K The data is then fed to the second matrix multiplication module, enabling the second matrix multiplication module to complete the temporary data matrix X. Q Temporary data matrix X K The matrix multiplication is performed, and the result of the matrix multiplication is sent to the softmax calculation module for softmax calculation. The result of the softmax calculation is then sent to the third matrix multiplication module.

[0050] The processor module configures the data scheduling module to store the temporary data matrix X in the matrix storage module. V The matrix X is fed into the third matrix multiplication module, where matrix X... V The processor module performs matrix multiplication with the output matrix of the softmax calculation module, and then processes the matrix multiplication result matrix X output by the third matrix multiplication module. softmax The data is transmitted back through the data scheduling module and stored in the DDR memory module;

[0051] The processor module configures the data scheduling module to allocate the temporary data matrix X in the DDR memory module. softmax Weight matrix W O With bias matrix B OThe data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively.

[0052] The processor module configures the data scheduling module to store matrix X in the matrix storage module. softmax Weight matrix W O With bias matrix B O The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete the multiplication of matrix X. softmax Weight matrix W O Matrix multiplication, the result of matrix multiplication and the bias matrix B O The matrix addition is performed, and the resulting matrix is ​​the final output matrix X, which serves as the multi-head attention layer. MultiHead The data scheduling module will output matrix X MultiHead It is then fed into the residual addition module.

[0053] According to a preferred embodiment of the present invention, performing the first residual summation and the second layer normalization function operation includes:

[0054] The processor module sends the input matrix X from the DDR memory module to the residual addition module, and the residual module completes the addition of matrix X. MultiHead The processor module performs matrix addition with the input matrix X, i.e., the first residual addition operation, and then adds the result X of the matrix addition. add The data is transmitted back via the data scheduling module and stored in the DDR memory module. The processor module then uses the matrix addition result X. add The weight matrix W and the bias matrix B complete the second layer normalization function operation, resulting in the matrix X. norm2 It is stored in the DDR memory module.

[0055] According to a preferred embodiment of the present invention, the feedforward layer calculation and the second residual addition operation include:

[0056] The weight matrices W1, W2, and B1 required for feedforward layer calculations are each divided into four parts, with W1 divided into W... 1·1 W 1·2 W 1·3 W 1·4 W2 is divided into W 2·1 W 2·2 W 2·3 W 2·4 B1 is divided into B 1·1 B 1·2 B 1·3 B 1·4 ;

[0057] First feedforward weight matrix Divide its second dimension into four equal parts to obtain W. 1·1 W 1·2 W 1·3 W 1·4 W 1·1 W 1·2 W 1·3 ,

[0058] Second feedforward weight matrix Divide its first dimension into 4 equal parts to obtain W. 2·1 W 2·2 W 2·3 W 2·4 W 2·1 W 2·2 W 2·3 ,

[0059] First feedforward bias matrix Divide its second dimension into 4 equal parts to obtain B. 1·1 B 1·2 B 1·3 B 1·4 B 1·1 B 1·2 B 1·3 ,

[0060] The processor module configures the data scheduling module to allocate the data matrix X in the DDR memory module respectively. norm2 Weight matrix W 1·1 With bias matrix B 1·1 The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively.

[0061] The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1·1 With bias matrix B 1·1 The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete matrix W. 1·1 With bias matrix B 1·1 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1·1 After completing the matrix addition, the processor module sends the result of the matrix addition to the GeLU operation module, which activates the GeLU function. Subsequently, the processor module processes the result matrix X output by the GeLU operation module. GeLU·1 The data is transmitted back through the data scheduling module and stored in the DDR memory module;

[0062] The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1·2 With bias matrix B 1·2 The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete matrix W. 1·2 With bias matrix B 1·2 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1·2 After completing the matrix addition, the processor module sends the result of the matrix addition to the GeLU operation module, which activates the GeLU function. Subsequently, the processor module processes the result matrix X output by the GeLU operation module. GeLU·2 The data is transmitted back through the data scheduling module and stored in the DDR memory module;

[0063] The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1·3 With bias matrix B 1·3 The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete matrix W. 1·3 With bias matrix B 1·3 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1·3 After completing the matrix addition, the processor module sends the result of the matrix addition to the GeLU operation module, which activates the GeLU function. Subsequently, the processor module processes the result matrix X output by the GeLU operation module. GeLU·3 The data is transmitted back through the data scheduling module and stored in the DDR memory module;

[0064] The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1·4 With bias matrix B 1·4 The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete matrix W. 1·4 With bias matrix B 1·4 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1·4 After completing the matrix addition, the processor module sends the result of the matrix addition to the GeLU operation module, which activates the GeLU function. Subsequently, the processor module processes the result matrix X output by the GeLU operation module. GeLU·4 The data is transmitted back through the data scheduling module and stored in the DDR memory module;

[0065] The processor modules respectively use the weight matrix W 2·1The bias matrix B2 performs the matrix multiplication and addition operations described above. The result of the matrix addition is sent to the residual addition module via the data scheduling module. This includes: the processor module transferring X from the DDR module... GeLU·1 The weight matrix W in the matrix storage module 2·1 The bias matrix B2 is used to calculate X according to the conformity matrix. GeLU·1 ×W 2-1 The data is sent serially or in parallel to the first matrix multiplication module via +B2 for matrix multiplication and addition, and the calculation results are then sent to the residual addition module.

[0066] At the same time, the processor module will transfer X from the DDR memory module. add The result is fed to the residual addition module and added to the matrix addition result input to the residual addition module. The matrix result of the residual addition is then sent back and stored in the DDR memory module.

[0067] The processor modules respectively use the weight matrix W 2·2 W 2·3 W 2·4 The bias matrix B2 performs the matrix multiplication and addition operations described above. The result of the matrix addition is sent to the residual addition module via the data scheduling module. This includes: the processor module transferring matrix X from the DDR module... GeLU·2 X GeLU·3 X GeLU·4 According to X GeLU·1 Similar methods are used with W respectively 2·2 W 2·3 W 2·4 Perform matrix multiplication and then add the results to the bias matrix B2 to obtain X. GeLU·2 ×W 2-2 +B2、X GeLU·3 ×W 2-3 +B2、X GeLU·4 ×W 2-4 The calculation results of +B2 are then sent to the residual addition module.

[0068] At the same time, the processor module sends the result matrix of the previous residual addition in the DDR memory module to the residual addition module, and performs a matrix addition operation with the matrix addition result input to the residual addition module this time. The result matrix of the residual addition is sent back and stored in the DDR memory module. This result is the final result of the feedforward layer operation and the second residual addition.

[0069] The beneficial effects of this invention are as follows:

[0070] This invention discloses a hardware accelerator for a Vision Transformer neural network encoder and its computation and quantization methods. The hardware accelerator includes: a processor module, a DDR memory module, a data scheduling module, a matrix storage module, a residual addition module, a first matrix multiplication module, a GeLU operation module, a second matrix multiplication module, a third matrix multiplication module, and a softmax calculation module. The processor module dynamically schedules the accelerator and performs auxiliary computations. Simultaneously, the computation array employs a data-parallel computation strategy with fixed outputs, exhibiting strong network size adaptability, high input / output data reuse, and efficient on-chip and off-chip data transfer. This hardware accelerator is a dedicated accelerator for Vision Transformer neural networks, effectively improving the computation speed and efficiency of Vision Transformer neural networks while significantly reducing computational power consumption. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of the Vision Transformer neural network model architecture;

[0072] Figure 2 This is a schematic diagram of the hardware architecture of a hardware accelerator applied to the Vision Transformer neural network disclosed in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram of the internal structure of the softmax calculation module in the hardware accelerator for the Vision Transformer neural network disclosed in an embodiment of the present invention.

[0074] Figure 4 This is a schematic diagram of the first layer normalization operation in the computation method of the hardware accelerator applied to the Vision Transformer neural network disclosed in the embodiments of the present invention.

[0075] Figure 5 This is a schematic diagram of multi-head attention layer operation in the computation method of the hardware accelerator applied to the Vision Transformer neural network disclosed in the embodiments of the present invention.

[0076] Figure 6 This is a schematic diagram of residual addition and second layer normalization operation in the calculation method of the hardware accelerator applied to the Vision Transformer neural network disclosed in the embodiments of the present invention.

[0077] Figure 7 This is a schematic diagram of the feedforward layer and residual addition in the calculation method of the hardware accelerator applied to the Vision Transformer neural network disclosed in an embodiment of the present invention.

[0078] Figure 8 This is a schematic diagram of the quantization scheme for matrix multiplication operations in the computation method of the hardware accelerator for the Vision Transformer neural network disclosed in an embodiment of the present invention.

[0079] Figure 9 This is a schematic diagram of a lookup table creation scheme for complex operations in the computation method of the hardware accelerator for the Vision Transformer neural network disclosed in an embodiment of the present invention.

[0080] Figure 10 A schematic diagram showing the comparison of joint optimization (multi-channel quantization) results;

[0081] Figure 11 This is a schematic diagram comparing the results of single-layer optimization (multi-channel quantization);

[0082] Figure 12 A visual diagram illustrating the iterative method for finding the optimal quantization coefficients. Detailed Implementation

[0083] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0084] Example 1

[0085] A hardware accelerator for the encoder portion of a Vision Transformer, such as... Figure 2 As shown, it includes: a processor module, a DDR memory module, a data scheduling module, a matrix storage module, a residual addition module, a first matrix multiplication module, a GeLU operation module, a second matrix multiplication module, a third matrix multiplication module, and a softmax calculation module;

[0086] The processor module is connected to the DDR memory module, the data scheduling module, and the residual addition module. The data scheduling module is connected to the matrix storage module, the first matrix multiplication module, the GeLU operation module, the second matrix multiplication module, and the third matrix multiplication module. The second matrix multiplication module is connected to the softmax calculation module.

[0087] The processor module reads parameters and temporary data stored in the DDR memory module, performs layer normalization function calculations with high precision within the processor module, provides dynamic control and data to the data scheduling module during calculations other than layer normalization, receives data from the data scheduling module or residual addition module, and writes temporary data and calculation results to the DDR memory module. The DDR memory module stores parameters, input data, temporary data generated during calculations, and the final results. The data scheduling module controls the module's operation and schedules data during calculations other than layer normalization. The residual addition module performs residual addition operations. The matrix storage module is used for layer normalization... Other than functions, the module stores the data to be processed and parameters during other operations; the first matrix multiplication module is used to perform linear layer operations, multiplying two matrices simultaneously input in a specific manner and adding them to a third matrix input in a specific manner; the GeLU operation module is used to perform operations on the data in the feedforward layer using the GeLU activation function; the second matrix multiplication module is used to calculate the first matrix multiplication in the dot product attention function; the softmax calculation module is used to perform the softmax operation; the third matrix multiplication module is used to calculate the second matrix multiplication in the dot product attention function, multiplying two matrices simultaneously input in a specific manner and adding them to a third input matrix.

[0088] Example 2

[0089] The hardware accelerator for the Vision Transformer encoder section described in Embodiment 1 differs in that:

[0090] The data scheduling module includes a controller unit, an address generation unit, and a data gating unit; the controller unit is connected to the address generation unit and the data gating unit; the processor module is connected to the controller unit and the data gating unit; the controller unit, the address generation unit, and the data gating unit are all connected to the matrix storage module.

[0091] The controller unit is used to control the matrix storage module, the address generation unit, and the data gating unit; the address generation unit is used to generate the addresses of each data storage unit in the matrix storage module; the data gating unit is used to select the data path between the processor module and each data storage unit.

[0092] The matrix storage module includes a data signal gating unit, a first data storage unit, a second data storage unit, and a third data storage unit; the data signal gating unit is connected to the first data storage unit, the second data storage unit, and the third data storage unit; the controller unit, the address generation unit, and the data gating unit are all connected to the data signal gating unit;

[0093] The data signal gating unit is used to select the address and data of the first data storage unit, the second data storage unit, and the third data storage unit; the first data storage unit, the second data storage unit, and the third data storage unit are all used for buffering input, output, and intermediate data.

[0094] The first matrix multiplication module includes multiple parallel multiply-accumulate units; the multiply-accumulate units are used for matrix multiplication operations.

[0095] The second matrix multiplication module includes a multiplication unit and an addition tree unit; the multiplication unit is used to multiply the input data; the addition tree unit is used to add the results of the multiplication unit in parallel.

[0096] The softmax calculation module uses formula (I) to perform the softmax operation:

[0097]

[0098] In equation (I), z i Let z be the output value of the i-th node, C be the number of output nodes (i.e., the number of categories), and z be the number of categories. max The maximum value of all node outputs is obtained by using the softmax function to convert the multi-class output values ​​into a probability distribution ranging from [0,1] to 1; softmax(z i ) indicates that for z i Perform softmax calculation, z c This is the output value of the c-th node.

[0099] The third matrix multiplication module includes multiple parallel multiply-accumulate units.

[0100] The softmax calculation module includes a maximum value unit, a first cache unit, an adder unit, an exponentiation unit, an accumulator unit, a second cache unit, and a division unit; for example... Figure 3 As shown;

[0101] like Figure 7 As shown, the input data is first input into the maximum value unit and the first cache unit respectively. Then, the result of the maximum value unit and the result of the first cache unit are sent into the adder unit. The result of the adder unit is sent into the exponentiation unit. Then, the result of the exponentiation unit is sent into the accumulator unit and the second cache unit. Finally, the output of the accumulator unit and the output of the second cache unit are both input into the division unit to obtain the final softmax result.

[0102] The maximum value unit is used to iterate through the input data of the softmax function and select the largest value as the output. The first buffer unit is used to buffer the input data. The adder unit is used to subtract the result of the first buffer unit from the result of the maximum value unit. The exponentiation unit is used to perform exponentiation on the adder result. The accumulator unit is used to accumulate the result of the exponentiation operation. The second buffer unit is used to buffer the result of the exponentiation unit. The division unit is used to divide the result of the accumulator unit by the output of the second buffer unit.

[0103] Divide by in the multi-head attention layer The operations can be integrated into the exponential (e) lookup table unit in the softmax calculation module. Inserting the first and second buffer units enables pipelined operations, reducing the computational latency of the softmax module.

[0104] The GeLU arithmetic unit uses a lookup table to perform approximate calculations using the following formula:

[0105] Example 3

[0106] A method for operating a hardware accelerator for the Vision Transformer encoder section according to Embodiment 1 or 2 includes:

[0107] The input data is processed sequentially through the first layer normalization function operation, multi-head attention layer calculation, first residual summation, second layer normalization function operation, feedforward layer calculation and second residual summation, and finally the final calculation result is obtained.

[0108] The hardware accelerator for the Vision Transformer encoder works as follows: Before running, it performs the following operations: input matrix X, weight matrix W, W... Q W K W V W O W1, W2, bias matrices B, B Q B K B V B O B1 and B2 are all stored in the DDR memory module, and there is still extra space inside the DDR memory module for storing intermediate temporary data;

[0109] W, W Q W K W V W O W 1 W 2These are, respectively, the layer normalized weights, Q-weight matrix, K-weight matrix, V-weight matrix, linear mapping weight matrix, first feedforward weight matrix, and second feedforward weight matrix; B, B Q B K B V B O B 1 B 2 These are, respectively, the layer normalized bias, Q bias matrix, K bias matrix, V bias matrix, linear mapping bias matrix, first feedforward bias matrix, and second feedforward bias matrix;

[0110] W, W Q W K W V W O W1, W2, B, B Q B K B V B O B1 and B2 are parameters of the Vision Transformer, obtained by training the dataset through a neural network.

[0111] like Figure 4 As shown, the first layer normalization function operation is performed, including:

[0112] The processor module calculates the mean and variance of the input matrix X in the DDR memory module using equations (II) and (III), respectively:

[0113]

[0114]

[0115] In equations (II) and (III), i = 1, 2, ..., d k j = 1, 2, ..., d model Let X(i,j) represent the element in the i-th row and j-th column of the input matrix X, μ(i) represent the mean of the elements in the i-th row of the input matrix X, and σ(i) represent the mean of the elements in the input matrix X. The variance of the elements in the i-th row; d k The number of image markers in the input image; d model This refers to the size of the neural network model;

[0116] The processor module performs layer normalization on each element of the input matrix X, the mean of each row of elements in the input matrix X, the reciprocal of the square root of the variance of each row of elements in the input matrix X, the weight matrix W and the bias matrix B in the DDR memory module according to equation (IV), and obtains the final output matrix X of the layer normalization function operation. norm1 :

[0117]

[0118] In equation (IV), ε is an infinitesimal, W(i,j) represents the j-th element in the i-th row of the learned weight matrix W, B(j) represents the j-th element of the bias matrix B, and X norm1 (i,j) represents the element in the i-th row and j-th column of the resulting matrix, X. norm1 This is the output matrix of the first layer normalization function operation, which is stored in the DDR memory module.

[0119] like Figure 4 As shown, the multi-head attention layer calculation includes:

[0120] The processor module, through the configured data scheduling module, allocates data matrix X from the DDR memory module respectively. norm1 Weight matrix W Q With bias matrix B Q The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively.

[0121] X norm1 The processor module, acting as matrix Q, configures the data scheduling module to store matrix Q and weight matrix W in the matrix storage module. Q With bias matrix B Q The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete the multiplication of matrix Q and weight matrix W. Q Matrix multiplication, the result of matrix multiplication and the bias matrix B Q Perform matrix addition to obtain the result matrix X. Q The processor module will perform matrix addition on the resulting matrix X. Q The data is transmitted back via the data scheduling module and stored in the DDR memory module.

[0122] The processor module configures the data scheduling module to allocate the weight matrix W in the DDR memory module. K With bias matrix B K The data is fed into the matrix storage module and stored in the second and third data storage units, respectively. Matrix Q acts as matrix K. The processor module configures the data scheduling module to process matrix K and weight matrix W in the matrix storage module. K With bias matrix B K The data is fed into the first matrix multiplication module, enabling the first matrix multiplication module to perform matrix K and weight matrix W multiplication. K Matrix multiplication, the result of matrix multiplication and the bias matrix B K Perform matrix addition to obtain the result matrix X. K The processor module will perform matrix addition on the resulting matrix X. KThe data is transmitted back via the data scheduling module and stored in the DDR memory module.

[0123] The processor module configures the data scheduling module to allocate the weight matrix W in the DDR memory module. V With bias matrix B V The data is fed into the matrix storage module and stored in the second and third data storage units, respectively. Matrix K acts as matrix V. The processor module configures the data scheduling module to process matrix V and weight matrix W from the matrix storage module. V With bias matrix B V The data is fed into the first matrix multiplication module, enabling the first matrix multiplication module to perform matrix multiplication on matrix V and weight matrix W. V Matrix multiplication, the result of matrix multiplication and the bias matrix B V Perform matrix addition to obtain the result matrix X. V The processor module will perform matrix addition on the resulting matrix X. V The data is transmitted back via the data scheduling module and stored in the DDR memory module.

[0124] The processor module, through the configured data scheduling module, allocates temporary data matrix X from the DDR memory module. Q Temporary data matrix X K With temporary data matrix X V The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively.

[0125] The processor module configures the data scheduling module to process the temporary data matrix X in the matrix storage module. Q With temporary data matrix X K The data is then fed to the second matrix multiplication module, enabling the second matrix multiplication module to complete the temporary data matrix X. Q Temporary data matrix X K The matrix multiplication is performed, and the result of the matrix multiplication is sent to the softmax calculation module for softmax calculation. The result of the softmax calculation is then sent to the third matrix multiplication module.

[0126] The processor module configures the data scheduling module to process the temporary data matrix X in the matrix storage module. V The data is fed into the third matrix multiplication module, where matrix X is multiplied. V The processor module performs matrix multiplication with the output matrix of the softmax calculation module, and then processes the matrix multiplication result matrix X output by the third matrix multiplication module. softmax The data is transmitted back via the data scheduling module and stored in the DDR memory module.

[0127] The processor module, through the configured data scheduling module, allocates temporary data matrix X from the DDR memory module. softmax Weight matrix W O With bias matrix B O The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively.

[0128] The processor module configures the data scheduling module to store matrix X in the matrix storage module. softmax Weight matrix W O With bias matrix B O The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete the multiplication of matrix X. softmax Weight matrix W O Matrix multiplication, the result of matrix multiplication and the bias matrix B O The matrix addition is performed, and the resulting matrix is ​​the final output matrix X, which serves as the multi-head attention layer. MultiHead The data scheduling module will output matrix X MultiHead It is then fed into the residual addition module.

[0129] Perform the first residual summation and the second layer normalization function operation, such as... Figure 5 As shown, it includes:

[0130] The processor module sends the input matrix X from the DDR memory module to the residual addition module, and the residual module completes the addition of matrix X. MultiHead The matrix addition with the input matrix X, i.e., the first residual addition operation, is performed by the processor module. The result of this matrix addition, X, is... add The data is transmitted back via the data scheduling module and stored in the DDR memory module. The processor module then uses the matrix addition result X. add The weight matrix W and the bias matrix B complete the second layer normalization function operation. The calculation process for the second layer normalization function operation is the same as that for the first layer normalization function operation, and the resulting matrix X is... norm2 It is stored in the DDR memory module.

[0131] Perform feedforward layer calculations and add the second residual, such as... Figure 6 As shown, it includes:

[0132] The weight matrices W1, W2, and B1 required for feedforward layer calculations are each divided into four parts, with W1 divided into W... 1·1 W 1·2 W 1·3 W 1·4 W2 is divided into W 2·1 W 2·2 W 2·3 W 2·4 B1 is divided into B1·1 B 1·2 B 1·3 B 1·4 ;

[0133] First feedforward weight matrix Divide its second dimension into four equal parts to obtain W. 1·1 W 1·2 W 1·3 W 1·4 W 1·1 W 1·2 W 1·3 ,

[0134] Second feedforward weight matrix Divide its first dimension into 4 equal parts to obtain W. 2·1 W 2·2 W 2·3 W 2·4 W 2·1 W 2·2 W 2·3 ,

[0135] First feedforward bias matrix Divide its second dimension into 4 equal parts to obtain B. 1·1 B 1·2 B 1·3 B 1·4 B 1·1 B 1·2 B 1·3 ,

[0136] The processor module, through the configured data scheduling module, allocates data matrix X from the DDR memory module respectively. norm2 Weight matrix W 1·1 With bias matrix B 1·1 The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively.

[0137] The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1·1 With bias matrix B 1·1 X is calculated according to the conformity matrix. norm2 ×W 1·1 +B 1·1 The data is transmitted serially or in parallel to the first matrix multiplication module, enabling the first matrix multiplication module to complete the matrix W... 1·1 With bias matrix B 1·1 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1·1After completing the matrix addition, the processor module sends the result to the GeLU operation module, which activates the GeLU function. The processor module then processes the result matrix X output by the GeLU operation module. GeLU·1 The data is transmitted back via the data scheduling module and stored in the DDR memory module.

[0138] The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1·2 With bias matrix B 1·2 X is calculated according to the conformity matrix. norm2 ×W 1·2 +B 1·2 The data is transmitted serially or in parallel to the first matrix multiplication module, enabling the first matrix multiplication module to complete the matrix W... 1·2 With bias matrix B 1·2 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1·2 After completing the matrix addition, the processor module sends the result to the GeLU operation module, which activates the GeLU function. The processor module then processes the result matrix X output by the GeLU operation module. GeLU·2 The data is transmitted back via the data scheduling module and stored in the DDR memory module.

[0139] The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1·3 With bias matrix B 1·3 X is calculated according to the conformity matrix. norm2 ×W 1·3 +B 1·3 The data is transmitted serially or in parallel to the first matrix multiplication module, enabling the first matrix multiplication module to complete the matrix W... 1·3 With bias matrix B 1·3 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1·3 After completing the matrix addition, the processor module sends the result to the GeLU operation module, which activates the GeLU function. The processor module then processes the result matrix X output by the GeLU operation module. GeLU·3 The data is transmitted back via the data scheduling module and stored in the DDR memory module.

[0140] The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1·4 With bias matrix B 1·4 X is calculated according to the conformity matrix. norm2 ×W 1·4 +B 1·4The data is transmitted serially or in parallel to the first matrix multiplication module, enabling the first matrix multiplication module to complete the matrix W... 1·4 With bias matrix B 1·4 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1·4 After completing the matrix addition, the processor module sends the result to the GeLU operation module, which activates the GeLU function. The processor module then processes the result matrix X output by the GeLU operation module. GeLU·4 The data is transmitted back via the data scheduling module and stored in the DDR memory module.

[0141] The processor modules use weight matrix W respectively 2·1 The bias matrix B2 performs the matrix multiplication and addition operations described above. The result of the matrix addition is sent to the residual addition module via the data scheduling module. This includes: the processor module transferring X from the DDR module... GeLU·1 The weight matrix W in the matrix storage module 2·1 The bias matrix B2 is used to calculate X according to the conformity matrix. GeLU·1 ×W 2-1 The data is sent serially or in parallel to the first matrix multiplication module via +B2 for matrix multiplication and addition, and the calculation results are then sent to the residual addition module.

[0142] At the same time, the processor module will transfer X from the DDR memory module. add The result is sent to the residual addition module and added to the matrix addition result input to the residual addition module. The matrix result of the residual addition is returned and stored in the DDR memory module.

[0143] The processor modules use weight matrix W respectively 2·2 W 2·3 W 2·4 The bias matrix B2 performs the matrix multiplication and addition operations described above. The result of the matrix addition is sent to the residual addition module via the data scheduling module. This includes: the processor module transferring matrix X from the DDR module to the residual addition module. GeLU·2 X GeLU·3 X GeLU·4 According to X GeLU·1 Similar methods are used with W respectively 2·2 W 2·3 W 2·4 Perform matrix multiplication and then add the results to the bias matrix B2 to obtain X. GeLU·2 ×W 2-2 +B2、X GeLU·3 ×W 2-3 +B2、X GeLU·4 ×W 2-4 The calculation results of +B2 are then sent to the residual addition module.

[0144] At the same time, the processor module sends the result matrix of the previous residual addition in the DDR memory module to the residual addition module, and performs a matrix addition operation with the matrix addition result input to the residual addition module this time. The result matrix of the residual addition is sent back and stored in the DDR memory module. This result is the final result of the feedforward layer operation and the second residual addition.

[0145] A quantization scheme for deploying the aforementioned accelerators and computational methods mainly includes two cases: matrix multiplication operations or computational parts that can be decomposed into matrix multiplication operations, and complex computational parts that cannot be decomposed into matrix multiplication operations.

[0146] For matrix multiplication operations or operations that can be decomposed into matrix multiplication operations, the optimal quantization coefficient matrix is ​​solved using an iterative method with the constraint of minimizing the quantization error, and the result is recorded and saved. Specifically, the model is decoupled into Python and layered. The maximum absolute value (max) and minimum absolute value (min) of the 32-bit floating-point matrix `data` to be quantized are calculated. The step size of the loop is determined based on the obtained `max` and `min`. The left side of the loop is defined as `left = step`, the right boundary as `right = max`, and the initial threshold of the loop is `th = left`. If `th <= right`, the inner loop is entered: the 32-bit floating-point matrix `data` is quantized to 8 bits using `data_q = round(data / th * 128)`, and the current quantization coefficients are saved. For elements in the quantized result matrix, data exceeding the range of 8 signed numbers are defined as +127 (positive overflow) or -128 (negative overflow). The quantized weight matrix is ​​multiplied by the quantized input matrix and added to the quantized bias matrix. After dequantization based on the quantization coefficients, the result is compared with the accurate value, and the mean squared error is selected as the error function. When th = right, the inner loop has ended, resulting in five mean squared error values. Find the two points adjacent to the minimum value among these five mean squared errors, and assign their corresponding thresholds to left and right respectively. If the minimum value is a boundary point, assign the value itself to left or right. Save the quantization coefficient corresponding to the minimum value. Repeat the above steps for three iterations. If a smaller error value appears than in the previous outer loop, save the quantization coefficient corresponding to that error value. Finally, obtain the optimal quantization coefficient. Quantize according to this optimal threshold to obtain the quantization result of the matrix. A visualization of this process is shown below. Figure 8 As shown.

[0147] For complex operations that cannot be decomposed into matrix multiplication, a tabular method is used to approximate the solution. Simultaneously, to facilitate smooth pre-computation and subsequent operations, the input and output bit widths of the lookup table need to be limited. The 32-bit floating-point input matrix `data` is initialized with the optimal threshold point `th` obtained from the previous quantization layer, and the input bit width of the lookup table is initialized to `w = M`. A loop is then entered, first performing an N-bit quantization operation on the 32-bit floating-point input matrix `data` and saving its quantization coefficients `data_q = round(data / th * 2^(M-1))`. For the elements of the quantized matrix, data exceeding the range of M-bit signed numbers is defined as +2^(M-1)-1 (positive overflow) or -2^(M-1) (negative overflow). Integers from -2^(M-1) to 2^(M-1)-1 are dequantized according to the aforementioned quantization coefficients and then subjected to complex operations. The expression yields the quantization table output list. The quantized input matrix data_q is then processed through the quantization table list to obtain the output matrix data'. The optimal quantization threshold point for the output matrix data' is found using the aforementioned iterative method, and the mean square error value generated at this threshold point is saved. If the current error value is acceptable, the same N-bit optimal quantization threshold point quantization is performed on the quantization table output: list_q = round(list / th' * 2^(N-1)). The bit width w is gradually decreased until the error becomes unacceptable. At this point, the input bit width w, output bit width N, and the lookup table list_q are output. A visualization of this process is shown below. Figure 9 As shown.

[0148] For consecutive multiplication operations, layer fusion is performed. Taking the last multiplication operation of the first layer normalization operation in the encoder and the first matrix multiplication operation of the multi-head attention layer as an example: Y norm =X×W1+B1 and It can be converted into in: This allows the original two-step operation to be merged into a single operation, reducing the amount of computation.

[0149] Multichannel quantization is used for the matrix multiplication operation; including:

[0150] Let X represent the m*n input matrix, W represent the m*n weight matrix, B represent the 1*n bias matrix, and Y represent the m*n output matrix; where the weight matrix has completely determined values ​​and distribution after training, and is subjected to multi-channel quantization as shown in equations (V), (VI), (VII), and (VIII):

[0151] Y = X × W T +B(V)

[0152]

[0153]

[0154]

[0155] Where round() represents the nearest integer operation, Q x Represents the uniform quantization coefficients of the input matrix X, x m·n w represents the element in the m-th row and n-th column of the input matrix X. m·n The element in the m-th row and n-th column of the weight matrix W, b n Represents the element in the nth column of the bias matrix B, Q wn Represents the quantization coefficient of the nth channel of the weight matrix W, Q bn X represents the quantization coefficient of the nth channel in the bias matrix B; p W q B q These are the quantization coefficients for the input matrix, weight matrix, and bias matrix, respectively. The quantization coefficients for each column of the final output matrix are shown in equation (IX):

[0156]

[0157] like Figure 1 as well as Figure 2 As shown, joint optimization means taking the quantization result of the previous layer as the input of the current layer and comparing the quantized result of the current layer with the standard result. Figure 10 A schematic diagram showing the comparison of joint optimization (multi-channel quantization) results; Figure 11 This is a schematic diagram comparing the results of single-layer optimization (multi-channel quantization); Figure 12 A visual diagram illustrating the iterative method for finding the optimal quantization coefficients.

[0158] For complex operations that cannot be decomposed into matrix multiplication, linear tabulation is used for quantization. If the complex operation is preceded or followed by operations on matrix elements, layer fusion can be performed, placing the operation within the quantization table. Since a division operation on matrix elements occurs before the softmax operation, this division operation can be incorporated into the quantization table to reduce computational load.

[0159] The order of the three linear operations in the multi-head attention layer computation can be changed.

Claims

1. A hardware accelerator for the encoder portion of a Vision Transformer, characterized in that, include: Processor module, DDR memory module, data scheduling module, matrix storage module, residual addition module, first matrix multiplication module, GeLU operation module, second matrix multiplication module, third matrix multiplication module, softmax calculation module; The processor module is connected to the DDR memory module, the data scheduling module, and the residual addition module. The data scheduling module is connected to the matrix storage module, the first matrix multiplication module, the GeLU operation module, the second matrix multiplication module, and the third matrix multiplication module. The second matrix multiplication module is connected to the softmax calculation module; The processor module is used to read parameters and temporary data stored in the DDR memory module, perform layer normalization function calculations with high precision within the processor module, provide dynamic control and data to the data scheduling module when performing other calculations besides the layer normalization function, receive data returned by the data scheduling module or the residual addition module, and write temporary data and calculation results to the DDR memory module; the DDR memory module is used to store parameters, input data, temporary data generated by calculation, and final results; the data scheduling module is used to control the module's operation and schedule data when performing other calculations besides the layer normalization function; the residual addition module is used to perform residual addition operations; the matrix storage module is used to perform... The system stores data and parameters to be processed during operations other than the layer normalization function; the first matrix multiplication module is used to perform linear layer operations, multiplying two matrices simultaneously input in a specific manner and adding them to a third matrix input in a specific manner; the GeLU operation module is used to perform operations on the data in the feedforward layer using the GeLU activation function; the second matrix multiplication module is used to calculate the first matrix multiplication in the dot product attention function; the softmax calculation module is used to perform the softmax operation; the third matrix multiplication module is used to calculate the second matrix multiplication in the dot product attention function, multiplying two matrices simultaneously input in a specific manner and adding them to a third input matrix. The data scheduling module includes a controller unit, an address generation unit, and a data gating unit; the controller unit is connected to the address generation unit and the data gating unit; the processor module is connected to the controller unit and the data gating unit; the controller unit, the address generation unit, and the data gating unit are all connected to the matrix storage module. The controller unit is used to: control the matrix storage module, the address generation unit, and the data gating unit; the address generation unit is used to: generate the addresses of each data storage unit in the matrix storage module; the data gating unit is used to: select the data path between the processor module and each data storage unit; The matrix storage module includes a data signal gating unit, a first data storage unit, a second data storage unit, and a third data storage unit; the data signal gating unit is connected to the first data storage unit, the second data storage unit, and the third data storage unit; the controller unit, the address generation unit, and the data gating unit are all connected to the data signal gating unit. The data signal gating unit is used to: select the addresses and data of the first data storage unit, the second data storage unit, and the third data storage unit; the first data storage unit, the second data storage unit, and the third data storage unit are all used for: buffering input, output, and intermediate data; The first matrix multiplication module includes multiple parallel multiply-accumulate units; The multiply-accumulate unit is used for matrix multiplication operations; The second matrix multiplication module includes a multiplication unit and an addition tree unit; The multiplication unit is used to multiply the input data; the addition tree unit is used to add the results of the multiplication unit in parallel. The softmax calculation module uses formula ( Complete the softmax operation: （ ） Mode( )middle, Let C be the output value of the i-th node, and C be the number of output nodes, i.e., the number of categories. The maximum value of all node outputs is used to convert the multi-class output values ​​into a probability distribution ranging from [0,1] to 1 using the softmax function; Indicates to Perform softmax calculation. This is the output value of the c-th node; The third matrix multiplication module includes multiple parallel multiply-accumulate units.

2. The hardware accelerator for the Vision Transformer encoder section according to claim 1, characterized in that, The softmax calculation module includes a maximum value unit, a first cache unit, an adder unit, an exponentiation unit, an accumulator unit, a second cache unit, and a division unit; First, the input data is fed into the maximum value unit and the first cache unit respectively. Then, the results of the maximum value unit and the first cache unit are fed into the adder unit. The result of the adder unit is fed into the exponentiation unit. Next, the result of the exponentiation unit is fed into the accumulator unit and the second cache unit. Finally, the outputs of the accumulator unit and the second cache unit are both fed into the division unit to obtain the final softmax result. The maximum value unit is used to: traverse the input data of softmax and select the largest value as the output; the first cache unit is used to: cache the input data; the adder unit is used to: subtract the result of the first cache unit from the result of the maximum value unit; the exponentiation unit is used to: perform exponentiation on the adder result; the accumulator unit is used to: accumulate the result of the exponentiation operation; the second cache unit is used to: cache the result of the exponentiation unit; the division unit is used to: divide the result of the accumulator unit by the output of the second cache unit.

3. A hardware accelerator for the Vision Transformer encoder section according to claim 1, characterized in that, The GeLU arithmetic unit uses a lookup table to perform approximate calculations using the following formula: .

4. The method for operating the hardware accelerator for the Vision Transformer encoder section according to claim 1, characterized in that, include: The input data is processed sequentially through the first layer normalization function operation, multi-head attention layer calculation, first residual summation, second layer normalization function operation, feedforward layer calculation and second residual summation, and finally the final calculation result is obtained.

5. The method for operating the hardware accelerator for the Vision Transformer encoder section according to claim 4, characterized in that, The hardware accelerator for the Vision Transformer encoder works as follows: Before running, it performs the following operations: input matrix X, weight matrix W, W... Q W K W V W O W1, W2, bias matrices B, B Q B K B V B O B1 and B2 are all stored in the DDR memory module, and there is still extra space inside the DDR memory module for storing intermediate temporary data; W, W Q W K W V W O W 1 W 2 These are the layer normalized weights, Weight matrix, Weight matrix, Weight matrix, linear mapping weight matrix, first feedforward weight matrix, and second feedforward weight matrix; B, B Q B K B V B O B 1 B 2 These are respectively layer normalized bias, Bias matrix, Bias matrix, Bias matrix, linear mapping bias matrix, first feedforward bias matrix, and second feedforward bias matrix; W, W Q W K W V W O W1, W2, B, B Q B K B V B O B1 and B2 are parameters of the Vision Transformer, obtained by training the dataset through a neural network.

6. The method for operating the hardware accelerator for the Vision Transformer encoder section according to claim 4, characterized in that, Perform the first layer normalization function operation, including: The processor modules respectively adopt formula ( ),Mode( Calculate the mean and variance of the input matrix X in the DDR memory module: ； ； Mode( ),Mode( )middle, Represents the input matrix X, i.e., the first... Line 1 Column elements, Represents the input matrix No. The mean of the row elements. Represents the input matrix No. The variance of row elements; Enter the number of image tags in the image; This refers to the size of the neural network model; The processor module is based on formula ( For each element of the input matrix X, the mean of each row of elements in the input matrix X, the reciprocal of the square root of the variance of each row of elements in the input matrix X, the weight matrix W and the bias matrix B in the DDR memory module, layer normalization is performed to obtain the final output matrix of the layer normalization function operation. : （ ） Mode( )middle, It is an infinitesimal quantity. Weight matrix representing learning The element in the i-th row and j-th position, Represents the bias matrix The One element, The matrix representing the result of the operation contains the first element. Line 1 The elements of the column, the resulting matrix This is the output matrix of the first layer normalization function operation, which is stored in the DDR memory module.

7. The method for operating the hardware accelerator for the Vision Transformer encoder section according to claim 4, characterized in that, Perform multi-head attention layer calculations, including: The processor module configures the data scheduling module to allocate the data matrices in the DDR memory module respectively. Weight matrix W Q With bias matrix B Q The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively. Acting as matrix Q, the processor module configures the data scheduling module to store the matrix in the matrix storage module. Weight matrix W Q With bias matrix B Q The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete the matrix multiplication. Weight matrix W Q Matrix multiplication, the result of matrix multiplication and the bias matrix B Q Perform matrix addition to obtain the result matrix X. Q The processor module will perform matrix addition on the resulting matrix X. Q The data is transmitted back through the data scheduling module and stored in the DDR memory module; The processor module configures the data scheduling module to allocate the weight matrix W in the DDR memory module respectively. K With bias matrix B K The data is transmitted to the matrix storage module and stored in the second data storage unit and the third data storage unit, respectively. Acting as a matrix The processor module configures the data scheduling module to store the matrix in the matrix storage module. Weight matrix W K With bias matrix B K The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete the matrix multiplication. Weight matrix W K Matrix multiplication, the result of matrix multiplication and the bias matrix B K Perform matrix addition to obtain the result matrix X. K The processor module will perform matrix addition on the resulting matrix X. K The data is transmitted back through the data scheduling module and stored in the DDR memory module; The processor module configures the data scheduling module to allocate the weight matrix W in the DDR memory module respectively. V With bias matrix B V The data is transmitted to the matrix storage module and stored in the second data storage unit and the third data storage unit, respectively. Acting as a matrix The processor module configures the data scheduling module to store the matrix in the matrix storage module. Weight matrix W V With bias matrix B V The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete the matrix multiplication. Weight matrix W V Matrix multiplication, the result of matrix multiplication and the bias matrix B V Perform matrix addition to obtain the result matrix X. V The processor module will perform matrix addition on the resulting matrix X. V The data is transmitted back through the data scheduling module and stored in the DDR memory module; The processor module configures the data scheduling module to allocate the temporary data matrix X in the DDR memory module. Q Temporary data matrix X K With temporary data matrix X V The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively. The processor module configures the data scheduling module to store the temporary data matrix X in the matrix storage module. Q With temporary data matrix X K The data is then fed to the second matrix multiplication module, enabling the second matrix multiplication module to complete the temporary data matrix X. Q Temporary data matrix X K The matrix multiplication is performed, and the result of the matrix multiplication is sent to the softmax calculation module for softmax calculation. The result of the softmax calculation is then sent to the third matrix multiplication module. The processor module configures the data scheduling module to store the temporary data matrix X in the matrix storage module. V The matrix X is fed into the third matrix multiplication module, where matrix X... V The processor module performs matrix multiplication with the output matrix of the softmax calculation module, and then processes the matrix multiplication result matrix X output by the third matrix multiplication module. softmax The data is transmitted back through the data scheduling module and stored in the DDR memory module; The processor module configures the data scheduling module to allocate the temporary data matrix X in the DDR memory module. softmax Weight matrix W O With bias matrix B O The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively. The processor module configures the data scheduling module to store matrix X in the matrix storage module. softmax Weight matrix W O With bias matrix B O The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete the multiplication of matrix X. softmax Weight matrix W O Matrix multiplication, the result of matrix multiplication and the bias matrix B O The matrix addition is performed, and the resulting matrix is ​​the final output matrix X, which serves as the multi-head attention layer. MultiHead The data scheduling module will output matrix X MultiHead It is then fed into the residual addition module.

8. The method for operating the hardware accelerator for the Vision Transformer encoder section according to claim 4, characterized in that, The process involves the first residual summation and the second layer normalization function operation, including: The processor module will input the matrix from the DDR memory module. The data is fed to the residual addition module, which completes the matrix X. MultiHead With input matrix The matrix addition, i.e., the first residual addition operation, is performed by the processor module, which then performs the matrix addition result X. add The data is transmitted back via the data scheduling module and stored in the DDR memory module. The processor module then uses the matrix addition result X. add The weight matrix W and the bias matrix B complete the second layer normalization function operation, resulting in the matrix X. norm2 It is stored in the DDR memory module.

9. The method of operating the hardware accelerator for the Vision Transformer encoder section according to any one of claims 4-8, characterized in that, The calculation of the feedforward layer and the addition of the second residual are performed, including: The weight matrices W1, W2, and B1 required for feedforward layer calculations are each divided into four parts, with W1 divided into W... 1∙1 W 1∙2 W 1∙3 W 1∙4 ; W2 is divided into W 2∙1 W 2∙2 W 2∙3 W 2∙4 B1 is divided into B 1∙1 B 1∙2 B 1∙3 B 1∙4 ; First feedforward weight matrix Divide its second dimension into four equal parts to obtain , ; Second feedforward weight matrix Divide its first dimension into 4 equal parts to obtain , ; First feedforward bias matrix Divide its second dimension into four equal parts to obtain , ; The processor module configures the data scheduling module to allocate the data matrix X in the DDR memory module respectively. norm2 Weight matrix W 1∙1 With bias matrix B 1∙1 The data are transported to the matrix storage module and stored in the first data storage unit, the second data storage unit, and the third data storage unit, respectively. The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1∙1 With bias matrix B 1∙1 The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete matrix W. 1∙1 With bias matrix B 1∙1 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1∙1 After completing the matrix addition, the processor module sends the result of the matrix addition to the GeLU operation module, which activates the GeLU function. Subsequently, the processor module processes the result matrix X output by the GeLU operation module. GeLU∙1 The data is transmitted back through the data scheduling module and stored in the DDR memory module; The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1∙2 With bias matrix B 1∙2 The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete matrix W. 1∙2 With bias matrix B 1∙2 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1∙2 After completing the matrix addition, the processor module sends the result of the matrix addition to the GeLU operation module, which activates the GeLU function. Subsequently, the processor module processes the result matrix X output by the GeLU operation module. GeLU∙2 The data is transmitted back through the data scheduling module and stored in the DDR memory module; The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1∙3 With bias matrix B 1∙3 The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete matrix W. 1∙3 With bias matrix B 1∙3 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1∙3 After completing the matrix addition, the processor module sends the result of the matrix addition to the GeLU operation module, which activates the GeLU function. Subsequently, the processor module processes the result matrix X output by the GeLU operation module. GeLU∙3 The data is transmitted back through the data scheduling module and stored in the DDR memory module; The processor module configures the data scheduling module to store matrix X in the matrix storage module. norm2 Weight matrix W 1∙4 With bias matrix B 1∙4 The data is fed to the first matrix multiplication module, enabling the first matrix multiplication module to complete matrix W. 1∙4 With bias matrix B 1∙4 Matrix multiplication, the result of matrix multiplication and the bias matrix B 1∙4 After completing the matrix addition, the processor module sends the result of the matrix addition to the GeLU operation module, which activates the GeLU function. Subsequently, the processor module processes the result matrix X output by the GeLU operation module. GeLU∙4 The data is transmitted back through the data scheduling module and stored in the DDR memory module; The processor modules respectively use the weight matrix W 2∙1 The bias matrix B2 performs the matrix multiplication and addition operations described above. The result of the matrix addition is sent to the residual addition module via the data scheduling module. This includes: the processor module transferring X from the DDR module... GeLU∙1 The weight matrix W in the matrix storage module 2∙1 The bias matrix B2 is used to calculate X according to the conformity matrix. GeLU∙1 ×W 2-1 The data is sent serially or in parallel to the first matrix multiplication module via B2 for matrix multiplication and addition, and the calculation results are then sent to the residual addition module. At the same time, the processor module will transfer X from the DDR memory module. add The result is fed to the residual addition module and added to the matrix addition result input to the residual addition module. The matrix result of the residual addition is then sent back and stored in the DDR memory module. The processor modules respectively use the weight matrix W 2∙2 W 2∙3 W 2∙4 The bias matrix B2 performs the matrix multiplication and addition operations described above. The result of the matrix addition is sent to the residual addition module via the data scheduling module. This includes: the processor module transferring matrix X from the DDR module... GeLU∙2 X GeLU∙3 X GeLU∙4 According to X GeLU∙1 Similar methods are used with W respectively 2∙2 W 2∙3 W 2∙4 Perform matrix multiplication and then add the results to the bias matrix B2 to obtain X. GeLU∙2 ×W 2-2 + B2、X GeLU∙3 ×W 2-3 + B2、X GeLU∙4 ×W 2-4 The calculation results of B2 are then sent to the residual addition module. At the same time, the processor module sends the result matrix of the previous residual addition in the DDR memory module to the residual addition module, and performs a matrix addition operation with the matrix addition result input to the residual addition module this time. The result matrix of the residual addition is sent back and stored in the DDR memory module. This result is the final result of the feedforward layer operation and the second residual addition.

Images