Vision transformer neural network acceleration system and method based on cpu and fpga

Abstract

This invention proposes a visual Transformer neural network acceleration system and method based on CPU and FPGA. The method's implementation steps are as follows: an embedding module constructs a feature matrix; a preprocessing module preprocesses the feature matrix and weight matrix; the preprocessed feature matrix and weight matrix are moved and cached; an acceleration unit constructs a self-attention matrix; the self-attention matrix is ​​cached and moved; a post-processing unit constructs a global feature matrix; and an MLP Head module obtains the classification result. The preprocessing module in the CPU reduces the additional data processing time of the FPGA acceleration unit by preprocessing the feature matrix and weight matrix, effectively improving the neural network's computation speed. Simultaneously, the normalization operation module in the acceleration unit utilizes exponential and logarithmic approximation results during approximation calculations, eliminating floating-point exponentiation and division operations, effectively reducing hardware resource consumption.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a neural network acceleration system and method, specifically a visual Transformer neural network acceleration system and method based on CPU and FPGA. Background Technology

[0002] Visual Transformer neural networks consist of embedding layers, encoding layers, and MLPHead layers. The encoding layers include stacked self-attention layers, multi-head attention layers, residual connection layers, layer normalization layers, multilayer perceptron layers, fully connected layers, Gelu activation layers, and Dropout layers. Visual Transformer neural networks typically have an extremely large parameter scale and very high computational complexity when performing image classification, object detection, and semantic segmentation. In particular, the dense matrix multiplications and numerous nonlinear operations in the self-attention layers place extremely high demands on memory bandwidth and computing units.

[0003] Accelerating the visual Transformer effectively breaks through computing power bottlenecks, making it more suitable for deployment in real-world scenarios with stringent latency and energy efficiency requirements. FPGA-based visual Transformer acceleration systems are a type of solution that leverages hardware customizability to achieve high-concurrency computing. Compared to traditional general-purpose processors (CPUs), FPGA acceleration systems significantly improve model inference throughput and reduce power consumption by constructing customized matrix computation arrays and on-chip data multiplexing logic, making them promising for applications in edge computing and industrial vision.

[0004] For example, application CN116484928A discloses a hardware accelerator and acceleration method based on the VisionTransformer neural network. This invention deploys the VisionTransformer neural network on the ZYNQ development platform; it acquires feature images through the ARM processor and stores them in DDR memory; the read data is distributed to the input buffer and weight buffer; the processed feature images are input to the on-chip cache unit; the processed data is sent to the PL terminal; the hardware IP of the PL terminal is configured, and read and write operations are performed simultaneously to obtain the final calculation result; the probability calculation result is transmitted to the PC; the PetaLinux operating system is used to port the hardware accelerator system of the VisionTransformer neural network to obtain the inference prediction result from the PC. This invention boasts high computing speed, low hardware resource consumption, and high recognition accuracy. However, due to the stringent requirements of its FPGA array on the PL side regarding the spatial arrangement and parallel timing of the input data stream, and the lack of an effective data scheduling mechanism, additional data processing operations are required, which hinders further improvements in computing speed. Furthermore, when the PL side of this invention executes the visual Transformer neural network algorithm calculation process, the normalization processing of the Softmax layer involves complex floating-point exponentiation and division operations, resulting in the acceleration system still consuming a significant amount of hardware resources. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and propose a visual Transformer neural network acceleration system and method based on FPGA, which improves the calculation speed and reduces the hardware resource consumption while ensuring high recognition accuracy.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A visual Transformer neural network acceleration system based on CPU and FPGA includes a visual Transformer neural network composed of a cascaded embedding module, an encoding module, and an MLP Head module, as well as a DMA module, an on-chip temporary storage module, and an acceleration unit implemented on the FPGA and loaded between the embedding module and the encoding module; a preprocessing module is loaded between the embedding module and the DMA module; the encoding module includes a self-attention module and a post-processing unit, wherein the self-attention module serves as an acceleration unit on the FPGA side and includes two matrix calculation acceleration modules and a normalization operation module loaded between them; the post-processing unit, as well as the embedding module, preprocessing module, and MLP Head module, are deployed on the CPU side.

[0008] As an optimization, the visual Transformer neural network, wherein:

[0009] The embedding module is used to construct the feature matrix, including a cascaded linear embedding module, a category labeling module, and a position encoding module;

[0010] The post-processing unit is used for global feature extraction based on the feature matrix, including a stacked multi-head attention module, residual connection module, layer normalization module, multilayer perceptron module, fully connected module, Gelu activation module and Dropout module; the MLP Head module includes a cascaded linear transformation module and classification decision module.

[0011] The MLP Head module is used to obtain classification results through global features, including a cascaded linear transformation module and a classification decision module.

[0012] As an optimization, the preprocessing module is used to preprocess the feature matrix, including a cascaded edge filling module, a block processing module, and a skew processing module.

[0013] As an optimization, the normalization operation module includes a cascaded data rearrangement unit and an approximation calculation unit. The approximation calculation unit includes multiple approximation calculation branches arranged in parallel and composed of a cascaded extreme value search unit, a logarithmic field accumulation unit, and a probability generation unit. The output of the data rearrangement unit is also connected to the input of the logarithmic field accumulation unit and the probability generation unit. The output of the extreme value search unit is also connected to the input of the probability generation unit.

[0014] A method for accelerating a visual Transformer neural network based on CPU and FPGA includes the following steps:

[0015] (1) Constructing the feature matrix using embedded modules:

[0016] The embedding module divides the RGB image into various parts. 1 pixel Image patches, and construct rows containing target category labels with dimensions of . Feature matrix ,in , ;

[0017] (2) The preprocessing module preprocesses the feature matrix and weight matrix:

[0018] The encoding preprocessing module processes the feature matrix. The corresponding RGB image includes the visual Transformer query weight matrix. Key weight matrix Sum weight matrix weight matrix Preprocessing was performed separately to obtain the preprocessed product. eigensubmatrix and weight submatrix ;

[0019] (3) Transfer and cache the preprocessed feature matrix and weight matrix:

[0020] The DMA module will preprocess the eigensubmatrix and weight submatrix The data is transferred from the CPU to the FPGA via an on-chip temporary storage module for caching.

[0021] (4) Acceleration unit constructs self-attention matrix:

[0022] The normalization module in the acceleration unit normalizes the attention score matrix constructed by the first matrix calculation acceleration module, which includes each preprocessed feature submatrix, query weight submatrix, and key weight submatrix. It then uses the sub-feature vectors corresponding to the attention score matrix during normalization to calculate the self-attention matrix, including the global maximum, exponential approximation, and logarithmic approximation results. ;

[0023] (5) Cache and move the self-attention matrix:

[0024] On-chip temporary storage module for self-attention matrix Caching is performed, and the DMA module simultaneously performs self-attention matrix caching. Transferred from the FPGA to the CPU;

[0025] (6) The post-processing unit constructs the global feature matrix:

[0026] The post-processing unit performs self-attention matrix processing. Perform global feature extraction and construct a global feature matrix. ;

[0027] (7) The MLP Head module obtains the classification results:

[0028] The MLP Head module accesses the global feature matrix. Perform linear transformation and classification decision to obtain the classification result corresponding to the target RGB image.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] 1. This invention preprocesses the feature matrix constructed by the embedded module and the query, key and value weight matrix of the visual Transformer by setting a preprocessing module on the CPU side of the acceleration system, so as to meet the spatial arrangement and parallel timing requirements of the input data stream of the FPGA acceleration unit. This can reduce the running time of the acceleration unit and, compared with the prior art, effectively improve the calculation speed of the visual Transformer while ensuring a high recognition accuracy.

[0031] 2. This invention splits the encoding module of the visual Transformer neural network into a self-attention module and a post-processing unit, and uses the self-attention module as an acceleration unit on the FPGA side. The normalization operation module uses the global maximum value, exponential approximation result, and logarithmic approximation result of the sub-feature vector corresponding to the attention score matrix during the normalization process to calculate the self-attention matrix. The global maximum value can avoid the numerical overflow problem in the normalization calculation, and the exponential and logarithmic approximation results can eliminate the floating-point exponentiation and division operations in the normalization calculation, while preserving the relative size of the attention score and the original probability distribution characteristics, effectively reducing the hardware resource consumption. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the acceleration system of the present invention.

[0033] Figure 2 This is a schematic diagram of the normalization operation module of the present invention.

[0034] Figure 3 This is a flowchart illustrating the implementation of the acceleration method of the present invention. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Reference Figure 1 The acceleration system of the present invention includes a visual Transformer neural network composed of a cascaded embedding module, an encoding module, and an MLP Head module, as well as a DMA module, an on-chip temporary storage module, and an acceleration unit implemented on an FPGA and loaded between the embedding module and the encoding module; a preprocessing module is loaded between the embedding module and the DMA module; the encoding module includes a self-attention module and a post-processing unit, wherein the self-attention module serves as an acceleration unit on the FPGA side and includes two matrix calculation acceleration modules and a normalization operation module loaded between them; the post-processing unit, as well as the embedding module, the preprocessing module, and the MLP Head module, are deployed on the CPU side.

[0037] The encoding module in a visual Transformer neural network includes a computationally intensive self-attention module. This invention decomposes the encoding module into a self-attention module, a multi-head attention module, a residual connection module, a layer normalization module, a multilayer perceptron module, a fully connected module, a Gelu activation module, and a Dropout module. The self-attention module is loaded into the acceleration unit of an FPGA, while the multi-head attention module, residual connection module, layer normalization module, multilayer perceptron module, fully connected module, Gelu activation module, and Dropout module are loaded into the post-processing unit on the CPU. The FPGA has high computational parallelism, enabling rapid computation with lower power consumption, while the CPU possesses efficient data processing capabilities, can be programmed and controlled via software, and optimized according to different needs. Therefore, using the FPGA for computationally intensive tasks and the CPU for data processing tasks makes the entire system more efficient.

[0038] To meet the spatial arrangement and parallel timing requirements of the input data stream for the FPGA-side acceleration unit, the running time of the acceleration unit can be reduced. Compared with existing technologies, while ensuring high recognition accuracy, the computation speed of the visual Transformer is effectively improved.

[0039] The visual Transformer neural network, wherein:

[0040] The embedding module is used to construct a feature matrix, including a cascaded linear embedding module, a category labeling module, and a position encoding module;

[0041] The post-processing unit is used to perform global feature extraction based on the feature matrix, including a stacked multi-head attention module, a residual connection module, a layer normalization module, a multilayer perceptron module, a fully connected module, a Gelu activation module, and a Dropout module.

[0042] The MLP Head module is used to obtain classification results through global features, and includes a cascaded linear transformation module and a classification decision module.

[0043] The preprocessing module is used to preprocess the feature matrix, including a cascaded edge filling module, a block processing module, and a skew processing module.

[0044] Since the FPGA-side acceleration unit has stringent requirements for the spatial arrangement and parallel timing of the input matrix, this invention uses a preprocessing module set on the CPU side of the acceleration system to preprocess the feature matrix constructed by the embedded module, as well as the query, key, and value weight matrices of the visual Transformer. This satisfies the spatial arrangement and parallel timing requirements of the input data stream for the FPGA-side acceleration unit, thereby reducing the running time of the acceleration unit. Compared with the prior art, this invention effectively improves the computation speed of the visual Transformer while ensuring a high recognition accuracy.

[0045] The normalization operation module has the following structure: Figure 2 As shown, it includes a cascaded data rearrangement unit and an approximation calculation unit. The approximation calculation unit includes multiple approximation calculation branches arranged in parallel and composed of a cascaded extremum search unit, a logarithmic field accumulation unit, and a probability generation unit. The output of the data rearrangement unit is also connected to the input of the logarithmic field accumulation unit and the probability generation unit. The output of the extremum search unit is also connected to the input of the probability generation unit.

[0046] Reference Figure 3 The acceleration method of the present invention includes the following steps:

[0047] Step 1) Construct the feature matrix using the embedding module:

[0048] The embedding module divides the RGB image into various parts. 1 pixel Image patches, and construct rows containing target category labels with dimensions of . Feature matrix ,in , ;

[0049] Step 2) The preprocessing module preprocesses the feature matrix and weight matrix:

[0050] The encoding preprocessing module processes the feature matrix. The corresponding RGB image includes the visual Transformer query weight matrix. Key weight matrix Sum weight matrix weight matrix Preprocessing is performed separately, and the steps are as follows:

[0051] (2a) Edge filling module for initial feature matrix and weight matrix The edges are padded with zero data to obtain the padded initial feature matrix and weight matrix. , ;

[0052] (2b) The block processing module processes the padded initial feature matrix and weight matrix. , The partitioning is performed to obtain the dimension as follows: of There are eigenvalue submatrices and weight submatrices, where ;

[0053] (2c) Skew processing module for each feature submatrix and each weight submatrix The Insert at the beginning of a line Zero data, to achieve , The skewness was obtained after preprocessing. eigensubmatrix and weight submatrix ,in, , ;

[0054] Step 3) Transfer and cache the preprocessed feature matrix and weight matrix:

[0055] The DMA module will preprocess the eigensubmatrix and weight submatrix The data is transferred from the CPU to the FPGA via an on-chip temporary storage module for caching.

[0056] Step 4) Accelerating the construction of the self-attention matrix:

[0057] The normalization module in the acceleration unit normalizes the attention score matrix constructed by the first matrix calculation acceleration module, which includes each preprocessed feature submatrix, query weight submatrix, and key weight submatrix. It then uses the sub-feature vectors corresponding to the attention score matrix during normalization to calculate the self-attention matrix, including the global maximum, exponential approximation, and logarithmic approximation results. The implementation steps are as follows:

[0058] (4a) The first matrix calculation acceleration module preprocesses each feature submatrix Query weight submatrix Bond weight submatrix Construct attention score matrix ;

[0059] ;

[0060] in, Indicates the transpose operation;

[0061] (4b) Normalization module on attention score matrix Normalization is performed to obtain the normalized attention score matrix. ;

[0062] (4b1) Data rearrangement unit will apply attention score matrix Split by row The attention vectors are concatenated to obtain a length of [length missing]. One-dimensional feature vector ,in Indicates the first Sub-feature vectors, ;

[0063] (4b2) The extreme value search unit in each approximate calculation branch is through the sub-eigenvector. Calculate the global maximum value And cache it, where This indicates the operation of retrieving the maximum value;

[0064] (4b3) The logarithmic field accumulation unit is obtained through the sub-eigenvector. and global maximum value Calculated integer median value and decimal midpoint Calculation of exponential approximation results and through Calculate the logarithmic approximation results :

[0065] ;

[0066] ;

[0067] ;

[0068] in, Indicates rounding down; To express summation; Represents the calculation of the natural logarithm;

[0069] Since the normalization operation in the attention module involves complex floating-point exponentiation and division operations, the acceleration system requires a lot of FPGA hardware resources. This invention splits the encoding module contained in the visual Transformer neural network into a self-attention module and a post-processing unit, and uses the self-attention module as the acceleration unit on the FPGA side. The normalization operation module uses the global maximum value, exponential approximation result, and logarithmic approximation result calculated by the sub-feature vectors corresponding to the attention score matrix during the normalization process to calculate the self-attention matrix. The global maximum value can avoid the numerical overflow problem in the normalization calculation, and the exponential and logarithmic approximation results can eliminate the floating-point exponentiation and division operations in the normalization calculation, while preserving the relative size of the attention score and the original probability distribution characteristics, effectively reducing the hardware resource consumption.

[0070] (4b4) The probability generation unit approximates the result using logarithmic approximation. Calculate each sub-feature vector probability output value And reorganize all output values ​​in row-major order into a dimension of Attention score matrix :

[0071] ;

[0072] in ;

[0073] (4c) The second matrix calculation acceleration module uses the normalized attention score matrix. and each feature submatrix after preprocessing Sum Weight Submatrix Calculate the self-attention matrix :

[0074] ;

[0075] Step 5) Cache and move the self-attention matrix:

[0076] On-chip temporary storage module for self-attention matrix Caching is performed, and the DMA module simultaneously performs self-attention matrix caching. Transferred from the FPGA to the CPU;

[0077] Step 6) The post-processing unit constructs the global feature matrix:

[0078] The post-processing unit performs self-attention matrix processing. Perform global feature extraction and construct a global feature matrix. ;

[0079] Step 7) The MLP Head module obtains the classification results:

[0080] The MLP Head module accesses the global feature matrix. Perform linear transformation and classification decision to obtain the classification result corresponding to the target RGB image.

Claims

1. A visual Transformer neural network acceleration system based on CPU and FPGA, comprising a visual Transformer neural network composed of a cascaded embedding module, an encoding module, and an MLP Head module, and a DMA module, an on-chip temporary storage module, and an acceleration unit implemented on the FPGA and loaded between the embedding module and the encoding module; characterized in that, A preprocessing module is loaded between the embedding module and the DMA module; the encoding module includes a self-attention module and a post-processing unit, wherein the self-attention module serves as an acceleration unit on the FPGA side, including two matrix calculation acceleration modules and a normalization operation module loaded between them; the post-processing unit, as well as the embedding module, the preprocessing module and the MLP Head module, are deployed on the CPU side.

2. The system according to claim 1, characterized in that, The visual Transformer neural network, wherein: The embedding module is used to construct the feature matrix, including a cascaded linear embedding module, a category labeling module, and a position encoding module; The post-processing unit is used to perform global feature extraction based on the feature matrix, including a stacked multi-head attention module, residual connection module, layer normalization module, multilayer perceptron module, fully connected module, Gelu activation module and Dropout module; The MLP Head module is used to obtain classification results through global features, including a cascaded linear transformation module and a classification decision module.

3. The system according to claim 1, characterized in that, The preprocessing module is used to preprocess the feature matrix, including a cascaded edge filling module, a block processing module, and a skew processing module.

4. The system according to claim 1, characterized in that, The normalization operation module includes a cascaded data rearrangement unit and an approximation calculation unit. The approximation calculation unit includes multiple approximation calculation branches arranged in parallel and composed of a cascaded extreme value search unit, a logarithmic field accumulation unit, and a probability generation unit. The output of the data rearrangement unit is also connected to the input of the logarithmic field accumulation unit and the probability generation unit. The output of the extreme value search unit is also connected to the input of the probability generation unit.

5. A method for accelerating the system according to any one of claims 1-4, characterized in that... Includes the following steps: (1) Constructing the feature matrix using embedded modules: The embedding module divides the RGB image into various parts. 1 pixel Image patches, and construct rows containing target category labels with dimensions of . Feature matrix ,in , ; (2) The preprocessing module preprocesses the feature matrix and weight matrix: The encoding preprocessing module processes the feature matrix. The corresponding RGB image includes the visual Transformer query weight matrix. Key weight matrix Sum weight matrix weight matrix Preprocessing was performed separately to obtain the preprocessed product. eigensubmatrix and weight submatrix ; (3) Transfer and cache the preprocessed feature matrix and weight matrix: The DMA module will preprocess the eigensubmatrix and weight submatrix The data is transferred from the CPU to the FPGA via an on-chip temporary storage module for caching. (4) Acceleration unit constructs self-attention matrix: The normalization module in the acceleration unit normalizes the attention score matrix constructed by the first matrix calculation acceleration module, which includes each preprocessed feature submatrix, query weight submatrix, and key weight submatrix. It then uses the sub-feature vectors corresponding to the attention score matrix during normalization to calculate the self-attention matrix, including the global maximum, exponential approximation, and logarithmic approximation results. ; (5) Cache and move the self-attention matrix: On-chip temporary storage module for self-attention matrix Caching is performed, and the DMA module simultaneously performs self-attention matrix caching. Transferred from the FPGA to the CPU; (6) The post-processing unit constructs the global feature matrix: The post-processing unit performs self-attention matrix processing. Perform global feature extraction and construct a global feature matrix. ; (7) The MLP Head module obtains the classification results: The MLP Head module accesses the global feature matrix. Perform linear transformation and classification decision to obtain the classification result corresponding to the target RGB image.

6. The method according to claim 5, characterized in that, The preprocessing described in step (2) is implemented as follows: (2a) Edge filling module for initial feature matrix and weight matrix The edges are padded with zero data to obtain the padded initial feature matrix and weight matrix. , ; (2b) The block processing module processes the padded initial feature matrix and weight matrix. , The partitioning is performed to obtain the dimension as follows: of There are eigenvalue submatrices and weight submatrices, where ; (2c) Skew processing module for each feature submatrix and each weight submatrix The Insert at the beginning of a line Zero data, to achieve , The skewness was obtained after preprocessing. eigensubmatrix and weight submatrix ,in, , .

7. The method according to claim 6, characterized in that, The construction of the self-attention matrix described in step (4) The implementation steps are as follows: (4a) The first matrix calculation acceleration module preprocesses each feature submatrix Query weight submatrix Bond weight submatrix Construct attention score matrix ; ； in, Indicates the transpose operation; (4b) Normalization module on attention score matrix Normalization is performed to obtain the normalized attention score matrix. ; (4c) The second matrix calculation acceleration module uses the normalized attention score matrix. and each feature submatrix after preprocessing Sum Weight Submatrix Calculate the self-attention matrix : 。 8. The method according to claim 7, characterized in that, The attention score matrix described in step (4b) The normalization process involves the following steps: (4b1) Data rearrangement unit will apply attention score matrix Split by row The attention vectors are concatenated to obtain a length of [length missing]. One-dimensional feature vector ,in Indicates the first Sub-feature vectors, ; (4b2) The extreme value search unit in each approximate calculation branch is through the sub-eigenvector. Calculate the global maximum value And cache it, where This indicates the operation of retrieving the maximum value; (4b3) The logarithmic field accumulation unit is obtained through the sub-eigenvector. and global maximum value Calculated integer median value and decimal midpoint Calculation of exponential approximation results and through Calculate the logarithmic approximation results : ； ； ； in, Indicates rounding down; To express summation; Represents the calculation of the natural logarithm; (4b4) The probability generation unit approximates the result using logarithmic approximation. Calculate each sub-feature vector probability output value And reorganize all output values ​​in row-major order into a dimension of Attention score matrix : ； in .

Images