A hardware accelerator and acceleration method based on a vision transformer neural network

Abstract

A hardware accelerator and acceleration method based on a VisionTransformer neural network, the accelerator is to deploy the VisionTransformer neural network on a ZYNQ development platform; the acceleration method is: an ARM processor stores a feature picture into a DDR memory, the read data is dispersed to an input cache and a weight cache, the processed feature picture is input to an on-chip cache unit, the processed data is sent to a PL end, the hardware IP of the PL end is configured, and read-write operation is performed at the same time, the final calculation result is obtained, written into the DDR memory, the data in the DDR memory is taken out and probability operation is completed, the probability operation result is transmitted to a PC, a PetaLinux operating system is transplanted to the hardware accelerator system of the VisionTransformer neural network, and a prediction result of inference is obtained from the PC; the neural network structure is optimized through the parallel method of multiple input and multiple output channels, the calculation speed is fast, the hardware resource occupation is low, the recognition accuracy is high, and the image classification task can be efficiently completed.

Description

Technical Field

[0001] This invention relates to the field of hardware acceleration technology, specifically to a hardware accelerator and acceleration method based on the Vision Transformer neural network. Background Technology

[0002] With the development of science and technology, artificial intelligence (AI) has gradually attracted attention. Deep learning-based AI is currently a key research focus in the scientific community. Traditional Convolutional Neural Networks (CNNs), due to their inherent algorithmic structure, have a receptive field limited to a local area, unable to capture the entire target. Therefore, to a certain extent, the development of traditional CNNs has reached a bottleneck. However, the Transformer, upon its release, topped leaderboards on various datasets. Its self-attention mechanism expands its receptive field to the entire target, surpassing traditional CNNs in image classification.

[0003] VisionTransformer has demonstrated performance comparable to or even better than state-of-the-art convolutional neural network models on some image classification tasks, while also exhibiting better interpretability and transferability. However, VisionTransformer has a relatively high training cost, requiring more computational resources and longer training time. Therefore, utilizing FPGAs to implement hardware acceleration of Transformer neural networks is of significant research value. Existing Transformer neural network acceleration designs use HLS (High-Level Synthesis) to describe the computational flow of the neural network, based on the Xilinx ZYNQ development platform. In ZYNQ, recurring neurons in the Transformer network are designed as fixed hardware acceleration IPs. The software processor configures ZYNQ according to the Transformer algorithm hierarchy, executing the hardware acceleration IPs multiple times at different times based on the Transformer network model structure, thereby achieving inference computation of the Transformer model, and finally deploying it into the Xilinx ZCU102 development kit.

[0004] ZYNQ is a series of SoC (System-on-a-Chip) products from Xilinx, based on the ARM Cortex-A9 processor and FPGA. The ZYNQ series chips combine the advantages of traditional processors and programmable logic, providing high-performance, low-power, programmable, and scalable processing capabilities. The Cortex-A9 processor provides high-performance application processing capabilities, while the programmable logic (FPGA) offers high flexibility and scalability, allowing for optimization and adaptation to different application scenarios. ZYNQ chips are also equipped with various interfaces and peripherals, such as PCIe, USB, Ethernet, and SD cards, facilitating connection to various peripheral devices. ZYNQ chips are primarily used in high-performance, low-power, real-time computing and signal processing fields, and are widely applied in machine vision, drones, industrial control, and network communications. The ZYNQ series also offers different levels of models to meet the needs of various application scenarios.

[0005] Existing hardware acceleration solutions for Transformers typically employ HLS (High-Level Synthesis) for neural network hardware deployment. Part of this is implemented in software on the software processor (PS part, mainly Arm cores and CPU cores), while the other part is implemented in hardware within the programmable logic unit (PL part). Because this approach involves both software and hardware resources, and due to the inherent advantage of hardware resources—that their computational speed is far greater than that of software resources—a mismatch in software and hardware execution time occurs within the entire system. This leads to frequent idle and waiting states for hardware resources. Therefore, this approach is suitable for scenarios with a single, fixed model, certain constraints on time, power consumption, and area, where ensuring the rational use of system software and hardware is crucial.

[0006] Patent application number [CN202110807193.1], entitled "Parallel Acceleration Processing Method of VGG16 Network Based on ZYNQ Platform," provides a parallel acceleration processing method for VGG16 network based on the ZYNQ platform. This method deploys a convolutional neural network on a ZYNQ development board, utilizes convolutional computation modules to extract feature images, and processes the extracted information through pooling and fully connected layers to ultimately classify the input feature images. However, as research progresses, the number of layers in convolutional neural networks, represented by VGG, is increasing. Only by stacking computational layers can better classification results be achieved. However, current hardware resources are clearly insufficient to support the unlimited increase in computational layers. Therefore, convolutional neural networks suffer from drawbacks such as high hardware resource consumption and low accuracy in classifying input feature maps. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention aims to provide a hardware accelerator based on the VisionTransformer neural network and its implementation method. By deploying the VisionTransformer neural network on the ZYNQ development platform, adopting a parallel method with multiple input and multiple output channels, and optimizing the VisionTransformer neural network structure, it has the characteristics of fast computing speed, low hardware resource consumption, and high feature map recognition accuracy, thus enabling efficient completion of image classification tasks.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A hardware accelerator based on the Vision Transformer neural network deploys the Vision Transformer neural network on the ZYNQ development platform, which includes a PS end and a PL end.

[0010] The PS terminal includes:

[0011] ARM processor: responsible for reading and writing to the off-chip SD card, driving the hardware IP on the PL side, and preprocessing the input image, including slice encoding and pixel format conversion;

[0012] On-chip cache unit: used to store computational data from the PS or PL side so that the ARM processor can read the data;

[0013] The PL terminal includes:

[0014] Data input module: The ARM processor sends the input data of the feature image in the DDR memory to the PL terminal for calculation through the input buffer via the AXI high-speed bus, and the weight data of the feature image is sent to the PL terminal for calculation through the weight buffer;

[0015] Input buffer: Temporarily stores input data;

[0016] Weight caching: Temporarily storing weight data;

[0017] The logic computation unit includes processing the input data in the input buffer and the weight data in the weight buffer;

[0018] Output buffer: Temporarily stores the calculation results of the logical calculation unit;

[0019] Data output module: The ARM processor transmits the data calculated at the PL end to the PS end through the AXI high-speed bus.

[0020] The slice encoding involves slicing the input feature image and then encoding the resulting sliced ​​images.

[0021] Pixel format conversion: Convert RGB pixel format images to BRG;

[0022] Convolutional slicing module: slices a two-dimensional sequence of images into a one-dimensional arrangement of images;

[0023] LayNorm layer: performs regularization processing on the input data;

[0024] SoftMax layer: Normalizes the input data;

[0025] Multi-head multiplication QK: Multiplying the input Q matrix and the K matrix;

[0026] Multi-head multiplication QKV: Self-attention computation of the Q matrix, K matrix, and V matrix to obtain the self-attention result;

[0027] 2D fully connected layer: performs linear transformation on the feature image;

[0028] 2D addition: performing addition operations on matrices and vectors in a two-dimensional plane;

[0029] Gelu activation layer: Performs activation calculations on the input data using Gelu activation functions;

[0030] 3D addition: performing addition operations on matrices and vectors in three-dimensional space;

[0031] Maximum / minimum extraction layer: Extracts the maximum calculated result.

[0032] The data output module can adopt either a data aggregation mode or a data distribution mode:

[0033] The data aggregation mode is as follows: the data calculated at the PL end is written sequentially to DDR via multiple output channels using direct memory DMA and the AXI bus through the output buffer. The cumulative BRAM resources consumed are calculated as follows:

[0034]

[0035] Among them, Buffer out It outputs the size of a set of data, C out It is the number of output channels, W data This refers to the data width; 36Kb is the storage size of a single BRAM. rout The calculation formula depends on the parameters of different layers:

[0036] Buffer out ≥C

[0037] Where C and S are the width and length of the output data of this layer, respectively;

[0038] The data distribution mode is as follows: data read from the DDR memory is distributed to the input buffer and weight buffer in a row-oriented manner via direct memory DMA and AXI bus through different data input channels.

[0039] The cumulative BRAM resources consumed are calculated as follows:

[0040]

[0041] Among them, Buffer in It is the size of the input set of data, C in For the number of input channels, W data This refers to the data width; 36Kb is the storage size of a single BRAM. Buffer in The specific calculation method depends on the parameters of different layers, and the formula is as follows.

[0042] Buffer in ≥C×S-C+S

[0043] Where C and S are the width and length of the output data of this layer, respectively.

[0044] A hardware acceleration method based on Vision Transformer neural network includes the following steps:

[0045] Step 1: The ARM processor on the PS side obtains the feature image from the SD card and stores it in the DDR memory;

[0046] Step 2: The ARM processor uses the AXI high-speed bus to distribute the data read from the DDR memory into the input buffer and weight buffer at the PL end in row-order.

[0047] Step 3: The ARM processor reads the feature images obtained in Step 1 and Step 2 through the AXI high-speed bus and preprocesses the feature images, including pixel conversion, slicing, slice encoding, adding a classification header, and adding position encoding. After processing, the images are input into the on-chip cache unit at the PS end.

[0048] Step 4: The ARM processor sends the preprocessed data obtained in Step 3 to the input buffer and weight buffer of the PL terminal. At the same time, it allocates storage space for the data input module and weight input module of the PL terminal and performs read and write operations to complete the calling of the PL terminal hardware IP and data transmission.

[0049] Step 5: Configure the hardware IP on the PL side using the ARM processor, and control the direct memory DMA to read and write to the DDR memory.

[0050] Step 6: According to the Vision Transformer neural network algorithm calculation process, repeat steps 2-5 to obtain the final calculation result, and save the calculation result to the output buffer of the PL end;

[0051] Step 7: The data output module writes the data in the output buffer into the DDR memory in the previously arranged order via the AXI high-speed bus using direct memory DMA;

[0052] Step 8: The ARM processor retrieves the data from the DDR memory, performs probability calculations, transmits the results to the PC, and uses the PetaLinux operating system to port the hardware accelerator system of the Transformer neural network.

[0053] Step 9: Obtain the prediction results from the PC.

[0054] In step 5, the memory DMA performs read and write operations on data via the AXI bus.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0056] 1. This invention utilizes the ZYNQ development platform to implement image classification tasks and proposes an FPGA hardware acceleration method based on the VisionTransformer neural network, which features high prediction accuracy, fast computation speed, and low power consumption.

[0057] 2. The ARM processor in this invention configures and calls hardware resources by controlling the AXI high-speed bus, which coordinates the timing consistency of hardware IPs and improves computing efficiency and accuracy.

[0058] 3. The VisionTransformer neural network hardware accelerator proposed in this invention re-divides hardware and software resources according to the computational characteristics of the algorithm and the advantages and disadvantages of the PS and PL ends, thereby improving the overall computational efficiency and accuracy of the algorithm and having good portability.

[0059] 4. This invention deploys a VisionTransformer neural network on the ZYNQ development platform, adopts a parallel method with multiple input and multiple output channels, and optimizes the VisionTransformer neural network structure. Compared with the VGG16 convolutional neural network, it improves the data processing speed, performance, and versatility of the neural network in image classification tasks while achieving lower resource consumption and power consumption, thus enhancing acceleration performance and efficiency. Attached Figure Description

[0060] Figure 1 This is the top-level architecture diagram of the VisionTransformer hardware accelerator of this invention.

[0061] Figure 2 This is a schematic diagram of the data aggregation method of the present invention.

[0062] Figure 3 This is a diagram illustrating the data distribution method of the present invention.

[0063] Figure 4 A schematic diagram of the ARM processor and SD card read / write process of this invention.

[0064] Figure 5 The image to be predicted is the VisionTransformer accelerator in this invention example.

[0065] Figure 6 This is a schematic diagram of the data input module of the present invention.

[0066] Figure 7 This is a schematic diagram of the image preprocessing process of the present invention.

[0067] Figure 8 This is a schematic diagram of the data input module of the present invention.

[0068] Figure 9 This is a prediction result diagram of the Vision Transformer accelerator in an example of the present invention.

[0069] Figure 10 This is a diagram of the convolutional slicing architecture of the present invention.

[0070] Figure 11 This is a diagram showing the effect of convolution slicing in this invention.

[0071] Figure 12 This is a schematic diagram of the application flow under PetaLinux in this invention. Detailed Implementation

[0072] The working principle of the invention will be described in detail below with reference to the accompanying drawings.

[0073] See Figure 1 A hardware accelerator based on the Vision Transformer neural network deploys the Vision Transformer neural network on the ZYNQ development platform. The ZYNQ development platform includes a PS end and a PL end, which transmit bidirectionally through an AXI bus.

[0074] The PS terminal includes:

[0075] ARM processor: responsible for reading and writing to the off-chip SD card, driving the hardware IP on the PL side, and preprocessing the input image, including slice encoding and pixel format conversion;

[0076] On-chip cache unit: used to store computational data from the PS or PL side for the ARM processor to read;

[0077] The PL terminal includes:

[0078] Data input module: The ARM processor sends the input data of the feature image in the DDR memory to the PL terminal for calculation through the input buffer via the AXI high-speed bus, and the weight data of the feature image is sent to the PL terminal for calculation through the weight buffer;

[0079] Input buffer: Temporarily stores input data;

[0080] Weight caching: Temporarily storing weight data;

[0081] Logical computation unit: processes the input data in the input buffer and the weight data in the weight buffer;

[0082] Output buffer: Temporarily stores the calculation results of the logical calculation unit;

[0083] Data output module: The ARM processor transmits the data calculated at the PL end to the PS end through the AXI high-speed bus.

[0084] Slice encoding: The input feature image is sliced, and the resulting sliced ​​images are encoded.

[0085] Pixel format conversion: Convert RGB pixel format images to BRG;

[0086] Convolutional slicing module: slices a two-dimensional sequence of images into a one-dimensional arrangement of images;

[0087] LayNorm layer: performs regularization processing on the input data;

[0088] SoftMax layer: Normalizes the input data;

[0089] Multi-head multiplication QK: Multiplying the input Q matrix and the K matrix;

[0090] Multi-head multiplication QKV: Self-attention computation of the Q matrix, K matrix, and V matrix to obtain the self-attention result;

[0091] 2D fully connected layer: performs linear transformation on the feature image;

[0092] 2D addition: performing addition operations on matrices and vectors in a two-dimensional plane;

[0093] Gelu activation layer: Performs activation calculations on the input data using Gelu activation functions;

[0094] 3D addition: performing addition operations on matrices and vectors in three-dimensional space;

[0095] Maximum / minimum extraction layer: Extracts the maximum calculated result.

[0096] The data output module can adopt either a data aggregation mode or a data distribution mode;

[0097] Reference Figure 2 The data aggregation mode is as follows: the data calculated at the PL end is written sequentially into DDR via multiple output channels using direct memory DMA and the AXI bus through the output buffer. The cumulative BRAM resources consumed are calculated as follows:

[0098]

[0099] Among them, Buffer out It outputs the size of a set of data, C out It is the number of output channels, W data This refers to the data width; 36Kb is the storage size of a single BRAM. out The calculation formula depends on the parameters of different layers:

[0100] Buffer out ≥C

[0101] Where C and S are the width and length of the output data of this layer, respectively;

[0102] See Figure 3 The data distribution mode is as follows: data read from the DDR memory is distributed to the input buffer and weight buffer in a row-oriented manner via direct memory DMA and AXI bus through different data input channels.

[0103] The cumulative BRAM resources consumed are calculated as follows:

[0104]

[0105] Among them, Buffer in It is the size of the input set of data, C in For the number of input channels, W data This refers to the data width; 36Kb is the storage size of a single BRAM. Buffer in The specific calculation method depends on the parameters of different layers, and the formula is as follows.

[0106] Buffer in ≥C×S-C+S

[0107] Where C and S are the width and length of the output data of this layer, respectively.

[0108] The accelerator operates as follows: The ARM processor on the PS side acquires the feature image from the SD card and saves it to the DDR memory. The ARM processor then sends the feature image data to the corresponding preprocessing module on the PL side via the AXI bus for preprocessing. The preprocessed image is then returned to the on-chip cache on the PS side. Simultaneously, the logic computing unit on the PL side is reset. The ARM processor then sends the preprocessed data to the input cache and weight cache on the PL side via the AXI bus. According to the Vision Transformer neural network algorithm, the logic computing unit IP on the PL side is called and the data is transmitted sequentially. Finally, the calculated data is saved to the DDR memory. The ARM processor retrieves the calculation results from the DDR memory and completes the probability calculation. Finally, the calculation results are transmitted to the PC.

[0109] A hardware acceleration method based on Vision Transformer neural network includes the following steps:

[0110] See Figure 4 Step 1: The ARM processor on the PS side obtains the input data and weight data of the feature image from the SD card and stores them in the DDR memory; the feature image refers to... Figure 5 ;

[0111] See Figure 6 Step 2: The ARM processor sends the data in DDR to the data input module at the PL end through the AXI high-speed bus. Under the control of the controller, the data is distributed to the data input interfaces 1-4 in the order of row arrangement. Under the control of the controller, the data is also transferred to the intermediate buffer 1-4 for storage. The controller adopts the data distribution mode. The buffer refers to the input module of the next level.

[0112] See Figure 7Step 3: The ARM processor reads the feature images obtained in Step 1 and Step 2 through the AXI high-speed bus and preprocesses the feature images, including converting RGB format pixels into BRG format images, converting two-dimensional feature maps into one-dimensional sequence vectors, slicing and encoding the sliced ​​one-dimensional sequence vectors, adding a classification head to each one-dimensional sequence vector, and adding position encoding operations to the one-dimensional sequence vectors with added classification heads.

[0113] Step 4: The ARM processor sends the preprocessed data obtained in Step 3 to the input buffer and weight buffer of the PL end of the FPGA. At the same time, it allocates the storage space of the data input module and weight input module of the PL end and performs read and write operations to complete the calling of the PL end hardware IP and data transmission.

[0114] Step 5: Configure the hardware IP resources of the PL side using the ARM processor, and control the direct memory DMA to read and write to the DDR memory. Configure the PL side on the PS side through the AXI bus, and generate specific signals to control the direct memory DMA of the PL side to perform read and write operations, so as to control the operation of each computing module on the PL side; the memory DMA performs data read and write operations through the AXI bus.

[0115] Step 6: According to the VisionTransformer neural network algorithm calculation process, repeat steps 2-5 to obtain the final calculation result, and save the calculation result to the output buffer of the PL end;

[0116] refer to Figure 8 Step 7: The data output module obtains data from the output buffer at the PL end, and transmits the obtained data to intermediate buffer 1 to intermediate buffer 4 for storage under the control of the controller. Then, under the control of the controller, it transmits the data to output interface 1 to data output interface 4. The controller adopts a data aggregation mode so that the data in the four output channels are transmitted to the DDR in the order previously arranged. Buffer refers to the input data at the PL end.

[0117] Step 8: The ARM processor retrieves the data from the DDR memory in Step 7 and completes the probability calculation. It then transmits the result of the probability calculation to the PC and uses the PetaLinux operating system to port the hardware accelerator system of the Transformer neural network.

[0118] See Figure 9 Step 9: Obtain the prediction results from the PC. The three results with the highest probability of input image are house finch (94.98%), brambling (3.56%), and goldfinch (0.18%).

[0119] See Figure 10 , Figure 11 In step 3, the two-dimensional feature map is converted into a one-dimensional sequence vector. The specific process is as follows:

[0120] Feature data is transmitted to the input buffer through the data input channel, and weight data is transmitted to the weight buffer through the weight input channel. Then, the feature data and weight data are transmitted to the multiply-accumulate array, where convolution operations are performed.

[0121] Data input channels 1, 2, 3, and 4 represent the input data of different channels; weight input channels 1, 2, 3, and 4 represent the weights of different convolutional kernels within different convolutional kernel groups; data output channels 1, 2, 3, and 4 represent the output data of different output channels; PE represents processing unit. Each input data participates in the calculation of N PE processing units simultaneously, and each output data corresponds to only one PE processing unit. No other calculation units are required during the calculation process.

[0122] The size of the convolution kernel is equal to the stride of the convolution kernel. This allows the convolution kernel to sequentially cut out a small piece of the image each time it slides. The cut image data is then stored in the cache according to the order of the cut pieces, which realizes the function of converting a two-dimensional image into a one-dimensional sequence. At the same time, the cut one-dimensional sequence image data is transmitted back to the on-chip cache unit of the PS through the AXI bus, and then image preprocessing is performed. The processed image data is used as the input part of the subsequent calculation module.

[0123] kernel size = stride

[0124] The calculation result is that a 14*14 pixel image is sliced, a classification header is added, and a position encoding operation is performed, turning it into a 197*768 matrix data.

[0125] See Figure 12The application design and operation flowchart under the PetaLinux system: Starting from the program, each step is executed according to the flowchart shown in the figure to finally obtain the result; During the testing phase, the Xilinx ZCU102 development board is used to access the ZYNQ platform through the PC and implement the accelerator function, compile and start the VisionTransformer embedded system; The application design and operation flowchart under the PetaLinux system includes: Initialization part: program start, hardware driver initialization, all parameters initialization, loading feature images; Preprocessing and PatchEmbedding part: slice encoding and data input, pixel format conversion, feature image slicing, slice encoding, adding classification head, adding position encoding; Transformer encoder: Transformer data input, Norm layer, Q encoding, K encoding, V encoding, multi-head self-attention algorithm QK, SoftMax, multi-head self-attention algorithm QKV, fully connected layer, 2D addition, Norm layer, fully connected layer, Gelu, fully connected layer, 2D addition; MLP: Norm layer, fully connected layer, SoftMax, extract results.

Claims

1. A hardware accelerator based on the VisionTransformer neural network, characterized in that, The VisionTransformer neural network is deployed on the ZYNQ development platform, which includes a PS end and a PL end. The PS terminal includes: ARM processor: responsible for reading and writing to the off-chip SD card, driving the hardware IP on the PL side, and preprocessing the input image, including slice encoding and pixel format conversion; On-chip cache unit: used to store computational data from the PS or PL side so that the ARM processor can read the data; The PL terminal includes: Data input module: The ARM processor sends the input data of the feature image in the DDR memory to the PL terminal for calculation through the input buffer via the AXI high-speed bus, and the weight data of the feature image is sent to the PL terminal for calculation through the weight buffer; Input buffer: Temporarily stores input data; Weight caching: Temporarily storing weight data; Logical computation unit: processes the input data in the input buffer and the weight data in the weight buffer; Output buffer: Temporarily stores the calculation results of the logical calculation unit; Data output module: The ARM processor transmits the data calculated at the PL end to the PS end via the AXI high-speed bus; The aforementioned logical calculation unit specifically includes: Slice encoding: The input feature image is sliced, and the resulting sliced ​​images are encoded. Pixel format conversion: Convert RGB pixel format images to BRG; Convolutional slicing module: slices a two-dimensional sequence of images into a one-dimensional arrangement of images; LayNorm layer: performs regularization processing on the input data; SoftMax layer: Normalizes the input data; Multi-head multiplication QK: Multiplying the input Q matrix and the K matrix; Multi-head multiplication QKV: Self-attention computation of the Q matrix, K matrix, and V matrix to obtain the self-attention result; 2D fully connected layer: performs linear transformation on the feature image; 2D addition: performing addition operations on matrices and vectors in a two-dimensional plane; Gelu activation layer: Performs activation calculations on the input data using Gelu activation functions; 3D addition: performing addition operations on matrices and vectors in three-dimensional space; Maximum / minimum extraction layer: Extracts the maximum calculated result; The data output module can adopt either a data aggregation mode or a data distribution mode: The data aggregation mode is as follows: the data calculated at the PL end is written sequentially to DDR via multiple output channels using direct memory DMA and the AXI bus through the output buffer. The cumulative BRAM resources consumed are calculated as follows: in, It outputs the size of a set of data. It is the number of output channels. This refers to the data width; 36Kb is the storage size of a single BRAM. The calculation formula depends on the parameters of different layers: Where C and S are the width and length of the output data of this layer, respectively; The data distribution mode is as follows: data read from the DDR memory is distributed to the input buffer and weight buffer in a row-oriented manner via direct memory DMA and AXI bus through different data input channels. The cumulative BRAM resources consumed are calculated as follows: in, It is the size of the input set of data. Input the number of channels. This refers to the data width; 36Kb is the storage size of a single BRAM. The specific calculation method depends on the parameters of different layers, and the formula is as follows. Where C and S are the width and length of the output data of this layer, respectively.

2. A hardware acceleration method based on Vision Transformer neural network, characterized in that, Includes the following steps: Step 1: The ARM processor on the PS side obtains the feature image from the SD card and stores it in the DDR memory; Step 2: The ARM processor uses the AXI high-speed bus to distribute the data read from the DDR memory into the input buffer and weight buffer at the PL end in row-order. Step 3: The ARM processor reads the feature images obtained in Step 1 and Step 2 through the AXI high-speed bus and preprocesses the feature images, including pixel conversion, slicing, slice encoding, adding a classification header, and adding position encoding. After processing, the images are input into the on-chip cache unit at the PS end. Step 4: The ARM processor sends the preprocessed data obtained in Step 3 to the input buffer and weight buffer of the PL terminal. At the same time, it allocates storage space for the data input module and weight input module of the PL terminal and performs read and write operations to complete the calling of the PL terminal hardware IP and data transmission. Step 5: Configure the hardware IP on the PL side using the ARM processor, and control the direct memory DMA to read and write to the DDR memory. Step 6: According to the VisionTransformer neural network algorithm calculation process, repeat steps 2-5 to obtain the final calculation result, and save the calculation result to the output buffer of the PL end; Step 7: The data output module writes the data in the output buffer into the DDR memory in the previously arranged order via the AXI high-speed bus using direct memory DMA; Step 8: The ARM processor retrieves the data from the DDR memory, performs probability calculations, transmits the results to the PC, and uses the PetaLinux operating system to port the hardware accelerator system of the Transformer neural network. Step 9: Obtain the prediction results from the PC.

3. The hardware acceleration method based on the Vision Transformer neural network according to claim 2, characterized in that, In step 5, the memory DMA performs read and write operations on data via the AXI bus.

Images