A method for designing and deploying an FPGA-based VGG-16 convolution accelerator

Abstract

The application discloses a kind of VGG-16 convolution accelerator design and deployment method based on FPGA, comprising the following steps: deployment aspect: quantization convolution neural network model, in the case where part precision is sacrificed, the quantization of entire network is 8bit data type, and is reduced to 1 / 4 of original network size;According to the common characteristics of the input feature map size of each convolution layer of VGG-16 model, the input size of the convolution accelerator is fixed, and the convolution accelerator is repeatedly called until the current convolution neural network calculation is completed;Convolution accelerator design aspect: increase the input feature map blocking parallel computation on convolution accelerator on convolution accelerator, realize the way of flow parallel in convolution process.The application is the blocking strategy of VGG-16 convolution accelerator design, and the calculation throughput of convolution accelerator per unit time is improved.

Description

Technical Field

[0001] This invention belongs to the field of CNN acceleration computing based on FPGA, and in particular relates to the design and deployment method of a VGG-16 convolution accelerator based on FPGA. Background Technology

[0002] Convolutional Neural Networks (CNNs) are deep learning models particularly well-suited for processing image data. They automatically extract and learn features from images through a combination of convolutional layers, pooling layers, and fully connected layers. When a CNN performs classification tasks, the input image is first processed through multiple convolutional kernels to extract different features. (The parameters of the convolutional kernels are learned from the training data.) The results of the convolution calculations are then processed with a non-linear activation function (such as ReLU) to increase the model's non-linear expressive power. Pooling is then performed after the activation function to reduce data dimensionality, retain important features, and increase the model's robustness. This completes one layer of convolution and pooling operations. Depending on the deployed CNN model, the convolutional kernel and pooling operations are repeated multiple times to extract higher-level, more abstract features layer by layer. Finally, the final feature map is unfolded and input into a fully connected layer for classification.

[0003] VGG-16 is a classic convolutional neural network architecture, renowned for its depth and simple network design. Although not specifically designed for computational electromagnetics and antenna orientation, its deep learning capabilities can be applied to these fields. Some studies have shown that VGG-16 can be used to effectively predict electromagnetic field distributions in complex structures. Through model training, it can quickly obtain high-accuracy predictions without extensive numerical simulations. Furthermore, researchers have used deep learning models like VGG-16, trained on large amounts of antenna design data, to achieve rapid prediction and optimization of antenna performance. However, due to the high power consumption and poor real-time performance of CPUs and GPUs, the deployment of VGG-16 has become a challenge. Summary of the Invention

[0004] To overcome the problem that the VGG-16 network model is too large and has low real-time deployment performance on edge devices, this invention provides a design and deployment method for a VGG-16 convolutional accelerator based on FPGA.

[0005] The objective of this invention is achieved through the following technical solutions, including:

[0006] S100. Pre-train the model parameters of the VGG-16 network, including the weights of the convolutional layers, bias data, and weight data of the fully connected layers;

[0007] S200. Quantize the entire VGG-16 network model and derive the quantized weights, biases, and quantization factors of each layer;

[0008] S300. Given input / output samples, use the High-Level Synthesis (HLS) tool to perform simulation testing and end the quantization process when the given error range is reached.

[0009] Further, S300 includes the following steps:

[0010] S310. Given multiple sets of input / output sample pairs;

[0011] S320. Import the quantized data into a high-level synthesis tool for simulation testing;

[0012] S330. Determine whether the simulation test result error is within the given error range. If yes, end the quantization process; otherwise, re-perform training quantization.

[0013] S400 allocates a data area on the off-chip DDR to store the quantized model data, and allocates a certain address space to store the intermediate data generated during the convolution calculation and the final output results;

[0014] S500. Based on the different sizes of each convolutional layer involved in VGG-16, a general convolutional accelerator is deployed to be repeatedly called until the overall network computation is completed.

[0015] Furthermore, S500 includes the following steps:

[0016] S510. Fix the input feature map size of the general convolution accelerator, ensuring that the input feature map size of all convolutional layers in the VGG-16 network model is greater than or equal to the input feature map size of the general convolution accelerator. If it is greater, call the convolution accelerator multiple times until the convolution calculation of this layer is completed;

[0017] S520. Determine the number of input and output feature map channels of the general convolutional accelerator, making them less than or equal to the number of input and output feature map channels of each convolutional layer in the VGG-16 network model. If the number of channels of the convolutional accelerator is less than the number of channels of a certain convolutional layer, the convolutional accelerator needs to be called multiple times to complete the convolution calculation of this layer;

[0018] S600. Design of a VGG-16 convolutional accelerator based on FPGA;

[0019] Furthermore, S600 includes the following steps:

[0020] S610. Allocate additional storage area on the convolution accelerator to process the feature image of the input buffer in blocks, so that there is no data dependency between blocks;

[0021] S620. The input channels of the convolution accelerator are completely flattened and allocated computing resources to achieve high-performance convolution calculation;

[0022] S630. The output channels of the convolution accelerator are completely flattened out, making full use of the parallelism of the FPGA to complete the convolution calculation;

[0023] S700. The calculation results of each convolution block after being divided on the convolution accelerator are stitched together to form the original output feature image, which is saved to the output buffer and then output to the off-chip DDR through the AXI bus for use in the next layer of convolution calculation;

[0024] S800. Repeat 500 to 700 until the entire network convolution calculation is completed.

[0025] The beneficial effects of this invention are as follows: This invention proposes a deployment method for VGG-16 that facilitates deployment. Specifically, in each convolutional layer of VGG-16, the larger portions of the input feature map are divided into blocks, and the convolutional accelerator is repeatedly invoked to complete the computation. Furthermore, a VGG-16 convolutional accelerator is designed, which divides the input feature image to the convolutional accelerator into blocks and executes the convolutional computation of each block in parallel. The convolutional accelerator designed in this invention increases the parallelism of convolutional computation by implementing an on-chip block strategy, fully utilizing the parallelism of the FPGA and improving the utilization rate of the DSP computing resources. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the overall deployment process of the present invention.

[0027] Figure 2 This is a schematic diagram of the accelerator data stream structure provided by the present invention.

[0028] Figure 3 This is a schematic diagram of the accelerator provided by the present invention.

[0029] Figure 4 This is a schematic diagram of the feature map being cyclically tiled according to the present invention.

[0030] Figure 5 This is a schematic diagram of the block division method for the convolution accelerator provided by the present invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The embodiments described are only one example of the present invention and not all examples.

[0032] The deployment process of the FPGA-based VGG-16 convolution accelerator designed in this embodiment is as follows: Figure 1 As shown, steps S100 to S400 are included.

[0033] Step S100. Pre-train the model parameters of the VGG-16 network and obtain the weights, bias data of the convolutional layers and the weight data of the fully connected layers.

[0034] Specifically, the model parameters include the convolution kernel parameters and fully connected layer parameters in each convolutional layer.

[0035] S200. Quantize the entire VGG-16 network model and export the quantized weights, biases, and quantization factors of each layer.

[0036] The specific quantization method is as follows: quantize the trained network model into 8-bit data to generate the input feature map quantization factor. Weight quantization factor Bias quantization factor Output feature map quantization factor (input feature map of the adjacent next convolutional layer) A n+1 It is the input to the next convolutional layer, A n W is the input of the current convolutional layer. n B is the current convolutional layer weight. n This is the current convolutional layer bias. This represents convolution. Its specific quantization formula is as follows:

[0037]

[0038] Further simplified to

[0039]

[0040] The quantization factors generated after quantizing the neural network model in this invention can all be quantized to an exponent of 2; therefore, the above formula can be simplified to...

[0041]

[0042] Therefore, when choosing to deploy VGG-16 on an FPGA, quantization can be completed simply through shift operations, which is more conducive to hardware deployment.

[0043] In some embodiments, S300 includes the following steps:

[0044] Step S310. Given multiple sets of input / output sample pairs.

[0045] Step S320. Import the quantized data into the high-level synthesis and perform simulation testing.

[0046] Step S330. Determine whether the simulation test result error is within the given error range. If yes, end the quantization process; otherwise, re-perform training quantization.

[0047] Step S400. Allocate a data area on the off-chip DDR to store the quantized model data, and allocate a certain address space to store the intermediate data generated during the convolution calculation and the final output result.

[0048] The specific allocation of data areas on the DDR requires the use of VITIS software. The convolution accelerator interacts with the DDR for data exchange, such as... Figure 2 As shown, when the convolution accelerator is enabled, the corresponding inputs and weights are first read from the additional memory (DDR) at the PS end into the buffer at the PL end. Then, convolution begins and the calculation result is stored in the output buffer. Finally, the output result is saved in DDR through control at the PS end.

[0049] Step S500. Since VGG-16 involves different sizes of each convolutional layer, the general convolutional accelerator is repeatedly called until the overall network computation is completed.

[0050] In some embodiments, such as Figure 3 As shown, S500 includes the following steps:

[0051] Step S510. Fix the input feature map size of the general convolution accelerator, ensuring that the input feature map size of all convolutional layers in the VGG-16 network model is greater than or equal to the input feature map size of the general convolution accelerator. If it is greater, call the convolution accelerator multiple times until the convolution calculation of this layer is completed.

[0052] Specifically caused by Figure 4 As shown, VGG16 consists of 13 convolutional layers and 3 fully connected layers. All convolutional layers use small 3×3 kernels with a stride of 1 and padding of 2, and each convolutional layer is followed by a ReLU activation function to increase the network's non-linearity. The 13 convolutional layers of VGG16 have five different input feature maps: 224×224, 112×112, 56×56, 28×28, and 14×14. To enable the convolutional accelerator to complete the computation of all convolutional layers without changing the output, this embodiment defines the input of the convolutional accelerator as a 16×16 pixel image. For convolutional layers with input feature maps larger than the input of the convolutional accelerator, a block-based strategy is adopted, dividing them into multiple 16×16 blocks of input feature maps.

[0053] After segmentation, there is no data dependency between the convolution results of adjacent spatial blocks. For simplicity, this embodiment uses... Figure 3Let's illustrate the principle of block-based convolution with an example. In this example, the input tensor contains three 8×8 feature maps, the convolution filter size is 3×3×3, and the convolution stride and padding are set to 1. The input feature maps are spatially divided into four blocks, each of which is a 4×4×3 three-dimensional tensor. In the original convolution, for an input feature map of size [H, W], the output [R, C] can be determined as follows:

[0054]

[0055] In this example, the output feature map size is [8,8]. In block convolution, each input block with a feature map of [5,5] is convolved with a 3×3 kernel to obtain an output block of [3,3]. Concatenating four [3,3] output blocks together results in an output block of [6,6], which is inconsistent with the original convolution output block size. To solve this problem, this embodiment pads adjacent blocks before convolution. That is, the [5,5] input block is symmetrically padded to become a [6,6] input block, where the central kernel map comes from the original feature map, such as... Figure 3 As shown. In this way, four [4,4] output blocks can be computed independently and then connected to form an [8,8] output feature map.

[0056] Step S520. Determine the number of input and output feature map channels of the general convolutional accelerator, making them less than or equal to the number of input and output feature map channels of each convolutional layer in the VGG-16 network model. If the number of channels of the convolutional accelerator is less than the number of channels of a certain convolutional layer, the convolutional accelerator needs to be called multiple times to complete the convolution calculation of this layer.

[0057] In some embodiments, such as Figure 5 As shown, step S600 includes the following steps:

[0058] Step S610. Allocate additional storage areas to the input feature image of the convolution accelerator, perform block partitioning, and enable multiple blocks to perform convolution in parallel at the same time.

[0059] Specifically, according to the formula for calculating the output feature map using two-dimensional convolution, in the VGG-16 model, padding, kernel size, and stride are all fixed, and the output height is only related to the input height. In this embodiment, the input feature map is divided into blocks of 4 rows each, resulting in four 4×14 feature images. The first image contains rows 1-4 of the input feature image, the second image contains rows 5-8, the third image contains rows 9-12, and the fourth image contains rows 13-16. To avoid data dependencies between adjacent blocks, the data dependency portion between adjacent blocks is copied and stored in an additional feature image. Specifically, this invention copies rows 3-6 of the data dependency portion between the first and second images as a separate block; rows 7-10 of the data dependency portion between the second and third images as a separate block; and rows 11-14 of the data dependency portion between the third and fourth images as a separate block. Therefore, the input feature image of the general convolution accelerator is divided into 7 feature images of size 4×14 with no data dependency between adjacent blocks. During convolution calculation, the 7 input feature image blocks can be fully expanded and calculated in parallel, achieving higher computational throughput.

[0060] Step S620. Fully flatten out the input channels of the convolution accelerator, allocate computing resources, and achieve high-performance convolution computation.

[0061] Step S630. Fully flatten the output channels of the convolution accelerator to fully utilize the parallelism of the FPGA to complete the convolution calculation.

[0062] Step S700. The calculation results of each convolution block after being divided on the convolution accelerator are stitched together to form the original output feature image, which is then output to the off-chip DDR via the AXI bus for use in the next layer of convolution calculation;

[0063] Specifically, the block-parallel convolution computation complexity (FLOPs) used in this invention is exactly the same as that of the original convolution, which is different from model compression techniques aimed at reducing computation or network parameters.

[0064] Step S800. Repeat steps S500 to S700 until the convolution calculation of the entire network is completed.

[0065] Specifically, by repeatedly calling the designed convolution accelerator, different input data and weight data are loaded, and finally the convolution calculation of the entire network is completed.

[0066] The above description is merely a preferred embodiment of the invention. It should be understood that the invention is not limited to the forms disclosed herein, nor is it intended to exclude other embodiments. Rather, it can be used in various other combinations, variations, and environments, and can be modified within the scope of the concept described herein by the teachings above or by the technology or knowledge in the relevant field. However, any changes and modifications made by those skilled in the art without departing from the spirit and scope of the invention will fall within the protection scope of the appended claims.

Claims

1. A design and deployment method for a VGG-16 convolutional accelerator based on FPGA, characterized in that, Includes the following steps: S100. Pre-train the model parameters of the VGG-16 network, including the weights of the convolutional layers, bias data, and weight data of the fully connected layers; S200. Quantize the entire VGG-16 network model and derive the quantized weights, biases, and quantization factors of each layer; S300. Given input / output samples, import the quantized model using a high-level synthesis tool, perform simulation testing, and end the quantization process when the given error range is reached; Further, S300 includes the following steps: S310. Given multiple sets of input / output sample pairs; S320. Import the quantized data into a high-level synthesis tool for simulation testing; S330. Determine whether the simulation test result error is within the given error range. If yes, end the quantization process; otherwise, re-perform training quantization. S400 allocates a data area on the off-chip DDR to store the quantized model data, and allocates a certain address space to store the intermediate data generated during the convolution calculation and the final output results; S500. Based on the different sizes of each convolutional layer involved in VGG-16, the general convolutional accelerator is repeatedly called until the overall network computation is completed. The S500 includes the following steps: S510. Fix the input feature map size of the general convolution accelerator to ensure that the input feature map size of all convolutional layers in the VGG-16 network model is greater than or equal to the input feature map size of the general convolution accelerator; if it is greater, call the general convolution accelerator multiple times until the convolution calculation of this layer is completed. S520. Determine the number of input feature map channels and output feature map channels of the general convolutional accelerator, making them less than or equal to the number of input feature map channels and output feature map channels of each convolutional layer in the VGG-16 network model; if the number of channels of the general convolutional accelerator is less than the number of channels of a certain convolutional layer, the convolutional accelerator needs to be called multiple times to complete the convolution calculation of this layer. S600. Implements the convolution calculation design of a convolution accelerator; The S600 includes the following steps: S610. Allocate additional storage area on the general convolution accelerator to process the feature image of the input buffer in blocks, so that there is no data dependency between blocks; S620. The input channels of the general convolution accelerator are completely flattened and distributed to allocate computing resources, so as to achieve high-performance convolution calculation; S630. The output channels of the general convolution accelerator are completely flattened out, making full use of the parallelism of the FPGA to complete the convolution calculation; S700. The calculation results of each convolution block after being divided on the general convolution accelerator are stitched together to form the original output feature image, which is saved to the output buffer and then output to the off-chip DDR through the AXI bus for use in the next layer of convolution calculation; S800. Repeat S500~S700 until the entire network convolution calculation is completed; The general convolutional accelerator has an input feature map size of 16×16 and an output feature map size of 14×14. It employs a row-based block strategy, dividing the input feature map into blocks of 4 rows each, resulting in four 4×14 feature images. The first image contains rows 1-4, the second contains rows 5-8, the third contains rows 9-12, and the fourth contains rows 13-16. To avoid data dependencies between adjacent blocks, the data dependencies between adjacent blocks are copied and stored in an additional feature image. The first image is copied... The input feature map in rows 3-6 of the data dependency portion between the second and third images is a single block; the input feature map in rows 7-10 of the data dependency portion between the second and third images is a single block; the input feature map in rows 11-14 of the data dependency portion between the third and fourth images is a single block; thus, the input feature map of the general convolution accelerator is divided into 7 feature maps of size 4×14 with no data dependency between adjacent blocks; during convolution calculation, the 7 input feature map blocks can be fully parallelized for computation, achieving higher computational throughput.

Images