Parallel acceleration processing method of VGG16 network based on ZYNQ platform

Abstract

The application belongs to the technical field of artificial intelligence and FPGA design, and specifically discloses a VGG16 network parallel acceleration processing method based on a ZYNQ platform; the application reduces the huge calculation amount caused by input and output data, weight data and bias through fixed-point quantization of data, thereby reducing power consumption, reducing calculation amount and solving the problem of limited on-chip resources; through the resource parallel mode, on the one hand, the problem of parallel calculation of data correlation between convolutional neural network layers is avoided, and on the other hand, the demand for hardware circuit resources is reduced; the application designs a hardware acceleration architecture capable of realizing high parallelism of the VGG16 network through the ZYNQ platform, thereby improving acceleration performance and acceleration efficiency on the basis of realizing small resource consumption and low power consumption.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and FPGA design technology, and particularly to radar imaging technology, specifically a parallel acceleration processing method for VGG16 networks based on the ZYNQ platform. Background Technology

[0002] With the rapid development of artificial intelligence technology in recent years, machine learning algorithms have become a hot research topic for researchers. Among them, convolutional neural network (CNN) algorithms have significant application value and research significance in fields such as image recognition and classification, speech analysis and retrieval, and object detection and monitoring. VGGNet, in particular, has a very simple structure, using the same kernel size (3x3) and max-pooling size (2x2) throughout the entire network. While maintaining the same receptive field, it increases the network depth, thus improving the performance of the neural network to some extent. Initially, CNNs were implemented through software programming; however, with the development of CNN research, researchers have gradually begun to implement them using hardware methods.

[0003] The ZYNQ platform is a system-on-a-chip (SoC) product launched by Xilinx that supports hardware and software co-design. Development based on the ZYNQ platform allows us to benefit from both the rich resources of ARM and the scalability and flexibility of FPGA.

[0004] In recent years, with the significant improvement in computer computing power, the research on convolutional neural networks has also developed rapidly. The complexity, computational load, and data volume of convolutional neural networks have also increased, which means that the design of convolutional neural networks will be constrained by various factors such as hardware computing power, hardware resource utilization, and energy consumption. Summary of the Invention

[0005] To address the problems existing in the prior art, the purpose of this invention is to provide a parallel acceleration processing method for VGG16 networks based on the ZYNQ platform, which improves the data processing speed and performance of VGG16 networks while achieving lower resource consumption and power consumption, thereby enhancing acceleration performance and efficiency.

[0006] To achieve the above objectives, the present invention employs the following technical solutions.

[0007] (I) A parallel acceleration method for VGG16 networks based on the ZYNQ platform, including the following steps:

[0008] Step 1: The ZYNQ PS terminal obtains the feature map data and weight data of the dataset to be processed through the host computer, and stores them in the DDR storage module. Then, it drives the AXI DMA module to load the feature data and weight data into the cache module.

[0009] Step 2: The ARM controller transmits the feature data and weight data of the cache module to the convolution module at the PL end of the ZYNQ for parallel convolution operation, and stores the operation result in the cache module.

[0010] Step 3: The data is then quantized and pooled in the post-processing module. After completing one layer of computation, the PS end of ZYNQ is reconfigured to provide the register information required for the next layer of convolutional module.

[0011] Step 4: Repeat steps 1-3 until all layer operations are completed. Then, the final result is transferred to DDR for storage via AXI_DMA. The ZYNQ PS terminal transmits the result to the host computer via serial port.

[0012] Furthermore, the convolution module performs parallel convolution operations, specifically including:

[0013] A high-parallelism PE array is used to perform parallel operation of convolutions with multiple input channels and multiple output channels;

[0014] The height of the input feature map and the direction of the output channel are optimized by data slicing. The total memory access requirement is reduced by reusing the input feature data and weight data.

[0015] Furthermore, the parallel operation of convolution with multiple input and output channels using a highly parallel PE array specifically includes:

[0016] Parallel convolution of multiple input channels in one dimension, i.e., input features Figure 1 The operation involves multiplying and accumulating the data from multiple input channels of a feature point and the corresponding channel data from a convolutional kernel. On the other dimension, the operation involves parallel convolution of multiple output channels, i.e., multiplying and accumulating the data from all input feature maps and multiple convolutional kernels.

[0017] Furthermore, the data slicing optimization processing for the height of the input feature map and the direction of the output channel is specifically as follows: for several layers of the VGG16 network input, a slicing optimization scheme is adopted, which first slices the output channel direction and then slices the height direction of the input feature map. This reduces the total memory access requirement by maximizing the reuse of input feature data.

[0018] For several layers of the VGG16 network near the output end, a slicing optimization scheme is adopted, which first slices the input feature map in the height direction and then slices it in the output channel direction, so as to reduce the total memory access requirement through weight data reuse.

[0019] Furthermore, the slicing optimization scheme, which first slices the output channel directions and then slices the input feature map in the height direction, is as follows: First, calculate the convolution of all output channel directions with the first slice along the height direction; then calculate the convolution of all output channel directions with the second slice along the height direction; and so on, until the convolution of all output channel directions with the last slice along the height direction is calculated. Finally, the convolution results of different slices along the height direction are concatenated in order.

[0020] Furthermore, the slicing optimization scheme, which first slices the input feature map in the height direction and then slices it in the output channel direction, is as follows: First, calculate the convolution of all slices in the height direction with the first output channel, then switch to the second output channel slice and continue to complete the convolution operation with all slices in the height direction, and continue to switch the output channel slice, and so on, until the convolution operation of all output channel slices is completed.

[0021] Furthermore, the quantization of the data specifically involves using INT8 quantization, and the quantization process is as follows:

[0022]

[0023] Where x represents the original FP32 value; scales represents the scaling factor of FP32; zero_point represents the offset of the value; round represents rounding to the nearest integer or rounding up or down; Q represents the quantized INT8 value.

[0024] Furthermore, the data pooling process is as follows: the two-dimensional pooling operation is decomposed into a one-dimensional operation with two dimensions, horizontal and vertical.

[0025] First, perform a horizontal one-dimensional pooling operation. Take out a row of data to be pooled in the order of width-height-channel. Perform a one-dimensional pooling operation according to the pooling parameters to obtain the first row of pooling results. Then, locally cache the first row of pooling results in the order of width-height-channel. Repeat this process to complete the one-dimensional pooling of all rows of data. Finally, arrange all the horizontal one-dimensional pooling results into a temporary matrix in order.

[0026] Next, a vertical one-dimensional pooling operation is performed, with the input being the result of the horizontal pooling operation, i.e., the temporary matrix. A one-dimensional pooling operation is then performed on the temporary matrix in the height direction to obtain the output feature map.

[0027] (II) A parallel acceleration device for VGG16 networks based on the ZYNQ platform, comprising: an FPGA, an ARM, and an AXI bus. The FPGA is used to implement hardware acceleration of the convolutional neural network and includes a convolution module, a post-processing module, a fully connected module, and a Softmax module. The ARM is used to preload input feature data, biases, and weights, and to configure CNN register information. It includes the ARM processor and its controlled data input port, classification result output port, DDR storage module, and parameter configuration table. The AXI bus is used to communicate with peripheral modules and includes AXI4 and AXI_Lite.

[0028] Furthermore, the convolution module is a plurality of PE arrays, which perform convolution operations on the feature data and weights retrieved from the cache module and output them through an activation function;

[0029] The post-processing module is a non-linear processing unit used for data quantization and pooling operations, and stores the results in the cache module. After completing one layer of convolution operation, it is written back to the DDR storage module.

[0030] The fully connected module is located at the end of the network. It first caches all feature data in RAM, and then reads weight data into RAM in stages to perform fully connected operations with the feature data.

[0031] The Softmax module is used to probabilize the calculation results and provide probabilities and labels, and then transmit the probability results back to the ARM through the DDR storage module.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0033] (1) The INT8 data quantization scheme adopted in this invention reduces the network model size and memory bandwidth requirements, thereby accelerating the processing.

[0034] (2) This invention optimizes the VGG16 network structure by optimizing on-chip cache, parallelizing multiple input and output channels, optimizing slicing scheme, and optimizing pooling module, thereby improving the computing speed, reducing the cost of moving each data, reducing the number of memory accesses with higher power consumption, and improving the computing performance of convolutional neural networks. Attached Figure Description

[0035] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0036] Figure 1 This is a network structure diagram of the VGG16 neural network algorithm used in this invention;

[0037] Figure 2 This is a block diagram of the overall architecture of the present invention;

[0038] Figure 3 This is the PE array used in the convolution calculation unit of the convolution module in this invention;

[0039] Figure 4 This refers to the single PE unit structure used in the convolution calculation unit of the convolution module in this invention;

[0040] Figure 5 These are two data scheduling sequences in the data slicing optimization scheme adopted in this invention;

[0041] Figure 6 This is a flowchart of the hardware quantization process used in this invention. Detailed Implementation

[0042] The embodiments of the present invention will be described in detail below with reference to examples. However, those skilled in the art will understand that the following examples are only for illustrating the present invention and should not be regarded as limiting the scope of the present invention.

[0043] refer to Figure 1 The VGG16 neural network algorithm accelerated by this invention consists of 13 convolutional layers, 5 downsampling layers, and 3 fully connected layers. The original input image is 224×224 three-channel data. The entire network uses 3×3 convolutional kernels with a stride of 1 to construct the convolutional layers, and performs non-linear processing on the results of each convolutional layer through the ReLU activation function. After the convolutional layers and before the activation function, a Batch Normalization (BN) layer is used to adjust the transfer of feature maps between layers of the neural network. After every 2 or 3 convolutional layers, a 2×2 max pooling layer with a stride of 2 is used to reduce the network size. After pooling, the width and height of the feature map are reduced by half, while the number of channels remains unchanged. The last three layers are fully connected layers, and the output is 1000 categories and their probabilities. The computational load of the network is mainly concentrated in the convolutional layers and fully connected layers. The core of the hardware acceleration design for convolutional neural networks lies in accelerating large-scale multiply-accumulate operations.

[0044] Example 1

[0045] This invention provides a parallel acceleration processing method for VGG16 networks based on the ZYNQ platform, comprising the following steps:

[0046] Step 1: The ZYNQ PS terminal obtains the feature map data and weight data of the dataset to be processed through the host computer, and stores them in the DDR storage module. Then, it drives the AXI_DMA module to load the feature data and weight data into the cache module.

[0047] Step 2: The ARM controller transmits the feature data and weight data of the cache module to the convolution module at the PL end of the ZYNQ for parallel convolution operation, and stores the operation result in the cache module.

[0048] In this invention, the convolution module employs a highly parallel PE array and data slicing optimization to improve throughput, bandwidth, and latency performance.

[0049] The convolutional computation unit employs a rectangular multiply-accumulate array to achieve a 32×32 parallel design, meaning it achieves parallel processing of 32 input channels and 32 output channels. This means it processes input features along one dimension. Figure 1 The operation involves multiplying and accumulating the 32 input channels of a feature point with the 32 channels of its corresponding convolutional kernel; on the other dimension, the operation involves multiplying and accumulating the input feature map with the 32 convolutional kernels.

[0050] refer to Figure 3 The PE array of the convolutional computation unit in this invention consists of 32 PE units, performing 32-output parallel and 32-input parallel computations. Weight data and feature map data are taken from on-chip caches `weight_input_buffer` and `image_input_buffer`, respectively. There are 32 `weight_input_buffers` storing weight data, each caching data from different convolutional kernels. Reading data from all 32 buffers simultaneously yields 32 sets of weight data, each from a different convolutional kernel. Feature map data reading employs a pipelined design; a trigger delay is added after each PE unit. The feature data is pipelined and alternately multiplied with the weight data from different convolutional kernels, achieving 32-output parallelism, i.e., simultaneously calculating the results of 32 output feature maps.

[0051] refer to Figure 4 The single PE unit structure of the convolutional computation unit of this invention consists of 32 DSPs, and each clock cycle completes 32 16-bit × 16-bit multiplication operations. The data stored in the weight_input_buffer and image_input_buffer both have a bit width of 32 × 16 bits. The retrieved data is divided according to the bit width, with each 16-bit segment representing one channel of data, and then sent to the 32 DSPs to complete the multiplication operation, thus realizing parallel computation of 32 input channels.

[0052] refer to Figure 5The data slicing optimization scheme adopted in this invention has two scheduling orders: N->M and M->N. The M->N scheduling scheme first calculates the convolution operation between all slices in the height direction and the first output channel, then switches to the second output channel slice, continues to perform convolution operations with all slices in the height direction, and continues switching output channel slices, sequentially completing the convolution operation of all output channel slices. The N->M scheduling scheme first calculates the convolution in all output channel directions, then calculates and concatenates the convolution results of different slices along the height direction. From the perspective of the output feature map, the former calculates one set of output feature maps before calculating other output feature maps, while the latter calculates 1 / M range of all output feature map planes first, then sequentially calculates other positions on the plane. In the first scheme, the input feature data needs to be loaded N times repeatedly, while the weight data does not need to be loaded repeatedly; in the second scheduling scheme, the weight data needs to be loaded M times repeatedly, and since there is overlapping data in the input feature data, the overlapping data also needs to be loaded repeatedly. To maximize data reuse, two slicing schemes are combined. For the case where the amount of feature data in the first few layers of the network is relatively large, the N→M scheduling method can be used to reduce the total memory access requirement by maximizing the reuse of input feature data. For the case where the amount of weight data in the last few layers of the network is particularly large, the M→N scheduling method can reduce the total memory access requirement by reusing weight data.

[0053] Step 3: The data is then quantized and pooled in the post-processing module. After completing one layer of computation, the PS end of ZYNQ is reconfigured to provide the register information required for the next layer of convolutional module.

[0054] Specifically, the post-processing module is mainly responsible for data quantization and pooling operations, where data quantization uses INT8 quantization. INT8 quantization generally employs linear quantization, and a common linear quantization process is as follows:

[0055]

[0056] Where x represents the original FP32 value; scales represents the scaling factor of FP32; zero_point represents the offset of the value; round represents the mathematical function for rounding to the nearest integer, and in addition to rounding to the nearest integer, it can also be used to round up or down; Q represents a quantized INT8 value.

[0057] Quantization involves not only quantizing the weighted data and output results, but also processing intermediate results. In the hardware implementation of accelerated design, since backward inference is not performed, only forward propagation is used. Therefore, the scaling factors (scales) and weights obtained from software training are directly adopted. However, some special processing is required in conjunction with the network. Through network training, the scaling factors (scales) and quantized weights have been obtained, where the weight data incorporates the results of the Batch Normalization (BN) layer. In the process of hardware acceleration using FPGA, the quantization method must be consistent with that of the software to ensure the accuracy of the results.

[0058] The pooling module decomposes the two-dimensional pooling operation into one-dimensional operations in both horizontal and vertical dimensions to reduce computational load and on-chip storage requirements. First, horizontal pooling is performed. Data is retrieved in a WHC (width-height-channel) format, and a one-dimensional pooling operation is performed according to the pooling parameters to obtain the first row's pooling result. This result is then temporarily stored in a local cache, still sorted according to the WHC format. All the results from the horizontal pooling operations are combined into a temporary matrix. The size of this matrix is ​​determined by the pooling parameters; for 2×2 pooling, the stride is 2, and the width of the temporary matrix is ​​half the width of the input feature map, with a height of H. Since the data is transmitted to the pooling module using the WHC method, there is no need for temporary storage of channel-oriented data, and the horizontal pooling module does not need to cache excessive data. Next, vertical pooling is performed, with the input being the result of the horizontal pooling operation—the temporary matrix. After performing height-oriented pooling on the temporary matrix, the output feature map is obtained.

[0059] refer to Figure 6 The hardware quantization process used in this invention is as follows: the quantized weight data obtained from software training is convolved with the image data; the convolution result is then multiplied by a scaling factor, which increases the bit width of the result; next, the clamp function is used to limit the result to the range of INT8; after accumulating the bias result, the bit width needs to be limited again; finally, the result is output after passing through the ReLU activation function.

[0060] Step 4: Repeat steps 1-3 until all layer operations are completed. Then, the final result is transferred to DDR for storage via AXI_DMA. The ZYNQ PS terminal transmits the result to the host computer via serial port.

[0061] Example 2

[0062] refer to Figure 2The present invention also provides a parallel acceleration device for VGG16 networks based on the ZYNQ platform, comprising: an FPGA, an ARM, and an AXI bus. The FPGA is used to implement hardware acceleration of the convolutional neural network and includes a convolution module, a post-processing module, a fully connected module, and a Softmax module. The ARM is used to preload input feature data, biases, and weights, and to configure CNN register information, including the ARM processor and its controlled data input port, classification result output port, DDR storage module, and parameter configuration table. The AXI bus is used to communicate with peripheral modules and includes AXI4 and AXI_Lite.

[0063] Furthermore, the convolution module contains 32 PE arrays, which perform convolution operations on the feature data and weights extracted from the input buffer and output them through an activation function.

[0064] The post-processing module is a non-linear processing unit, mainly responsible for quantizing data and pooling operations, storing the results in the output buffer within the module, and writing them back to DDR after completing one layer of convolution operation.

[0065] The fully connected module performs the operations of the last three fully connected layers. Since the fully connected layers are the later layers in the network, there is less feature data and more weight data. In the design process, all feature data is first cached in the on-chip RAM of the module, and then the weight data is read into the feature data in the on-chip and RAM in batches for operation.

[0066] The Softmax module performs probability calculations and outputs probabilities and labels, and finally transmits the probabilities back to the ARM via DDR memory.

[0067] The DDR storage module is used to store input image feature data, bias, weight, and temporary storage of intermediate results after each convolution operation.

[0068] The CNN register module mainly completes the settings of the convolution register, such as controlling the current number of layers, feature map and convolution kernel size, number of image blocks, padding control, starting address of data and weight reading, and number of weight transfers.

[0069] The AXI4 bus is responsible for data transmission, while the AXI_Lite bus is responsible for signal transmission.

[0070] This invention reduces the enormous computational load caused by input / output data, weight data, and bias by using fixed-point quantization of data, thereby reducing power consumption, computational load, and solving the problem of limited on-chip resources. By adopting a resource parallelism approach, it avoids the parallel computation problem of data correlation between convolutional neural network layers on the one hand, and reduces the demand for hardware circuit resources on the other hand.

[0071] Although the present invention has been described in detail in this specification with general description and specific embodiments, some modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention are within the scope of protection claimed by the present invention.

Claims

1. A parallel acceleration processing method for VGG16 networks based on the ZYNQ platform, characterized in that, Includes the following steps: Step 1: The ZYNQ PS terminal obtains the feature map data and weight data of the dataset to be processed through the host computer, and stores them in the DDR storage module. Then, it drives the AXI_DMA module to load the feature data and weight data into the cache module. Step 2: The ARM controller transmits the feature data and weight data of the cache module to the convolution module at the PL end of the ZYNQ for parallel convolution operation, and stores the operation result in the cache module. Step 3: The data is then quantized and pooled in the post-processing module. After completing one layer of computation, the PS end of ZYNQ is reconfigured to provide the register information required for the next layer of convolutional module. Step 4: Repeat steps 1-3 until all layer operations are completed. Then, the final result is transferred to DDR for storage via AXI_DMA. The PS end of ZYNQ transmits the result to the host computer via serial port. The convolution module performs parallel convolution operations, specifically including: A high-parallelism PE array is used to perform parallel operation of convolutions with multiple input channels and multiple output channels; Data slicing optimization is performed on the height of the input feature map and the direction of the output channel. The total memory access requirement is reduced by reusing the input feature data and weight data. The parallel operation of convolution with multiple input and output channels using a high-parallelism PE array is specifically as follows: Parallel convolution of multiple input channels is performed in one dimension, which is the multiplication and accumulation operation of multiple input channels of a feature point in the input feature map and the corresponding channel data of a convolution kernel; parallel convolution of multiple output channels is performed in another dimension, which is the multiplication and accumulation operation of all input feature maps and multiple convolution kernels. The data slicing optimization process for the height of the input feature map and the direction of the output channel is as follows: For several layers of the VGG16 network input, a slicing optimization scheme is adopted, which first slices the output channel direction and then slices the height direction of the input feature map. This reduces the total memory access requirement by maximizing the reuse of input feature data. For several layers of the VGG16 network near the output end, a slicing optimization scheme is adopted, which first slices the input feature map in the height direction and then slices it in the output channel direction, so as to reduce the total memory access requirement through weight data reuse.

2. The parallel acceleration processing method for VGG16 networks based on the ZYNQ platform according to claim 1, characterized in that, The slicing optimization scheme, which first slices the output channel directions and then slices the input feature map in the height direction, is as follows: First, calculate the convolution of all output channel directions with the first slice along the height direction; then calculate the convolution of all output channel directions with the second slice along the height direction; and so on, until the convolution of all output channel directions with the last slice along the height direction is calculated. Finally, the convolution results of different slices along the height direction are concatenated in order.

3. The parallel acceleration processing method for VGG16 networks based on the ZYNQ platform according to claim 1, characterized in that, The slicing optimization adopts a scheduling scheme that first slices the input feature map in the height direction and then slices it in the output channel direction. Specifically, it calculates the convolution of all slices in the height direction with the first output channel, then switches to the second output channel slice and continues to perform convolution operations with all slices in the height direction. It then switches to the output channel slice again and so on, until the convolution operation of all output channel slices is completed.

4. The parallel acceleration processing method for VGG16 networks based on the ZYNQ platform according to claim 1, characterized in that, The quantization of the data specifically involves using INT8 quantization, and the quantization process is as follows: Where x represents the original FP32 value; scales represents the scaling factor of FP32; zero_point represents the offset of the value; round represents rounding to the nearest integer or rounding up or down; Q represents the quantized INT8 value.

5. The parallel acceleration processing method for VGG16 networks based on the ZYNQ platform according to claim 1, characterized in that, The data pooling process is as follows: the two-dimensional pooling operation is decomposed into a one-dimensional operation with two dimensions, horizontal and vertical. First, perform a horizontal one-dimensional pooling operation. Take out a row of data to be pooled in the order of width-height-channel. Perform a one-dimensional pooling operation according to the pooling parameters to obtain the first row of pooling results. Then, locally cache the first row of pooling results in the order of width-height-channel. Repeat this process to complete the one-dimensional pooling of all rows of data. Finally, arrange all the horizontal one-dimensional pooling results into a temporary matrix in order. Next, a vertical one-dimensional pooling operation is performed, with the input being the result of the horizontal pooling operation, i.e., the temporary matrix. A one-dimensional pooling operation is then performed on the temporary matrix in the height direction to obtain the output feature map.

6. A parallel acceleration device for VGG16 networks based on the ZYNQ platform, characterized in that, include: The system comprises an FPGA, an ARM, and an AXI bus. The FPGA is used to implement hardware acceleration of the convolutional neural network and includes a convolution module, a post-processing module, a fully connected module, and a Softmax module. The ARM is used to preload input feature data, biases, and weights, and to configure CNN register information. It includes the ARM processor and its controlled data input port, classification result output port, DDR storage module, and parameter configuration table. The AXI bus is used to communicate with peripheral modules and includes AXI4 and AXI_Lite.

7. The VGG16 network parallel acceleration device based on the ZYNQ platform according to claim 6, characterized in that, The convolution module consists of multiple PE arrays, which perform convolution operations on the feature data and weights retrieved from the cache module and output them through an activation function; The post-processing module is a non-linear processing unit used for data quantization and pooling operations, and stores the results in the cache module. After completing one layer of convolution operation, it is written back to the DDR storage module. The fully connected module is located at the end of the network. It first caches all the feature data in RAM, and then reads the weight data into RAM in batches to perform fully connected operations with the feature data. The Softmax module is used to probabilize the calculation results and provide probabilities and labels, and then transmit the probability results back to the ARM through the DDR storage module.

Images