Shared DSP convolutional neural network accelerator implementation method based on FPGA
By designing a shared DSP convolutional neural network accelerator on FPGA, using data quantization and deep dynamically configurable row buffering mechanisms, the problems of convolutional neural network computing complexity and storage requirements in marginalized deployment environments are solved, and efficient and real-time computing processing is achieved.
Patent Information
- Application Number
- CN202411891716.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-04-11
AI Technical Summary
In resource-constrained marginalized deployment environments, the increased computational complexity and storage requirements of convolutional neural networks have led to huge challenges in real-time computing processing.
The shared DSP convolutional neural network accelerator implementation method is adopted based on FPGA, and the storage resource occupation is reduced through data quantization and block rearrangement, a deep dynamically configurable line buffering mechanism is designed, and DSP resource management is optimized to achieve efficient computing.
Effectively utilize transmission bandwidth, reduce hardware costs, improve computing efficiency, meet real-time computing needs, and adapt to the development of convolutional neural networks.
Smart Images
Figure CN119808856A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of hardware accelerated convolutional neural network calculations based on FPGA, and in particular to a convolutional neural network accelerator design and deployment method based on FPGA. Background Art
[0002] In recent years, Convolutional Neural Network (CNN) has gained wide attention and achieved remarkable success in computer vision fields such as image classification, semantic segmentation, object detection, and autonomous driving. However, with the continuous expansion of the application scenarios of CNN and its continuous development, its computational complexity and storage requirements are also increasing accordingly, which brings huge challenges to the deployment of CNN in resource-constrained edge environments.
[0003] At present, the mainstream deployment platforms of convolutional neural networks include graphics processing units (GPUs), application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). GPUs accelerate the calculation of convolutional neural networks by using multithreading to achieve highly parallel computing, but the power consumption is relatively high. ASICs design specific acceleration architectures for specific networks, which can achieve the goals of high performance and low power consumption, but the design architecture cannot be flexibly adjusted, and the development cycle is long and the cost is high. FPGAs can achieve accelerated computing of convolutional neural networks in resource-constrained edge deployment environments through parallel computing and resource reuse. FPGAs have the advantage of low power consumption compared to GPUs. In addition, FPGAs can be laid out and wired an unlimited number of times, which can adapt to the development of convolutional neural networks and have the advantages of flexibility and low cost compared to ASICs. At the same time, FPGAs can perform highly parallel computing and meet real-time computing needs. Therefore, the deployment of convolutional neural network accelerators using FPGAs has good application prospects. Summary of the invention
[0004] The present invention proposes a method for implementing a shared DSP convolutional neural network accelerator based on FPGA in order to deploy convolutional neural networks in resource-constrained edge scenarios that require real-time computing and processing.
[0005] The object of the present invention is achieved through the following technical solutions, including:
[0006] S100. According to the scenario to be deployed, a data set is constructed, a convolutional neural network is trained, and a network model is obtained; parameters of the trained network model are read, and integer 8-bit (INT8) quantization is performed, and quantized weight parameters, scaling factors, and numerical offsets are saved;
[0007] S200. The quantized weight parameters are divided into blocks and rearranged, and the weight data and bias data are hierarchically saved as binary files according to the convolutional neural network model, and the image data is quantized and rearranged in blocks, and finally stored in the SD card;
[0008] S300.PS (Processing System) reads image data and weight data from the SD card, and transmits the image data and weight data required for the convolutional neural network to the PL (Programmable Logic) side. The PL side uses the parallel computing capability of FPGA to design a shared DSP convolutional neural network accelerator based on FPGA for accelerated computing, and returns the calculation results to the PS side;
[0009] Further, the S300 includes the following steps:
[0010] S310.PS reads the data in the SD card, and then uses DMA (Direct Memory Access) to send image data, weight data and bias data to the PL through the AXI (Advanced eXtensible Interface) protocol. The PL uses a ping-pong cache design to store the image data, weight data and bias data that need to be calculated;
[0011] S320. The PS configures the control module of the PL through the AXI protocol, starts the calculation process of the convolutional neural network and controls it;
[0012] S330. The accelerator uses a deeply configurable row cache design to implement the output of the corresponding feature map convolution window according to different convolution layer configurations in the control module;
[0013] S340. The output feature map convolution window and weight data convolution kernel are convolved through a shared DSP module based on integer 8-bit quantization, and then the calculation results are accumulated in the accumulation module;
[0014] S350. After the calculation results of all input channels are accumulated, the accumulated results are output to the quantization module to quantize the results;
[0015] S360. Input the quantization result to the pooling module for maximum pooling operation, and return the result to the PS end for storage;
[0016] S370. Continuously loop the operations of S310-S360 until all calculations of this layer of the convolutional neural network are completed, and the calculation results are used as inputs for the next layer;
[0017] S400. Repeat S300 until the calculation of all layers of the entire convolutional neural network is completed, save the calculation results to the PS end and print the results on the host computer using the serial port.
[0018] Beneficial Effects
[0019] The beneficial effects of the present invention are as follows: the present invention proposes a method for implementing a shared DSP convolutional neural network accelerator based on FPGA. Firstly, by means of data quantization and data block rearrangement, the occupation of storage resources and the use of addressing resources are reduced, and the transmission bandwidth is effectively utilized; secondly, a deeply dynamically configurable row buffer mechanism is designed, which can dynamically adjust the depth of the row buffer according to the data flow characteristics and computing requirements of different convolutional layers, thereby avoiding the problem of over-allocation or insufficient storage resources that may exist in the traditional fixed-depth buffer design; finally, a shared DSP module is proposed, which realizes the function of a single DSP supporting two signed INT8 multiplications at the same time by optimizing the internal resource management of the DSP, reducing the number of DSPs required to complete the same computing task, thereby reducing hardware costs and improving overall computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is a deployment flow chart of the present invention based on FPGA shared DSP convolutional neural network accelerator;
[0021] Figure 2 A schematic diagram of weight data rearrangement provided by the present invention;
[0022] Figure 3 A schematic diagram of the block rearrangement of image data provided by the present invention;
[0023] Figure 4 A schematic diagram of the overall framework of the FPGA-based shared DSP convolutional neural network accelerator provided by the present invention;
[0024] Figure 5 A schematic diagram of the control register configuration provided by the present invention;
[0025] Figure 6 A schematic diagram of a design of a deeply dynamically configurable row buffer provided by the present invention;
[0026] Figure 7 A schematic diagram of a shared DSP module based on INT8 quantization provided by the present invention;
[0027] Figure 8 This is a schematic diagram of the quantization module provided by the present invention. DETAILED DESCRIPTION
[0028] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. The embodiment described is only one of the embodiments of the present invention, not all embodiments.
[0029] The FPGA-based shared DSP convolutional neural network accelerator designed in this embodiment deploys the VGG-16 network. The deployment process is as follows: Figure 1 As shown, the following steps are included:
[0030] S100. According to the data set and the VGG-16 network structure, a specific VGG-16 network model is trained, which includes weight parameters and bias parameters; the weight parameters and bias parameters of each layer of the network are obtained according to the trained network model, and the weight parameters are linearly quantized using INT8, and the quantized weight parameters, scaling factors, and numerical offsets are saved;
[0031] The specific quantification process is as follows: The quantification formula used is:
[0032]
[0033] Among them, w represents the weight parameter of the original 32-bit floating point number, scale represents the quantized scaling factor, zero_point represents the numerical offset, the round function implements the rounding function, and Q represents the quantized INT8 weight parameter;
[0034] S200. Figure 2 The quantized weight parameters are rearranged into groups of 8 convolution kernels, and the 8-bit data of each 8 input channels of each convolution kernel are grouped into a 64-bit data for storage; the image data is rearranged in blocks, the 3-channel image data is filled with 0 to 8 channel data, and the data is combined into 64-bit data according to 8 channels, such as Figure 3 As shown in the figure, the combined 64-bit data can make full use of the 64-bit transmission bandwidth of the PS and PL ends to achieve efficient data transmission. The divided image data does not need to be rearranged in the subsequent layers, reducing the use of addressing resources. The processed weight parameters and image data are saved as binary files and stored in the SD card on the PS end.
[0035] S300. Read the data in the SD card and transmit it to the FPGA-based shared DSP convolutional neural network accelerator designed by the PL for calculation. The overall framework of the FPGA-based shared DSP convolutional neural network accelerator is as follows: Figure 4 As shown, in this embodiment, the S300 includes the following steps:
[0036] S310.PS reads the weight and other parameters and image binary files in the SD card through the file system; then controls the memory controller and uses DMA to send the divided image data, weight data and bias data to the cache of the PL accelerator using the AXI protocol. The transmission adopts a ping-pong cache design and opens up two storage areas, so that one is receiving data while the other is outputting the calculation data stream, which reduces the calculation pause time caused by transmission and improves the calculation efficiency.
[0037] S320.PS uses the AXI protocol to control the accelerator on the PL side using control signals. The control module is designed as follows: Figure 5 As shown in the figure, multiple control signals with smaller bit widths are combined into a 32-bit control signal, which fully utilizes the control signal bandwidth, realizes the control of the PL-side accelerator and starts the convolutional neural network calculation process;
[0038] S330. The accelerator first outputs the corresponding image convolution window through a deeply dynamically configurable row buffer design, such as Figure 6 As shown in the figure, using the characteristic that the input of each convolutional layer of the VGG-16 network is 1 / 2 of the input of the previous convolutional layer, the maximum 224 is used as the total depth, and the shift registers are divided into 14, 14, 28, 56, and 112 lengths, so that each register depth after division can be combined to meet each situation from the minimum image size to the maximum image size, so as to achieve the purpose of reusing the resources in each layer and reduce the resource utilization rate;
[0039] S340. The output image convolution window and the weight convolution window are calculated using a shared DSP module, such as Figure 7 As shown; the image data is input as a common multiplier to the lower 8 bits of the B port of the DSP module, and the remaining bits of the B port are filled with the image data sign bit, and the two weight data are respectively input into the upper 8 bits of the A port and the lower 8 bits of the D port of the DSP module, and the remaining bits of the A port are filled with 0, and the remaining bits of the D port are filled with the sign bit; the A port and the D port are first pre-added and then multiplied with the B port, and then the results of A×B and D×B are taken out from the high and low bits of the DSP output port P result respectively; that is, a single DSP is used to implement two signed INT8 multiplications, which reduces the use of multiplier resources and improves the accelerator computing efficiency;
[0040] S350. The calculation results of all input channels are accumulated by the accumulation module, and the accumulated results are quantized by the quantization module, such as Figure 8 As shown, the bias parameter is first added to the channel accumulation result, and then the result is limited. Then, the intermediate calculation result is quantized into 8-bit data through shift quantization and limit operations.
[0041] S360. Input the quantized data into the pooling module, perform a maximum pooling operation with a step size of 2 and a window of 2×2, and return the result to the PS end for storage;
[0042] S370. Continuously loop the operations of S310-S360 until all calculations of the current layer of the convolutional neural network are completed, and the calculation results are used as inputs for the next layer;
[0043] S400. Repeat S300 until the calculation of all layers of the entire convolutional neural network is completed, save the calculation results to the PS end and print the results on the host computer using the serial port.
[0044] The above is only a specific implementation of the present invention. It should be noted that the present invention is not limited to the specific form disclosed, nor does it exclude other possible implementations. The present invention can be applied to various combinations, variations and environments, and can be adjusted and modified according to the above description or the technology and knowledge in the relevant field. However, as long as it does not deviate from the spirit and scope of the present invention, any changes and improvements made by those skilled in the art should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A method for implementing a shared DSP convolutional neural network accelerator based on FPGA, characterized in that: The following steps are involved: S100. According to the data set and the network structure, a network model is trained, including weight parameters and bias parameters; the weight parameters and bias parameters of each layer of the network are obtained according to the trained network model, and the weight parameters are linearly quantized into 8-bit integers (INT8), and the quantized weight parameters, scaling factors and numerical offsets are saved; S200. Rearrange the quantized weight parameters, divide them into a group according to every 8 convolution kernels, and divide the weight data of each convolution kernel according to every 8 input channels, combine 8 8-bit data at the same position into a 64-bit data for storage, and complete the rearrangement of all weight data in sequence; rearrange the image data in blocks, fill the 3-channel image data with data '0' to become 8-channel image data, and combine the image data according to the 8 channels at the same position to become 64-bit single-channel data; save the processed weight parameters and image data as binary files and store them in the SD card at the processing system (PS) end; S300. Read the data in the SD card and transmit it to the efficient convolutional neural network accelerator based on FPGA designed at the programmable logic (PL) end for calculation; the S300 includes the following steps: S310.PS reads the weight and other parameters and image binary files in the SD card through the file system, and then controls the memory controller to use DMA (Direct Memory Access) to send the divided image data, weight data and bias data to the cache of the PL accelerator using the AXI (Advanced eXtensible Interface) protocol; the transmission adopts a ping-pong cache design and sets two buffer storage areas, so that when one buffer storage area is receiving data, the other buffer storage area is outputting the calculation data stream, reducing the calculation pause time caused by data transmission and improving the calculation efficiency; S320. The PS side uses the control signal to control the accelerator on the PL side through the AXI protocol, combines different control signals into a 32-bit control signal, realizes the control of the accelerator on the PL side and starts the convolutional neural network calculation process; S330. The accelerator first uses a deeply dynamically configurable row buffer module to configure the output of the corresponding image convolution window using different image sizes; S340. The output image convolution window and the weight convolution window share a DSP module and use a single DSP to implement two signed INT8 multiplications for convolution calculation; S350. The calculation results of all input channels are accumulated in the accumulation module, and the accumulated results are quantized by the quantization module, that is, the bias parameter is first added to the channel accumulation result, and then the result is limited; then the intermediate calculation result is quantized into 8-bit data through shift quantization and limit operation; S360. Input the quantized data into the pooling module, perform a maximum pooling operation with a step size of 2 and a window of 2×2, and return the result to the PS end for storage; S370. Continuously loop the operations of S310-S360 until all calculations of the current layer of the convolutional neural network are completed; and the calculation results are used as inputs of the next layer; S400. Repeat S300 until the calculation of all layers of the entire convolutional neural network is completed, save the calculation results to the PS end and print the results on the host computer using the serial port protocol.
2. According to claim 1, a method for implementing a shared DSP convolutional neural network accelerator based on FPGA, characterized in that: The row buffer module with dynamically configurable depth utilizes the pooling operation of the convolutional neural network pooling layer to make the input image size of the next convolutional layer 1 / 2 of the image size of the previous convolutional neural network. The maximum image size MAX_LENGTH is used as the total shift register depth, and the total register depth is symmetrically divided into registers of multiple depths in a loop until it is divided into the minimum image size MIN_LENGTH, so that each register depth after division can meet each situation from the minimum image size to the maximum image size in a combined manner; after the image data passes through this module, the correct image data convolution window can be output.
3. According to claim 1, a method for implementing a shared DSP convolutional neural network accelerator based on FPGA, characterized in that: The shared DSP module inputs the image data as a common multiplier to the lower 8 bits of the DSP module B port, and the remaining bits of the B port are filled with the image data sign bit; the two weight data are respectively placed in the upper 8 bits of the DSP module A port and the lower 8 bits of the DSP module D port, the remaining bits of the A port are filled with 0, and the remaining bits of the D port are filled with the sign bit; the A port and the D port data are first pre-added and then multiplied with the B port data, and then the A×B and D×B results are taken out from the high and low bits of the DSP output port P result respectively.