A general convolutional neural network accelerator implementation method based on FPGA

Abstract

The application relates to the fields of deep learning and embedded machine vision, in particular to a general convolutional neural network accelerator implementation method based on FPGA, which comprises a processor system and programmable logic, a hardware platform adopts a ZYNQ series FPGA of XILINX, a control algorithm is designed on the processor system side of the FPGA, an RTL convolutional neural network accelerator IP is designed on the programmable logic side of the FPGA, a camera and a VGA interface are contained in the hardware system, images collected by the camera are identified, and results are output to an external screen through the VGA interface, the FPGA resources are finely optimized according to the application, resources are greatly saved under the condition that the precision meets the requirements, the forward inference speed of the convolutional neural network is greatly improved, and the application has a higher energy efficiency ratio, and the architecture of the method can be expanded and cut according to the application, so as to correspond to higher and lower resources of the FPGA.

Description

TECHNICAL FIELD

[0001] The application relates to the fields of deep learning and embedded machine vision, and in particular to a general convolutional neural network accelerator implementation method based on FPGA. BACKGROUND

[0002] At present, deep learning, especially convolutional neural network, has been widely applied in target detection and recognition fields. With the increase of complexity of the convolutional neural network model structure and the improvement of recognition accuracy, the computing power of the deployment platform is challenged. At present, the deployment of this kind of algorithm is almost dominated by GPU, and it is difficult to run on embedded platforms and low-end CPUs, so that high-performance models are difficult to implement on mobile terminals based on embedded devices. On the other hand, although ASIC chips have the advantages of high computing power and high energy efficiency, they can replace GPUs in specific fields, but the development cycle of ASIC is long, and the architecture of neural networks is updated faster, so that the iteration cycle of hardware cannot meet the requirements of algorithms. Therefore, FPGA provides a new idea for the deployment of deep learning algorithms with its parallelism and fast reconfigurability of computing structure, as well as low power consumption and high energy efficiency. At the same time, the convolutional neural network has the characteristics of single computation, which makes it easy for FPGA to do parallel acceleration, thus meeting the requirement of high computing power. SUMMARY

[0003] To solve the above problems, the application provides a general convolutional neural network accelerator implementation method based on FPGA, which is characterized by heterogeneous computing and scalability. The heterogeneous computing means that the method designs a general convolutional neural network accelerator architecture in a soft and hard cooperative manner. The control algorithm is designed on the processor system side of the FPGA, and the RTL convolutional neural network accelerator IP is designed on the programmable logic side of the FPGA. Different network models can be accelerated by configuration.

[0004] The technical scheme of the application is as follows:

[0005] A general convolutional neural network accelerator implementation method based on FPGA includes a processor system (PL) and a programmable logic (PL). The hardware platform adopts ZYNQ series FPGA of XILINX. The control algorithm is designed on the processor system side of the FPGA, and the RTL convolutional neural network accelerator IP is designed on the programmable logic side of the FPGA. The hardware system includes a camera and a VGA interface. The images collected by the camera are recognized, and the results are output to the external screen through the VGA interface.

[0006] The processor system comprises a CORTEX-M0 processor core, a DMA, a DDR controller, a DDR, an SDIO controller, and a processor running a control algorithm and a camera driver; the processor running the control algorithm comprises calculating a corresponding parameter address according to a network layer relationship in an interrupt service function, a DMA sending and receiving address and an enable signal, and a write operation on an RTL convolutional neural network accelerator IP control register, all parameters are saved in respective arrays, required parameters for each calculation are a segment of data in the array, and the parameters are sent to the RTL convolutional neural network accelerator IP through the DMA, so the address corresponding to the required data is given to the DMA before each calculation, the DMA sends the data of the segment address to the RTL convolutional neural network accelerator IP, and the calculation mode of the accelerator is further configured through an AXI_lite protocol, the AXI_lite protocol uses four 32-bit registers, the registers are decoded into required control information, and the control information is transmitted to the RTL convolutional neural network accelerator IP.

[0007] The RTL convolutional neural network accelerator IP comprises a matrix generation module, a padding module, a DSP array module, a weight quantization module, a pooling module, an AXI bus module, and a plurality of on-chip caches; the on-chip caches comprise an input feature map cache, a weight cache, a bias cache, an intermediate result cache, and an output feature map cache, wherein the weight cache, the input image cache, and the output image cache are 8 bits, the bias cache is 16 bits, and the intermediate result cache is 32 bits; input and output of the accelerator are AXI_lite protocol and AXI_stream protocol interfaces; the matrix generation module extracts a sliding window on the input feature map; the padding module adopts a hardware padding mode and is used for zero padding around an image; the DSP array module is a calculation unit of the accelerator and is used for parallel acceleration of multiplication and accumulation operations; the weight quantization module quantizes high-bit data after calculation to 8 bits, so that the next layer can directly calculate; the pooling module is a downsampling module and is used for compressing a feature map and extracting a feature; the AXI bus module comprises an AXI_lite module and an AXI_stream module, the AXI_lite module is used for transmitting control information, and the AXI_stream module is used for transmitting a data stream; and the plurality of on-chip caches are used for storing quantized parameters and caching input feature maps, intermediate calculation results, and output feature maps.

[0008] The matrix generation module is composed of two synchronous FIFOs and a peripheral control circuit; the FIFOs are connected in a head-to-tail manner, data enters a first FIFO in sequence, output of the first FIFO controls start of a next row, and the read data is written into a second FIFO, so that two rows of data stored in the FIFOs and input of a third row of data can generate a pixel matrix at each clock.

[0009] The padding module is to make the feature map size of the convolution output same as the input, and to pad zeros around the image before the convolution calculation and the pooling calculation; the padding module works with the input feature map or the intermediate layer result buffer; in the padding module, the rows and columns are counted, the position of the zero padding is calculated according to the required zero padding format, and when the corresponding zero padding position is counted, the read enable is pulled low, while the read enable is pulled high at the remaining positions; the padding module supports two zero padding formats, one of which is to add a circle of zeros around the image, and the other of which is to add a circle of zeros on the right side and the bottom edge of the image, for example, the size of the input image is 28*28, after the first zero padding, the size becomes 30*30, and after the second zero padding, the size should become 29*29.

[0010] The input data of the DSP array module are 8BIT feature map, 8BIT weight data and 16BIT bias data; among them, the weight data and the feature map are multiplied and accumulated, and finally the bias is added, so the output data width is 19BIT, and after the quantization module, the bit width becomes 32BIT; in the DSP array module, since the data bit width of the DSP is fixed, the convolution data is 8BIT, and one DSP can perform two multiplications by using the multiplication law, so 288 DSPs are used to perform 576 multiplications, which can realize 64-way convolution parallel calculation, and the multiplication result is 8BIT; after the multiplication, a sum tree is designed to accumulate the multiplication results, for example, for a 3*3 window, 9 16BIT numbers are accumulated, and the accumulation result is saved in 20BIT, and the data here enters the quantization module.

[0011] The quantization module is to make the 20BIT result of the previous layer calculation become 8BIT, and then 8BIT is saved in 32BIT, and the subsequent bit will be truncated to 8BIT, which can be directly used for the input of the next layer, and the quantization principle is the same as that of the PC end, and the scale of the quantization module is obtained by observation in the PC end quantization algorithm, which only needs to be sent to the accelerator by the processor system through the AXI_lite protocol; in addition, due to the different number of convolution layers between different networks, an intermediate result buffer is set after quantization, which is used for accumulation of multiple batches of calculation results, and the quantization module also controls the read enable of the intermediate result buffer, so that the data read from the intermediate layer result buffer is aligned in time sequence with the calculation output data of the quantization module, so as to facilitate the accumulation of multiple batches; rounding operation is involved in the quantization calculation process, and two rounding methods are selected for configuration, namely rounding to even and rounding to nearest, which can be flexibly selected according to different software algorithm versions, and a low BIT correction algorithm is designed for hardware implementation of rounding to even.

[0012] The pooling module refers to maximum value pooling, which is composed of a synchronous FIFO and an external control circuit, and the principle is to compare the size of the input data one by one, and store the result in the FIFO, the second row of data is operated in the same way, when the result of the second row appears, it is compared with the data of the previous row, and the maximum value pooling result is obtained.

[0013] The plurality of on-chip caches are all based on dual-port BRAM design; wherein, in the weight cache, the input weight data stream is transposed to meet the format requirements of parallel calculation; in the bias cache, 64BIT data are input per clock, that is, 4 bias parameters (4*16BIT), two are clock input a group (8*16BIT), and are respectively present in 8 BRAMs; and in the caches of the input feature map and the output feature map, the ping-pong strategy is adopted, so that the calculation results of the previous calculation can still be read while the calculation under the flow is carried out.

[0014] Compared with the prior art, the beneficial effects of the present application are: the present application is based on FPGA, and a general convolutional neural network accelerator implementation method is designed, the general convolutional neural network accelerator designed based on the method is optimized in a fine-grained manner based on FPGA resources, and the storage space occupied by parameters is reduced to 1 / 4 under the condition that the precision meets the requirements, a large amount of resources are saved, the parallel degree of convolution is improved to 64 ways through efficient use of DSP, the forward inference speed of the convolutional neural network is greatly improved, the peak example reaches 60GOPS, the power consumption is 3.105W, the power consumption of the accelerator part is 0.526W, 18167 LUTs, 297 DSPs and 100 BRAMs are occupied, compared with desktop CPUs and GPUs, the present application has a higher energy efficiency ratio.In addition, the skilled person in the art can expand and tailor the architecture according to the method, so as to correspond to higher and lower resource FPGAs. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The accelerator architecture and single-layer convolution calculation flowchart.

[0016] Figure 2 The model calculation flowchart. DETAILED DESCRIPTION

[0017] The specific embodiments of the present application will be further described below in combination with the drawings and technical solutions.

[0018] The network trained on the PC end cannot be directly deployed on the FPGA, and needs to be quantized first, in the present application, the weight is quantized to INT8 data type, and the bias is quantized to 16BIT data, the specific method is to insert an observer in each layer, which is used to observe the scale of the parameter, the observation process needs to use the data of the test set to infer at least one hundred times, so that a set of appropriate parameters can be observed, and the scale is used to quantize the weight and bias, it needs to be mentioned that PYTORCH does not quantize the bias, but performs floating point operation, since the floating point operation unit is not designed in the accelerator, the bias needs to be quantized to 16BIT, and the scale of the bias is directly replaced by the scale of the weight, and the quantization principle is:

[0019] q=round(r / s+zero)

[0020] Wherein q is the quantized data of INT8 type, r is the floating point type, s is the scale, representing the data range of the floating point number, and zero is the zero point position after quantization.

[0021] Then the parameters are extracted. The quantized data is written into the bin file layer by layer, and when writing, according to the design of the cache area in the accelerator, the tensor format exported by PYTORCH needs to be transposed correspondingly, so as to facilitate the subsequent data transmission and storage, then the data is written into the BIN file and placed in the SD card.

[0022] The quantized weight data exists in the SD card, which is first cached in the DDR through the processor system, and then the weight parameters of each layer are transmitted into the weight cache area of the accelerator through the control algorithm, and the process is completed by DMA.

[0023] The quantized bias data exists in the SD card, which is first cached in the DDR through the processor system, and then the weight parameters of each layer are transmitted into the weight cache area of the accelerator through the control algorithm, and the process is completed by DMA.

[0024] For the input feature map, it can be a picture in the SD card or pixel data of the camera. Here, the camera data is used as an example. The camera is initialized using the SCCB protocol, which can be configured using hardware and software. SCCB is similar to IIC. The principle of software implementation is to simulate the IIC timing and write configuration information. The camera interface uses the DVP interface, which transmits 8-bit data per clock. The initial data format transmitted by the camera is RGB565, which is 16 bits. Therefore, two clocks are required to accept a complete pixel data. Then, RGB565 is converted to RGB888. The usual method is to add high-bit data to the low bit. For the R channel, add the high three bits to the low three bits. For the G channel, add the high two bits to the low bit. The B channel is the same as the R channel. The data after padding is cached in the DDR. The accelerator input channel is 8, so the remaining channels need to be padded with zeros before calculation. This process is completed by the processor system.

[0025] As shown in Figure 1 After the data is ready, enter the loop task. The single-layer convolution calculation process will be described in the following steps:

[0026] First step: send weight data. The quantized parameters, including weights, biases, and quantization parameters, are stored in the DDR in the previous step. Then, send the weights required for one calculation to the weight buffer area in the accelerator IP. The weight buffer area is specially designed to meet the requirements of multi-channel parallel calculation. After the transmission is completed, the parameter update is controlled by the processor system. If the number of convolution layer weights exceeds the number of cacheable weights, the weights should be sent in batches.

[0027] Second step: send bias data. Send the weights required for one calculation to the bias buffer area in the accelerator IP. The bias buffer area is similar to the weight buffer, except that the bit width is different. After the transmission is completed, the parameter update is controlled by the processor system. Similarly, if the number of bias parameters exceeds the number of cacheable parameters, the bias should be sent in batches.

[0028] Third step: initialize the accelerator and transfer control information to the register group of the accelerator. The register group is four 32-bit registers, which are transmitted using the AXI_lite protocol. The accelerator parameters for this calculation are configured according to the model parameters. The registers that can be configured are as follows:

[0029]

[0030] Fourth step: send feature map. The calculation of the accelerator IP is mainly divided into two stages of pipeline. The first stage of pipeline is to transfer data into the input buffer area on the RTL accelerator through DMA. The padding module reads the data in the buffer area, judges whether padding operation is needed, and transmits the data to the matrix generation module. Then the sliding window of convolution is generated, and the data in the window is transmitted to the DSP array module. In the DSP array module, the received parameters are multiplied and accumulated. At the same time, it is judged whether the results of the last calculation need to be accumulated according to the control information, and the results are output to the quantization module. The quantization module re-quantizes the accumulated high-bit-width data to 8 bits, and stores it in the intermediate result buffer area. The second stage of pipeline is that the padding module reads the data from the intermediate result buffer area, judges whether padding operation is needed, and transmits the processed data to the pooling module. The pooling module judges whether pooling operation is needed, and the type of pooling operation is executed. In this accelerator, maximum value pooling is supported. In this accelerator architecture, users can add average value pooling and the like. The pooled data is written to the output data buffer area. All data of the above-mentioned pipeline is given in the combination of data and valid, which is convenient for the design and debugging of each module.

[0031] Fifth step: receive calculation results. The output feature map after calculation is cached in the output buffer area, and then the accelerator pulls up the sending enable. At this time, the AXI_stream master on the accelerator side initiates handshake, and the slave is the receiving channel of DMA. After successful handshake, the data in the output buffer area is transmitted back to the DDR. During this period, the processor system has calculated the address space where the data to be transmitted back is to be stored. When the transmission process is completed, the DMA pulls up the interrupt of the processor system. In the interrupt service function, some flag bits are processed. Thus, one calculation is completed. If the number of channels of the convolution layer is greater than the number of channels that can be calculated by the accelerator at a time, the above-mentioned process can be repeated.

[0032] The above is the process of single-layer convolution calculation. However, a convolutional neural network usually has many layers. Since the method proposed is a general accelerator, it can be slightly configured to realize the acceleration of various convolutions, such as Figure 2 As shown, if single-time calculation of a convolution layer cannot be completed, it needs to be calculated repeatedly. For a complete network, it is calculated repeatedly according to the method of single layer. The subsequent layers can use the same calculation method, and the control registers need to be configured according to the number of layers.

Claims

1. A method for implementing a general-purpose convolutional neural network accelerator based on FPGA, characterized in that, It consists of two parts: a processor system and programmable logic. The hardware platform uses Xilinx's ZYNQ series FPGA. The control algorithm is designed on the processor system side of the FPGA, and the RTL convolutional neural network accelerator IP is designed on the programmable logic side of the FPGA. The hardware system includes a camera and a VGA interface, which recognizes the images captured by the camera and outputs the results to an external screen through the VGA interface. The processor system includes a Cortex-M0 processor core, DMA, DDR controller, DDR chip, SDIO controller, processor operation control algorithm, and camera driver. The processor operation control algorithm includes calculating the corresponding parameter address based on the network layer relationship in the interrupt service function, the address and enable signal for DMA transmission and reception, and write operations to the control register of the RTL convolutional neural network accelerator IP. All parameters are stored in their respective arrays. Each calculation requires a segment of data from the array. Since the parameters need to be sent to the RTL convolutional neural network accelerator IP via DMA, the address corresponding to the required data must be given to the DMA before each calculation. The DMA then sends the data at that address to the RTL convolutional neural network accelerator IP. In addition, the accelerator's calculation mode is configured through the AXI_lite protocol. The AXI_lite protocol uses four 32-bit registers to decode the registers into the required control information and transmit it to the RTL convolutional neural network accelerator IP. The RTL convolutional neural network accelerator IP includes a matrix generation module, a padding module, a DSP array module, a weighting module, a pooling module, an AXI bus module, and multiple on-chip caches. The on-chip caches include an input feature map cache, a weight cache, a bias cache, an intermediate result cache, and an output feature map cache. The weight cache, input image cache, and output image cache are all 8-bit, the bias cache is 16-bit, and the intermediate result cache is 32-bit. The accelerator's input and output are via AXI_lite and AXI_stream protocol interfaces. The matrix generation module extracts a sliding window from the input feature map; the padding module... The block uses hardware padding to add zeros around the image; the DSP array module is the accelerator's computing unit, used for parallel acceleration of multiply-accumulate operations; the weighting module quantizes the computed high-bit data into 8-bit data for direct computation in the next layer; the pooling module, or downsampling, is used to compress feature maps and extract features; the AXI bus module includes the AXI_lite module and the AXI_stream module, with the AXI_lite module used to transmit control information and the AXI_stream module used to transmit data streams; multiple on-chip caches are used to store quantized parameters and to cache input feature maps, intermediate computation results, and output feature maps; The matrix generation module consists of two synchronous FIFOs and peripheral control circuits. The FIFOs are connected end to end, and the data enters the first FIFO in sequence. The output of the first FIFO will be controlled to start from the next row, and the read data will be written to the second FIFO. In this way, the two rows of data buffered in the FIFO and the input third row of data will generate a pixel matrix in each clock cycle. The padding module is used to ensure that the size of the feature map output by the convolution is the same as that of the input. Zeros are padded around the image before the convolution and pooling calculations. The padding module works in conjunction with the input feature map or the intermediate layer result buffer. In the padding module, rows and columns are counted, and the positions where zero padding is required are calculated according to the zero padding format set. When the count reaches the corresponding zero padding position, the read enable is pulled low, and the read enable is pulled high at other positions. The padding module supports two zero-padding formats: one is to add a ring of zeros around the image, and the other is to add a ring of zeros on the right side and bottom edge of the image. The DSP array module inputs 8-bit feature map, 8-bit weight data, and 16-bit bias data. The weight data and feature map are multiplied and accumulated, and the bias is added at the end, so the output data width is 19-bit. After passing through the weighting module, the bit width becomes 32-bit. In the DSP array module, since the data bit width of the DSP is fixed and the convolution data is 8 bits, the associative law of multiplication is used to enable one DSP to perform two multiplications; after the multiplication, an addition tree is designed to accumulate the multiplication results. The weighting module transforms the result of the previous layer's calculation into an 8-bit value. This 8-bit value is then stored in a 32-bit format and subsequently truncated back to 8 bits, allowing it to be directly used as input for the next layer. The quantization principle is the same as that on a PC. The scale of the weighting module is obtained through observation in the PC-side quantization algorithm and only needs to be sent to the accelerator via the AXI_lite protocol through the processor system. In addition, due to the different numbers of convolutional layers in different networks, an intermediate result buffer is set up after weighting for accumulating multiple batches of calculation results. The weighting module also controls the read enable of the intermediate result buffer to ensure that the data read from the intermediate layer result buffer is time-aligned with the calculation output data of the weighting module, facilitating the accumulation of multiple batches. The weighting calculation process involves rounding operations. Two rounding methods are selected for configuration: rounding to even numbers and rounding to the nearest whole number. The rounding method varies depending on the software algorithm version. At the same time, a low-bit correction algorithm is designed for hardware implementation of rounding to even numbers. The pooling module refers to max pooling, which consists of a synchronous FIFO and an external control circuit. The principle is to compare the input data one by one and store the result in the FIFO. The same operation is performed on the second row of data. When the result of the second row appears, it is compared with the data of the previous row to obtain the result of max pooling. All the on-chip caches are based on a dual-port BRAM design. In the weight cache, the input weight data stream is transposed to meet the format requirements of parallel computing. In the bias cache, 64 bits of data are input for each clock cycle, i.e., four bias parameters are 4*16 bits each, and two are clock inputs in a group of 8*16 bits each, which are stored in eight BRAMs respectively. In the caches of input feature maps and output feature maps, a ping-pong strategy is adopted, which allows the previous calculation results to be read while pipelined computing is being performed.

Images