A block convolution method for pre-reading data by channel based on FPGA platform
By adopting the convolution method by channel by channel on the FPGA platform, the problem of high memory footprint of the CNN model is solved, and more efficient computing speed and memory utilization is achieved, which is suitable for CNN acceleration for embedded devices.
Patent Information
- Application Number
- CN202110253729.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-09
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-03-09
Smart Images

Figure CN112950656B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a block convolution method for pre-reading data by channel based on an FPGA platform, and belongs to the field of customized optimization acceleration of neural networks based on hardware. Background Art
[0002] In recent years, convolutional neural networks (CNNs) have made great progress in image classification, target detection, and speech recognition. CNNs are far more accurate than other machine learning algorithms, but as the accuracy improves, the model network parameters of CNNs continue to increase. In addition, convolution operations in CNNs are the main method of calculation, and convolution consumes a lot of computing resources. Therefore, when using neural networks to complete a specific task, not only will the amount of calculation and memory required increase, but the time cost and power consumption will also increase. Therefore, optimizing the design of CNNs is of great significance for network acceleration and terminal node deployment.
[0003] GPU is the most popular solution to the CNN acceleration problem. It mainly optimizes and accelerates the convolution process. However, due to the high power consumption of GPU, it is difficult to apply it in many embedded devices. In order to promote CNN to more application areas, especially embedded devices, many hardware acceleration solutions based on FPGA and ASIC have been proposed. FPGA-based design has attracted more and more attention because it can achieve better energy-saving performance than GPU; it also provides more flexibility than ASICs; more importantly, the emergence of high-level synthesis (HLS) tools Vivado HLS and OpenCL has greatly reduced development time and programming complexity.
[0004] Compared with GPU, FPGA has lower energy consumption, but it still lags far behind GPU in performance. This is mainly because the existing CNN model relies on intensive calculations of floating-point data. However, the limited computing units and on-chip buffers of FPGA greatly limit its acceleration capabilities. In order to reduce the demand for computing memory and communication bandwidth, the size of the CNN model can be reduced by using model compression methods, such as using low-precision weights, activations, and even gradients to train the CNN network model. However, in the above method, the memory still has a high occupancy, and the customizable advantage of FPGA cannot be fully utilized. Therefore, the present invention takes reducing the memory usage of FPGA as the starting point, and proposes a block convolution method for pre-reading data by channel, thereby reducing the memory usage of FPGA and optimizing the entire neural network. Summary of the invention
[0005] The purpose of the present invention is to propose a block convolution method for pre-reading data by channel based on FPGA platform. The innovation lies in using the high flexibility of FPGA to customize the entire calculation process to improve the calculation speed of the entire inference process. In terms of data pre-reading, the feature map is divided into blocks to eliminate the dependency between the layers in the traditional convolution process, and the data is read by channel reading to reduce the cache occupancy. This method uses Xilinx's PYNQ-Z2 chip as an experimental platform. First, the feature map is loaded through the processing system (PS, Processing System, a processing system based on dual ARM Cortex A9 cores, which integrates memory storage, external memory interface and a large number of peripheral interfaces) of PYNQ-Z2, and the data is transferred to the programmable logic (PL, Programmable Logic, a programmable logic unit based on Xilinx 7 series architecture) part by direct address access (DMA, Direct Memory Access) using AXI4 protocol. The PS part is mainly responsible for data input and result output, as well as completing the configuration of the PS side; while the PL part is responsible for customizing the design of the entire network, dividing the feature map transmitted from the PS part into blocks, and changing the data reading direction when reading data to reduce memory usage. When facing multi-layer loop calculations, the pipeline design will be carried out in the form of loop expansion. Then, a customized IP core is generated through Vivado HLS and Vivado tools and deployed to the FPGA development board. Finally, the data results of the network can be viewed through Jupyter Notebook (online editing tool) based on Web access on the PS side.
[0006] The technical solution adopted by the present invention is a block convolution method for pre-reading data by channel based on an FPGA platform, which belongs to a method of software and hardware collaborative design. The implementation steps of the method are as follows:
[0007] Step 1: Process the data in chunks.
[0008] The input feature map is divided into 4 small blocks of the same size. In order to ensure that the output feature map after convolution and rearrangement of each small block is the same size as the output feature map obtained after the input feature map is convolved, it is necessary to fill the four sides of each small block. In the present invention, 0 values are selected for filling. If the input feature map size is 8*8, the convolution kernel size is 3*3, and the step size is 1, the input feature map is now divided into 4 small blocks of equal size. In order to ensure that the output feature map sizes obtained before and after segmentation are the same, the size of the 4 small blocks needs to be filled to 6*6.
[0009] Step 2: Read data by channel.
[0010] Generally speaking, the dimensions of the input feature map are arranged in the order of batch size, input channels, height, weight, and the dimensions of the convolution kernel are arranged in the order of output channels, input channels, height, weight. Changing the reading direction of the data requires transposing these dimensions so that the input channel becomes the last dimension. In this way, the data will be read by channel, and the occupied cache size is at most K*W*C (K is the convolution kernel size, W is the input feature map width, and C is the number of input feature map channels), which will significantly reduce the use of buffer memory.
[0011] Step 3. Convolution operation optimization.
[0012] Convolution can be regarded as a matrix multiplication operation of the input feature map and the weight. The formula is as follows:
[0013] y=g(W T x) (1)
[0014] Among them, g represents the activation function, x represents the input feature map, and W T Represents the transpose of the convolution kernel, y represents the features learned by the convolution kernel W on the input feature map x, that is, the output feature map.
[0015] During the convolution optimization process, parallel computing and pipeline methods are used to accelerate matrix operations. In FPGA, both the input feature map and the convolution kernel are stored in smaller memory units so that data can be obtained in parallel; during the calculation process, it consists of many register stages, and instead of waiting for the current calculation to be completed, it continues to obtain memory in the next clock cycle, thereby creating a data processing pipeline. Therefore, the output data will be generated at a high throughput, thereby increasing the overall data processing speed.
[0016] Step 4. According to step 3, predict the key code of the operation and generate the IP core.
[0017] The convolution IP core includes the following structures:
[0018] a.Input Buffer: input data buffer;
[0019] b.Weight Buffer: weight data buffer;
[0020] c.Output Buffer: output data buffer;
[0021] d.AXI-Stream: allows unlimited burst transmission of data and is a high-performance data transmission protocol;
[0022] e.AXI-Lite: A lightweight address-mapped single-transfer protocol suitable for control signal transmission of hardware computing units;
[0023] f.Memory Interconnect: data path interconnection;
[0024] g.DMA: Direct memory access, responsible for data transfer between the accelerator and memory;
[0025] h.Control Interconnect: control signal line interconnection;
[0026] i.PE: multiplication unit;
[0027] The present invention designs a large number of parallel processing units (PEs) to improve computing performance. Each PE is responsible for the multiplication calculation of the pixels of the input feature map and the corresponding weights, followed by an addition tree structure to accumulate the intermediate results of the convolution.
[0028] Step 5. Under the operating system, burn the hardware bitstream file into the development board and write the drivers for the IP core and hardware devices.
[0029] In the driver writing, each hardware device is accessed in the way of accessing Linux character devices; in the DMA driver writing, a mapping mechanism is used to fill data; the mapping mechanism is to reserve a continuous physical memory in the memory, map it to an address in the kernel space, and then map the kernel space address to the user space.
[0030] Beneficial effects of the present invention:
[0031] In the present invention, by dividing the feature map into blocks and filling the divided feature map, the dependency between feature maps is eliminated; by pre-reading data by channel, the use of buffer memory is effectively reduced; in the convolution process, the loop pipelining and loop expansion are used to fully utilize the large amount of computing resources of FPGA and increase the parallel strength of calculation. At the same time, in order to make the pipeline interval of the pipeline 1, the present invention adjusts the parallel loop expansion layer to the innermost loop. This method fully utilizes data locality, reduces the data bandwidth required by the hardware, and improves the computing throughput. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 For the overall architecture.
[0033] Figure 2 Traditional convolution and block convolution
[0034] Figure 3It is the block filling of the feature map. Where a and b are traditional convolution and block convolution respectively.
[0035] Figure 4 Pre-read the data.
[0036] Figure 5 is matrix multiplication. a and b are the default reading mode and channel reading mode respectively.
[0037] Figure 6 Optimized design for matrix multiplication.
[0038] Where a and b are matrix multiplication and pipeline design respectively. DETAILED DESCRIPTION
[0039] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of this specification, the technical solutions in the embodiments of this specification will be described in detail below in conjunction with the drawings in the embodiments of this specification.
[0040] The technical solution adopted by the present invention is a block convolution method for pre-reading data by channel based on FPGA platform, which belongs to a method of software and hardware collaborative design. The overall architecture is as follows Figure 1 As shown, the implementation steps of this method are as follows:
[0041] Step 1: Process the data in chunks.
[0042] exist Figure 2 In (a), we can see that in the traditional convolution process, the data dependency between consecutive layers is not eliminated. When B1 is calculated, C1 is not calculated immediately because B2, B3 and B4 are required to participate in the C1 calculation. Therefore, in the present invention, a block design scheme is adopted to reduce the FPGA memory usage, such as Figure 2 (b) as shown.
[0043] The input feature map is divided into 4 small blocks of the same size. In order to ensure that the output feature map after convolution and rearrangement of each small block is the same size as the output feature map obtained after the input feature map is convolved, it is necessary to fill the four sides of each small block. In this invention, 0 values are used for filling. If the input feature map size is 8*8, the convolution kernel size is 3*3, and the step size is 1, the input feature map is now divided into 4 small blocks of equal size. In order to ensure that the output feature map sizes obtained before and after segmentation are the same, the size of the 4 small blocks needs to be filled to 6*6. Figure 3 shown.
[0044] Step 2: Read data by channel.
[0045] exist Figure 4In the above example, the data reading direction is changed accordingly. Generally speaking, the dimensions of the input feature map are arranged in the order of batch size, input channels, height, and weight, and the dimensions of the convolution kernel are arranged in the order of output channels, input channels, height, and weight. In both arrangements, the input channels are arranged before the height and width. If these matrices are decomposed into data streams, almost the entire feature map needs to be stored in the buffer, which is obviously not a recommended operation for the limited memory of FPGA. Therefore, it is necessary to change the data reading direction by transposing these dimensions to make the input channel the last dimension. In this way, the data will flow between channels, which means that the buffer size is at most K*W*C, which will significantly reduce the usage of buffer memory.
[0046] Step 3. Convolution operation optimization.
[0047] The main operation of the convolutional layer is matrix multiplication, as shown in formula (1). Figure 5 In the matrix C = A*B, each value in C is obtained by multiplying the corresponding row A and column B.
[0048] y=g(W T x) (1)
[0049] Among them, g represents the activation function, x represents the input feature map, and W T Represents the transpose of the convolution kernel, y represents the features learned by the convolution kernel W on the input feature map x, that is, the output feature map.
[0050] Figure 6 It shows how FPGA can accelerate matrix addition through parallel computing and pipelining. Assume that the dimension of matrix multiplication is (M*3)*(3*N). Figure 6 (a) is the digital hardware generated on the FPGA. Memory blocks (1,3,5) store input feature maps, and memory blocks (2,4,6) store weights. Both the input feature maps and weights are divided into smaller memory blocks so that multiple input data can be accessed in parallel. In each clock cycle, three new operand pairs are fetched into registers A1-3 and B1-3 for parallel calculation. The calculation process consists of many register stages. Instead of waiting for the current calculation to complete, it continues to fetch memory in the next clock cycle, creating a data processing pipeline (such as Figure 6 (b)). Therefore, the output data will be generated at a high throughput, thereby increasing the overall data processing speed.
[0051] Step 4. According to step 3, predict the key code of the operation and generate the IP core.
[0052] The convolution IP core includes the following structures:
[0053] a.Input Buffer: input data buffer;
[0054] b.Weight Buffer: weight data buffer;
[0055] c.Output Buffer: output data buffer;
[0056] d.AXI-Stream: allows unlimited burst transmission of data and is a high-performance data transmission protocol;
[0057] e.AXI-Lite: A lightweight address-mapped single-transfer protocol suitable for control signal transmission of hardware computing units;
[0058] f.Memory Interconnect: data path interconnection;
[0059] g.DMA: Direct memory access, responsible for data transfer between the accelerator and memory;
[0060] h.Control Interconnect: control signal line interconnection;
[0061] i.PE: multiplication unit;
[0062] The present invention designs a large number of parallel processing units (PEs) to improve computing performance. Each PE is responsible for the multiplication calculation of the pixels of the input feature map and the corresponding weights, followed by an addition tree structure to accumulate the intermediate results of the convolution.
[0063] Step 5. Under the operating system, burn the hardware bitstream file into the development board and write the drivers for the IP core and hardware devices.
[0064] In the driver writing, each hardware device is accessed in the way of accessing Linux character devices; in the DMA driver writing, a mapping mechanism is used to fill data; the mapping mechanism is to reserve a continuous physical memory in the memory, map it to an address in the kernel space, and then map the kernel space address to the user space.
Claims
1. A block convolution method for pre-reading data by channel based on FPGA platform, Features: Through block convolution and data pre-reading by channel; at the same time, the convolution process is optimized by using parallel pipelining method; Step 1. Process the data in blocks; The input feature map is divided into 4 small blocks of the same size. In order to ensure that the output feature map after convolution and rearrangement of each small block is the same size as the output feature map obtained after convolution of the input feature map, it is necessary to fill the four sides of each small block with 0 values. Step 2. Read data by channel; The dimensions of the input feature map are arranged in the order of batch size, input channels, height, and weight, and the dimensions of the convolution kernel are arranged in the order of output channels, input channels, height, and weight. Changing the reading direction of the data means transposing these dimensions so that the input channel becomes the last dimension. The data will be read by channel, and the occupied cache size is at most K*W*C, where K is the convolution kernel size, W is the input feature map width, and C is the number of input feature map channels; Step 3. Convolution operation optimization; The matrix multiplication operation of the convolution input feature map and the weight is as follows: Among them, g represents the activation function, represents the input feature map, W T Represents the transpose of the convolution kernel, and y represents the convolution kernel W in the input feature map The features learned above are output feature maps; In the convolution optimization process, parallel computing and pipeline methods are used to accelerate matrix operations. In FPGA, both the input feature map and the convolution kernel are stored in memory units so that data can be obtained in parallel. In the calculation process, it consists of many register stages, and instead of waiting for the current calculation to be completed, it continues to obtain memory in the next clock cycle, thus creating a data processing pipeline. Step 4. According to step 3, predict the key code of the operation and generate the IP core; The convolution IP core includes the following structures: a.Input Buffer: input data buffer; b.Weight Buffer: weight data buffer; c.Output Buffer: output data buffer; d.AXI-Stream: allows unlimited burst transmission of data and is a high-performance data transmission protocol; e.AXI-Lite: A lightweight address-mapped single-transfer protocol suitable for control signal transmission of hardware computing units; f.Memory Interconnect: data path interconnection; g.DMA: Direct memory access, responsible for data transfer between the accelerator and memory; h.Control Interconnect: control signal line interconnection; i.PE: multiplication unit; Each PE is responsible for the multiplication of the pixels of the input feature map and the corresponding weights, followed by an addition tree structure to accumulate the intermediate results of the convolution; Step 5. Under the operating system, burn the hardware bitstream file into the development version and write the drivers for the IP core and hardware devices; In the driver writing, each hardware device is accessed in the way of accessing Linux character devices; in the DMA driver writing, a mapping mechanism is used to fill data; the mapping mechanism is to reserve a continuous physical memory in the memory, map it to an address in the kernel space, and then map the kernel space address to the user space.
Citation Information
Patent Citations
SOC-based data reuse convolutional neural network accelerator
CN108171317A
Deep neural network acceleration platform based on FPGA
CN108229670A
