A general standard convolution operator accelerator based on ARM and FPGA

Abstract

The application relates to a general standard convolution operator accelerator based on ARM and FPGA, which adopts a software and hardware cooperative processing mode; parallel computation and multi-layer fusion modes of the convolution operator accelerator are designed on the FPGA side; a convolution neural network is built by calling the convolution operator accelerator on the FPGA side through the ARM; configuration parameters are transmitted to the convolution operator accelerator on the FPGA side through the ARM to set the structure of each layer network, and network model parameters are stored into an SD card; in the program running, the ARM reads the network model parameters in the SD card and writes the network model parameters into a DDR memory, and the ARM and the FPGA read the model parameters in the DDR memory to perform operation; the ARM transmits the network inference result to a PC end through a UART; the processor system adopted comprises an ARM processor kernel, a DDR controller, an SDIO controller and a UART controller.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence embedded technology, specifically a general standard convolution operator accelerator based on ARM and FPGA, used for accelerating edge inference of convolutional neural networks. Background Technology

[0002] Convolutional neural networks (CNNs) have been widely used in image processing as AI algorithms. With the development of technologies such as 5G, IoT, and embedded systems, CNNs are increasingly appearing in mobile applications. The number of layers and parameters in deep neural network models is constantly increasing, placing higher demands on hardware computing power, memory bandwidth, and data storage. Traditional computers use serial computing, which cannot adapt to complex parallel network structures. General-purpose processors cannot meet the low-power, high-performance requirements of mobile scenarios, making it impossible to deploy high-precision AI algorithms at resource-constrained edge devices. The reason for this is that existing edge chips cannot support the huge computational load generated by AI operations. Most existing edge AI chips are based on GPU or ASIC architectures. GPUs have a large number of stream processors that can perform parallel operations to reduce computation time, but GPUs increase power consumption by increasing the number of cores and clock speed. Since stable power supplies cannot be provided in industrial and complex combat environments, GPUs are not suitable for mobile applications. ASICs are custom-designed chips to meet specific needs. Although they have superior performance, ASICs have high design and manufacturing costs, long development cycles, and are not reconfigurable. FPGA is a semi-custom circuit that possesses both data parallelism and task parallelism computing capabilities. It processes data in a hardware pipeline manner, allowing for customized AI accelerators to map AI algorithms onto FPGAs to achieve hardware acceleration and maximize the exploitation of the parallelism characteristics of network models. This method offers advantages such as low cost, low power consumption, and portability, and is of great significance for the deployment of large-scale AI algorithm models, improving the edge computing performance of AI algorithms, and accelerating the process of domestically developed and controllable key equipment. Summary of the Invention

[0003] The purpose of this invention is to propose a general-purpose standard convolution operator accelerator based on ARM and FPGA, building upon existing technologies. The technical solution is as follows:

[0004] A general-purpose standard convolution operator accelerator based on ARM and FPGA employs a hardware-software co-processing approach. Parallel computation and multi-layer fusion are implemented on the FPGA side of the convolution operator accelerator. A convolutional neural network is built by the ARM processor calling the FPGA-side convolution operator accelerator. Configuration parameters are passed from the ARM processor to the FPGA-side convolution operator accelerator to set the structure of each layer, and the network model parameters are stored in an SD card. During program execution, the ARM processor reads the network model parameters from the SD card and writes them to DDR memory. The ARM and FPGA then read the model parameters from the DDR memory for computation. The ARM processor transmits the network inference results to a PC via UART. The processor system includes an ARM processor core, a DDR controller, an SDIO controller, and a UART controller. The DDR controller controls data access to memory; the SDIO controller controls data reading and writing on the SD card; and the UART controller transmits data to the PC. The ARM processor core acts as the core controller, controlling the DDR controller, SDIO controller, UART controller, and convolution operator accelerator.

[0005] The convolution operator accelerator is implemented by an FPGA and includes a convolution module, an activation module, a pooling module, a ping-pong buffer module, an AXI bus module, and an on-chip cache module. The convolution module is used to extract features from the input feature map; the activation module is used to improve the nonlinear modeling capability of the network; the pooling module is used for feature dimensionality reduction; the AXI bus module is used to transfer data between DDR, ARM, and FPGA; the on-chip cache module includes an input feature map cache, an output feature map cache, a weight parameter cache, and a bias parameter cache.

[0006] The parallel computing involves processing the input and output layers in parallel. The input feature map, output feature map, weights, and bias parameters are decomposed into matrices according to the degree of parallelism. The computational operations on the decomposed data are regarded as a processing unit (PE). Multiple PEs are designed to execute the data computation operations divided according to the degree of parallelism, and the PEs are pipelined in parallel. The data computation operations include convolution, activation, and pooling operations, which are implemented through convolution modules, activation modules, and pooling modules, respectively.

[0007] Furthermore, the ping-pong buffer module is used to improve data transmission bandwidth and access speed. It divides the on-chip buffer into two groups of different sizes, called buffer 0 and buffer 1. When the accelerator performs convolution operation on the data in buffer 0, buffer 1 loads the data required for the next convolution operation into the DDR memory. After the accelerator finishes processing the data in buffer 0, buffer 1 stores the calculation result. When the next clock cycle arrives, the accelerator directly processes the data in buffer 1, and buffer 0 loads the input data for the next convolution operation into the DDR, and so on.

[0008] Furthermore, the multi-layer fusion method is as follows: the convolutional layer, activation layer and pooling layer are fused together. When the convolutional layer and the pooling layer are adjacent, the data is directly transmitted to the pooling module after the convolutional module and the activation module have completed their calculations. After the pooling module has completed its calculations, the data is transmitted to the DDR memory. When the convolutional layer and the pooling layer are not adjacent, the data is directly transmitted to the DDR memory after the convolutional module and the activation module have completed their calculations.

[0009] Furthermore, the hardware and software co-processing involves the ARM and FPGA working together. The FPGA side designs a multi-PE parallel pipeline hardware architecture for the convolution operator and a ping-pong buffered data processing flow, and encapsulates it as an IP core with an AXI bus interface. The ARM calls the IP core to build a neural network. The ARM drives the SDIO controller to read the input feature map, weights, and bias parameters from the SD card and drives the DDR controller to transfer them to the DDR memory. The ARM drives the IP core to read the input feature data from the DDR memory through the AXI bus and stores the output feature data of the IP core operation into the DDR memory.

[0010] The advantages of this invention are as follows:

[0011] 1) Generalization: This accelerator can implement standard convolution operators of arbitrary size by adjusting the parallelism, and can adapt to FPGAs with different hardware resource sizes;

[0012] 2) Low power consumption: This accelerator has a dedicated hardware structure designed for algorithms, reducing power consumption from unnecessary circuits, making it suitable for mobile scenarios with high power consumption requirements, such as drones.

[0013] 3) High integration: This accelerator integrates convolution, activation and pooling modules, and the required layers can be selectively configured as needed. Attached Figure Description

[0014] Figure 1 A diagram illustrating convolution operations.

[0015] Figure 2 A schematic diagram of pooling operations

[0016] Figure 3 A schematic diagram of parallel convolution operations

[0017] Figure 4 Overall architecture diagram of the accelerator

[0018] Figure 5 Data flow strategy diagram

[0019] Figure 6 Diagram of a hardware and software co-processing system

[0020] Figure 7 For hardware and software co-design diagrams

[0021] Figure 8 A comparison chart of accelerator power consumption Detailed Implementation

[0022] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

[0023] The steps of the present invention are described in detail below as embodiments of the invention and are not the only forms that can be manufactured or utilized. Other embodiments that can achieve the same function should also be included within the scope of the present invention.

[0024] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, taking the acceleration of the LeNet network to achieve image recognition of the MNIST handwritten digit dataset as an example.

[0025] Step 1:

[0026] To address the issue of insufficient channel count per layer in LeNet, which hinders the full utilization of FPGA parallelism, three major improvements were made to the traditional LeNet network. The improved i-LeNet network structure is shown in Table 1. Two fully connected layers were removed to reduce network size; two convolutional layers were added to improve feature extraction capabilities; and to accommodate the parallelism of the convolution operator accelerator, the number of channels was set to 32, the convolution kernel size to 5×5, the max-pooling kernel size to 2×2, the convolution stride to 1, the max-pooling stride to 2, and the padding to 2. ReLU was used as the activation function. The i-LeNet network shown in Table 1 was built on a PC using the PyTorch framework. The model was trained on the MNIST training set using backpropagation. After training, the MNIST test set, the weights and bias parameters of the i-LeNet network were written to a bin file, which was then stored on an SD card.

[0027] Table 1 i-LeNet Network Structure

[0028]

[0029] Step Two:

[0030] Design of a convolution operator accelerator IP using High-Level Synthesis (HLS) tool. (See attached image.) Figure 1As shown, the principle of a convolutional layer is as follows: input feature data is stored in the input feature map, weight data is stored in the convolution kernel, and output feature data is stored in the output feature map. The data at the corresponding position in the input feature map is multiplied by the corresponding weight parameter in the convolution kernel. Then, all multiplication results in the convolution kernel are accumulated, and the accumulated result is mapped to the corresponding position in the output feature map. Then, the layer slides with a stride to compute the feature at the next position. Its mathematical expression is:

[0031]

[0032] cho∈[0,CH out -1]

[0033] chi∈[0,CH in -1]

[0034] kr∈[0,K r -1]

[0035] kc∈[0,K c -1]

[0036] co∈[0,C out -1]

[0037] ro∈[0,R out -1]

[0038] Where Out is the output feature parameter, In is the input feature parameter, W is the weight, and CH is the input feature parameter. in CH is the number of input channels. out K is the number of output channels. c K is the kernel length. r C is the kernel width. out R is the output feature length. out The output feature width is defined by s, the stride is cho, the layer index is ro, the row index is co, and the column index is co. Following the convolutional layer, an activation layer activates the feature parameters using the ReLU function, whose mathematical expression is:

[0039] f(x) = max(0,x)

[0040] Where x is the feature parameter. Following the activation layer, as shown in the appendix... Figure 2 As shown, pooling layers reduce the dimensionality of feature maps. Pooling operations are similar to convolution operations. The pooling kernel slides across the input feature map, taking the maximum or average value of the input feature data covered by the kernel, and mapping the result to the corresponding position in the output feature map. Then it continues to slide, calculating the feature at the next position. Since there is no data dependency between different input and output feature maps, as shown in the attached diagram... Figure 3As shown, the input and output layers are processed in parallel using HLS optimization instructions. The input feature map, output feature map, weights, and bias parameters are decomposed into matrices according to the degree of parallelism. The operation of processing one layer of convolution is treated as a processing unit (PE). Multiple PEs are used to process the convolution, activation, and pooling operations after the parallelism partitioning, and the PEs are pipelined. (See attached diagram) Figure 4 and attached Figure 5 As shown, a double buffer is initialized, and a ping-pong buffer algorithm is designed. An AXI bus IP core is encapsulated, where configurable parameters include input feature map, output feature map, weight matrix, bias matrix, number of input channels, number of output channels, parallelism, and a pooling flag.

[0041] Step 3:

[0042] As attached Figure 6 As shown, a hardware and software co-processing system is built in Vivado's Block Design. The processor system enables the SDIO controller and UART controller, and the accelerator IP is connected to the processor through the AXI-SmartConnect IP, and then a bit stream file is generated.

[0043] Step Four:

[0044] As attached Figure 7 As shown, the ARM program is designed in the SDK. The first step is to drive the SDIO controller to read the bin file from the SD card, force-cast the data in the bin file to INT16 type, and then store the data (input feature data, weight data, bias data) in the bin file into off-chip DDR memory. The second step is to define the parameter matrices (weight matrix, bias matrix, output feature matrix) for each layer of the network and initialize the convolution operator accelerator IP. The third step is to call the convolution operator accelerator IP to build the i-LeNet network. As shown in Table 1, the convolutional layer and pooling layer are adjacent in the second and third layers, so the second and third layers are merged. The convolutional layer and pooling layer are adjacent in the fifth and sixth layers, so the fifth and sixth layers are merged. The convolution operator accelerator IP is called for the first time when the network executes the first layer, for the second time when the network executes the second and third layers, for the third time when the network executes the fourth layer, and for the fourth time when the network executes the fifth and sixth layers. The fully connected layer in the seventh layer is designed by the ARM side. The fourth step is to load the data in the DDR into the parameter matrix defined in the second step, run the i-LeNet network to perform forward propagation, and send the results to the PC through the UART controller.

[0045] The invention was tested on a ZYNQ chip with an ARM+FPGA architecture. The accuracy of predicting 10,000 MNIST data points was 98.57%, the processing time for each image was 16.09ms, and the power consumption was 1.969W.

[0046] Table 2 Comparison of Existing Edge Products

[0047]

[0048] As shown in Table 2, compared with existing edge computing products, the power consumption of this invention is only 1.969W, which has a significant advantage in terms of power consumption.

[0049] As shown in Table 3, compared with accelerators in the academic field, this invention requires the fewest resources, including 33 BRAM resources, 38 DSP resources, 11345 FF resources, and 30201 LUT resources, making it friendly to edge devices with limited resources. Since the clock frequency of this invention is only 100MHz, its advantages in speed and computing power are not obvious. It also requires the lowest power consumption.

[0050] Table 3 Comparison of Accelerators in the Academic Circle

[0051]

[0052] [1]Yali Shi, Tong Gan, Shaobo Jiang, Yali Shi, Tong Gan and Shaobo Jiang, Design of Parallel Acceleration Method of Convolutional Neural Network Based on FPGA, 2020 IEEE 5th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), 2020: 133-137.

[0053] [2]Wei Wang, Kaili Zhou, Yichang Wang, Guang Wang, ZhengLin Yang, and JunYuan, FPGA Parallel Structure Design of Convolutional Neural Network (CNN) Algorithm, Microelectronics and Computer, 2019:36(4):57-62.

[0054] [3]Gan Feng, Zuyi Hu, Song Chen and Feng Wu, Energy-efficient and high-throughput FPGA-based accelerator for Convolutional Neural Networks, 2016 13th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT), 2016: 624-626.

[0055] [4]Huimin Li, Xitian Fan, Li Jiao, Wei Cao, Xuegong Zhou and Lingli Wang, A high performance FPGA-based accelerator for large-scale convolutionalneural networks, 201626th International Conference on Field Programmable Logicand Applications (FPL), Lausanne, 2016: 1-9.

[0056] [5] Liqiang Lu, Jiaming Xie, Ruirui Huang, Jiansong Zhang, Wei Lin and YunLiang, An Efficient Hardware Accelerator for Sparse Convolutional NeuralNetworks on FPGAs, 2019IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), 2019: 17-25.

[0057] like Figure 8 As shown, the accelerator of the present invention has a significant advantage in terms of power consumption.

[0058] Experiments have shown that accelerators can effectively speed up the operation of convolution operators and have extremely low power consumption, making them an effective tool for edge deployment of convolutional neural networks.

Claims

1. A general-purpose standard convolution operator accelerator based on ARM and FPGA, employing a hardware-software co-processing approach; parallel computing and multi-layer fusion are designed on the FPGA side for the convolution operator accelerator; a convolutional neural network is built by the ARM calling the FPGA-side convolution operator accelerator, and configuration parameters are passed from the ARM to the FPGA-side convolution operator accelerator to set the structure of each layer of the network, storing the network model parameters in an SD card; during program execution, the ARM reads the network model parameters from the SD card and writes them into DDR memory, and the ARM and FPGA read the model parameters from the DDR memory for computation; the ARM transmits the network inference results to the PC via UART; the processor system includes an ARM processor core, a DDR controller, an SDIO controller, and a UART controller, where the DDR controller controls data memory access; the SDIO controller controls data reading and writing in the SD card; the UART controller transmits data to the PC; and the ARM processor core acts as the core controller, controlling the DDR controller, SDIO controller, UART controller, and convolution operator accelerator. The convolution operator accelerator is implemented by an FPGA and includes a convolution module, an activation module, a pooling module, a ping-pong buffer module, an AXI bus module, and an on-chip cache module. The convolution module is used to extract features from the input feature map; the activation module is used to improve the nonlinear modeling capability of the network; the pooling module is used for feature dimensionality reduction; the AXI bus module is used to transfer data between DDR, ARM, and FPGA; the on-chip cache module includes an input feature map cache, an output feature map cache, a weight parameter cache, and a bias parameter cache. The parallel computing involves processing the input and output layers in parallel. The input feature map, output feature map, weights, and bias parameters are decomposed into matrices according to the degree of parallelism. The computational operations on the decomposed data are treated as a processing unit (PE). Multiple PEs are designed to execute the data computation operations divided according to the degree of parallelism, and the PEs are pipelined in parallel. The data computation operations include: Convolution, activation, and pooling operations are implemented through convolution modules, activation modules, and pooling modules, respectively. The multi-layer fusion method is as follows: the convolutional layer, activation layer and pooling layer are fused together. When the convolutional layer and the pooling layer are adjacent, the data is directly transmitted to the pooling module after the convolutional module and the activation module have completed their calculations. After the pooling module has completed its calculations, the data is transmitted to the DDR memory. When the convolutional layer and the pooling layer are not adjacent, the data is directly transmitted to the DDR memory after the convolutional module and the activation module have completed their calculations. The aforementioned hardware and software co-processing involves the collaborative work of ARM and FPGA. On the FPGA side, a multi-PE parallel pipeline hardware architecture for the convolution operator and a ping-pong buffered data processing flow are designed and encapsulated as an IP core with an AXI bus interface. The ARM calls the IP core to build a neural network. The ARM drives the SDIO controller to read the input feature map, weights, and bias parameters from the SD card and drives the DDR controller to transfer them to the DDR memory. The ARM drives the IP core to read the input feature data from the DDR memory through the AXI bus and stores the output feature data of the IP core operation into the DDR memory. The improved i-LeNet network structure is adopted, removing two fully connected layers to reduce network size; adding two convolutional layers to improve feature extraction capability; and setting the number of channels to 32 and the kernel size to accommodate the parallelism of the convolution operator accelerator. Set the max pooling kernel size to The convolution stride is set to 1, the max pooling stride is set to 2, the padding is set to 2, and ReLU is used as the activation function.

2. The universal standard convolution operator accelerator according to claim 1, characterized in that, The ping-pong buffer module is used to improve data transmission bandwidth and access speed. It divides the on-chip buffer into two groups, called buffer 0 and buffer 1. When the accelerator performs convolution operation on the data in buffer 0, buffer 1 loads the data required for the next convolution operation into the DDR memory. After the accelerator finishes processing the data in buffer 0, buffer 1 stores the calculation result. When the next clock cycle arrives, the accelerator directly processes the data in buffer 1, and buffer 0 loads the input data for the next convolution operation into the DDR, and so on.

Images