Convolutional neural network software and hardware collaborative accelerator based on ARM and FPGA

Abstract

The invention relates to a convolutional neural network software and hardware collaborative accelerator based on an ARM and an FPGA, which is characterized in that a convolutional neural network is disassembled according to a network layer based on the essential characteristics of mathematical operation, a convolutional layer and a full connection layer which are essentially multiplied and added operation are deployed on the FPGA, a Softmax layer which is essentially exponential and division operation is deployed on the ARM, and the convolutional neural network software and hardware collaborative accelerator based on the ARM and the FPGA is obtained. The pooling layer is deployed on the FPGA; parameters on a convolution layer, a pooling layer and a full-connection layer deployed on an FPGA are quantized into a fixed-point number format and stored in an on-chip ROM of the FPGA, and a Softmax layer deployed on an ARM adopts a floating-point number operation mode; a channel parallel strategy is adopted for a convolution layer and a pooling layer deployed on the FPGA, and convolution, pooling and full-connection hardware operation units are designed respectively.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence embedded technology, specifically a hardware and software co-accelerator for convolutional neural networks based on ARM and FPGA, used for edge deployment of convolutional neural networks. Background Technology

[0002] Convolutional neural networks (CNNs) are computationally intensive models. With the continuous expansion and deepening of applications such as computer vision, natural language processing, and speech recognition, the demand for high-performance, low-power, and low-latency algorithm acceleration is constantly growing. The computing power of edge embedded devices themselves is insufficient to handle the massive computational load and storage space required by neural networks. FPGA-based CNN processors are one effective way to address this demand. However, different layers of a neural network have different operational characteristics. Convolutional and fully connected layers perform multiplication and addition operations, pooling layers perform maximum (comparison) or average (addition and division) operations, and the Softmax layer performs division and exponentiation operations. For FPGAs, exponentiation and division operations result in complex circuit structures and consume significant hardware resources. Many researchers have approximated the Softmax layer on FPGAs using mathematical transformations (addition and shifting to replace exponentiation and division), but this incurs a loss of accuracy. Furthermore, many researchers deploy neural networks using an ARM+FPGA architecture. The FPGA is designed with parallel network accelerators for the network, while the ARM is only responsible for program initialization and accelerator invocation, without participating in network computation, thus failing to leverage the advantages of heterogeneity and resulting in low acceleration efficiency. Based on this, the present invention proposes a hardware and software co-accelerator for convolutional neural networks based on ARM and FPGA, which fully leverages the advantages of heterogeneous architecture to achieve efficient deployment of convolutional neural networks. Summary of the Invention

[0003] The purpose of this invention is to propose a hardware-software co-accelerator for convolutional neural networks based on ARM and FPGA, building upon existing technologies. The technical solution is as follows:

[0004] A hardware-software co-accelerator for convolutional neural networks based on ARM and FPGA decomposes the convolutional neural network into layers based on the essential characteristics of mathematical operations. Convolutional and fully connected layers, which are essentially multiplication and addition operations, are deployed on the FPGA; Softmax layers, which are essentially exponential and division operations, are deployed on the ARM; and pooling layers are also deployed on the FPGA. The parameters of the convolutional, pooling, and fully connected layers deployed on the FPGA are quantized into fixed-point format and stored in the on-chip ROM of the FPGA. The Softmax layer deployed on the ARM uses floating-point operations. A channel-parallel strategy is adopted for the convolutional and pooling layers deployed on the FPGA, with separate hardware operation units designed for convolution, pooling, and fully connected layers.

[0005] When the FPGA has sufficient hardware resources, multiple hardware computing units are built according to the number of channels to execute the operations of convolutional layers, pooling layers and fully connected layers in parallel. Otherwise, a grouping operation strategy is adopted. When the number of channels is too large and the FPGA's hardware resources cannot build computing units for each channel, a grouping strategy is adopted to group all channels. The FPGA only builds hardware computing units for each channel within the group. The computing units are time-division multiplexed to implement the operation of all channels. Parallel computing is performed within the group, and serial computing is performed between groups.

[0006] The hardware operation units are as follows: the convolutional layer hardware operation unit includes a multiplier, an adder, and a comparator, wherein the multiplier and adder are used for convolution operations, and the comparator is used for ReLU activation function operations; the pooling layer hardware operation unit is composed of comparators and is used for max pooling operations; the fully connected layer hardware operation unit is composed of multiple multipliers and multiple adders, and the adders are constructed into an addition tree structure.

[0007] Furthermore, the ARM and FPGA perform collaborative computing, and the FPGA and ARM communicate with each other via the AXI bus.

[0008] The method for constructing the LeNet network model using the aforementioned convolutional neural network hardware / software co-accelerator includes:

[0009] The LeNet network was constructed and trained using Matlab to obtain the network model parameters;

[0010] Using the obtained network model parameters, a LeNet network is constructed using HLS. The network parameters are quantized at fixed points and stored in the on-chip ROM of the FPGA. The convolutional layer, pooling layer, and fully connected layer are optimized using pragma statements.

[0011] The hardware operation of a single-channel convolution is as follows: the input features and weight parameters are multiplied by a multiplier, the bias parameters are added by an adder, and the ReLU activation function is constructed by comparing with 0 by a comparator to obtain the convolution result. When the convolution is expanded according to the channel dimension, the hardware resources exceed the limit, so a grouped parallel strategy is adopted.

[0012] The pooling layer is expanded according to the channel dimension, and pooling operations are performed in parallel between different channels;

[0013] The operations of the fully connected layer are converted into matrix multiplication operations. A multiplication-addition tree is constructed for matrix multiplication. The output neurons of the fully connected layer are grouped. An addition tree is constructed within each group. The multiplication operations and addition trees within each group are processed in parallel pipeline.

[0014] Furthermore, the LeNet network built in HLS is encapsulated into an AXI interface IP core. Then, the LeNet network IP core and ZYNQ PS processor are added to Vivado. An HP interface and a USART interface are added to the ZYNQ PS processor to build a hardware and software co-processing system.

[0015] The advantages of this invention are as follows:

[0016] 1) High efficiency: The accelerator of this invention is split according to the characteristics of the algorithm. ARM and FPGA are each responsible for the algorithm part they are good at. ARM and FPGA work together to implement the deployment of the algorithm, giving full play to the advantages of the heterogeneous architecture.

[0017] 2) Low power consumption: The accelerator of this invention quantizes complex floating-point operations into fixed-point operations, and allocates complex exponentiation and division operations to the ARM, reducing FPGA circuit resource overhead and further reducing the power consumption generated by the hardware circuit, making it suitable for mobile scenarios with high power consumption requirements such as drones. Attached Figure Description

[0018] Figure 1 This diagram illustrates the allocation of accelerator resources (the example image to be recognized in the diagram comes from the MNIST handwritten digit dataset and is a handwritten "5").

[0019] Figure 2 Schematic diagram of parallel unfolding of convolutional layers

[0020] Figure 3 A schematic diagram of convolutional layer operations

[0021] Figure 4 Diagram of hardware computing units for convolutional layers

[0022] Figure 5 Schematic diagram of parallel expansion of pooling layers

[0023] Figure 6 Schematic diagram of pooling layer operation

[0024] Figure 7 Schematic diagram of the pooling layer hardware computing unit operation

[0025] Figure 8 Schematic diagram of fully connected layer operations

[0026] Figure 9 Schematic diagram of the fully connected layer hardware computing unit

[0027] Figure 10 Diagram of the hardware and software co-processing system in Vivado Detailed Implementation

[0028] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

[0029] The basic scheme of this invention will be described below. This invention proposes a hardware-software co-deployment strategy for convolutional neural networks (CNNs). The CNN is decomposed into layers according to the essential characteristics of mathematical operations. Convolutional layers and fully connected layers, which are essentially multiplication and addition operations, are deployed on an FPGA, while the Softmax layer, which is essentially exponential and division operations, is deployed on an ARM processor. Since pooling layers are usually placed after convolutional layers, they are also deployed on the FPGA to reduce latency caused by data flow. To reduce the circuit complexity of the FPGA and improve computational speed, a fixed-point quantization strategy is adopted. The parameters of the convolutional, pooling, and fully connected layers deployed on the FPGA are quantized into fixed-point number format and stored in the on-chip ROM of the FPGA to reduce latency and power consumption caused by off-chip memory access. The Softmax layer deployed on the ARM processor still uses floating-point arithmetic. A channel parallel strategy is adopted for the convolutional and pooling layers on the FPGA. Dedicated hardware computation units are designed for convolution, pooling, and fully connected layers. Multiple hardware computation units are constructed based on the number of channels to execute the operations of the convolutional, pooling, and fully connected layers in parallel. To address the resource shortage issue in FPGAs caused by the channel parallel strategy, a grouping operation strategy is adopted, with parallel computation within a group and serial computation between groups.

[0030] The proposed hardware-software co-deployment strategy involves collaborative computing between ARM and FPGA. ARM is the software component, while FPGA is the hardware component. Each component performs its respective computational strengths, and communication between them is via the AXI bus. Based on the computational characteristics of each layer in a convolutional neural network (CNN), the CNN primarily consists of convolutional layers, pooling layers, fully connected layers, and softmax layers. Convolutional layers essentially perform multiplication and addition operations, pooling layers perform averaging or maximum value operations, fully connected layers perform multiplication and addition operations, and softmax layers perform exponentiation and division operations. FPGA excels at repetitive operations like multiplication and addition, but the circuitry for exponentiation and division is very complex. Therefore, the operations of convolutional, pooling, and fully connected layers are assigned to the FPGA. ARM excels at exponentiation and division operations, so the softmax layer is assigned to the ARM. Data transmission between the FPGA and ARM is achieved via the AXI high-speed bus.

[0031] The parameter quantization strategy is as follows: floating-point operations in FPGAs generate complex circuit structures and consume a lot of hardware resources, while fixed-point operations generate relatively simple circuits on FPGAs. In order to improve the computing efficiency of FPGAs, the parameters of convolutional layers, pooling layers and fully connected layers operated on FPGAs are quantized into fixed-point format for operation, while the Softmax layer operated on ARM is still operated in floating-point format.

[0032] The described channel grouping parallel strategy, since there is no data dependency between different input and output channels of the convolutional and pooling layers, expands the convolutional and pooling layers according to the channel dimension. Each channel uses a separate computation unit for computation, constructing multiple identical computation units to perform operations on all channels. When the number of channels is too large and the FPGA's hardware resources cannot build a computation unit for each channel, a grouping strategy is adopted. All channels are grouped, and the FPGA only builds its own hardware computation unit for each channel within the group, time-division multiplexing the computation units of the group to perform operations on all channels.

[0033] The hardware computation units for the convolutional layers include a multiplier, an adder, and a comparator. The multiplier and adder are used for convolution operations, and the comparator is used for ReLU activation function operations. The hardware computation units for the pooling layers consist of comparators (max pooling operations), where the comparators are used to extract the maximum value. The hardware computation units for the fully connected layers include multiple multipliers and multiple adders. To improve the efficiency of addition operations, the adders are constructed into an addition tree structure.

[0034] The steps of the present invention are described in detail below as embodiments of the invention and are not the only forms that can be made or utilized. Other embodiments that can achieve the same function should also be included within the scope of the present invention.

[0035] This invention uses the LeNet network as an example for specific implementation.

[0036] Step 1: Neural Network Algorithm Analysis

[0037] The mathematical model of convolution operation is

[0038]

[0039]

[0040] The output feature parameter is O, the input feature parameter is I, and the number of channels in the output feature map is C. o The number of channels in the input feature map is C. i The weight parameter is W, and the kernel length is K. l The kernel width is K w The output feature map has a length of L.o The output feature map width is W o .

[0041] The mathematical model for operations in fully connected layers and softmax layers is as follows:

[0042]

[0043] The input features of the fully connected layer are X = [x1, x2, ..., x...]. n The weight parameters of the fully connected layer are W = [w1, w2, ..., w m The output of the fully connected layer is w·x, and the categories are H={h1,h2,...,h...}. m}, category h i The probability is P k (y=h i ;x k ).

[0044] As can be seen from formulas (1) and (2), the operations of convolutional layers and fully connected layers are essentially multiplication and addition operations, while the operations of Softmax layers are actually exponential and division operations.

[0045] Step 2: Neural Network Hardware and Software Allocation

[0046] As attached Figure 1 As shown, since FPGAs excel at repetitive computations and addition and multiplication operations, the network layers involved in addition and multiplication (convolutional layers and pooling layers) are allocated to the FPGA. Because pooling layers typically follow convolutional layers and are usually comparison operations (max pooling), they are also allocated to the FPGA. Since FPGAs generate complex circuit structures for exponentiation and division operations, resulting in high hardware resource overhead, the Softmax layer is allocated to the ARM. To reduce latency and power consumption caused by off-chip memory access, the model parameters on the convolutional and fully connected layers are stored in the FPGA's on-chip ROM. The ARM and FPGA communicate via the AXI bus.

[0047] Step 3: Network Model Construction

[0048] The LeNet network was constructed using Matlab and trained on the MNIST dataset to obtain the network model parameters. The LeNet network was then constructed using HLS (High-Level Synthesis), and the network parameters were quantized into 32-bit fixed-point quantization (16 bits for integers and 16 bits for decimals) and stored in the on-chip ROM of the FPGA. Finally, pragma statements were used to optimize the convolutional layers, pooling layers, and fully connected layers.

[0049] Step 4: LeNet Network Acceleration Design

[0050] Design to accelerate convolutional layer operations:

[0051] As attached Figure 2 As shown, since there is no data dependency between different input and output channels in the convolutional layer, the convolutional layer is expanded according to the dimensions of input and output channels, and convolution operations are performed in parallel between different channels.

[0052] As attached Figure 3 The diagram shows a single-channel convolution operation. The construction for a single-channel convolution is shown in the attached diagram. Figure 4 The convolutional hardware operation unit shown in the diagram multiplies the input features and weight parameters using a multiplier, then adds the bias parameters using an adder, and finally compares them with 0 using a comparator (the comparator compares with 0 to construct the ReLU activation function), thus obtaining the convolution result. The pseudocode for constructing the convolutional layer using HLS is shown in Algorithm 1. Here, OUT_SIZE is the length and width of the output feature map in each channel, KERNEL_SIZE is the length and width of the convolutional kernel in each channel, and KERNEL_CHANNEL is the number of channels in the convolutional kernel. Parallel unrolling and pipelined processing are performed on the input and output channels (corresponding to line 10 of the algorithm). The third dimension of the input feature map is fully partitioned (corresponding to line 1 of the algorithm). The first and second dimensions of the convolutional kernel are fully partitioned (corresponding to lines 2 and 3 of the algorithm). The bias is fully partitioned (corresponding to line 4 of the algorithm). The third dimension of the output feature map is fully partitioned (corresponding to line 5 of the algorithm).

[0053]

[0054]

[0055] When the hardware resources exceed the limit after expansion along the channel dimension, a grouped parallel strategy is adopted. The grouped parallel strategy is an improvement on Algorithm 1, specifically modifying lines 1, 3, 10-15, and 22-27. The pseudocode for the convolutional layer hardware operation under the grouped parallel strategy is shown in Algorithm 2. The output channels after parallel expansion are grouped, with N output channels in each group (corresponding to lines 10 and 12 of the algorithm). Parallel pipeline expansion is performed on the N output channels in each group first (corresponding to line 11 of the algorithm). Different groups are executed serially. The full partitioning of the first dimension array of the convolution kernel is changed to the partitioning of the array with a factor of N (corresponding to line 1 of the algorithm). The full partitioning of the bias array is changed to the partitioning of the array with a factor of N (corresponding to line 3 of the algorithm). The full partitioning of the third dimension array of the output feature map is changed to the partitioning of the array with a factor of N (corresponding to line 5 of the algorithm).

[0056]

[0057]

[0058] Pooling layer computation acceleration design:

[0059] As attached Figure 5 As shown, since the pooling layer only compresses the feature map and there is no correlation between different channels (the input and output channels of the pooling layer correspond one-to-one, so the input and output channels are collectively referred to as channels), the pooling layer is expanded according to the channel dimension, and the pooling operation is performed in parallel between different channels.

[0060] As attached Figure 6 The diagram shows the pooling operation for one channel. The construction for single-channel pooling is shown in the attached diagram. Figure 7 The pooling hardware unit shown in the diagram obtains the maximum value from the input layer data through a comparator. The pseudocode for constructing the pooling layer using HLS is shown in Algorithm 3. Here, IN_SIZE is the length and width of the input feature map in each channel, IN_CHANNEL is the number of channels in the input feature map, OUT_SIZE is the length and width of the output feature map in each channel, and OUT_CHANNEL is the number of channels in the output feature map. During pooling, IN_CHANNEL and OUT_CHANNEL have the same value. The channels are expanded in parallel and pipelined (corresponding to line 6 of the algorithm). The third dimension of the input feature map is fully partitioned (corresponding to line 1 of the algorithm), and the third dimension of the output feature map is fully partitioned (corresponding to line 2 of the algorithm).

[0061]

[0062]

[0063] Fully connected layer computation acceleration design:

[0064] As attached Figure 8 As shown, the operations of the fully connected layer are transformed into matrix multiplication operations, and then a matrix multiplication mechanism is constructed as shown in the attached diagram. Figure 9The multiplication and addition tree shown is illustrated in the pseudocode for constructing a fully connected layer using HLS, as shown in Algorithm 4. Here, IN_SIZE represents the number of all input neurons in the fully connected layer, OUT_SIZE represents the number of all output neurons in the fully connected layer, the output neurons are grouped, unit_OUT_SIZE represents the number of output neurons in each group, IN_SIZE / unit_OUT_SIZE represents the number of groups after grouping the output neurons, add_unit_0 and add_unit_1 are the two branches of the addition tree, and an addition tree is constructed within each group (corresponding to lines 7-15 of the algorithm; only a two-level addition tree example is shown here). The multiplication operations and addition trees within each group are processed in parallel pipeline (corresponding to line 3 of the algorithm). The second dimension of the weights is partitioned into an array with a factor of unit_OUT_SIZE (corresponding to line 1 of the algorithm), and the first dimension of the input neurons is partitioned into an array with a factor of unit_OUT_SIZE (corresponding to line 2 of the algorithm).

[0065]

[0066] Step 5: Model Deployment

[0067] The LeNet network built in HLS is encapsulated into an AXI interface IP core. Then, the LeNet network IP core and a ZYNQ PS processor are added to Vivado. An HP interface and a USART interface are added to the ZYNQ PS processor to build a hardware-software co-processing system. Figure 10 As shown, ARM code is written in the SDK. First, the program is initialized, then the input image is transmitted to the FPGA accelerator, and then the FPGA's calculation results are obtained. After that, the probability of each category is obtained through the Softmax layer, and then transmitted to the computer through USART to view the final inference result.

[0068] Experimental results:

[0069] Table 1 shows the network model quantization experiment. After training the LeNet network on the MNIST dataset, 100 images were used to test the inference accuracy. The inference accuracy using floating-point operations was 99%. When the number of decimal places was greater than 9, the accuracy loss was only 1%. When the number of decimal places was less than 9, the accuracy loss was severe.

[0070] Table 1. Network Model Quantization Experiment (Fixed <Decimal Places>)

[0071]

[0072] Table 2. Deployment Results of LeNet Network Hardware and Software Co-processing

[0073]

[0074] The LeNet network was deployed on two development boards, ZYNQ7020 and ZU4EV. Table 2 shows the deployment results of the LeNet network hardware and software co-processing, where the parameter quantization is a 32-bit fixed-point number (16 bits for integers and 16 bits for fractions). The results show that on the ZYNQ7020, the processing time for a single image is 1.066ms with a power consumption of 2.188W, which is 32 times faster than the pure ARM implementation. On the ZU4EV, the processing time for a single image is 0.818ms with a power consumption of 2.246W, which is 36 times faster than the pure ARM implementation.

Claims

1. A hardware-software co-accelerator for convolutional neural networks based on ARM and FPGA, which decomposes the convolutional neural network into layers based on the essential characteristics of mathematical operations. Convolutional layers and fully connected layers, which are essentially multiplication and addition operations, are deployed on the FPGA; Softmax layers, which are essentially exponential and division operations, are deployed on the ARM; and pooling layers are deployed on the FPGA. The parameters of the convolutional, pooling, and fully connected layers deployed on the FPGA are quantized into fixed-point format and stored in the on-chip ROM of the FPGA. The Softmax layer deployed on the ARM uses floating-point arithmetic. A channel-parallel strategy is adopted for the convolutional and pooling layers deployed on the FPGA, and hardware operation units for convolution, pooling, and fully connected layers are designed separately. When the FPGA has sufficient hardware resources, multiple hardware computing units are built according to the number of channels to execute the operations of convolutional layers, pooling layers and fully connected layers in parallel. Otherwise, a grouping operation strategy is adopted. When the number of channels is too large and the FPGA's hardware resources cannot build computing units for each channel, a grouping strategy is adopted to group all channels. The FPGA only builds hardware computing units for each channel within the group. The computing units are time-division multiplexed to implement the operation of all channels. Parallel computing is performed within the group, and serial computing is performed between groups. The hardware operation units are as follows: the convolutional layer hardware operation unit includes a multiplier, an adder, and a comparator, wherein the multiplier and adder are used for convolution operations, and the comparator is used for ReLU activation function operations; the pooling layer hardware operation unit is composed of comparators and is used for max pooling operations; the fully connected layer hardware operation unit is composed of multiple multipliers and multiple adders, and the adders are constructed into an addition tree structure.

2. The convolutional neural network hardware / software co-accelerator according to claim 1, characterized in that, ARM and FPGA perform collaborative computing, and the FPGA and ARM communicate with each other via the AXI bus.

3. A method for constructing a LeNet network model using the convolutional neural network hardware / software co-accelerator as described in claim 1, characterized in that, include: The LeNet network was constructed and trained using Matlab to obtain the network model parameters; Using the obtained network model parameters, a LeNet network is constructed using HLS. The network parameters are quantized at fixed points and stored in the on-chip ROM of the FPGA. The convolutional layer, pooling layer, and fully connected layer are optimized using pragma statements. The hardware operation of a single-channel convolution is as follows: the input features and weight parameters are multiplied by a multiplier, the bias parameters are added by an adder, and the ReLU activation function is constructed by comparing with 0 by a comparator to obtain the convolution result. When the convolution is expanded according to the channel dimension, the hardware resources exceed the limit, so a grouped parallel strategy is adopted. The pooling layer is expanded according to the channel dimension, and pooling operations are performed in parallel between different channels; The operations of the fully connected layer are converted into matrix multiplication operations. A multiplication-addition tree is constructed for matrix multiplication. The output neurons of the fully connected layer are grouped. An addition tree is constructed within each group. The multiplication operations and addition trees within each group are processed in parallel pipeline.

4. The method for constructing the LeNet network model according to claim 3 involves encapsulating the LeNet network constructed in HLS into an AXI interface IP core, then adding the LeNet network IP core and a ZYNQ PS processor to Vivado, adding an HP interface and a USART interface to the ZYNQPS processor, and building a hardware and software co-processing system.

Images