A convolutional neural network computing method based on heterogeneous platforms

Abstract

The present invention discloses a convolutional neural network operation method based on a heterogeneous platform. The heterogeneous platform is composed of a CPU end and an FPGA end, and the convolutional layer and the fully connected layer of the convolutional neural network are divided. The FPGA end designs an operation array to calculate matrix multiplication. The CPU end initializes the FPGA end and rearranges the data. The CPU end reads the operation result of the FPGA end, and converts the convolutional neural network operation into a matrix multiplication operation. The large-scale convolution or fully connected calculation is converted into a cyclic iterative calculation of multiple small matrices. It only needs to design according to the current layer size parameters and matrix parameters, and directly change the CPU code to complete the convolutional neural network operations of different scales. Compared with the traditional development method based on a single CPU or FPGA, the efficiency and flexibility are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of convolutional neural networks, and in particular relates to a heterogeneous platform technology. Background Art

[0002] Convolutional neural network is a feedforward neural network with advantages in image processing. With the continuous improvement of hardware platform computing power, convolutional neural network has achieved remarkable results in image recognition, target detection and speech recognition. With the improvement of hardware capabilities, the complexity of convolutional neural network is also increasing. For some large-scale networks, the huge amount of computing has posed a great challenge to development and implementation.

[0003] At present, compared with the traditional use of CPU processors for network reasoning, most researchers in the field of convolutional neural networks will use heterogeneous development methods to complete network reasoning, which can greatly improve development efficiency. OpenCL is a language for writing programs on heterogeneous platforms. Heterogeneous platforms can be composed of CPUs and GPU / DSP / FPGA hardware accelerators. At present, most mature heterogeneous development platforms use a combination of CPU+GPU.

[0004] As a programmable logic device, FPGA has great advantages due to its rich on-chip resources and development flexibility. Through the combination of CPU + FPGA, the online reconfiguration feature of FPGA as a programmable device can be fully utilized to improve the development efficiency of convolutional neural networks of different types and sizes. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention proposes a convolutional neural network operation method based on a heterogeneous platform. According to the characteristics of different network types and different network layers with different feature map sizes, convolution kernel sizes and numbers of neurons, the reconfigurable accelerated calculation of convolutional neural networks of different scales by FPGA is completed by only setting parameters of the CPU. In order to achieve the above purpose, the present invention adopts the following technical solutions.

[0006] A heterogeneous platform is composed of the CPU and FPGA ends. The convolutional layer and the fully connected layer of the convolutional neural network are divided. The FPGA end designs the operation array to calculate the matrix multiplication. The CPU end initializes the FPGA end and rearranges the data. The CPU end reads the operation results of the FPGA end and converts the convolutional neural network operation into a matrix multiplication operation.

[0007] In the Linux environment, according to the OpenCL specification, the CPU side is compiled in C++ language to generate an executable file, and the FPGA side is compiled in C language to generate a binary bit stream file. The CPU side writes the binary bit stream file to the FPGA side through the PCIe bus, and the CPU side memory reads and writes the FPGA side memory through the DMA module.

[0008] Divide the convolutional neural network into convolutional layers conv1 to conv X , fully connected layers fc1 to fc Y , define the corresponding parameters Para of each convolutional layer conv1～X , including image length H, image width W, number of channels N, step size S, zero padding P, convolution kernel size K, number of convolution kernels M, define the corresponding parameters Para of each fully connected layer fc1～Y , including the number of units Q and the number of neurons R.

[0009] The operation array uses the matrix multiplication kernel function GEMM-Kernel, which inputs two matrix data and outputs the result of matrix multiplication. The static parameters BS1, BS2, and BS3 are defined. The matrix operation size of a single GEMM-Kernel single memory read is [BS1, BS2] × [BS2, BS3]. The memory is read in a loop to complete the multiplication of two matrices.

[0010] Furthermore, G1×G2 independent GEMM units are designed on the FPGA side. Each GEMM unit independently accesses the memory on the FPGA side, reads the corresponding blocks of the matrix to be processed to the local storage in batches, completes the multiplication operation of the block matrix, accumulates the calculation results read in the loop, and writes them into the memory on the FPGA side.

[0011] Instead, each GEMM unit can simultaneously process the small-dimensional matrix multiplication of [BS1, BS2] × [BS2, BS3], and the CPU-side control operation array can process the convolution layer [K 2 ×N,M] and large dimensional matrix multiplication of the fully connected layer [Q,R].

[0012] The CPU configures BS1, BS2, and BS3 of the GEMM-Kernel function according to the size parameters of the current convolutional neural network and the design parameters of the FPGA-side operation array. According to the parameter settings of the current layer, the data to be processed is rearranged into a format that satisfies matrix multiplication, and is written into the memory of the FPGA side in sequence through DMA. The operation array on the FPGA side is started for the GEMM unit to read.

[0013] Furthermore, if the feature map size of the convolutional layer is large or the number of input and output neurons of the fully connected layer is large, the CPU divides the data to be processed and sends it to different GEMM units in blocks. Multiple GEMM units process them independently and in parallel, and the CPU reads and summarizes the processing results of each GEMM unit.

[0014] Each time the FPGA completes a layer of calculation, the CPU reads the calculation results of the FPGA, activates the next layer of calculation and rearranges or divides it. If the next layer is a convolutional layer, the CPU and FPGA read and calculate cyclically based on the zero-padding matrix. If the fully connected layer is calculated, the CPU performs Softmax classification processing.

[0015] The beneficial effects of the present invention are as follows: the calculation of the convolution layer and the fully connected layer with the largest computing demand in the network is converted into a general matrix multiplication calculation. Through the design idea of ​​the block matrix in the matrix multiplication, the larger-scale convolution or fully connected calculation process is converted into a loop iterative calculation of multiple small matrices. Through the CPU+FPGA heterogeneous platform, after completing the design of the FPGA matrix multiplication array, for the reasoning operation of convolutional neural networks of different scales, it is only necessary to design according to the current layer size parameters and the matrix multiplication array parameters. The accelerated processing of different network models and different layer parameters can be completed directly by changing the CPU code, which improves the efficiency and flexibility compared with the traditional development method based on a single CPU or FPGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a diagram of the heterogeneous platform structure. Figure 2 is the operation flow chart, Figure 3 This is the GEMM operation array structure diagram. Figure 4 This is the schematic diagram of data rearrangement. Figure 5 This is the principle diagram of matrix multiplication. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0018] The structure of heterogeneous platforms is as follows Figure 1 As shown, it consists of a CPU side and an FPGA side. The memory on the CPU side and the memory on the FPGA side can interact directly through DMA.

[0019] The operation process is as follows Figure 2 As shown in the figure, the GEMM-Kernel design is completed according to the characteristics of FPGA. A single Kernel has the function of completing the multiplication of matrix A and matrix B. A single Kernel supports matrix multiplication of size [BS1, BS2] × [BS2, BS3] when reading and writing memory at a single time. According to the actual FPGA resources, G1 × G2 independent GEMM units are designed, as shown in Figure 3As shown, a GEMM operation array is formed.

[0020] Compile the GEMM-Kernel module through the CPU, generate a binary bitstream file, burn it to the FPGA chip, and set the parameters of the target convolutional neural network on the CPU, including the size of each convolutional layer. conv1～X and the size of each fully connected layer Para fc1～Y .

[0021] The convolution layer is based on Para conv1～X The number of multiplication and addition elements that need to be completed by a single convolution kernel in a single sliding window is K 2 ×N, M convolution kernels simultaneously slide the window and the number of multiplication and addition elements that need to be completed is K 2 ×N×M, that is, this convolution layer operation needs to complete [K 2 ×N,M] dimensions.

[0022] The fully connected layer is based on Para fc1～Y The number of multiplication and addition elements required for a single neuron operation is Q, and the number of multiplication and addition elements required for R neurons to multiply weights at the same time is Q×R, that is, this fully connected layer operation needs to complete the benchmark matrix multiplication operation of [Q, R] dimensions.

[0023] The CPU rearranges the matrix multiplication data that needs to be completed according to the BS1, BS2, and BS3 parameters of the GEMM-Kernel function, such as Figure 4 As shown in the figure, the weight parameters of the M convolution kernels are rearranged, and the single row of the weight matrix is ​​arranged according to the number of channels of a single convolution kernel, and the single column is arranged according to the weight of each convolution kernel. The weight matrix and the data matrix are sent to the FPGA at the same time, so that the FPGA reads the memory data sequentially to achieve maximum read and write efficiency.

[0024] The CPU sends the dimensions of the matrix multiplication that needs to be completed to the FPGA through the Kernel parameter setting module that meets the OpenCL specification, and starts the GEMM array operation, such as Figure 5 As shown, A is read from the FPGA memory loop 11 B 11 , A 12 B 21 Complete the matrix multiplication that satisfies the size [BS1, BS2] × [BS2, BS3], accumulate the results, and get C 11 , written into the FPGA memory, the FPGA reads and calculates cyclically to obtain the entire matrix C, which is written into the memory in blocks for the CPU to read, and the CPU adds bias and activation operations.

[0025] Repeat the above steps until the convolutional layer and the fully connected layer are calculated, and the CPU performs the Softmax calculation to obtain the final classification result.

[0026] The above are embodiments of the present invention and do not limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims

1. A convolutional neural network computing method based on a heterogeneous platform, characterized in that: include: The CPU and FPGA are used to form a heterogeneous platform, and the convolutional layer and fully connected layer of the convolutional neural network are divided. The FPGA side designs a calculation array to calculate matrix multiplication. The CPU side initializes the FPGA side and rearranges the data of the initial image and intermediate results. The CPU side reads the calculation results of the FPGA side and converts the convolutional neural network operation into a matrix multiplication operation. The method of dividing the convolutional neural network into convolutional layers and fully connected layers comprises: dividing the convolutional neural network into convolutional layers conv1 to conv X , fully connected layers fc1 to fc Y , define the corresponding parameters Para of each convolution layer conv1～X , including image length H, image width W, number of channels N, step size S, zero padding P, convolution kernel size K, number of convolution kernels M, define the corresponding parameters Para of each fully connected layer fc1～Y , including the number of units Q and the number of neurons R; The FPGA end designs a calculation array to calculate matrix multiplication, including: the calculation array uses a matrix multiplication kernel function GEMM-Kernel, inputs two matrix data, outputs the result of matrix multiplication, defines static parameters BS1, BS2, and BS3, the matrix operation size of a single GEMM-Kernel single memory read is [BS1, BS2]×[BS2, BS3], and the memory is read cyclically to complete the multiplication of two matrices; The operation array adopts the matrix multiplication kernel function GEMM-Kernel, including: designing G1×G2 independent GEMM units on the FPGA side, each GEMM unit independently accesses the memory on the FPGA side, reads the corresponding blocks of the matrix to be processed to the local storage in batches, completes the multiplication operation of the block matrix, accumulates the calculation results of the cyclic reading, and writes them into the memory on the FPGA side; The CPU end initializes the FPGA end and rearranges the data of the initial image and the intermediate result, including: the CPU end configures BS1, BS2, and BS3 of the GEMM-Kernel function according to the size parameters of the current convolutional neural network and the design parameters of the FPGA end operation array, rearranges the data to be processed into a format that satisfies matrix multiplication according to the parameter settings of the current layer, writes it into the memory of the FPGA end in sequence through DMA, and starts the operation array of the FPGA end for the GEMM unit to read.

2. The convolutional neural network computing method based on a heterogeneous platform according to claim 1, characterized in that: The heterogeneous platform is composed of a CPU and an FPGA, including: in a Linux environment, using the OpenCL specification, using C++ language to compile the CPU to generate an executable file, and using C language to compile the FPGA to generate a binary bit stream file, the CPU writes the binary bit stream file to the FPGA through a PCIe bus, and the CPU memory reads and writes the FPGA memory through a DMA module.

3. The convolutional neural network computing method based on a heterogeneous platform according to claim 1, characterized in that: Also includes: Each GEMM unit can simultaneously process the small-dimensional matrix multiplication of [BS1, BS2] × [BS2, BS3], and the CPU-side control operation array can process the convolution layer [K 2 ×N,M] and large dimensional matrix multiplication of the fully connected layer [Q,R].

4. The convolutional neural network computing method based on a heterogeneous platform according to claim 1, characterized in that: The CPU reads the calculation results of the FPGA end, including: each time the FPGA end completes a layer of calculation, the CPU reads the calculation results of the FPGA end, activates the next layer of calculation and rearranges or divides it, if the next layer is a convolutional layer, the CPU reads and calculates cyclically according to the zero-padding matrix and the FPGA end, if the full connection layer is calculated, the CPU performs Softmax classification processing.

Images