Parallel processing scheme of spiking neural network on FPGA

Abstract

The invention relates to an image parallel method for accelerating a pulse neural network on an FPGA (Field Programmable Gate Array), which is characterized in that in the neural network, some operation steps are executed in parallel, so that the parallel operation of the processes can be realized on the FPGA, and pipeline operation can be realized on the part which cannot realize the parallel operation; the reasoning efficiency of the neural network is improved to the greatest extent; and meanwhile, FPGA resources are saved.

Description

Technical Field

[0001] The present invention relates to an image parallel method for accelerating a pulse neural network on an FPGA, which can realize high-speed computing of a neural network on an FPGA to the greatest extent according to network requirements while saving FPGA resources. Background Art

[0002] If the neural network needs to be accelerated on the hardware, it is necessary to realize the parallel operation of the neural network on the hardware to the greatest extent. This is the only way to shorten the reasoning time of the neural network. In the neural network, some operation steps are originally executed in parallel. We can realize the parallel operation of these processes on the FPGA, and realize pipeline operation in the parts that cannot be parallelized, so as to maximize the reasoning efficiency of the neural network.

[0003] For multi-layer neural networks, operations within the same layer are easy to perform in parallel, but it is difficult to perform in parallel between layers because there is close data exchange between the front and back layers of the neural network, and the operation of the latter layer requires the data input of the previous layer. Therefore, parallel development within a layer is feasible, while the space for parallel development between layers is small and data transmission scheduling is difficult. At the same time, the operation methods between different layers are similar, and some modules and operation resources can be reused between different layers. In this way, a neural network FPGA hardware accelerator with high operation efficiency, small memory space occupation, and strong versatility can be designed.

[0004] For the convolutional layer neurons in the spiking neural network, a large number of convolution operations are required during the calculation process, which consumes a lot of computing resources. Therefore, the parallelism of convolutional layer operations can greatly improve the operation speed of this layer and save computing time. The parallel operations of the same convolutional layer are also divided into parallel operations at different levels. Summary of the invention

[0005] The present application invents an image parallel method for accelerating a neural network on an FPGA, which can realize high-speed computing of a neural network on an FPGA to the greatest extent possible according to network needs while saving FPGA resources.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] S1. Convolution window internal parallelism. The multiplication and accumulation operations of all fxf image pixels covered by a convolution window can be performed in parallel. Figure 1 As shown, Input is the input feature map, Filter is the convolution kernel (filter), and Output is the output feature map.

[0008] If the convolution window covers the data of i11-i33, the fxf multiplication operations required to calculate o11 on the output feature map can be performed in parallel.

[0009] S2: Parallelism between input feature map convolution windows. The same input feature map convolution window is convolved at different positions. When the same convolution window slides to different positions on the image, the output values ​​at different positions of the output feature map can be calculated. When the convolution window is at different positions, the multiplication operations at the same position of the convolution window can be performed in parallel. Figure 2 As shown, for different positions o11 and o33 on the output feature map, the values ​​of the positions corresponding to w11 on their corresponding convolution windows can be executed in parallel.

[0010] S3: Input feature maps are executed in parallel. For feature maps with multi-channel input, the number of channels of the convolution kernel is the same as that of the input feature map. When the positions covered by the convolution windows on different feature maps are the same, the multiplication operations at the same positions can be executed in parallel. Figure 3 As shown, for an input feature map with three channels, the convolution kernel also has three channels, and the three channels are calculated together to obtain o11. At this time, the multiplication operations corresponding to the positions of i111, i211, and i311 can be executed in parallel.

[0011] S4: Output feature maps are executed in parallel. For single-channel input, when multiple convolution kernels correspond to multiple output feature maps, the calculation of the same convolution window of the input feature map corresponding to different convolution kernels can be executed in parallel. Figure 4 As shown, the three convolution kernels are all covered at the convolution window in the upper left corner of the input channel. The three channels 1w11, 2w11, and 3w11 correspond to the parallel multiplication operation of the first element i11 of the input feature map, and O111, o211, and o311 can be obtained at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical method of the present application, a brief introduction is given below to the drawings required in the description of the specific implementation method.

[0013] Figure 1 : Convolution window internal parallelism.

[0014] Figure 2 : Input feature map convolution windows are parallelized.

[0015] Figure 3 : Input feature maps are executed in parallel.

[0016] Figure 4 : Output feature maps are executed in parallel.

[0017] Figure 5 : Convolution window moving method.

[0018] Figure 6 :Input and output feature maps correspond DETAILED DESCRIPTION

[0019] This application invents a method for solving the problem that a large number of convolution operations are required in the calculation process of convolution layer neurons in a neural network, and a large amount of computing resources are consumed in the convolution operation process. A parallel scheme for convolution layer operations is designed, which can greatly improve the operation speed of this layer and save operation time, and the parallel operations of the same convolution layer are also divided into parallel operations at different levels.

[0020] Embodiment: This parallel method is characterized by comprising the following steps:

[0021] S1. Convolution window internal parallel implementation For the convolution of the spiking neural network, no multiplication operation is required, only f×f addition operations need to be performed, because the pulses of the spiking neural network only have states of 0 and 1. For the spiking neural network, only one input pulse state recognition is required to decide whether to add the weight value to replace the multiplication operation. However, the multiplication and addition operation will continue to be used to refer to the operation within the convolution window. The implementation of parallel execution within the convolution window requires the addition of pipeline technology.

[0022] When calculating the neurons on an output feature map corresponding to an input feature map, one convolution kernel is needed to perform a convolution operation on the input feature map at a time. Therefore, when the convolution kernel performs a convolution operation on this input feature map, the loaded convolution kernel weight will not change. When the calculation of this input feature map is completed, the output data obtained are all partial sums, because the partial sums calculated on other channels need to be added to finally obtain the output feature map. When the calculation of an input feature map is completed, it is necessary to calculate the partial sum corresponding to the next input feature map. At this time, a new convolution kernel needs to be reloaded for convolution calculation.

[0023] Therefore, when a convolution kernel of size f×f moves on a feature map and performs a convolution operation, the data that needs to be loaded is f input neuron values ​​and a partial sum; when switching feature maps, f×f input neuron values ​​and f×f convolution kernel weight parameters need to be loaded.

[0024] If each neuron and weight requires B bytes to store, the side length of the convolution kernel is f, the number of convolution windows is p×p, the number of input feature maps is n, the side length of the input feature map is nw, the height of the input feature map is nh, and the number of output feature maps is m.

[0025] The amount of data that needs to be stored in the register during the calculation process is: f*f+f*f f*f+f*n W(If the image cache uses a line buffer) The amount of data required to complete the convolution of an input feature map is: B*(f*(p*p-1)+f*f) Switching the input feature map requires loading the weights: f*f The clock cycle required for calculation is: n*m*p*p Since the partial sums of different input feature maps need to be stored for subsequent calculations, many intermediate values ​​need to be temporarily stored during the calculation process. The cache size needs to be able to store the convolution partial sum generated by the next input feature map.

[0026] S2. Parallelism between convolution windows of input feature maps. Parallelism between convolution windows of the same feature map can complete the operation of L convolution windows in each clock cycle, and each convolution window performs a multiplication and addition operation per cycle. When the operation of one input feature map is completed, the next input feature map needs to be loaded for calculation. When performing multiplication and addition operations on an input feature map, the convolution kernel will not change, only the convolution window will change, so the loaded convolution kernel data does not need to be loaded repeatedly, and only the neuron value to be calculated needs to be loaded.

[0027] However, due to the parallel calculation between different convolution windows, the loading of neuron data values ​​is different from the parallelism within the convolution window, and the neurons corresponding to different convolution windows need to be loaded. If there is overlap between the convolution windows, the neuron values ​​can be reused. If there is no overlap between the convolution windows, the input neuron data cannot be reused. In this case, the neuron data that needs to be loaded from the outside each time will increase.

[0028] If each neuron and weight requires B bytes to store, the convolution kernel side length is f, the number of convolution windows is p×p, the number of input feature maps is n, and the input feature map side length is n W , the input feature map height is n H , the number of output feature maps is m, and the number of convolution windows operating simultaneously is L. The amount of data that needs to be stored in the register during the calculation process is: L*2 The amount of data required to complete the convolution of an input feature map is: (f+p-1)*(f+p-1) Switching the input feature map requires loading the weights: f*f The clock cycle required for calculation is:

[0029] If you want to reuse data when convolution windows overlap when using input feature graph convolution windows in parallel, you need to add additional control logic to the data loading process, or add additional data storage copies to reduce the complexity of the control logic. However, these two methods require complex control logic and consume additional register resources.

[0030] S3. Parallel execution of different input feature maps. Parallel calculation of different input feature maps can make the convolution operation be performed simultaneously on the same convolution window position on L feature maps out of the total n input feature maps, and perform a cumulative multiplication and addition once per clock cycle. When f×f×L cumulative multiplication and addition operations are completed in a convolution window position, the convolution kernel is moved to the next convolution window to repeat the above cumulative multiplication and addition operations. The neuron corresponding to an output feature map is the result of the multiplication and addition operation of the convolution kernel n channels, so after the parallel calculation is completed, the results of the L channels need to be added when outputting. This parallel structure needs to load the convolution kernel weight parameters of the L channels and the input neuron parameters corresponding to the L channels.

[0031] There are two options for moving the convolution window in parallel for different input feature maps. Assume that the input feature Figure 1 There are four images in total, and the number of channels for parallel convolution is 2.

[0032] The first convolution window movement method is that the convolution kernel is initially calculated on the first convolution window of the first set of input feature maps to obtain a partial sum for temporary storage, and then the convolution window is moved to the first convolution window of the second set of feature maps and added to the partial sum obtained by the first operation to obtain the first output neuron value, and then the convolution window is moved again to the second convolution window of the first set of feature maps for calculation, and so on. In this case, switching the input feature map requires loading f×f×L convolution kernel weight data and f×f×L input neuron data.

[0033] The second convolution window moving method is to calculate the convolution kernel on the first convolution window of the first set of input feature maps to obtain partial sums for temporary storage, and then move the convolution window to the second convolution window of the first set of feature maps to obtain partial sums again for temporary storage, until the first set of input feature maps are calculated, and then move the convolution window to the first convolution window of the second set of feature maps for calculation. This window moving method can reuse data well. Each time the convolution window is moved, there is no need to reload the convolution kernel weight value, and part of the neuron input can also be reused. However, this method will generate a lot of partial sum data to be used in the calculation of the next set of input feature maps, and the storage space size is the size of the partial sum data of L input feature maps.

[0034] If each neuron and weight requires B bytes to store, the convolution kernel side length is f, the number of convolution windows is p×p, the number of input feature maps is n, and the input feature map side length is n W , the input feature map height is n H , the number of output feature maps is m, and the number of input feature maps operated simultaneously is L. The amount of data that needs to be stored in the register during the calculation process is: f*f*L+f*f*L The amount of data required to complete the convolution of an input feature map is: (f*(p*p-1)+f*f)*L Switching the input feature map requires loading the weights: f*f*L The clock cycle required for calculation is:

[0035] S4. Output feature map parallel execution. Output feature map parallel execution can make multiple convolution kernels corresponding to a convolution window corresponding to an input feature map be calculated simultaneously, so as to obtain the calculation results of the same position of multiple output feature maps. Therefore, for the case of multiple output feature maps being executed in parallel, if the number of output feature maps calculated in parallel is Q, the number of input convolution kernels is Q f×f convolution kernels, and each time the convolution window slides on an input feature map, f new input neuron data needs to be input. After the convolution of all convolution kernels on the same input feature map is completed, the convolution window is moved to the next input feature map for convolution operation. The difference between output feature map parallel execution and convolution window internal parallel execution is that output feature map parallel execution can complete L multiplication and addition operations in one clock cycle, and complete the multiplication and accumulation calculation of all convolution kernels at one convolution window after f×f times; while convolution window internal parallel execution can complete f×f multiplication and accumulation calculations of one convolution kernel in one clock cycle.

[0036] If each neuron and weight requires B bytes to store, the convolution kernel side length is f, the number of convolution windows is p×p, the number of input feature maps is n, and the input feature map side length is n W , the input feature map height is n H , the number of output feature maps is m, and the number of output feature maps calculated simultaneously is Q. The amount of data that needs to be stored in the register during the calculation process is: f*f+f*f*Q The amount of data required to complete the convolution of an input feature map is: f*f*p+f*(p-1) Switching the input feature map requires loading the weights: f The clock cycle required for calculation is:

[0037] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. This application invents an image parallel method for accelerating a neural network on an FPGA, which can realize high-speed computing of a neural network on an FPGA to the greatest extent possible according to network needs while saving FPGA resources. It is characterized in that: Specifically include the following: S1. In the first aspect, the embodiments of the present disclosure provide a convolution window internal parallelism. The multiplication and accumulation operations of all fxf image pixels covered by a convolution window can be performed in parallel. If the convolution window covers the data of i11-i33, the fxf multiplication operations required for the calculation of o11 on the output feature map can be performed in parallel; S2. In the second aspect, the embodiment of the present disclosure provides a parallel operation between convolution windows of an input feature map. The same input feature map convolution window is convolved at different positions. When the same convolution window slides to different positions on the image, the output values ​​at different positions of the output feature map can be calculated, and the multiplication operations at the same position of the convolution window can be performed in parallel when the convolution window is at different positions; S3. In a third aspect, the embodiments of the present disclosure provide a parallel execution of input feature maps. For feature maps with multi-channel inputs, the number of channels of the convolution kernel is the same as that of the input feature map. When the positions covered by the convolution windows on different feature maps are the same, the multiplication operations of the same positions can be executed in parallel. For an input feature map with three channels, the convolution kernel also has three channels. The three channels are calculated together to obtain o11. At this time, the multiplication operations of the positions corresponding to i111, i211, and i311 can be executed in parallel; S4. In the fourth aspect, the embodiments of the present disclosure provide a parallel execution of output feature maps. For single-channel input, when multiple convolution kernels correspond to multiple output feature maps, the calculation of the same convolution window of the input feature map corresponding to different convolution kernels can be executed in parallel. The three convolution kernels are all covered at the convolution window in the upper left corner of the input channel. The three channels 1w11, 2w11, and 3w11 correspond to the parallel multiplication operation of the first element i11 of the input feature map, and O111, o211, and o311 can be obtained at the same time.

Images