A spiking neural network acceleration system and method based on input and coordinate encoding

Abstract

A pulse neural network acceleration system and method based on input and coordinate coding belongs to the field of FPGA hardware accelerator design. Compared with the existing coding mode, the event-based input channel parallel sparse coding proposed by the present invention effectively reduces the storage space and bandwidth requirements of the feature map. At the same time, the encoding module and the decoding module consume very few FPGA resources. At the same time, the channel-parallel coding method provides a feasible direction for the accelerator's input channel-parallel calculation. The present invention serializes the channel direction calculation process of the input feature map, and multiple inputs are calculated in parallel, achieving the effect of parallelizing the channel direction calculation of the input feature map. At the same time, the results of the parallel calculation are used to update the membrane potential in a serial manner, solving the conflict of simultaneous reading and writing of the calculation results to the BRAM during general parallel calculation.

Description

Technical Field

[0001] The present invention belongs to the field of FPGA (Field-Programmable Gate Array) hardware accelerator design, and in particular designs a pulse neural network acceleration system and method based on input and coordinate encoding. Background Art

[0002] Spiking neural networks, such as Spike-based LeNet5 (LeNet network proposed by LeCun et al.), Spike-based VGG16 (Visual Geometry Group Network), etc., have natural sparse characteristics. There is a considerable amount of zero pulse transmission in the process of layer-to-layer propagation, which contains a large number of invalid calculations. At the same time, spiking neural networks all contain a large number of pulse-based convolutional layers, and most of the network's calculations are concentrated in the convolutional layers. These pulse-based convolutional layers account for the main part of the network's calculations. Therefore, by accelerating the sparsity of spiking neural networks, accelerated calculations and low-power calculations of these algorithms can be achieved. In the field of computing, CPU (Central Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), NPU (Neural Network Processing Unit), TPU (Tensor Processing Unit), DSP (Digital Signal Processing) and FPGA are common computing platforms, each of which has unique advantages and applicable scenarios. FPGA has the characteristics of high flexibility and reconfigurability, powerful parallel processing capability, low power consumption and high efficiency, which is suitable as an accelerated computing platform for spiking neural networks. Therefore, according to the sparsity characteristics of spiking neural networks, it is of practical significance to use FPGA to design a spiking neural network acceleration method based on input and coordinate encoding.

[0003] Existing pulse neural network accelerators have made full use of the sparsity and computing resources of pulses for accelerated calculations, but there are still some efficiency issues. If the original convolutional neural network acceleration algorithm is used to accelerate the calculation of the pulse neural network, it will be difficult to use sparsity; if the input feature map is encoded by direct binary mask, and then read into the FPGA for non-zero screening, there will be a large number of invalid bits in the mask, which will occupy a large storage bit width and increase the bandwidth requirement; if a separate position coordinate storage structure is used, although the logic of non-zero input screening is saved, the storage space and bandwidth requirements of the input feature map will also increase significantly. In addition, if the input 3×3 window of the traditional convolutional neural network is mapped to an output result, although some non-zero calculations in the convolution window can be skipped, the entire convolution window cannot be skipped; and the current accelerator algorithm based on one input mapped to a 3×3 output result does not yet have a method for parallel calculation of input channels. Therefore, if we want to effectively accelerate the calculation of spiking neural networks, we need to improve the encoding method to reduce the transmission bandwidth requirements, use the calculation method of mapping 3×3 output results based on events, and ensure the parallel calculation of input feature maps and the parallelism of convolution kernels. Therefore, we designed a spiking neural network feature map encoding method based on input and coordinate encoding based on FPGA, which reduces the storage space and transmission bandwidth requirements of feature maps, uses the mapping calculation method of mapping 3×3 output results based on events, effectively skips the calculation part of the input feature map with zero, reduces the number of times the intermediate results are transferred between DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory, hereinafter referred to as DDR) and on-chip BRAM (Block Random Access Memory), and the input feature map dimension and convolution kernel dimension can be calculated in parallel, greatly improving the calculation parallelism. Summary of the invention

[0004] In view of the above problems, the present invention proposes a pulse neural network acceleration method based on input and coordinate encoding. By applying a new pulse neural network feature map encoding method based on input and coordinate encoding, and applying a membrane potential storage method in BRAM, this encoding is decoded on the FPGA, the mask value and coordinates of the activation input of the non-zero input feature map are screened out, the non-zero input calculation of the pulse neural network is skipped, the weight is selected according to the input coordinate, the weight corresponding to the activation input of the screened non-zero input feature map is added and calculated, and the coordinates projected to the output are calculated. The membrane potential of the pulse neuron is stored in 9 BRAMs, which respectively store the membrane potential of the pulse neuron numbered at positions 0 to 8 of the 3×3 output mapping window, and the calculated output part and the coordinates according to the projection to the output are stored in the corresponding read-write BRAM task storage FIFO (First In First Out) queue. The above process can be instantiated with multiple parallelisms in two dimensions: input feature map and weight, to obtain multiple read-write BRAM task storage FIFO queues, and then use the cyclic priority method to cyclically read the tasks of these read-write BRAM task storage FIFO queues in sequence, and perform the corresponding coordinate reading, summing the read value and the original value, and writing back to the original BRAM operation on the subsequent BRAM neuron membrane potential storage array according to the read address and data. When the input feature map is calculated, the neuron membrane potentials in the 9 BRAMs are read out in sequence, and the threshold judgment and excitation calculation operations of the neurons are performed. The binary data of the output feature map is obtained, and the binary data of the output feature map is encoded by a pulse neural network feature map encoding method based on input and coordinate encoding proposed by the present invention to obtain the same encoded data as the input feature map and stored in the DDR for the next layer of pulse neural network calculation.

[0005] The technical solution of the present invention is as follows:

[0006] A pulse neural network acceleration system based on input and coordinate encoding includes a PS (Processing System) end and a PL (Programmable Logic) end-pulse neural network accelerator, and the PL end-pulse neural network accelerator includes an instruction interaction control unit and a pulse neural network convolution calculation unit. The pulse neural network convolution calculation unit includes a DMA (Direct Memory Access) module, a weight loading module, a weight cache module, a weight RAM module, a feature map loading module, a feature map cache module, a feature map FIFO module, an input feature map decoding and non-zero screening module, a weight index module, a coordinate calculation module, a weight accumulation module, a coordinate synchronization module, a read-write BRAM address and data FIFO queue module, a neuron membrane potential storage BRAM module, a neuron threshold excitation module, and an output feature map encoding module.

[0007] The instruction interaction control unit includes an instruction interaction control module.

[0008] The instruction interaction control module receives the control instruction data sent by the PS through the GP interface, and temporarily stores the instruction feedback information returned by each module for the PS to read and feedback. The instruction interaction control module is responsible for controlling the DMA module, weight loading module, weight cache module, weight RAM module, feature map loading module, feature map cache module, weight accumulation module, read and write BRAM address and data FIFO queue module, neuron membrane potential storage BRAM module, neuron threshold excitation module, and output feature map encoding module.

[0009] The composition and functions of each module are as follows:

[0010] 1) Instruction interaction control module: responsible for decoding and encoding control instruction data and instruction feedback information, realizing instruction interaction with the PS end through the GP (General Purpose) interface, and decoding the control instruction data into the control instruction signals and parameters required by each module of the pulse neural network convolution calculation unit according to the register values ​​of different addresses. After receiving the control instruction signals and parameters, each module starts to perform the corresponding calculation. After each module completes the corresponding calculation operation, it sends the instruction feedback information to the instruction interaction control module. The instruction interaction control module encodes the instruction feedback information returned by each module of the pulse neural network convolution calculation unit into feedback instructions for the PS end to read.

[0011] 2) DMA module: responsible for moving data between the PS-side DDR and the PL-side DDR without the full participation of the PS side, thus reducing the performance occupancy of the PS side. After receiving the control instruction signal and parameters from the instruction interaction control module, the DMA module is responsible for moving the instruction-length data from the specified address of the PS-side DDR to the PL-side DDR, or moving the instruction-length data from the specified address of the PL-side DDR to the PS-side DDR. After the data is moved, the instruction interaction control module sends data and signals to the instruction interaction control module, which then feeds back the execution status to the PS side, thus completing a complete data moving task.

[0012] 3) Weight loading module, weight cache module, weight RAM module: After the weight loading module receives the control command signal and parameters from the command interaction control module, it reads the weight parameter data of the pulse convolutional neural network from the DDR specified address of the PL end to the weight cache module, and feeds back the status information to the command interaction control module. The weight cache module temporarily stores the weight parameter data transmitted from the weight loading module, and updates the weight parameter data to the weight RAM module in turn according to the calculation progress and status. The weight RAM module uses a multi-group, intra-group ping-pong mode to cache the weight information in the calculation process for use by the weight index module.

[0013] 4) Feature map loading module, feature map cache module, feature map FIFO module: After receiving the control command signal and parameters from the command interaction control module, the feature map loading module reads the input feature map encoding data of the pulse convolutional neural network from the DDR specified address of the PL end to the feature map cache module, and then feeds back the status information to the command interaction control module at the same time. The feature map cache module temporarily stores the input feature map encoding data transmitted from the feature map loading module, and the feature map cache module writes the temporarily stored input feature map encoding data into the multiple feature map buffer FIFOs in the feature map FIFO module in a circular priority order. The feature map FIFO module sends the multiple feature map buffer FIFOs contained therein one by one to the corresponding feature map decoding and non-zero filtering circuits in the input feature map decoding and non-zero filtering module.

[0014] 5) Input feature map decoding and non-zero screening module: The internal multiple feature map decoding and non-zero screening circuits are responsible for decoding and non-zero screening of the input feature map coded data received from multiple feature map buffer FIFOs. During the non-zero screening, the first non-zero flag and the last non-zero flag of each coded input are marked as the beginning and end flags of the non-zero elements screened out from the input feature map coded data. The beginning and end flags of the non-zero elements are referred to as the position index of the non-zero elements. The input feature map coordinate data, non-zero elements and their position indexes obtained after decoding the input feature map coded data are sent to the coordinate calculation module and the weight index module respectively.

[0015] 6) Weight index module: According to the non-zero elements and their position indexes filtered out by the input feature map decoding and non-zero filtering module, the 3×3 weight window corresponding to the non-zero elements is selected from the weight RAM module. The selected 3×3 weight window is sent to the weight accumulation module together with the first non-zero flag and the last non-zero flag of each encoded input.

[0016] 7) Coordinate calculation module: Receives the input feature map coordinate data decoded by the input feature map decoding and non-zero screening module, and calculates according to the principle of one input feature map coordinate corresponding to a 3×3 output feature map coordinate, and obtains the 9 coordinates of the input corresponding output feature map and the corresponding membrane potential storage address in the BRAM, and sends them to the coordinate synchronization module as the coordinate calculation result.

[0017] 8) Weight accumulation module: Receives the 3×3 weight window and the first non-zero flag and the last non-zero flag sent by the weight index module, and adds the beginning and end flags of the non-zero elements to obtain the weight accumulation result. Finally, the weight accumulation result is sent to the read-write BRAM address and data FIFO queue module and the coordinate synchronization module.

[0018] 9) Coordinate synchronization module: aligns the weight accumulation result obtained by the weight accumulation module with the coordinate calculation result obtained by the coordinate calculation module through synchronization operation. Finally, the synchronized coordinates are sent to the read-write BRAM address and data FIFO queue module.

[0019] 10) Read and write BRAM address and data FIFO queue module: Receive the weight accumulation result output by the weight accumulation module and the synchronized coordinate output by the coordinate synchronization module, and store them in the FIFO after combination. Map the synchronized coordinates to the BRAM address of the neuron membrane potential storage BRAM module. At the same time, the read and write BRAM address and data FIFO queue module reads each FIFO in a circular priority order, and then reads the membrane potential data in the corresponding address from the neuron membrane potential storage BRAM module according to the read BRAM address and weight accumulation result, adds it to the data read from the FIFO, and writes it back to the corresponding BRAM address.

[0020] 11) Neuron membrane potential storage BRAM module: The membrane potential storage method in BRAM is used to store the output feature map in 9 BRAMs according to the coordinates in 3×3 windows. Each output feature map coordinate corresponds to a 3×3 window. The overlapping parts of the coordinates of two adjacent output feature maps are stored twice, and are updated separately when the membrane potential is updated. When the membrane potential threshold is read out, the membrane potential of the overlapping parts is added to obtain the output feature map pulse neuron membrane potential at the corresponding position. At the same time, the neuron membrane potential storage BRAM module also has the function of directly reading and writing PL-side DDR data. When the FPGA internal BRAM is not enough to store the complete output feature map, the PL-side DDR is used for block storage and reading and writing.

[0021] 12) Neuron threshold excitation module: After receiving the start signal and parameters of the instruction interaction unit, it reads the output characteristic graph pulse neuron membrane potential from the neuron membrane potential storage BRAM module in sequence, and performs threshold judgment and pulse excitation on the output characteristic graph pulse neuron after calculating the membrane potential.

[0022] 13) Output feature map encoding module: responsible for encoding the pulse data output by the neuron threshold excitation module according to the pulse neural network feature map encoding method based on input and coordinate encoding, and writing the encoded data into the PL end DDR.

[0023] Furthermore, the spiking neural network feature map encoding method based on input and coordinate encoding, i.e., the total encoding bit width is 32 bits, including 16 bits of input feature map coordinate data and 16 bits of input feature map channels 1 to 16 binary values. The high 8 bits of the 32 bits are the input feature map X coordinates, representing the number of rows of the input feature map, the middle 8 bits from the 24th bit to the 17th bit are the input feature map Y coordinates, representing the number of columns of the input feature map, and the low 16 bits are the binary values ​​of the input feature map channels 1 to 16.

[0024] A pulse neural network acceleration method based on input and coordinate encoding includes the following steps:

[0025] Step 1: Convolution calculation:

[0026] Step 1.1:

[0027] Before the PS side transmits the input feature map to the PL side - pulse neural network accelerator, it encodes the input feature map based on the pulse neural network feature map encoding method of input and coordinate encoding, and then starts the DMA module to transmit the input feature map encoding data and weight parameter data of the pulse convolutional neural network in the PS side DDR to the PL side DDR.

[0028] Step 1.2:

[0029] Feature map loading module, feature map cache module, feature map FIFO module: read out the input feature map encoding data of the pulse convolutional neural network in the DDR of the PL end in sequence, and after the read input feature map encoding data passes through the feature map loading module and the feature map cache module, the read input feature map encoding data is sequentially written into the multiple feature map buffer FIFOs in the feature map FIFO module in a cyclic priority order. The feature map FIFO module sends the multiple feature map buffer FIFOs contained therein one by one to the corresponding feature map decoding and non-zero filtering circuits in the input feature map decoding and non-zero filtering module.

[0030] Step 1.3:

[0031] Weight loading module, weight cache module, weight RAM module: read out the weight parameter data in the DDR of the PL end, and after the read weight parameter data passes through the weight loading module and the weight cache module, the read weight parameter data is updated to the weight RAM module in sequence as needed according to the calculation progress and status. The weight RAM module uses a multi-group, intra-group ping-pong mode to cache the weight information in the calculation process for use by the weight index module. The multi-group, intra-group ping-pong mode in the weight RAM module specifically refers to: according to the results decoded by the input feature map and the non-zero screening module, the weight RAM module is filled with weight parameter data. When the weight parameter data in one RAM is used up and the weight parameters in another RAM are used, the weight loading module and the weight cache module fill the new weight parameter data into the used RAM, thereby completing the weight loading and replacement in the calculation process in a cycle.

[0032] Step 1.4:

[0033] The multiple feature map decoding and non-zero screening circuits inside the input feature map decoding and non-zero screening module are responsible for decoding and non-zero screening of the input feature map coded data received from the multiple feature map buffer FIFOs. The first non-zero flag and the last non-zero flag of each coded input are marked as the beginning and end flags of the non-zero elements screened out by an input feature map coded data. The beginning and end flags of the non-zero elements are referred to as the position index of the non-zero elements. The input feature map coordinate data, non-zero elements and their position indexes obtained after decoding the input feature map coded data are sent to the coordinate calculation module and the weight index module respectively. The decoding is to decode the input feature map coded data from the feature map FIFO module into the horizontal, vertical and depth coordinates of the feature map, that is, the three-dimensional coordinates of the input feature map and the binary data in the channel depth direction according to the pulse neural network feature map encoding method based on input and coordinate encoding. The non-zero screening is to sequentially screen out the non-zero elements and their position indexes from low to high from the binary data through the non-zero screening circuit.

[0034] Step 1.5:

[0035] The weight index module selects the 3×3 weight window corresponding to the non-zero elements from the weight RAM module according to the non-zero elements and their position indexes filtered out by the input feature map decoding and non-zero filtering module. The selected 3×3 weight window is sent to the weight accumulation module together with the first non-zero flag and the last non-zero flag of each encoded input.

[0036] The weight accumulation module receives the 3×3 weight window and the first non-zero flag and the last non-zero flag from the weight index module, and adds the beginning and end flags of the non-zero elements to obtain the weight accumulation result. Finally, the weight accumulation result is sent to the read-write BRAM address and data FIFO queue module and the coordinate synchronization module.

[0037] The coordinate calculation module receives the input feature map coordinate data decoded by the input feature map decoding and non-zero screening module, and performs calculations based on the principle that one input feature map coordinate corresponds to a 3×3 output feature map coordinate, and obtains 9 coordinates of the input corresponding to the output feature map and the corresponding membrane potential storage address in the BRAM, which are sent to the coordinate synchronization module as the coordinate calculation result.

[0038] The coordinate synchronization module aligns the weight accumulation result obtained by the weight accumulation module with the coordinate calculation result obtained by the coordinate calculation module by performing synchronization operations, and finally sends the synchronized coordinates to the read-write BRAM address and data FIFO queue module.

[0039] Step 2: Update of neuronal membrane potential:

[0040] Step 2.1:

[0041] The read-write BRAM address and data FIFO queue module receives the weight accumulation result output by the weight accumulation module and the synchronized coordinate output by the coordinate synchronization module, and stores them in the FIFO after combination. The synchronized coordinate is mapped to the BRAM address of the neuron membrane potential storage BRAM module. At the same time, the read-write BRAM address and data FIFO queue module reads each FIFO in a circular priority order, and then reads the membrane potential data in the corresponding address from the neuron membrane potential storage BRAM module according to the read BRAM address and weight accumulation result, adds it to the data read from the FIFO, and writes it back to the corresponding BRAM address.

[0042] Step 3: Neuron threshold judgment pulse excitation:

[0043] Step 3.1:

[0044] The storage method of membrane potential in BRAM is adopted. The output feature map is stored in 9 BRAMs according to the coordinates in 3×3 windows. Each output feature map coordinate corresponds to a 3×3 window. The overlapping parts of the coordinates of two adjacent output feature maps are stored twice. When the membrane potential is updated, each is updated separately. When the membrane potential threshold is read out, the membrane potential of the overlapping parts is added to obtain the output feature map pulse neuron membrane potential at the corresponding position. At the same time, the neuron membrane potential storage BRAM module also has the function of directly reading and writing PL-side DDR data. When the internal BRAM of the FPGA is not enough to store the complete output feature map, the PL-side DDR is used for block storage and reading and writing.

[0045] Step 3.2:

[0046] After the neuron membrane potential is updated, the neuron threshold excitation module starts to judge the threshold and excite the neuron. The output characteristic graph pulse neuron membrane potential is read from the neuron membrane potential storage BRAM module in sequence. If the output characteristic graph pulse neuron membrane potential is greater than the threshold, the output characteristic graph pulse neuron membrane potential is reset to zero and written back to BRAM, and the neuron pulse output is 1; if it is less than the threshold, the output characteristic graph pulse neuron membrane potential is unchanged and written back to BRAM, and the neuron pulse output is 0.

[0047] Step 4: Output feature map encoding:

[0048] Step 4.1:

[0049] The output feature map encoding module encodes the pulse data output by the neuron threshold excitation module according to the pulse neural network feature map encoding method based on input and coordinate encoding, and writes the encoded data into the PL end DDR as the input feature map encoding data for the next layer of network calculation.

[0050] Beneficial effects of the present invention:

[0051] 1) Compared with the existing encoding mode, the pulse neural network feature map encoding method based on input and coordinate encoding proposed in the present invention effectively reduces the storage space and bandwidth requirements of the feature map. At the same time, the encoding module and the decoding module consume very little FPGA resources, and the channel-parallel encoding method provides a feasible direction for the input channel-parallel computing of the pulse neural network acceleration system based on input and coordinate encoding.

[0052] 2) After the present invention decodes the encoded input feature map into coordinates and data, parallel calculation can be performed in the channel direction of the input feature map. Combined with the parallel calculation of the convolution kernel dimension, the pulse neural network acceleration system based on input and coordinate encoding proposed by the present invention performs parallel calculation in two dimensions, which greatly improves the computing efficiency of the pulse neural network acceleration system based on input and coordinate encoding.

[0053] 3) The present invention serializes the channel direction calculation process of the input feature map, and calculates multiple inputs in parallel, thereby achieving the effect of parallelizing the channel direction calculation of the input feature map. At the same time, the results of the parallel calculation are used to update the membrane potential in a serial manner, solving the simultaneous reading and writing conflicts of the calculation results on the BRAM during general parallel calculations.

[0054] 4) The present invention proposes a method for storing membrane potential in BRAM, which stores the output feature map in 9 BRAMs in units of 3×3 windows according to coordinates. Figure 3 ×3 windows cannot be read simultaneously. This makes it possible to effectively implement the pulse neural network acceleration system based on input and coordinate encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a pulse neural network acceleration system architecture diagram based on input and coordinate encoding of the present invention. Figure 1 The thick arrows in the middle are data flows, and the thin arrows are command control signals.

[0056] Figure 2 It is a schematic diagram of the feature map cache module cyclically and preferentially writing feature map encoding data into the FIFO in the feature map FIFO module, the non-zero screening module, the weight RAM module, the weight index module, the weight accumulation module, the coordinate calculation module, the coordinate synchronization module, the read and write BRAM address, and the data FIFO queue module.

[0057] Figure 3 It is a schematic diagram of updating the neuron membrane potential of the neuron membrane potential storage BRAM module by the read-write BRAM address and data FIFO queue module.

[0058] Figure 4 This is the pulse neural network feature map encoding method based on input and coordinate encoding proposed by the present invention.

[0059] Figure 5 This is the membrane potential storage method proposed by the present invention.

[0060] Figure 6 is the feature map and the input non-zero filtering circuit. DETAILED DESCRIPTION

[0061] In order to make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0062] The present invention selects the VGG16 network model. The second convolution layer of the network is selected, and a feature map with an input height of 224, a width of 244, a channel number of 64, a convolution kernel size of 3×3, and a number of convolution kernels of 128 is taken as an example. The convolution step size is 1, and the algorithm of mapping an input to a 3×3 output feature map window does not require padding. The present invention is explained in detail through the convolution process. The feature map coordinate system is that the X coordinate represents the row number coordinate of the input feature map, and the Y coordinate represents the column number coordinate of the input feature map. The input feature map bit width is 32 bits, and its encoding, storage and loading methods are as follows Figure 4 As shown, the high 8 bits are the input feature map X coordinates, representing the number of rows of the input feature map, the middle 8 bits from the 24th bit to the 17th bit are the input feature map Y coordinates, representing the number of columns of the input feature map, and the low 16 bits are the binary values ​​of the input feature map channels 1 to 16. The weight input bit width is 32, and 4 8-bit weight parameter data are input each time. The feature map buffer FIFO queue selects 8, and the weight RAM uses 32, or 16 groups, with each two forming a weight ping-pong RAM. The input feature map calculation parallelism of the pulse neural network acceleration system based on input and coordinate encoding is depth 16, the parallelism of the XY direction dimension is 8, and the parallelism of the convolution kernel dimension is 16. The output feature map parallelism is 16. All resources and configurations support the maximum resource usage of all layers of the VGG16 network, that is, the pulse neural network acceleration system design based on input and coordinate encoding can support the calculation of all convolutional layers of the VGG16 network through resource reuse and flexible calling, and also supports other pulse neural network convolution calculations that do not exceed the scale of the VGG16 network.

[0063] like Figure 1 As shown, a pulse neural network acceleration system based on input and coordinate encoding includes a PS (Processing System) end and a PL (Programmable Logic) end-pulse neural network accelerator, and the PL end-pulse neural network accelerator includes an instruction interaction control unit and a pulse neural network convolution calculation unit. The pulse neural network convolution calculation unit includes a DMA (Direct Memory Access) module, a weight loading module, a weight cache module, a weight RAM module, a feature map loading module, a feature map cache module, a feature map FIFO module, an input feature map decoding and non-zero screening module, a weight index module, a coordinate calculation module, a weight accumulation module, a coordinate synchronization module, a read-write BRAM address and data FIFO queue module, a neuron membrane potential storage BRAM module, a neuron threshold excitation module, and an output feature map encoding module.

[0064] The instruction interaction control unit includes an instruction interaction control module.

[0065] The instruction interaction control module receives the control instruction data sent by the PS through the GP interface, and temporarily stores the instruction feedback information returned by each module for the PS to read and feedback. The instruction interaction control module is responsible for controlling the DMA module, weight loading module, weight cache module, weight RAM module, feature map loading module, feature map cache module, weight accumulation module, read and write BRAM address and data FIFO queue module, neuron membrane potential storage BRAM module, neuron threshold excitation module, and output feature map encoding module.

[0066] A spiking neural network acceleration method based on input and coordinate encoding, the steps are as follows:

[0067] Step 1: Convolution calculation:

[0068] Step 1.1:

[0069] like Figure 4 Before the PS end transmits the input feature map to the PL end, it encodes it according to the pulse neural network feature map encoding method based on input and coordinate encoding proposed by the present invention, and the total encoding bit width is 32 bits, of which the upper 8 bits are the input feature map X coordinate, representing the number of rows of the input feature map, the middle 8 bits from the 24th bit to the 17th bit are the input feature map Y coordinate, representing the number of columns of the input feature map, and the lower 16 bits are the binary values ​​of the input feature map channels 1 to 16. For example, if the values ​​at positions (2,2) of channels 1 to 16 of the input feature map are 1010111100000101 respectively, the encoded value is 32'b0000001000000010101010111100000101. The high 8bit-0010000 is the input feature map X coordinate of 2, the middle 8bit-01000000 from the 24th bit to the 17th bit is the input feature map Y coordinate of 128, and the low 16bit-1010111100000101 is the binary value of the input feature map channel 1 to 16. The DMA module is then started to transfer the input feature map encoding data and weight parameter data to the DDR on the PL side.

[0070] Step 1.2:

[0071] like Figure 2, feature map loading module, feature map cache module, feature map FIFO module: read out the input feature map encoding data of the pulse convolutional neural network in the DDR of the PL end in sequence, and after the read input feature map encoding data passes through the feature map loading module and the feature map cache module, the read input feature map encoding data is sequentially written into the multiple feature map buffer FIFOs in the feature map FIFO module in a cyclic priority order. For example: the first input is written into FIFO1, the second is written into FIFO2, and the eighth is written into FIFO8. If the FIFO to be written is full, this FIFO will not be written, and the next FIFO will be written. The feature map FIFO module sends the multiple feature map buffer FIFOs contained therein one by one to the corresponding feature map decoding and non-zero filtering circuits in the input feature map decoding and non-zero filtering module.

[0072] Step 1.3:

[0073] like Figure 2 , weight loading module, weight cache module, weight RAM module: read out the weight parameter data in the DDR of the PL end, and after the read weight parameter data passes through the weight loading module and the weight cache module, the read weight parameter data is updated to the weight RAM module in sequence as needed according to the calculation progress and status. The weight RAM module uses a multi-group, intra-group ping-pong mode to cache the weight information in the calculation process for use by the weight index module. The multi-group, intra-group ping-pong mode in the weight RAM module specifically refers to: according to the results decoded by the input feature map and the non-zero screening module, the weight RAM module is filled with weight parameter data. When the weight parameter data in one RAM is used up and the weight parameters in another RAM are used, the weight loading module and the weight cache module fill the new weight parameter data into the used RAM, thereby completing the weight loading and replacement in the calculation process in a cycle.

[0074] Step 1.4:

[0075] like Figure 5The multiple feature map decoding and non-zero filtering circuits inside the input feature map decoding and non-zero filtering module are responsible for decoding and non-zero filtering of the input feature map coded data received from multiple feature map buffer FIFOs. During the non-zero filtering, the first non-zero flag and the last non-zero flag of each coded input are marked as the beginning and ending flags of the non-zero elements filtered out from an input feature map coded data. The beginning and ending flags of the non-zero elements are referred to as the position index of the non-zero elements. The input feature map coordinate data, non-zero elements and their position indexes obtained after decoding the input feature map coded data are sent to the coordinate calculation module and the weight index module respectively. The decoding is to decode the input feature map coded data from the feature map FIFO module into the horizontal, vertical and depth coordinates of the feature map, that is, the three-dimensional coordinates of the input feature map and the binary data in the channel depth direction according to the pulse neural network feature map encoding method based on input and coordinate encoding. The non-zero filtering is performed by a non-zero filtering circuit (such as Figure 6 As shown in the figure, the binary data is filtered out from low to high non-zero elements and their position indexes. For example, the values ​​of the input feature map at positions (2,2) from channel 1 to 16 are 1010111100000101, and the encoded value is 32'b0010000001000000101010111100000101. After decoding, the data sent to the coordinate calculation unit is X-8'd2(8'b00000010), Y-8'd2(8'b00000010) and the data sent to the 16-channel input is 16'b1010111100000101. Specifically, channel 1 input is 1, channel 2 input is 0, channel 3 input is 1, channel 4 input is 0, channel 5 input is 1, channel 6 input is 1, channel 7 input is 1, channel 8 input is 1, channel 9 input is 0, channel 10 input is 0, channel 11 input is 0, channel 12 input is 0, channel 13 input is 0, channel 14 input is 1, channel 15 input is 0, and channel 16 input is 1. The non-zero filtering module screening results are 0000000000000001(1), 00000000000000100(3), 0000000100000000(9), 0000001000000000(10), 0000010000000000(11), 0000100000000000(12), 00100000000000000(14), 10000000000000000(16). The output feature map after calculation by the coordinate calculation module maps the 3×3 window with coordinates of 11, 12, 13, 21, 22, 23, 31, 32, 33.

[0076] Step 1.5:

[0077] The weight index module selects the 3×3 weight window corresponding to the non-zero elements from the weight RAM module according to the non-zero elements and their position indexes filtered out by the input feature map decoding and non-zero filtering module. The selected 3×3 weight window is sent to the weight accumulation module together with the first non-zero flag and the last non-zero flag of each encoded input.

[0078] The weight accumulation module receives the 3×3 weight window and the first non-zero flag and the last non-zero flag from the weight index module, and adds the beginning and end flags of the non-zero elements to obtain the weight accumulation result. Finally, the weight accumulation result is sent to the read-write BRAM address and data FIFO queue module and the coordinate synchronization module.

[0079] The coordinate calculation module receives the input feature map coordinate data decoded by the input feature map decoding and non-zero screening module, and performs calculations based on the principle that one input feature map coordinate corresponds to a 3×3 output feature map coordinate, and obtains 9 coordinates of the input corresponding to the output feature map and the corresponding membrane potential storage address in the BRAM, which are sent to the coordinate synchronization module as the coordinate calculation result.

[0080] The coordinate synchronization module aligns the weight accumulation result obtained by the weight accumulation module with the coordinate calculation result obtained by the coordinate calculation module by performing synchronization operations. Finally, the synchronized coordinates are sent to the read-write BRAM address and data FIFO queue module. Figure 2 , the weight index module selects the corresponding 3×3 weight window from the weight RAM module according to the position index (2, 2) of the non-zero element and sends it to the weight accumulation module. The coordinate calculation module and the coordinate synchronization module calculate the three-dimensional coordinates of the 3×3 window mapped to the output feature map according to the three-dimensional coordinates of the input feature map and synchronize with the weight accumulation module. Specifically, as described in step 1.5, the weight index module selects the convolution kernel 3×3 window of the corresponding channel in the weight RAM module according to the channel of the non-zero element (1, 3, 9, 10, 11, 12, 14, 16), and then adds the 3×3 window weights of these channels according to the corresponding position to obtain the membrane potential increment of a single channel, and stores the increment and position coordinates in the read-write BRAM address and data FIFO queue module.

[0081] Step 2: Update of neuronal membrane potential:

[0082] Step 2.1:

[0083] like Figure 2, the read-write BRAM address and data FIFO queue module receives the weight accumulation result output by the weight accumulation module and the synchronized coordinates output by the coordinate synchronization module, and stores them in the FIFO after combination. The synchronized coordinates are mapped to the BRAM address of the neuron membrane potential storage BRAM module. At the same time, the read-write BRAM address and data FIFO queue module reads each FIFO in a circular priority order, and then reads the membrane potential data in the corresponding address from the neuron membrane potential storage BRAM module according to the read BRAM address and weight accumulation result, adds it to the data read from the FIFO, and writes it back to the corresponding BRAM address. The update of the neuron membrane potential is completed. Specifically, the BRAM storage address corresponding to the output window is first calculated according to the position index (2, 2) of the non-zero element, the address of BRAM1 is 4, the address of BRAM2 is 1, and the address of BRAM3 is 1. Then the neuron membrane potential is read from the neuron membrane potential storage BRAM, and then the weighted accumulation and the 3×3 window are rotated 180 degrees and added to the original membrane potential. The corresponding relationship between the weighted and membrane potential is 33-11, 32-12, 31-13, 23-21, 22-22, 21-23, 13-31, 12-32, 11-33. After the sum, the result is written back to the BRAM address to complete the membrane potential update.

[0084] Step 3: Neuron threshold judgment pulse excitation:

[0085] Step 3.1:

[0086] The storage method of membrane potential in BRAM is adopted. The output feature map is stored in 9 BRAMs according to the coordinates in 3×3 windows. Each output feature map coordinate corresponds to a 3×3 window. The overlapping parts of the coordinates of two adjacent output feature maps are stored twice. When the membrane potential is updated, each is updated separately. When the membrane potential threshold is read out, the membrane potential of the overlapping parts is added to obtain the output feature map pulse neuron membrane potential at the corresponding position. At the same time, the neuron membrane potential storage BRAM module also has the function of directly reading and writing PL-side DDR data. When the internal BRAM of the FPGA is not enough to store the complete output feature map, the PL-side DDR is used for block storage and reading and writing.

[0087] Step 3.2:

[0088] like Figure 3After the neuron membrane potential is updated, the neuron threshold excitation module starts to perform threshold judgment and pulse excitation on the neuron. Specifically, the neuron membrane potential is synchronously read out from the 16 BRAM arrays. For example, if the threshold judgment pulse excitation is to be performed on the neuron at position 01, the data with addresses 0 and 1 are read out from BRAM1, and the membrane potential of 01 at the corresponding position is intercepted and summed to obtain the real membrane potential of 01. If the threshold judgment pulse excitation is to be performed on the neuron at position 12, the data with addresses 0, 1 and 2 are read out from BRAM2, and the membrane potential of 12 at the corresponding position is intercepted and summed to obtain the real membrane potential of 12. Then, it is processed according to the neuron threshold judgment pulse excitation mode and the binary pulse data is output.

[0089] Step 4: Output feature map encoding:

[0090] Step 4.1:

[0091] The output feature map encoding module receives the binary data sent by the neuron threshold excitation module, counts according to the number of rows and columns, obtains the X and Y coordinates of the output neuron, and then integrates the 16-bit neuron output pulse into the output feature map obtained by the pulse neural network feature map encoding method based on input and coordinate coding of the present invention. The output parallelism is designed to be 16, which is compatible with the 16-channel mode of the input feature map, achieving the effect of output compatibility with input. After encoding, it can be directly used as the input of the next layer of feature map without the help of other processing. When the calculation results of the last layer are calculated, the PS end is notified through the instruction interaction control unit, and the PS end reads the final calculation results in the DDR of the PL end through the DMA module to complete the accelerated calculation of the pulse neural network acceleration system based on input and coordinate coding.

Claims

1. A pulse neural network acceleration system based on input and coordinate encoding, characterized in that: It includes a PS end and a PL end-pulse neural network accelerator, and the PL end-pulse neural network accelerator includes an instruction interaction control unit and a pulse neural network convolution calculation unit; the pulse neural network convolution calculation unit includes a DMA module, a weight loading module, a weight cache module, a weight RAM module, a feature map loading module, a feature map cache module, a feature map FIFO module, an input feature map decoding and non-zero screening module, a weight index module, a coordinate calculation module, a weight accumulation module, a coordinate synchronization module, a read-write BRAM address and data FIFO queue module, a neuron membrane potential storage BRAM module, a neuron threshold excitation module, and an output feature map encoding module; The instruction interaction control unit includes an instruction interaction control module; The instruction interaction control module receives the control instruction data sent by the PS end through the GP interface, and temporarily stores the instruction feedback information returned by each module for the PS end to read and feedback; the instruction interaction control module is responsible for controlling the DMA module, weight loading module, weight cache module, weight RAM module, feature map loading module, feature map cache module, weight accumulation module, read and write BRAM address and data FIFO queue module, neuron membrane potential storage BRAM module, neuron threshold excitation module, and output feature map encoding module; The composition and functions of each module are as follows: 1) Instruction interaction control module: responsible for decoding and encoding control instruction data and instruction feedback information, realizing instruction interaction with the PS end through the GP interface, and decoding the control instruction data into the control instruction signals and parameters required by each module of the pulse neural network convolution calculation unit according to the register values ​​of different addresses; each module starts to perform corresponding calculations after receiving the control instruction signals and parameters, and after each module completes the corresponding calculation operation, it sends the instruction feedback information to the instruction interaction control module, and the instruction interaction control module encodes the instruction feedback information returned by each module of the pulse neural network convolution calculation unit into feedback instructions for the PS end to read; 2) DMA module: responsible for transferring data between the PS-side DDR and the PL-side DDR without the full participation of the PS-side, thus reducing the performance occupation of the PS-side; after receiving the control instruction signal and parameters from the instruction interaction control module, the DMA module is responsible for transferring the instruction-length data from the specified address of the PS-side DDR to the PL-side DDR, or transferring the instruction-length data from the specified address of the PL-side DDR to the PS-side DDR; after completing the data transfer, the instruction interaction control module sends data and signals to the instruction interaction control module, which then feeds back the execution status to the PS-side, thus completing a complete data transfer task; 3) Weight loading module, weight cache module, weight RAM module: After receiving the control command signal and parameters from the command interaction control module, the weight loading module reads the weight parameter data of the pulse convolutional neural network from the DDR specified address of the PL end to the weight cache module, and feeds back the status information to the command interaction control module; the weight cache module temporarily stores the weight parameter data transmitted from the weight loading module, and updates the weight parameter data to the weight RAM module in turn as needed according to the calculation progress and status; The weight RAM module uses a multi-group, intra-group ping-pong mode to cache the weight information during the calculation process for use by the weight index module; 4) Feature map loading module, feature map cache module, feature map FIFO module: after receiving the control command signal and parameters from the command interaction control module, the feature map loading module reads the input feature map encoding data of the pulse convolutional neural network from the DDR specified address of the PL end to the feature map cache module, and then feeds back the status information to the command interaction control module at the same time; the feature map cache module temporarily stores the input feature map encoding data transmitted from the feature map loading module, and the feature map cache module writes the temporarily stored input feature map encoding data into the multiple feature map buffer FIFOs in the feature map FIFO module in sequence according to the cyclic priority order; the feature map FIFO module sends the multiple feature map buffer FIFOs contained therein one by one to the corresponding feature map decoding and non-zero filtering circuits in the input feature map decoding and non-zero filtering module; 5) Input feature map decoding and non-zero screening module: The internal multiple feature map decoding and non-zero screening circuits are responsible for decoding and non-zero screening of the input feature map coded data received from multiple feature map buffer FIFOs. During the non-zero screening, the first non-zero flag and the last non-zero flag of each coded input are marked as the beginning and end flags of the non-zero elements screened out from the input feature map coded data. The beginning and end flags of the non-zero elements are referred to as the position index of the non-zero elements. The input feature map coordinate data, non-zero elements and their position indexes obtained after decoding the input feature map coded data are sent to the coordinate calculation module and the weight index module respectively. 6) Weight index module: According to the non-zero elements and their position indexes filtered out by the input feature map decoding and non-zero filtering module, the 3×3 weight window corresponding to the non-zero elements is selected from the weight RAM module, and the selected 3×3 weight window is sent to the weight accumulation module together with the first non-zero flag and the last non-zero flag of each encoded input; 7) Coordinate calculation module: receiving the input feature map coordinate data decoded by the input feature map decoding and non-zero screening module, and calculating according to the principle that one input feature map coordinate corresponds to mapping 3×3 output feature map coordinates, and obtaining 9 coordinates of the input corresponding output feature map and the corresponding membrane potential storage address in the BRAM, and sending them to the coordinate synchronization module as the coordinate calculation result; 8) Weight accumulation module: receives the 3×3 weight window and the first non-zero flag and the last non-zero flag sent by the weight index module, and adds the beginning and end flags of the non-zero elements to obtain the weight accumulation result; finally, sends the weight accumulation result to the read-write BRAM address and data FIFO queue module and the coordinate synchronization module; 9) Coordinate synchronization module: aligns the weight accumulation result obtained by the weight accumulation module with the coordinate calculation result obtained by the coordinate calculation module by performing synchronization operations; finally, the synchronized coordinates are sent to the read-write BRAM address and data FIFO queue module; 10) Read and write BRAM address and data FIFO queue module: receive the weight accumulation result output by the weight accumulation module and the synchronized coordinate output by the coordinate synchronization module, combine and store them in the FIFO; map the synchronized coordinate to the BRAM address of the neuron membrane potential storage BRAM module, and at the same time, read and write the BRAM address and data FIFO queue module to read each FIFO in a circular priority order, and then read the membrane potential data in the corresponding address from the neuron membrane potential storage BRAM module according to the read BRAM address and weight accumulation result, add it to the data read from the FIFO, and write it back to the corresponding BRAM address; 11) Neuron membrane potential storage BRAM module: The membrane potential storage method in BRAM is used to store the output feature map in 9 BRAMs according to the coordinates in units of 3×3 windows. Each output feature map coordinate corresponds to a 3×3 window. The overlapping parts of the coordinates of two adjacent output feature maps are stored twice, and are updated separately when the membrane potential is updated. When the membrane potential threshold is read out, the membrane potential of the overlapping parts is added to obtain the output feature map pulse neuron membrane potential at the corresponding position; at the same time, the neuron membrane potential storage BRAM module also has the function of directly reading and writing PL-side DDR data. When the FPGA internal BRAM is not enough to store the complete output feature map, the PL-side DDR is used for block storage and reading and writing; 12) Neuron threshold excitation module: after receiving the start signal and parameters of the instruction interaction unit, it reads the output characteristic graph pulse neuron membrane potential from the neuron membrane potential storage BRAM module in sequence, and performs threshold judgment and pulse excitation on the output characteristic graph pulse neuron after calculating the membrane potential; 13) Output feature map encoding module: responsible for encoding the pulse data output by the neuron threshold excitation module according to the pulse neural network feature map encoding method based on input and coordinate encoding, and writing the encoded data into the DDR at the PL end.

2. The spiking neural network acceleration system based on input and coordinate encoding according to claim 1, characterized in that: The pulse neural network feature map encoding method based on input and coordinate encoding, that is, the total encoding bit width is 32 bits, including 16 bits of input feature map coordinate data and 16 bits of input feature map channels 1 to 16 binary values; the upper 8 bits of the 32 bits are the input feature map X coordinates, representing the number of rows of the input feature map, the middle 8 bits from the 24th bit to the 17th bit are the input feature map Y coordinates, representing the number of columns of the input feature map, and the lower 16 bits are the binary values ​​of the input feature map channels 1 to 16.

3. A pulse neural network acceleration method using a pulse neural network acceleration system based on input and coordinate encoding as described in any one of claims 1-2, characterized in that: The steps include: Step 1: Convolution calculation: Step 1.1: Before the PS side transmits the input feature map to the PL side-spiking neural network accelerator, the PS side encodes the input feature map based on the input and coordinate encoding spiking neural network feature map encoding method, and then starts the DMA module to transmit the input feature map encoding data and weight parameter data of the spiking convolutional neural network in the PS side DDR to the PL side DDR; Step 1.2: Feature map loading module, feature map cache module, feature map FIFO module: read out the input feature map encoding data of the pulse convolutional neural network in the DDR of the PL end in sequence, and after the read input feature map encoding data passes through the feature map loading module and the feature map cache module, the read input feature map encoding data is sequentially written into the multiple feature map buffer FIFOs in the feature map FIFO module according to the cyclic priority order; the feature map FIFO module sends the multiple feature map buffer FIFOs contained therein one by one to the corresponding feature map decoding and non-zero filtering circuits in the input feature map decoding and non-zero filtering module; Step 1.3: Weight loading module, weight cache module, weight RAM module: read out the weight parameter data in the DDR of the PL end, and after the read weight parameter data passes through the weight loading module and the weight cache module, the read weight parameter data is updated to the weight RAM module in turn according to the calculation progress and status; The weight RAM module uses a multi-group, intra-group ping-pong mode to cache the weight information in the calculation process for use by the weight index module; the multi-group, intra-group ping-pong mode in the weight RAM module specifically refers to: according to the results of input feature map decoding and non-zero screening module decoding, the weight RAM module is filled with weight parameter data, and when the weight parameter data in one RAM is used up and the weight parameter in another RAM is used, the weight loading module and the weight caching module fill the new weight parameter data into the used RAM, so as to complete the weight loading and replacement in the calculation process in a cycle; Step 1.4: The multiple feature map decoding and non-zero screening circuits inside the input feature map decoding and non-zero screening module are responsible for decoding and non-zero screening of the input feature map coded data received from the multiple feature map buffer FIFOs. During the non-zero screening, the first non-zero flag and the last non-zero flag of each coded input are marked as the beginning and ending flags of the non-zero elements screened out from the input feature map coded data. The beginning and ending flags of the non-zero elements are referred to as the position index of the non-zero elements. The input feature map coordinate data, non-zero elements and their position indexes obtained after the decoding of the input feature map coded data are sent to the coordinate calculation module and the weight index module respectively. The decoding is to decode the input feature map coded data from the feature map FIFO module into the horizontal, vertical and depth coordinates of the feature map, that is, the three-dimensional coordinates of the input feature map and the binary data in the channel depth direction according to the pulse neural network feature map encoding method based on the input and coordinate encoding. The non-zero screening is to sequentially screen out the non-zero elements and their position indexes from low to high from the binary data through the non-zero screening circuit. Step 1.5: The weight index module selects the 3×3 weight window corresponding to the non-zero elements from the weight RAM module according to the non-zero elements and their position indexes filtered out by the input feature map decoding and non-zero filtering module. The selected 3×3 weight window is sent to the weight accumulation module together with the first non-zero flag and the last non-zero flag of each encoded input; The weight accumulation module receives the 3×3 weight window and the first non-zero flag and the last non-zero flag sent by the weight index module, and adds the beginning and end flags of the non-zero elements to obtain the weight accumulation result; finally, the weight accumulation result is sent to the read-write BRAM address and data FIFO queue module and the coordinate synchronization module; The coordinate calculation module receives the input feature map coordinate data decoded by the input feature map decoding and the non-zero screening module, and calculates according to the principle that one input feature map coordinate corresponds to a 3×3 output feature map coordinate, obtains 9 coordinates of the input corresponding to the output feature map and the corresponding membrane potential storage address in the BRAM, and sends them to the coordinate synchronization module as the coordinate calculation result; The coordinate synchronization module aligns the weight accumulation result obtained by the weight accumulation module with the coordinate calculation result obtained by the coordinate calculation module by performing synchronization operations; finally, the synchronized coordinates are sent to the read-write BRAM address and data FIFO queue module; Step 2: Update of neuronal membrane potential: Step 2.1: The read-write BRAM address and data FIFO queue module receives the weight accumulation result output by the weight accumulation module and the synchronized coordinates output by the coordinate synchronization module, and stores them in the FIFO after combining; the synchronized coordinates are mapped to the BRAM address of the neuron membrane potential storage BRAM module, and at the same time, the read-write BRAM address and data FIFO queue module reads each FIFO in a circular priority order, and then reads the membrane potential data in the corresponding address from the neuron membrane potential storage BRAM module according to the read BRAM address and the weight accumulation result, adds it to the data read from the FIFO, and writes it back to the corresponding BRAM address; Step 3: Neuron threshold judgment pulse excitation: Step 3.1: The method of storing membrane potential in BRAM is adopted. The output feature map is stored in 9 BRAMs according to the coordinates in units of 3×3 windows. Each output feature map coordinate corresponds to a 3×3 window. The overlapping parts of the coordinates of two adjacent output feature maps are stored twice. When the membrane potential is updated, each is updated separately. When the membrane potential threshold is read out, the membrane potential of the overlapping parts is added to obtain the output feature map pulse neuron membrane potential at the corresponding position. At the same time, the neuron membrane potential storage BRAM module also has the function of directly reading and writing PL-side DDR data. When the BRAM inside the FPGA is not enough to store the complete output feature map, the PL-side DDR is used for block storage and reading and writing. Step 3.2: After the neuron membrane potential is updated, the neuron threshold excitation module starts to perform threshold judgment and pulse excitation on the neuron; the output characteristic graph pulse neuron membrane potential is read from the neuron membrane potential storage BRAM module in sequence. If the output characteristic graph pulse neuron membrane potential is greater than the threshold, the output characteristic graph pulse neuron membrane potential is reset to zero and written back to BRAM, and the neuron pulse output is 1; if it is less than the threshold, the output characteristic graph pulse neuron membrane potential is unchanged and written back to BRAM, and the neuron pulse output is 0; Step 4: Output feature map encoding: Step 4.1: The output feature map encoding module encodes the pulse data output by the neuron threshold excitation module according to the pulse neural network feature map encoding method based on input and coordinate encoding, and writes the encoded data into the PL end DDR as the input feature map encoding data for the next layer of network calculation.

Images