Residual snn hardware accelerator and method

Abstract

The application relates to a residual SNN hardware accelerator and method, a new residual SNN hardware accelerator overall implementation scheme is designed by carrying out hardware implementation and acceleration on an FPGA chip for a core module in a residual SNN network, that is, a pulse residual module, cooperating with a DDR memory outside the chip, combining the characteristics of SNN multi-time step calculation, a technical means of image row-by-row input and calculation under multiple time steps is designed to overcome the realistic problem that feature map size is too large in the SNN network algorithm, and it is difficult to completely buffer the feature map and the neuron state of the SNN on the chip, and the data processing acceleration performance is greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of neural network circuit technology, and relates to a residual SNN hardware accelerator and method. Background Technology

[0002] In the field of computer vision, object recognition and detection are fundamental and crucial tasks. The former aims to determine the category of objects in an image, while the latter goes a step further, locating the position of objects in an image and identifying their category. Over the past decade, Artificial Neural Networks (ANNs) have achieved tremendous success in object recognition and detection, demonstrating revolutionary performance compared to traditional machine learning methods based on expert systems. They are widely used in scenarios such as security monitoring, autonomous driving, and medical diagnosis. However, ANN-based object detection requires significant computational resources and energy, and it also suffers from shortcomings in areas such as biological interpretability.

[0003] Spiking Neural Networks (SNNs), as an emerging computing paradigm that more closely resembles the working principles of the brain, have attracted attention due to their higher energy efficiency and biological plausibility. SNNs are considered the third generation of neural networks, and compared to ANNs, they are more biomimetic to the biological brain. SNNs simulate the propagation of biological signals in neural networks in the form of pulse sequences, exhibiting sparsity and event-driven characteristics, making them more energy-efficient than ANNs that use continuous values ​​to represent information. Furthermore, SNNs are built upon neuron models that more closely model neural dynamics, completely simulating biological neural network systems in terms of network construction. Based on this brain-inspired computing paradigm, SNNs offer new possibilities for achieving low-power, high-efficiency target recognition and detection, and for developing novel computer vision systems. However, in the process of realizing this invention, the inventors discovered that in practical target recognition and detection tasks, using traditional CPUs or GPUs to run spiking neural networks still suffers from poor data processing acceleration performance. Summary of the Invention

[0004] To address the problems existing in the above-mentioned traditional methods, this invention proposes a residual SNN hardware accelerator and a residual SNN hardware acceleration method, which can significantly improve the data processing acceleration performance.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a residual SNN hardware accelerator is provided, including DDR memory and a pulse residual module of the residual SNN network deployed on an FPGA chip. The pulse residual module includes an input buffer, a short-circuit path unit, a residual path unit, an adder, a convolution activation module, an output FIFO (first-in-first-out data storage buffer), and control logic and register units. The DDR memory is connected to the input buffer, short-circuit path unit, residual path unit and output FIFO respectively. The DDR memory is used to store the input feature map, output feature map and network weights of the pulse residual module according to the set data arrangement and read / write order. The input buffer, short-circuit path unit, residual path unit, adder, convolution activation module and output FIFO are connected in sequence and used to perform calculation processing on the input feature map according to the line-by-line multi-time step calculation strategy and output the corresponding output feature map. The control logic and register unit are used to configure the pulse residual module to work in the line-by-line multi-time step mode and configure the pulse residual module to the specifications of the residual module that needs to be accelerated. Both the short-circuit path unit and the residual path unit adopt a two-order pipelined convolutional computation structure. The short-circuit path unit includes a max pooling module, an intermediate output FIFO, an intermediate input buffer, and a 1×1 convolutional layer without an activation module connected in sequence. The residual path unit includes a 3×3 convolutional layer with an activation module, an intermediate output FIFO, an intermediate input buffer, and a 3×3 convolutional layer without an activation module connected in sequence. The input channels of the adder are respectively connected to the output channels of the 1×1 convolutional layer and the 3×3 convolutional layer without an activation module.

[0006] On the other hand, a residual SNN hardware acceleration method based on the above-mentioned residual SNN hardware accelerator is also provided, including the following steps: By configuring registers, the operating mode of the pulse residual module that needs to be accelerated in the residual SNN network is switched to the row-by-row multi-time-step mode, and the pulse residual module is configured to the specifications of the residual module that needs to be accelerated. Based on the set data arrangement and read / write order, the network weights of the pulse residual module and the input feature map to be processed are read from the DDR memory; The pulse residual module performs computational processing on the input feature map according to the row-by-row multi-time-step calculation strategy, and outputs the corresponding output feature map. Based on the set data arrangement and read / write order, the output feature map is written to DDR memory through the pulse residual module.

[0007] In another aspect, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described residual SNN hardware acceleration method.

[0008] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned residual SNN hardware accelerator and method implements and accelerates the core module of the residual SNN network, namely the pulse residual module, on an FPGA chip. Combined with off-chip DDR memory, a comprehensive implementation scheme for the residual SNN hardware accelerator is designed. Taking into account the multi-time-step computation characteristics of SNNs, a technique for line-by-line image input and computation at multiple time steps is designed to overcome the practical problem of the large feature map size in SNN network algorithms, making it difficult to cache all feature maps and SNN neuron states on-chip. This designed hardware accelerator achieves a data computation speed more than twice that of the CPU hardware implementation scheme for the pulse residual module in the residual SNN network, while significantly reducing power consumption compared to the CPU hardware implementation scheme, thus greatly improving data processing acceleration performance. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of the hardware structure of a residual SNN hardware accelerator in one embodiment. Figure 2 This is a schematic diagram of the hardware design of a residual SNN hardware accelerator in one embodiment. Figure 3 This is a schematic diagram of the row-by-row multi-timestep neuron update process in one embodiment; Figure 4 This is a schematic diagram illustrating the storage and retrieval of a large-size feature map in one embodiment; Figure 5 This is a schematic diagram of data window loading under row-by-row multi-timestep implementation in one embodiment; Figure 6 This is a schematic diagram of caching intermediate results under a row-by-row, multi-time-step approach in one embodiment; Figure 7 A schematic diagram of writing back the output feature map under row-by-row multi-timestep implementation in one embodiment; Figure 8 This is a schematic diagram of the control state machine for the input buffer in one embodiment; Figure 9 This is a schematic diagram of address rearrangement in one embodiment, where (a) is the data input window and (b) is the corresponding rule; Figure 10This is a schematic diagram of the control state machine for the intermediate input buffer in one embodiment; Figure 11 This is a schematic diagram of the design structure of a pulsed convolutional layer in one embodiment; Figure 12 This is a schematic diagram of the design structure of the max pooling layer in one embodiment; Figure 13 This is a schematic diagram of the module structure of a hardware acceleration method for residual SNN in one embodiment. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application.

[0012] It should be noted that the reference to "embodiment" in this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described in this application can be combined with other embodiments. The term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0013] Limited by the current research level, SNNs still face many challenges in practical applications, especially in complex target recognition and detection tasks. Due to the special computational paradigm of SNNs, the designed target detection algorithm based on residual spiking neural networks requires a dedicated hardware platform to truly achieve low-power target detection. However, due to the relatively short development time, existing FPGA (Field-Programmable Gate Array) hardware implementations of SNNs only implement a limited number of network types and small-scale networks, mainly focusing on simple image recognition tasks. For more complex computer vision tasks such as target detection, related research is still very limited. Moreover, target detection networks are usually characterized by large feature map sizes and numerous convolutional channels, posing a significant challenge to FPGA implementation. To address these issues, this invention conducts research on SNN hardware acceleration schemes for target detection tasks. A hardware accelerator design for the algorithm is proposed. This design incorporates a line-by-line image input and multi-time-step computation method, enabling the computation of large-size feature maps in the algorithm to be implemented on an FPGA. Finally, the corresponding module hardware design is completed based on the above scheme.

[0014] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0015] First, regarding the design of the SpikingRes block, a fully impulsive residual module: In the research results in this field, the core component of the Spiking-YOLO network model is the SpikingRes block with a fully impulsive structure designed in this paper. This module mainly consists of four components: impulsive convolutional layers, batch normalization (BN) layers, LIF neuron layers, and max pooling layers. In its internal structure, CnSm indicates that this layer is a convolutional layer with a kernel size of n×n and a stride of m, where n and m are both integers; Pn indicates that this layer is a max pooling layer with a size of n×n.

[0016] Based on these four network layers, the SpikingRes block internally designs two paths: a residual path and a short-circuit path (i.e., identity mapping). The calculation results of the two paths are added together before activation to obtain the output feature map. The residual path contains two convolutions and corresponding tdBN (threshold-dependent batch normalization) layers, where the first convolution is followed by activation via a LIF neuron layer. The short-circuit path contains one max pooling and a 1×1 convolutional layer, enabling scaling and dimensionality increase or decrease of the input feature map. To maintain the same output feature map size, the stride of the first convolutional layer in the residual path can be set to 1, and the max pooling layer in the short-circuit path can be disabled.

[0017] For the Batch Normalization (BN) layer in the SpikingRes block, the specific method used is tdBN, designed for SNNs. In CNNs, batch normalization is typically used to prevent gradient vanishing or exploding during deep network training. However, tdBN considers both time and space dimensions during batch normalization and assigns the normalized data distribution to the threshold of neurons, rather than a standard normal distribution. This improvement fully considers the structural characteristics of SNNs. The specific method is as follows: (1) (2) in, It is the output of the convolutional layer. The The feature maps of each channel at all time steps are... A four-dimensional matrix in terms of dimensions (i.e., batch, length, width, and time step). For hyperparameters, V th Threshold voltage, It is a constant that approaches 0. and These are the parameters that need to be trained for the tdBN layer. and They are The mean and variance. The final result... That is, the result of batch standardization, Replace the original As input to the LIF neuron layer.

[0018] It's understandable that the tdBN layer is only used during training. For network inference and hardware deployment, the tdBN layer can be fused into the weights of the convolutional layers, eliminating the need for batch normalization. The fused convolutional layer weights and biases are as follows: (3) (4) in, and They are respectively and The expectation over the entire dataset can be calculated using the existing method of moving average. and These represent the input channels of the convolutional layer. To output channel The weights and biases, and This refers to the weights and biases after fusion.

[0019] The SpikingRes block uses pulse-based information for both its input and output, significantly reducing the transmission load on the input and output feature maps. This makes it more hardware-friendly for network deployment. Furthermore, the convolution and pooling operations within the block only need to process pulse information, fully utilizing the characteristics of SNNs. In contrast, the EMS-ResNet and SEW-ResNet modules used in existing research still use continuous-value-based feature maps for input and output. Besides the higher transmission load, the SEW-ResNet module, because its input is not pulse-based, essentially includes multiply-accumulate operations in its convolution module, rather than low-power pulse convolution. The EMS-ResNet module, on the other hand, requires pooling and activation of the continuous-value-based input image before pulse convolution, and also includes concat operations, making the module more complex for hardware deployment.

[0020] At the network level, to ensure network lightweightness, the number of intermediate channels in the residual path of each SpikingRes block, i.e., the number of output channels of the first convolutional layer, is set to half the number of output channels of the SpikingRes block. Furthermore, for modules in the network that reduce dimensionality, because their input channels are relatively large, this results in a large number of weights in the first convolutional layer of the module, and it also requires a longer time to compute the first convolutional result, which is detrimental to pipelined operations during hardware deployment.

[0021] Therefore, preferably, the kernel size of the first convolutional layer in the residual path is 1×1. Specifically, for the SpikingRes block (hereinafter also referred to as the pulse residual module) where the number of output channels is less than the number of input channels, the first convolutional layer of its residual path is set to a 1×1 size convolution to reduce the weight and computational cost of this layer.

[0022] Then, please see Figure 1In one embodiment, a residual SNN hardware accelerator is provided, including DDR memory 11 and a pulse residual module 13 of a residual SNN network deployed on an FPGA chip. The pulse residual module 13 includes an input buffer 131, a short-circuit path unit 132, a residual path unit 133, an adder 134, a convolution activation module 135, an output FIFO 136, and a control logic and register unit 137. The DDR memory 11 is connected to the input buffer 131, the short-circuit path unit 132, the residual path unit 133, and the output FIFO 136, respectively. The DDR memory 11 is used to store the input feature map, output feature map, and network weights of the pulse residual module 13 according to a set data arrangement and read / write order. The input buffer 131, the short-circuit path unit 132, the residual path unit 133, the adder 134, the convolution activation module 135, and the output FIFO 136 are connected sequentially and are used to perform computation processing on the input feature map according to a row-by-row multi-time-step computation strategy and output the corresponding output feature map. The control logic and register unit 137 is used to configure the pulse residual module 13 to operate in line-by-line multi-time-step mode and to configure the pulse residual module to the specifications of the residual module that needs to be accelerated.

[0023] Both the short-circuit path unit 132 and the residual path unit 133 employ a two-order pipelined convolutional computation structure. The short-circuit path unit 132 comprises a max-pooling module, an intermediate output FIFO, an intermediate input buffer, and a 1×1 convolutional layer without an activation module, connected in sequence. The residual path unit 133 comprises a 3×3 convolutional layer with an activation module, an intermediate output FIFO, an intermediate input buffer, and a 3×3 convolutional layer without an activation module, connected in sequence. The input channels of the adder 134 are connected to the output channels of the 1×1 convolutional layer without an activation module and the 3×3 convolutional layer, respectively.

[0024] It is understandable that for networks designed for object detection tasks, the large network size and intermediate feature map dimensions, along with special topologies such as feature map fusion, make it difficult to directly implement the entire network on an FPGA. Furthermore, the YOLO network involves non-neural network operations such as Concat (function) fusion layers, upsampling, and non-maximum suppression. Implementing these operations on an FPGA would require significant resources and complex control logic, significantly increasing implementation complexity. In addition, the unique time step mechanism in the SNN model further amplifies the difficulty of implementing the entire network on an FPGA. Therefore, this paper chooses a more practical approach, focusing on accelerating the SpikingRes block on an FPGA. By configuring registers, the SpikingRes block is sequentially configured as the corresponding SpikingRes block in the network for computation. The SpikingRes block is the main SNN part of SpikingRes-YOLO and is the most critical part for performance improvement. This strategy allows resources to be concentrated on optimizing representative modules in the network and designing them as modular IPs for use and testing.

[0025] Due to the limitations of on-chip memory size in FPGAs, the network's input and output feature maps and weights can only be stored in off-chip DDR (Double Data Rate Synchronous Dynamic Random Access Memory) memory, and then read into the FPGA as needed. Considering that the residual path and the short-circuit path within each SpikingRes block share the same input, and that the computation results of the two paths need to be accumulated and activated before output, this paper implements a complete SpikingRes structure in the FPGA to reduce unnecessary DDR read operations.

[0026] The overall hardware architecture can be specifically described as follows: Figure 2 As shown, this includes an input buffer for the feature map, two parallel computation units consisting of pulsed convolutional layers and max-pooling layers, a convolutional activation module and an output FIFO (First-In-First-Out queue), as well as control logic and configuration registers for module configuration and scheduling. The two parallel computation units handle residual path and short-circuit path operations respectively. Both computation units are divided into a two-stage pipeline (stage 1 and stage 2 in the figure), with an intermediate output FIFO and an intermediate input buffer between the two stages for intermediate data transmission, reception, and temporary storage. Additionally, it includes channels for transmitting information to the off-chip DDR, used to read the input feature map, convolutional weights, and configuration information, and write back the output feature map. These channels are connected to the DDR via the Xilinx Memory Interface Generator (MIG) IP.

[0027] Based on this module, it is also necessary to consider configuring it as all SpikingRes blocks in the network model. For example, Table 1 lists some configurations of each SpikingRes block in one type of network model (where the input and output sizes are calculated based on time step T=3; the weights are calculated based on a fixed-point 8-bit width; rconv0 and rconv1 are the first and second convolutional layers in the residual path of the SpikingRes block, respectively; and sconv is the convolutional layer in the short-circuit path). It can be seen that, taking 3 time steps as an example, the input feature map size of the first two SpikingRes blocks reaches over 500kB. On the other hand, since the neuron state calculation at each time step in the SNN depends on the state of the previous time step, the membrane potential state value needs to be saved for each neuron at each coordinate during the sliding window process. For the first two SpikingRes blocks, if the entire feature map and its corresponding neuron state are directly saved on-chip, the required storage space obviously exceeds the available on-chip resources. Therefore, for modules with large feature map sizes, an operation mode of loading images line by line and calculating multiple time steps for each line is required. The specific design will be introduced below.

[0028] Table 1

[0029] The design of a row-by-row multi-timestep computation method for large feature maps includes two main parts: the row-by-row multi-timestep computation strategy and the design of data arrangement and read / write order. Line-by-line, multi-timestep computation strategy: Compared to the images used in traditional image classification tasks, the input images for object detection tasks are much larger, especially for networks primarily targeting small objects. Even after the encoding layer, the image size is still 384*384*32. Such images cannot be directly stored using on-chip cache. Therefore, the SpikingRes block can only store a portion of the image on the FPGA. Since the Spiking-YOLO network model typically only uses two types of convolutional kernels, 3*3 and 1*1, the data required for a single convolutional computation spans a maximum of three rows. Therefore, for modules with large input feature map sizes, only three rows of data are cached at a time. After the first convolutional layer is completed, one row of output feature map data is obtained and output to the next layer. This process continues until the pipeline is full, with each data write yielding a new row of results.

[0030] However, for SNNs, the issue of preserving neuron states across time steps also needs to be considered. While some research has designed dedicated RAM blocks to store the membrane potential of the neuron corresponding to each pixel position in the output feature map at the previous time step—the old membrane potential value is then used to calculate the activation result at that position in the current time step—storing a large number of neurons requires significantly more RAM resources when the output feature map size is large. Therefore, under this condition, the neuron state BRAM (a dedicated storage resource within the FPGA) is set to store only the neuron membrane potential values ​​required for one row of pixels in the output feature map. This also means that the pulse convolution calculation process for each row becomes a multi-time-step calculation, requiring caching three rows of input data across all time steps and calculating the convolution result for that row at each time step. Figure 3 As shown in (a), let the input features be... Figure 1 The line width is Output features Figure 1 The line width is Calculate the first feature map of the output feature map Walking When the result is obtained, the neuron state BRAM stores that row. The state of the nth neuron at the previous time step. Calculate the nth neuron's state. During the process, as the calculation results of the current neuron state are generated, the state values ​​cached in the BRAM are updated one by one. When the calculation of the output feature map begins... When traveling, as Figure 3 As shown in (b), the neuron's state BRAM stores the state of the last time step of the previous row. Therefore, the state values ​​in the BRAM will be used by the new row of neurons. The calculated membrane potential at that time is replaced.

[0031] As shown in Table 1, the feature map sizes of the first two SpikingRes blocks in the Spiking-YOLO network model are relatively large. Therefore, the large-size feature map multi-timestep scheme designed in this paper is selected for these two SpikingRes blocks. At the same time, the number of output channels in these two layers is relatively small, so they can be fully expanded and computed in parallel according to the number of channels during configuration. The weights only need to be written once at the beginning of the computation.

[0032] Data layout and read / write order design: In order to ensure that the entire process of reading data from DDR, calculating and writing back can run smoothly and correctly, a data layout and read / write order that facilitates pulse convolution operations needs to be designed.

[0033] Let the width, height, and depth (number of channels) of the input feature map be respectively... The width, height, and depth (number of channels) of the output feature map are: The time step is The kernel size is According to the rules of convolution, calculating one pixel of the output feature map requires... This means that all data needs to be traversed along the input channel dimension. Then, through parallel design at the output channel level, it can be obtained in one go. Each pixel. Therefore, this design prioritizes traversing the data based on the channel dimension. Then, according to the row-by-row, multi-time-step calculation scheme, it will sequentially move along... The dimension calculation outputs the pixels following the feature map.

[0034] Based on this calculation order, the image storage and retrieval order under this scheme is as follows: Figure 4 As shown. The image is... The sequence is stored in DDR. Fortunately, the SpikingRes block full-pulse design ensures that all input feature maps are in the form of binary pulses, and the bit width of the dual-channel DDR interface (taking the MIG (Memory Interface Generators) IP data bit width configuration of 256 bits as an example) is sufficient to cover... All data in the direction, if there is extra bit width, even the next... The data on the above are concatenated and transmitted together, such as Figure 4 The DDR section is shown in the dark box. Therefore, when loading data from DDR, it can be done directly according to... The data is read sequentially, one line at a time, into the input buffer. The input buffer consists of a BRAM with a width corresponding to the DDR interface width and its control logic. The BRAM contains an address field for each time step. The storage area for each time step caches the three lines of data required for the convolution operation, with the data arranged in order... Stored in order of size.

[0035] After the data is loaded, the input buffer will read the data at the corresponding address in the BRAM according to the coordinates of the current sliding window, and then fill the data window accordingly. Groups of data, each group consisting of... It consists of pulse data. After the data window is loaded, the coordinates of the sliding window are based on... The data is updated sequentially. For a 3×3 data window, all data within the window enters the residual path to complete the calculation of one output feature map pixel; while the lower right 2×2 portion of the data window is simultaneously input into the short-circuit path for pooling. The above process is as follows: Figure 5 As shown.

[0036] The caching method for the output of the first-level convolution or pooling operation is as follows: Figure 6As shown. The calculation results are collected in the intermediate output FIFO. When the intermediate FIFO is full of one row of data and the intermediate input buffer is free to receive data, the intermediate output FIFO reads one row of data and loads it into the intermediate input buffer. The storage order of the intermediate input buffer is the same as that of the module's input buffer, except that its width is set to match the maximum number of channels supported by the physical implementation of the convolutional layer. The bit width of the intermediate output FIFO is the same as that of the intermediate input buffer. If the currently configured number of channels in the convolutional layer is less than the bit width of the intermediate input buffer, the intermediate output FIFO will concatenate the calculation results of multiple pixels before storing them to make full use of the buffer's storage space. After the intermediate input buffer has written 3 rows of data, it will fill the data window in the same way as the input buffer, for use in the next layer of convolution.

[0037] Since the two convolutions in the second stage need to be summed before activation, to ensure that the output feature maps corresponding to the two results are at the same position, the two intermediate input buffers are filled with data windows based on the same sliding window address. The summed result is input into a separate convolution activation module 135, which has the same function as the activation part in the convolutional layer module. The final activation result is passed to the final output FIFO of the module. The output FIFO is then filled with the output features. Figure 1 After processing the data for all time steps, a write-back to DDR can be initiated. The order of the write-back data is consistent with the order in which the module read data from DDR at the beginning. The above output feature map write-back process is as follows: Figure 7 As shown.

[0038] The aforementioned residual SNN hardware accelerator implements and accelerates the core module of the residual SNN network, namely the pulse residual module 13, on an FPGA chip. Combined with off-chip DDR memory 11, a comprehensive implementation scheme for the residual SNN hardware accelerator is designed. Taking into account the multi-time-step computation characteristics of SNN, a technique for line-by-line image input and computation at multiple time steps is designed to overcome the practical problem of the large feature map size in SNN network algorithms, making it difficult to cache all feature maps and SNN neuron states on-chip. This designed hardware accelerator achieves a data computation speed more than twice that of the CPU hardware implementation scheme for the pulse residual module 13 in the residual SNN network, while significantly reducing power consumption compared to the CPU hardware implementation scheme, thus greatly improving data processing acceleration performance.

[0039] Each module in the aforementioned residual SNN hardware accelerator can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of a device with data processing capabilities, or stored in software within the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of image data computing and processing devices already existing in the art.

[0040] Further explanation is needed regarding the specific design of the functional modules of the residual SNN hardware accelerator: First, the input buffer module design: To support the aforementioned row-by-row, multi-time-step data input / output method, a configurable input buffer was designed, and its specific configuration parameters are shown in Table 2 below: Table 2

[0041] `kernel_size` represents the size of the convolutional layer in the current module, which can be either 3×3 or 1×1. `kernel_number` represents the number of convolutional kernels in the first-stage convolutional layer, i.e., the number of output channels. `stride` represents the stride of the first convolutional layer operation in the current module, which can be set to 2 or 1. `channel_in` is the channel depth of the input feature map; the input buffer will calculate the actual BRAM read / write address based on this configuration. `h_in`, `w_in`, and `time_step` are the height, width, and set time step of the input feature map, respectively, used to calculate the current window position and state. `padding` indicates whether the current convolution needs to perform zero-padding on the edges of the input image; a setting of 0 means no padding is needed, otherwise padding is required. `row_number` is the number of rows of feature map data stored in the input buffer. When `row_number` is configured to 3, it indicates that the input buffer is performing the row-by-row multi-timestep mode designed in this section. These parameters are also shared with other modules in the first-stage pipeline so that other modules can implement configurable designs.

[0042] In one embodiment, the working states of the input buffer include INIT, FIRST_WRITE, READ, WAIT, WRITE, MAP_END, and DONE states.

[0043] It is understandable that the input buffer control logic is as follows: the input buffer is controlled by, for example... Figure 8 The state machine shown in the diagram controls the operation. Only the state transition paths are indicated. If the input conditions do not meet the transition requirements, the state machine will maintain its current state. The input buffer has seven states: INIT, FIRST_WRITE, READ, WAIT, WRITE, MAP_END, and DONE. These states are explained in detail below: INIT: Initialization phase. During this phase, hardware initialization and SpikingRes block configuration are completed. After configuration, it waits for an enable signal from the host computer. Upon receiving the enable signal, it switches to the FIRST_WRITE state.

[0044] FIRST_WRITE: The initial loading stage. During this stage, data from the DDR is received, completing the initial loading of the input feature map. For the input buffer, depending on the operating mode, if in line-by-line multi-step mode, the input buffer will load data from the first three rows of the feature map across all time steps; if in full-input single-step mode, it will load the feature map data from all time steps. Figure 1 All data at each time step. After the buffered data is written, the acknowledgment signal on the DDR side is turned off, and the system enters the READ state.

[0045] READ: This stage reads data from the input buffer and loads it into the data window register. During this stage, the input buffer calculates the read address based on the sliding window coordinates, sequentially retrieves data from the buffer, loads it into the data window register, and enables the convolutional layer computation. After the data window is loaded, the system enters the WAIT state.

[0046] WAIT: Waiting state. During this stage, the input buffer maintains its internal state and the data window register data, waiting for the flag signal indicating completion of the convolution calculation. When the flag signal arrives, the following state transitions will occur based on the current conditions: (a) If the currently completed convolution operation is the last data point of the entire feature map, it will switch to the MAP_END state; (b) If the module is in a row-by-row multi-step mode, and the currently completed convolution operation is the last data point of the last time step in a row, it will switch to the WRITE state; (c) Otherwise, it will return to the READ state and load data for the next sliding window position.

[0047] WRITE: Buffer loading stage. This stage is only valid in line-by-line multi-step mode for the input buffer. In this stage, data is received from DDR again, the buffer is written with new data sufficient for the next line of convolution operation, the DDR-side acknowledgment signal is turned off, and the system re-enters the READ state.

[0048] MAP_END: ​​The final stage of the feature map. At this stage, the sliding window coordinates have reached the last pixel of the feature map, and the system will wait for the activation completion flag signal. When this flag signal arrives, the following state transitions occur based on the current conditions: (a) If the feature maps at all time steps have been calculated, the system will switch to the DONE state; (b) If the module is in full-input single-step mode and there are still output channel groups that have not been calculated within this time step, the system will return to the READ state and re-slide the window; (c) If the module is in full-input single-step mode and the output feature map for the next time step has not yet been calculated, the system will return to the FIRST_WRITE state and write the input feature map for the next time step into the buffer.

[0049] DONE: Computation completion stage. This stage indicates that the module has completed all computations on a feature map. It will clear all internal status registers and data windows, awaiting the enable signal to be turned off. After the enable signal is turned off, it returns to the INIT state.

[0050] In one embodiment, the input buffer reuses data according to a set address rearrangement mechanism during data read and write operations.

[0051] It's understandable that the input buffer can also reuse data: in a row-by-row multi-timestep mode, when the input buffer enters the WRITE state, it means that some data has already been stored in the buffer, and each row's convolution operation will use some data from the previous row's convolution. To avoid duplicate data readings, an address rearrangement mechanism is designed in the input buffer, allowing the already stored data to be effectively reused. The specific method is as follows: like Figure 9 As shown in (a), when performing convolution calculations on a new row, data will be read from the three cached rows B0, B1, and B2 to fill the corresponding row of the data window. Depending on the stride configuration, data loading falls into two categories: If stride = 1, the row with the smallest number (L0) from the data loaded in the previous calculation will no longer be used, and a new row of data (L3) needs to be written; if stride = 2, only the row with the largest number from the data loaded in the previous calculation needs to be retained, and two new rows of data are written. The addresses of the saved data in BRAM remain unchanged, meaning that the location of the data retained from the previous calculation does not correspond to the current cached row number.

[0052] Therefore, additional corresponding rules need to be designed, specifically, such as... Figure 9 As shown in (b), B0, B1, and B2 represent the virtual row numbers of the cache, and there is a one-to-one correspondence between the virtual row numbers and the rows of the data window; while the actual physical row addresses of the BRAM are represented by H0, H1, and H2. When entering the WRITE state for the first time (Cycle 0), the virtual row numbers and physical row addresses maintain a one-to-one correspondence, which is the first correspondence rule. At this time, the row to be written is stored in the row with the smallest original number (i.e., the dark part in the figure), and the virtual row number and physical address switch to the second correspondence rule: when stride=1, the actual physical address corresponding to the first row B0 of the data window is H1, B1 corresponds to H2; B2 corresponds to H0; the same applies when stride=2. When entering the WRITE state for the second time (Cycle 1), new data is also written to the corresponding positions B0 or ​​B0~B1. After writing, the virtual row number and physical address switch to the third correspondence rule, as shown below. Figure 9As shown in Cycle2 in (b). Similarly, it can be seen that the correspondence of Cycle3 and above returns to the first correspondence rule. Therefore, only three correspondence rules need to be set, and the rule can be switched after each write operation in the WRITE state is completed.

[0053] By using the address reordering mechanism, the input buffer can be addressed according to the current corresponding rules when writing and reading data, thus enabling data reuse. Each write only needs to be supplemented with a number of rows of data equal to the step size.

[0054] In one embodiment, the padding operation on the input feature map in the input buffer includes vertical padding operation and horizontal padding operation.

[0055] As can be understood, regarding the padding operation mentioned above: for a 3×3 convolution operation, to ensure that the output feature map size remains unchanged (when the stride is 1) or is half the size of the input (when the stride is 2), a padding operation needs to be performed on the input feature map, that is, to expand the feature map with a ring of pixels with a value of 0 around its perimeter. To ensure the continuity of reading and writing data to DDR, the padding operation is performed in the input buffer, and it is decomposed into two parts: top and bottom padding and left and right padding, which are performed separately.

[0056] The upper and lower padding operations are performed when data is written to the buffer. When the input buffer padding is configured to 1, the total number of rows in the input feature map is considered to be h_in+2. When loading the first and last rows, the DDR side acknowledgment of the input buffer is turned off. At this time, the DDR does not send data to the input buffer, but the input buffer writes a row of 0s to the BRAM itself; in other parts, it continues to accept DDR data written to the BRAM as usual.

[0057] The left and right padding operations are performed when loading the data window. This is because the width of a row after adding left and right padding (w_in+2) may not be divisible by the data merging amount (DDR_width / channel_in), resulting in the data of a row not being stored neatly in the BRAM. Therefore, the width of the input feature map is considered to be w_in+2 during left and right padding, but the actual width of the stored feature map is still w_in. However, when loading the data window, the first and last columns are not read from the BRAM, but are directly written to the corresponding positions in the data window. One zero.

[0058] Intermediate Input Buffer Design: The configurable parameters for the intermediate input buffer are largely the same as those for the input buffer. The intermediate input buffer interfaces with the intermediate output FIFO of the previous stage, while the input buffer is the MIG IP of the DDR. Therefore, compared to the input buffer, the BRAM width set for the intermediate input buffer is the maximum parallelism of the computing unit, while that for the input buffer is the DDR user interface width under the MIG.

[0059] In terms of control logic, the intermediate input buffer also basically follows the state machine design of the input buffer, such as... Figure 10 As shown. However, in the full-input single-timestep mode, the FIRST_WRITE state does not cache all data before outputting to the convolutional layer. Instead, it receives the data needed for the next convolutional layer's calculation and then starts outputting data. This also means that the intermediate input buffer will enter WRITE mode in single-timestep mode to store the next row of input data. When calculating the end of the feature map, a check is added to see if the current batch of input feature maps has been fully loaded. If there are still unloaded ones, it will return to the FIRST_WRITE state to wait for the next batch of feature maps generated by the first convolutional layer. If all feature maps have been cached, but there are still uncalculated convolutional groups, it will return to the READ state and re-window. It will only enter the DONE state after all output feature maps for all timesteps have been calculated. Apart from the above differences, the other characteristics of the intermediate input buffer are consistent with the input buffer.

[0060] Hardware module design for convolutional layers and max pooling layers: such as Figure 11 and Figure 12 As shown, the spiking convolutional layer and max pooling layer inherit the design of the spiking convolutional layer and max pooling layer modules in Spiking Convolutional Neural Networks (SCNN), but remove the input data buffer and output line buffer, as their functions are replaced by the input buffer (or intermediate input buffer) and output FIFO (intermediate output FIFO), respectively. Furthermore, since the spiking convolutional layer needs to read weights from the DDR (Data Retrieval Memory), it also includes a channel to write DDR data to the BRAM (Browser RAM) of each weight.

[0061] Because BN layers can be fused into convolutional layers during inference, and after fusion, the convolutional layers are directly connected to LIF activation layers, this pulsed convolutional layer can be used to implement the computation of consecutive convolutional layers and LIF activation layers in the SpikingRes block. Since the second-level activation requires summing the convolution results of the residual path and the short-circuit path before feeding them into the neuron, the second-level convolutional module needs to separate its convolution computation part from the LIF neuron part. The separated LIF neuron part is... Figure 2The convolution activation module in [the context].

[0062] As shown in Table 1, the convolutional layers in the residual path use both 3×3 and 1×1 kernels. This requires the 3×3 convolutional layers in the residual path to also support 1×1 convolution operations. Therefore, when a 3×3 convolutional layer is configured as a 1×1 layer, its 3×3 data window only contains... Only a portion of the data is effective, and the convolutional layer only accumulates weights for that portion. The number of rows where the weights are stored in BRAM is [number missing]. Each weight is stored in the lower-order part of a row in BRAM. For convolutional layers with short-circuit paths, only a 1×1 convolutional kernel is required.

[0063] For SpikingRes blocks with a stride of 1, no pooling operation is required. Therefore, in the hardware implementation, to support SpikingRes blocks with this configuration, the max pooling layer does not perform pooling operations when the stride is 1, but instead directly outputs the data from its 2×2 data window. This is equivalent to simply synchronizing the input feature map to the second-stage pipeline for computation by the convolutional layers with short-circuit paths.

[0064] Activation module design: The activation modules in the first-level convolutional layer and the convolutional activation modules before the output FIFO are also inherited from the convolutional activation design in SCNN.

[0065] Output FIFO Module Design: The output FIFO and intermediate output FIFO share a similar design. Both contain a FIFO with a configurable full-signal. When the data buffered in the FIFO reaches the configured number of rows, and the next stage can write, the configured number of rows of data are read from the FIFO and sent to the next stage. On the other hand, when receiving data from the convolutional layer, if the number of data channels is less than the FIFO's bit width, both modules will concatenate several received data points to the FIFO width before storing them in the FIFO. The bit width of the output FIFO is configured to match the DDR data interface bit width, while the bit width of the intermediate output FIFO matches the maximum parallelism of the convolutional layer. Furthermore, the output FIFO's output port is a DDR interface, therefore it also contains logic for generating DDR write addresses.

[0066] In some implementations, to more fully and intuitively illustrate the performance of the above-described solutions, experimental examples for the solutions given above are provided below: The designed hardware accelerator architecture was written in SystemVerilog as RTL code and mapped onto an FPGA development board for synthesis and implementation. In the designed architecture, the parallelism of each convolutional channel was set to 128, the time step to 3, the neuron reset voltage to 0, and the threshold voltage to 1. The weight quantization precision was set to 8-bit fixed-point numbers, including 1 bit for the sign, 3 bits for the integer part, and 4 bits for the fractional part, with a time constant... The clock frequency of the FPGA is set to 2. Since the required time step is very short and the neuron membrane potential resets once it exceeds the threshold voltage, only 4 bits of fixed-point data (including 1 bit for the sign, 1 bit for the integer, and 2 bits for the fractional) are needed to store the intermediate states of the neuron, saving space in the neuron state BRAM. The hardware resources consumed after mapping to the FPGA are shown in Table 3. The DSP is only used when calculating the BRAM address in the input buffer and intermediate input buffer modules, and no DSP resources are used in the pulse convolution, greatly reducing the demand for DSP resources.

[0067] Table 3

[0068] The hardware accelerator designed in this paper was configured for each SpikingRes block in SpikingRes10-YOLO. The acceleration effect of the accelerator on the SpikingRes blocks was tested, while the remaining modules continued to run on the main control unit. Simultaneously, this example also tested the running speed of the same SpikingRes block on a CPU platform for comparison. The acceleration effect for each SpikingRes block is shown in Table 4. As can be seen from Table 4, for most SpikingRes blocks, the computational scheme in this paper can achieve full load operation of computing resources, and the inference latency of each SpikingRes block is lower than that of the CPU platform. However, for the first SpikingRes block, due to its smaller number of channels, some computing resources are not utilized, and the image size is large, resulting in a relatively worse acceleration effect compared to other SpikingRes blocks.

[0069] Table 4

[0070] Table 5 shows the overall performance and power consumption comparison between the accelerator and the CPU platform. It can be seen that the average inference latency of the FPGA implementation is 32.72ms, and the average speed per SpikingRes block is 30.56fps, more than twice that of the CPU platform, while the power consumption is significantly lower. Regarding accuracy, the error of the hardware accelerator designed in this paper mainly comes from the fixed-point quantization of weights and neuron states. As shown in Table 4, under the condition of converting weights to 8-bit fixed-point numbers and neuron states to 4-bit fixed-point numbers, the accuracy loss of the network is within an acceptable range, and the required storage space and power consumption are greatly reduced.

[0071] Table 5

[0072] In one embodiment, such as Figure 13 As shown, a residual SNN hardware acceleration method based on the above-mentioned residual SNN hardware accelerator is also provided, which may include the following processing steps S12 to S18: S12, by configuring the register, switch the working mode of the pulse residual module that needs to be accelerated in the residual SNN network to the row-by-row multi-time-step mode and configure the pulse residual module to the specifications of the residual module that needs to be accelerated; S14, read the network weights of the pulse residual module and the input feature map to be processed from the DDR memory according to the set data arrangement and read / write order; S16, the pulse residual module performs calculation processing on the input feature map according to the row-by-row multi-time-step calculation strategy, and outputs the corresponding output feature map; S18, according to the set data arrangement and read / write order, writes the output feature map into DDR memory through the pulse residual module.

[0073] The aforementioned hardware acceleration method for residual SNNs implements and accelerates the core module of the residual SNN network, namely the pulse residual module, on an FPGA chip. Combined with off-chip DDR memory, a novel hardware accelerator implementation scheme for residual SNNs is designed. Taking into account the multi-time-step computation characteristics of SNNs, a technique for line-by-line image input and computation at multiple time steps is designed to overcome the practical problem of large feature map sizes in SNN network algorithms, making it difficult to cache all feature maps and SNN neuron states on-chip. This designed hardware accelerator achieves a data computation speed more than twice that of the CPU hardware implementation scheme for the pulse residual module in the residual SNN network, while significantly reducing power consumption, thus greatly improving data processing acceleration performance.

[0074] In one embodiment, when the number of convolutional layer channels currently configured in the pulse residual module is less than the bit width of the input buffer, the intermediate output FIFO in the pulse residual module concatenates the calculation results of multiple pixels before storing them.

[0075] In one embodiment, the two intermediate input buffers within the pulse residual module fill the data window based on the same sliding window address.

[0076] It is understood that the explanations of the features in the various embodiments of the residual SNN hardware acceleration method described above can be understood by referring to the explanations of the corresponding features in the various embodiments of the residual SNN hardware accelerator above, and will not be repeated here.

[0077] It should be understood that, although the above process Figure 13 The steps in the process are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this application, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Furthermore, the above process... Figure 13 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0078] In one embodiment, a computer device is also provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following processing steps: switching the operating mode of the pulse residual module that needs to be accelerated in the residual SNN network to a row-by-row multi-timestep mode by configuring registers and configuring the pulse residual module to the specifications of the residual module that needs to be accelerated; reading the network weights of the pulse residual module and the input feature map to be processed from the DDR memory according to the set data arrangement and read / write order; performing calculation processing on the input feature map by the pulse residual module according to the row-by-row multi-timestep calculation strategy and outputting the corresponding output feature map; and writing the output feature map into the DDR memory by the pulse residual module according to the set data arrangement and read / write order.

[0079] It is understood that, in addition to the memory and processor mentioned above, the computer equipment may also include other hardware and software components not listed in this specification. The specific components may be determined according to the model of the specific image data processing equipment in different application scenarios, and will not be listed and described in detail in this specification.

[0080] In one embodiment, when the processor executes the computer program, it can also implement the steps or sub-steps added in the various embodiments of the residual SNN hardware acceleration method described above.

[0081] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus DRAM (RDRAM), and interface DRAM (DRDRAM), etc.

[0082] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and all of these fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A residual SNN hardware accelerator, characterized in that, The pulse residual module includes DDR memory and a residual SNN network deployed on an FPGA chip. The pulse residual module includes an input buffer, a short-circuit path unit, a residual path unit, an adder, a convolution activation module, an output FIFO, and control logic and register units. The DDR memory is connected to the input buffer, the short-circuit path unit, the residual path unit and the output FIFO respectively. The DDR memory is used to store the input feature map, output feature map and network weights of the pulse residual module according to the set data arrangement and read / write order. The input buffer, the short-circuit path unit, the residual path unit, the adder, the convolution activation module, and the output FIFO are sequentially connected and used to perform computation processing on the input feature map according to the row-by-row multi-time-step computation strategy and output the corresponding output feature map. The control logic and register unit is used to configure the pulse residual module to work in the row-by-row multi-time-step mode and configure the pulse residual module to the specifications of the residual module that needs to be accelerated; wherein, when performing computation processing on the input feature map according to the row-by-row multi-time-step computation strategy and outputting the corresponding output feature map, the width of one row of the input feature map is assumed to be... The output feature map has a row width of 1. Calculate the first feature map of the output feature map Walking When the result is reached, the neuron state BRAM stores the result of the first time. OK Calculate the state of the nth neuron at the previous time step. During the process, as the calculation results of the current neuron state are generated, the state values ​​cached in the BRAM are updated one by one. When the calculation of the output feature map begins... During a row, the neuron's state BRAM stores the state of the last time step of the previous row; The short-circuit path unit and the residual path unit both adopt a two-order pipelined convolutional computation structure. The short-circuit path unit includes a max pooling module, an intermediate output FIFO, an intermediate input buffer, and a 1×1 convolutional layer without an activation module connected in sequence. The residual path unit includes a 3×3 convolutional layer with an activation module, an intermediate output FIFO, an intermediate input buffer, and a 3×3 convolutional layer without an activation module connected in sequence. The input channel of the adder is connected to the output channels of the 1×1 convolutional layer without an activation module and the 3×3 convolutional layer, respectively.

2. The residual SNN hardware accelerator according to claim 1, characterized in that, The input buffer's operating states include INIT, FIRST_WRITE, READ, WAIT, WRITE, MAP_END, and DONE.

3. The residual SNN hardware accelerator according to claim 1 or 2, characterized in that, The input buffer reuses data according to a set address rearrangement mechanism during data read and write operations.

4. The residual SNN hardware accelerator according to claim 3, characterized in that, The padding operations on the input feature map in the input buffer include vertical padding operations and horizontal padding operations.

5. A residual SNN hardware acceleration method based on the residual SNN hardware accelerator according to any one of claims 1 to 4, characterized in that, Including the following steps: By configuring registers, the working mode of the pulse residual module that needs to be accelerated in the residual SNN network is switched to the row-by-row multi-time-step mode, and the pulse residual module is configured to the specifications of the residual module that needs to be accelerated. The network weights of the pulse residual module and the input feature map to be processed are read from the DDR memory according to the set data arrangement and read / write order. The pulse residual module performs calculation processing on the input feature map according to the row-by-row multi-time-step calculation strategy, and outputs the corresponding output feature map. According to the set data arrangement and read / write order, the output feature map is written into the DDR memory through the pulse residual module.

6. The residual SNN hardware acceleration method according to claim 5, characterized in that, When the number of convolutional layer channels currently configured in the pulse residual module is less than the intermediate input buffer width, the intermediate output FIFO in the pulse residual module will concatenate the calculation results of multiple pixels before storing them.

7. The residual SNN hardware acceleration method according to claim 5, characterized in that, The two intermediate input buffers within the pulse residual module are filled with data windows based on the same sliding window address.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the residual SNN hardware acceleration method according to any one of claims 5 to 7.

Images