Pulse neural network inference apparatus and method supporting elastic inference

Abstract

This invention discloses a spiking neural network inference device and method supporting elastic inference, belonging to the field of artificial intelligence hardware acceleration. The device includes multiple neural cores and an on-chip network (NoC). Each neural core includes a processing unit (PE), a router, and an output scheduler. The router bundles multiple spiking events generated by the same spine / token in the same row into a bundled address event representation BAER data unit and transmits it in the NoC. After unpacking by the target neural core, the output scheduler immediately triggers the computation of subsequent layers according to the fine-grained pipeline at the spine / token level. The PE uses mini-batch spiking Gustavson product, reading and writing back the corresponding membrane potential / tracker row only once for each BAER batch, and accumulating multiple weight rows in parallel, supporting the addition and subtraction of polarity spiking updates. This scheme reduces initial response latency and reduces communication and memory access power consumption.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence computing and neuromorphic computing, and in particular to a device and method for accelerating spiking neural network (SNN) inference that supports resilient inference. It implements a spine / token-level fine-grained pipeline under Network-on-Chip (NoC) interconnection conditions, and combines the Bound Address Event Representation (BAER) communication protocol with mini-batch spiking Gustavson product resilient inference architecture and its implementation method. Background Technology

[0002] SNNs propagate information across multiple time steps using discrete impulse events. Their computations are primarily additive and feature sparse activation, making them suitable for implementing event-driven computation in hardware while reducing energy consumption. In many vision and sequence tasks, SNN outputs tend to form gradually over time steps: salient inputs may produce stable or high-confidence outputs in fewer time steps, while complex inputs require more. "Resilient inference" refers to the ability of the inference output to gradually converge and produce a usable response at an earlier time point within a given maximum allowed time step, with the final output consistent with the complete execution after sufficient time steps.

[0003] However, existing SNN inference hardware commonly employs two execution methods: First, layer-by-layer execution: This requires all time steps to be completed in a given layer before moving to the next, meaning output can only be generated after the entire network is complete, making early response difficult. Second, time-step execution: While each time step traverses the network to gradually generate output, this often uses a coarse-grained pipeline, requiring buffering and synchronization of all backbones / tokens within a layer before proceeding. This prevents preceding layers from immediately pushing forward completed local results, thus suppressing the first-response advantage of resilient inference.

[0004] Existing neuromorphic communication often uses Address Event Representation (AER) to transmit individual pulse events with added spatial location and temporal information. If transmitted in a "one event, one packet / one flit" manner, it will introduce a large header overhead and packet number, and amplify NoC traffic under multi-timestep conditions.

[0005] On the other hand, SNN inference involves state variables such as membrane potential and pulse tracker, whose bit width is often higher than that of the weights and the pulses themselves. If the membrane potential row is frequently switched under the condition of asynchronous event arrival, the number of state read and write will increase, offsetting the energy consumption advantage brought by sparsity.

[0006] Therefore, a new device and method are needed to achieve this without introducing a synchronization barrier that would disrupt elastic reasoning: Enables the spine / token to be instantly forwarded and advanced between layers; Reduce header overhead and packet count in pulse communication; Reduces the number of accesses to the membrane potential / tracker and is compatible with fine-grained flow.

[0007] To address the aforementioned issues, there is an urgent need for an SNN inference hardware architecture and method that can achieve fine-grained pipelined propagation, efficient communication, and state memory access without introducing synchronization barriers, so as to fully leverage the energy efficiency and latency advantages of SNNs in elastic inference scenarios. Summary of the Invention

[0008] This invention aims to solve at least the above three technical problems existing in current spiking neural network (SNN) hardware inference devices, including: 1) Existing SNN inference accelerators suffer from high initial response latency due to hierarchical synchronization and coarse-grained pipelines, making it difficult to utilize elastic inference. 2) Traditional AER protocol has a large single-event transmission header overhead and a large number of packets, which leads to an increase in on-chip network communication volume and energy consumption; 3) When the conventional sparse matrix multiplication execution mode is directly adopted, the membrane potential and tracker state are read and written frequently, resulting in high memory access energy consumption and conflict with the arrival order of asynchronous pulses.

[0009] The technical solution of the present invention is as follows: A spiking neural network inference device supporting flexible inference, characterized in that it includes: an on-chip network and a plurality of neural nuclei interconnected through the on-chip network; Each of the neural nuclei includes: a router, a processing unit, and an output scheduler, wherein: The router has a communication interface connected to the on-chip network for receiving, sending, routing, and processing pulse event data units. The processing unit, connected to the router, is used to perform pulse-driven matrix multiplication calculations and to store and update neuron states. The output scheduler is connected to the router and the processing unit respectively, and is used to control the calculation triggering timing of the processing unit according to the unpacked pulse event; The router is configured to bundle multiple pulse events originating from the same data path to form a bundled address event representation data unit with a shared header, and transmit it to the target neural nucleus through the on-chip network; wherein the same data path is defined by the same fine-grained control unit identifier and the same target neuron state row index; The output scheduler is configured to use fine-grained control units as the smallest scheduling unit. After a single fine-grained control unit completes the calculation of the current layer, it immediately triggers the forwarding of the calculation result corresponding to that fine-grained control unit to the subsequent layers, without needing to synchronize with the execution progress of other fine-grained control units in the same layer.

[0010] Furthermore, the fine-grained control unit is defined in a convolutional neural network as the spine of all channel data of the feature map at the same spatial location, or in a Transformer-based spiking neural network as a token in a sequence.

[0011] Furthermore, the data structure of the bundled address event representing the data unit includes: A shared header field containing destination information, type identifier, fine-grained control unit identifier, and neuron state row index; An event list field contains at least one event entry, each event entry containing at least column position information for addressing weights and polarity information indicating the direction of pulse action.

[0012] Furthermore, the processing unit is specifically configured as follows: The entirety of the event entries contained in a single received bundled address event representation data unit is processed as a single computation batch. For this computation batch, a single read operation is performed on the neuron state; The weights of each event entry within the calculation batch are accumulated in parallel. Based on the accumulation and pulse firing results, a single write-back update operation is performed on the neuron state.

[0013] Furthermore, the processing unit includes: A state memory is used to store neuronal membrane potentials and / or pulse tracker states; A weight memory is used to store synaptic weights associated with neuron state rows in the state memory; control logic is used to parse the bundled address event representation data unit and generate an access control sequence for the state memory and the weight memory, the access control sequence ensuring that access to the same neuron state row satisfies a single read / write constraint. The computing array, connected to the control logic, the state memory, and the weight memory, is used to perform parallel reading of the weights, accumulation operations, and neuron firing determination.

[0014] Furthermore, the router further includes: A local input merger is used to receive output pulse events generated by the processing unit and merge them according to the fine-grained control unit identifier and the target neuron state row index to form a set of events to be bundled. The flit generator, connected to the local input merger, is used to encapsulate the event set into the bundled address event representation data unit; A routing engine is used to determine the forwarding path of the bundled address event representation data unit in the on-chip network based on the header information of the bundled address event representation data unit. The flit decoder is used to parse the data units arriving at this neural nucleus and recover the impulse event data; At least one first-in-first-out queue is connected between the flit decoder and the output scheduler as a pipeline register across neural nuclei.

[0015] Furthermore, the router also integrates at least one non-matrix multiplication operator processing unit, which is connected between the local input merger and the flit generator, and / or between the flit decoder and the first-in-first-out queue, for performing specific operations of the spiking neural network; the specific operations include at least one of the following: convolution im2col transformation, spiking softmax operation, and spiking layer normalization operation.

[0016] Furthermore, the output scheduler is further configured to: Maintain the dependency state of each output spine in the output feature map on the set of input spines it depends on; Monitor the arrival events of the input spine; When it is determined that all dependent input spines of a certain output spine have been reached, the calculation for that output spine is immediately triggered, and this triggering action is independent of the dependency satisfaction state of other output spines.

[0017] Furthermore, the output scheduler is further configured to: In the process of token-level asynchronous pipelined implementation, a synchronization point is set for specific operators that require cross-token synchronization; When the pipeline reaches the synchronization point, the forward push of relevant tokens is paused. After all the necessary tokens arrive at the synchronization point, synchronization is desynchronized and the pipeline execution continues.

[0018] The present invention also provides a spiking neural network inference method supporting elastic inference, applied to the device described above, characterized in that the method includes the following steps: Receive input pulse event stream; Pipeline scheduling is implemented between layers of the neural network at the level of fine-grained control units, so that the completion event of the computation of a single fine-grained control unit can independently trigger the computation of its successor layer. During pulse event transmission, multiple pulse events belonging to the same fine-grained control unit and targeting the same neuron state row are bundled and aggregated into a data packet with a shared header for communication. In the processing unit, the event entries within a single data packet are treated as a computation batch. A single read and a single write-back operation are performed on the neuron state rows involved in the batch, and the weights corresponding to each event in the batch are accumulated in parallel.

[0019] Furthermore, the method also includes the following in the flow scheduling step: For convolutional layers, the dependency of the output spine on the input spine is dynamically maintained; A mechanism of triggering upon dependency satisfaction is adopted. Once all the dependent input spines of a certain output spine are detected to be ready, the computation and forwarding of that output spine are immediately started.

[0020] Furthermore, the method also includes the following in the flow scheduling step: For self-attention operations in Transformer class models, insert synchronization points in the token-level asynchronous pipeline; At the synchronization point, after all relevant tokens involved in the attention calculation have arrived, the pulse softmax calculation is performed and the token pipeline continues.

[0021] Furthermore, the bundling and aggregation step also includes: When a fine-grained control unit detects that the event stream has ended at the current time step or calculation phase, if the number of merged event entries is insufficient to form a complete standard data packet, a fixed-length final data packet is generated and sent using a padding method.

[0022] Compared with the prior art, the present invention has at least the following beneficial effects: 1) Reduced initial response latency: Fine-grained pipelines at the spine / token level enable completed spines / tokens to be immediately advanced across layers, reducing synchronization wait time; 2) Reduced communication volume: BAER reduces the number of flits and redundant field transmissions by sharing header fields and aggregating multiple event entries; 3) Reduced memory access power consumption: Small-batch pulse Gustavson integration enables the membrane potential / tracker to perform only one read / write operation within a small batch, reducing access to high-bit-width states; 4) Compatible with multiple model operators: Adapts to convolutional SNNs and Transformer / MLP SNNs, and can support necessary non-matrix multiplication operators on the router side. Attached Figure Description

[0023] Figure 1 : Schematic diagram of the overall structure of a spiking neural network inference device supporting elastic inference (including multiple neural nuclei, NoC, router, PE, and data flow direction of the output scheduler).

[0024] Figure 2A schematic diagram comparing fine-grained spine / token-level flow and hierarchical coarse-grained flow.

[0025] Figure 3 A schematic diagram of the computational process / state update of a spiking neuron (e.g., ST-BIF) (integration, firing decision, update).

[0026] Figure 4 : Schematic diagram of flow granularity definition (spine in CNN, token in Transformer).

[0027] Figure 5 Comparison between traditional AER and BAER, and a schematic diagram of BAER field structure.

[0028] Figure 6 : Microarchitecture diagram of the processing unit PE (weights / membrane potential / tracker buffer, control module, addition tree, distribution and update components).

[0029] Figure 7 : A schematic diagram of the data flow of small-batch pulse Gustavson product (reading and writing the same row once, reading multiple weighted rows in parallel, and adding and subtracting negative pulses).

[0030] Figure 8 : Schematic diagram of the tiling strategy for weighted / membrane potential segmentation and pulse broadcasting.

[0031] Figure 9 : Router microarchitecture diagram (local input merging, flit generation, routing, arbitration, decoding, FIFO, output scheduling, im2col / softmax / layernorm units).

[0032] Figure 10 : Schematic diagram of CNN convolutional layer spine-level pipeline scheduling (dependency fulfillment trigger) and output scheduler control logic flow.

[0033] Figure 11 : Schematic diagram of Transformer token-level pipeline scheduling (softmax pre-synchronization point).

[0034] Figure 12 : Schematic diagram of partition-mapping-multipath routing process.

[0035] Explanation of the labels in the diagram 200: On-Chip Network NoC 300: Neural nuclei 310: Router 311: Local Input Reducer 312: flit generator 313: Routing Engine 314: Arbitrator 315: Flit Decoder 316: FIFO queue (pipeline register) 317: Output Scheduler 318: im2col unit 319: Pulse softmax unit 320: Pulse layernorm unit 330: Processing Unit PE 331: Control Module 332: Weighted Buffer (N-way / Multi-way) 333: Membrane potential buffer 334: Pulse Tracker Buffer 335: Addition Tree 336: Issuance Judgment Component 337: State Update Component 338: ST-BIF Neuron Circuit Array 400: BAER data unit 401: Destination field 402: Type field 403: Spine / Token Identifier Field 404: Row index field 405: Event list field 406: Event Entry 407: Column position field 408: Polarity Field 409: Validation field Detailed Implementation To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0036] 1. Overall Structure of the Device like Figure 1 As shown, a spiking neural network inference device supporting flexible inference includes multiple neural cores 300 and an on-chip network NoC 200. Each neural core 300 includes: Router 310: Responsible for packetizing / unpacking and routing of pulse events, and can implement some non-matrix multiplication operators (such as pulse softmax, pulse layernorm, convolution unrolling broadcast, residual addition, etc.). Processing unit PE 330: responsible for the core calculations of matrix multiplication-type impulse operators, and storing and updating neuron membrane potentials and impulse tracker states; Output scheduler 317: works in conjunction with router 310 to enable fine-grained pipeline control and triggering at the spine / token level.

[0037] 2. Spine / token-level fine-grained pipeline (elastic inference mechanism) like Figure 2 As shown, this invention uses the spine / token as the flow propulsion particle size: Once a spine / token has been calculated in the current layer, the output scheduler 317 does not wait for other spines / tokens in the same layer to complete, but instead triggers the calculation of that spine / token in the subsequent layer. This mechanism eliminates the "full spine / token synchronization barrier" in the hierarchical coarse-grained pipeline, enabling the network to form continuous streaming processing, thereby reducing first response latency.

[0038] in: For CNNs, the spine is defined as a 1×1×C vector corresponding to a spatial location on the feature map (see...). Figure 4 ); For Transformer, tokens are defined as 1×D vectors in the sequence (see...). Figure 4 ).

[0039] 3. The bundled address event indicates BAER (Communication Optimization). like Figure 5 As shown, router 310 bundles and packages multiple pulse events of the same spine / token into BAER data units 400: In a BAER data cell, the destination field 401, the type field 402, the spine / token identifier field 403, and the row index field 404 appear only once; The event list field 405 contains multiple event entries 406, and each event entry contains at least a column position field 407 and a polarity field 408; Optionally, the type field 402 indicates the start / middle / end position of the BAER data unit within the same spine / token event sequence; By bundling header fields across multiple event entries, the number of flits / packets and redundant field transmissions are reduced, thereby lowering NoC traffic and energy consumption. In one embodiment, when the computation is complete and the number of event entries is insufficient, zero-padding is used to form fixed-length BAER data units.

[0040] 4. Mini-batch pulse Gustavson product (memory access optimized, and compatible with pipelining) like Figure 7As shown, the processing unit PE 330 utilizes the row-aligned mini-batch provided by BAER to adapt the Gustavson product: A list of event entries within a BAER data unit of 400 is used as a mini-batch; For this small batch, the corresponding membrane potential line is read only once and accumulated in parallel, then written back in one go; For negative polarity pulse events, perform weight inversion / complement negation or subtraction in the accumulator; thereby reducing the number of line switching and read / write operations of the membrane potential and tracker without introducing cross-spine / token synchronization.

[0041] 5. Router / PE Collaboration and Operator Support like Figure 6 and Figure 9 As shown: Router 310 includes a local input merger 311, a flit generator 312, a routing engine 313, an arbitrator 314, a flit decoder 315, a FIFO queue 316, and an output scheduler 317; Router 310 may include internal router operator units such as im2col unit 318, pulse softmax unit 319, and pulse layernorm unit 320; FIFO queue 316 serves as a pipeline register between adjacent neural nuclei, supporting non-blocking, sequential, fine-grained pipeline propagation.

[0042] 6. Mapping and Route Optimization (Optional) like Figure 12 As shown, a three-stage method of partitioning, mapping, and routing is provided: Partitioning: Under resource capacity constraints, merge adjacent layers / operators with high communication volume to reduce inter-core communication; Mapping: Generate an initial layout using space-filling curves (such as Hilbert curves) and iteratively optimize the communication potential; Routing: Employs multi-path traffic splitting and assigns probabilities to paths to balance link load and reduce congestion.

[0043] Example 1: Overall Device Structure and Data Flow like Figure 1 As shown, the inference device of this embodiment includes multiple neural cores 300 interconnected via NoC 200. Each neural core 300 includes at least a router 310, one or more processing units PE 330, and an output scheduler 317. Input data (e.g., images, sequence features) is first encoded into a multi-timestep pulse event stream, with each timestep corresponding to a set of pulse events; the pulse events are transmitted between neural cores in the form of BAER data units 400.

[0044] To support fine-grained pipelined flow, neural nuclei are connected via a FIFO queue 316 as a pipelined register: Router 310 unpacks BAER data units that arrive at the core and belong to the core's responsible layer / operator into pulse events and writes them into FIFO queue 316; The output scheduler 317 retrieves pulse events from the FIFO queue 316 and triggers the PE 330 or the operator unit within the router to execute; The generated output pulse events are then packaged by router 310 and sent to the neural nucleus where the successor layer is located via NoC 200.

[0045] Example 2: Spiking Neuron Model This embodiment uses a bipolar integral-distributed neuron with a pulse tracker (ST-BIF) as an example to illustrate neuron state updates. The neuron outputs three-valued pulses {-1, 0, 1}, where only non-zero pulses form event entries for transmission. Figure 3 As shown, a neuron involves three steps: integration, firing decision, and updating. (1) Integration A neuron receives several presynaptic impulse events at time step t, the polarity of which can be positive or negative; the corresponding weights are accumulated to the membrane potential.

[0046] in For non-zero pulse polarity, This corresponds to the synaptic weights.

[0047] (2) Fire decision Based on the membrane potential and the tracker state S_t, output a firing pulse. In one embodiment, a dual threshold and tracker upper and lower bound constraints can be used for determination: when The output is +1 when the positive threshold is exceeded and the tracker has not reached the upper bound; when Output -1 if the value is below 0 and the tracker has not reached the lower bound; otherwise, output 0.

[0048] (3) Update The membrane potential and tracker are updated using a soft reset method:

[0049] when The state is maintained or updated as needed.

[0050] The neuron states (membrane potential V and tracker S) are stored in membrane potential buffer 333 and tracker buffer 334, and are read and written by PE 330 according to row address.

[0051] Example 3: BAER Data Format and Packing / Unpacking (corresponding to) Figure 5 ) like Figure 5 As shown, the BAER data unit 400 in this embodiment is used to transmit multiple pulse event entries 406 on the NoC 200. In one embodiment, the BAER data unit 400 uses a fixed-length flit (e.g., 256 bits), and its fields include: Destination field 401: Records the relative number of jumps in the target nucleus (e.g., the number of jumps m in the x-direction and the number of jumps n in the y-direction in a two-dimensional grid). Type field 402: Indicates the position type (start / middle / end) of this flit within the same spine / token sequence; Spine / token identifier field 403: Identifies which spine or token this batch of events belongs to; Row index field 404: Identifies the row address shared by this batch of events (corresponding to the membrane potential / tracker row); Event list field 405: Contains K event entries 406, each event entry contains at least a column position field 407 and a polarity field 408; Validation field 409: Used for error detection / correction.

[0052] Packaging (sender): The local input merger 311 of router 310 collects pulse events from local PE 330 or operator units within the router, aggregating event entries with the same spine / token and the same row index. When K entries are reached or the end of the spine / token phase is detected, the flit generator 312 forms BAER data units 400 and sends them. If there are fewer than K entries and the phase ends, zero padding is performed.

[0053] Forwarding (intermediate router): The routing engine 313 selects the output port based on the hop count m and n in the destination field 401; the hop count in the corresponding direction can be decreased with each forward until the target core is reached.

[0054] Unpacking (receiving end): When the arbitrator 314 detects that both m and n meet the "arrival condition" (e.g., 0), the flit decoder 315 parses the BAER and outputs the event entry, writing it to the FIFO queue 316. The output scheduler 317 retrieves the event from the FIFO queue 316 and drives the subsequent calculation.

[0055] Example 4: Mini-batch pulse Gustavson product of PE like Figure 6As shown, PE 330 includes a control module 331, a weight buffer 332, a membrane potential buffer 333, a tracker buffer 334, an addition tree 335, a firing decision component 336, and an update component 337, and may include a neuron circuit array 338.

[0056] 4.1 Single Event Handling Abstraction (Row / Column Address Meaning) A pulse event entry 406 can be abstracted as : Row index x: Indicates the address of the membrane potential row to be updated; Column position y: Indicates the address of the weight row / weight vector to be read; Polarity q: Indicates whether the pulse is positive or negative.

[0057] For each event, PE 330 performs the following: reads the weight vector corresponding to address y in weight buffer 332 and adds it to the membrane potential row corresponding to address x in membrane potential buffer 333; if q is negative, it negates the weight vector or performs subtraction.

[0058] 4.2 Mini-batch processing (BAER-driven Gustavson product) like Figure 7 As shown, event entries within a BAER data unit 400 share the same row index x, but differ in column position y_i. PE 330 processes in mini-batch: 1) Control module 331 reads the x-th row of membrane potential buffer 333 and the x-th row of tracker buffer 334; 2) Control module 331 will Input into weight buffer 332; read K weight rows / vectors in parallel from weight buffer 332; 3) For each i, if q_i is negative, then perform inversion / two's complement negation on the corresponding weight; 4) Addition tree 335 accumulates the K weight vectors to the membrane potential row to obtain the integral result; 5) The issuance determination component 336 combines the integration result with the tracker status to perform the issuance determination and output a pulse; 6) Update component 337 to perform a write-back update on the membrane potential and tracker (only one write-back for this small batch); 7) If the output pulse is non-zero, the output event will be sent to router 310 for further packaging and forwarding.

[0059] This process ensures that the state of the "same row index" resides within small batches, thereby reducing frequent row switching of the membrane potential / tracker.

[0060] 4.3 Tiling and Broadcasting like Figure 8As shown, to improve area / storage utilization, weights and membrane potentials can be distributed in column direction to multiple PEs 330 within the same neural nucleus; input pulse events are broadcast to all relevant PEs via router 310; each PE only updates the membrane potential portion within its responsible segment and performs reduction / merging via local input merger 311 when necessary.

[0061] Example 5: Router Structure and Five-Channel Router like Figure 9 As shown, router 310 includes: Local Input Merger 311: Collects events from the local PE and aggregates them according to the binding conditions; flit generator 312: Packs event entries into BAER; Router Engine 313: Selects forwarding ports based on hop count and port probability; Arbitrator 314: Arbitrates flits from different directions; flit decoder 315: Unpacks into event entries upon arrival at the target kernel; FIFO queue 316: buffers unpacking events as a pipeline register; Output scheduler 317: Determines when to trigger the subsequent layer at the spine / token granularity; Optional router internal operator units include: im2col unit 318, pulse softmax unit 319, pulse layernorm unit 320, etc.

[0062] Routers can be configured with multiple physical data paths (e.g., two local and three remote) to reduce structural contention between local event packaging and remote flit reception.

[0063] Example 6: Fine-grained pipeline scheduling (CNN and Transformer) 6.1 CNN Convolutional Layers: Spinal Corresponding Pipeline Scheduling (corresponding to) Figure 10 ) like Figure 10 As shown, convolutional layers exhibit local dependencies: a certain output spine in a later layer depends on multiple input spines within a single convolutional window of the previous layer. The output scheduler 317 maintains the dependency arrival status, and when it detects that all the input spines required for a certain output spine have arrived, it immediately triggers the computation of that output spine in the subsequent layer. To trigger it as early as possible, the input spines can be processed in a non-traditional scanning order (e.g., along the diagonal, right to left, or bottom to top combination) to increase the probability of "dependencies being satisfied first".

[0064] In one embodiment, the output scheduler 317 may execute the following control flow: Input: Convolution kernel height Hk, width Wk, stride S, padding P, input feature map size HI×WI, current input spine coordinates (i,j); step: 1) Perform padding correction on (i,j): i←i+P, j←j+P; 2) Determine which output coordinates (u,v) have their dependencies satisfied after reaching (i,j) based on the convolution window coverage relationship; 3) Add the satisfied party to the trigger list L, and trigger the subsequent layer according to the preset priority order; 4) For output coordinates involving the filled area, the trigger is delayed until the last valid input spine arrives.

[0065] 6.2 Transformer: Token-based Pipeline Scheduling like Figure 11 As shown, in the token-level pipeline, operators such as linear layers, MLP, and normalization can proceed independently per token. The output scheduler 317 immediately triggers token k to enter the next layer after token k completes the current layer. However, for operators such as pulse softmax that require a full query / key token set, a synchronization point is set: before entering pulse softmax, the required token set is waited for to be completed, and then the pipeline continues.

[0066] Example 7: Partition-Mapping-Multipath Routing like Figure 12 As shown, this embodiment provides a three-stage mapping method: 1) Partitioning: Using layers as the basic unit, the inter-layer communication volume c_ij and the resource requirements of each layer (storage a_i, number of neuron circuits d_i) are statistically analyzed. Under the constraint of the neural nucleus capacity (A,D), layer pairs are merged in descending order of communication volume to reduce inter-nucleus communication and improve intra-nucleus reuse. 2) Mapping: Map the partitions to the locations of the 2D mesh neural nuclei. The initial placement can be generated using Hilbert space-filling curves, and then the communication potential can be reduced iteratively. 3) Routing: Set multiple candidate paths for hot communication pairs and assign sending probabilities to achieve load balancing and reduce peak bandwidth requirements.

[0067] The detailed descriptions of the five embodiments above fully elucidate the implementation details of the apparatus and method described in this invention. This solution systematically solves the three major problems of latency, communication, and memory access raised in the background art through collaborative innovation in device architecture (connection and coordination of routers, PEs, and schedulers within the neural nucleus), data format (BAER), computation method (mini-batch Gustavson product), and scheduling strategy (fine-grained pipelined architecture). It provides a complete hardware solution for achieving efficient, low-latency elastic inference in spiking neural networks.

Claims

1. A spiking neural network inference device supporting flexible inference, characterized in that, include: On-chip network and multiple neural nuclei interconnected through the on-chip network; Each of the neural nuclei includes: a router, a processing unit, and an output scheduler, wherein: The router has a communication interface connected to the on-chip network for receiving, sending, routing, and processing pulse event data units. The processing unit, connected to the router, is used to perform pulse-driven matrix multiplication calculations and to store and update neuron states. The output scheduler is connected to the router and the processing unit respectively, and is used to control the calculation triggering timing of the processing unit according to the unpacked pulse event; The router is configured to bundle multiple pulse events originating from the same data path to form a bundled address event representation data unit with a shared header, and transmit it to the target neural nucleus through the on-chip network; wherein the same data path is defined by the same fine-grained control unit identifier and the same target neuron state row index; The output scheduler is configured to use fine-grained control units as the smallest scheduling unit. After a single fine-grained control unit completes the calculation of the current layer, it immediately triggers the forwarding of the calculation result corresponding to that fine-grained control unit to the subsequent layers, without needing to synchronize with the execution progress of other fine-grained control units in the same layer. The fine-grained control unit is defined in a convolutional neural network as the spine of all channel data of the feature map at the same spatial location, or in a Transformer-based spiking neural network as a token in a sequence.

2. The spiking neural network inference device supporting flexible inference according to claim 1, characterized in that, The data structure of the bundled address event representation data unit includes: A shared header field containing destination information, type identifier, fine-grained control unit identifier, and neuron state row index; An event list field contains at least one event entry, each event entry containing at least column position information for addressing weights and polarity information indicating the direction of pulse action.

3. The spiking neural network inference device supporting flexible inference according to claim 1, characterized in that, The processing unit is specifically configured as follows: The entirety of the event entries contained in a single received bundled address event representation data unit is processed as a single computation batch. For this computation batch, a single read operation is performed on the neuron state; The weights of each event entry within the calculation batch are accumulated in parallel. Based on the accumulation and pulse firing results, a single write-back update operation is performed on the neuron state.

4. The spiking neural network inference device supporting flexible inference according to claim 3, characterized in that, The processing unit includes: A state memory is used to store neuronal membrane potentials and / or pulse tracker states; A weight memory is used to store synaptic weights associated with neuron state rows in the state memory; control logic is used to parse the bundled address event representation data unit and generate an access control sequence for the state memory and the weight memory, the access control sequence ensuring that access to the same neuron state row satisfies a single read / write constraint. The computing array, connected to the control logic, the state memory, and the weight memory, is used to perform parallel reading of the weights, accumulation operations, and neuron firing determination.

5. The spiking neural network inference device supporting flexible inference according to claim 1, characterized in that, The router further includes: A local input merger is used to receive output pulse events generated by the processing unit and merge them according to the fine-grained control unit identifier and the target neuron state row index to form a set of events to be bundled. The flit generator, connected to the local input merger, is used to encapsulate the event set into the bundled address event representation data unit; A routing engine is used to determine the forwarding path of the bundled address event representation data unit in the on-chip network based on the header information of the bundled address event representation data unit. The flit decoder is used to parse the data units arriving at this neural nucleus and recover the impulse event data; At least one first-in-first-out queue is connected between the flit decoder and the output scheduler as a pipeline register across neural nuclei.

6. The spiking neural network inference device supporting flexible inference according to claim 5, characterized in that, The router also integrates at least one non-matrix multiplication operator processing unit, which is connected between the local input merging unit and the flit generator, and / or between the flit decoder and the first-in-first-out queue, for performing specific operations of the spiking neural network; the specific operations include at least one of the following: convolution im2col transformation, spiking softmax operation, and spiking layer normalization operation.

7. The spiking neural network inference device supporting flexible inference according to claim 1, characterized in that, The output scheduler is further configured to: Maintain the dependency state of each output spine in the output feature map on the set of input spines it depends on; Monitor the arrival events of the input spine; When it is determined that all dependent input spines of a certain output spine have been reached, the calculation for that output spine is immediately triggered, and this triggering action is independent of the dependency satisfaction state of other output spines.

8. The spiking neural network inference device supporting flexible inference according to claim 1, characterized in that, The output scheduler is further configured to: In the process of token-level asynchronous pipelined execution, a synchronization point is set for specific operators that require cross-token synchronization; When the pipeline reaches the synchronization point, the forward push of relevant tokens is paused. After all the necessary tokens arrive at the synchronization point, synchronization is desynchronized and the pipeline execution continues.

9. A spiking neural network inference method supporting flexible inference, applied to the apparatus as described in any one of claims 1-8, characterized in that, The method includes the following steps: Receive input pulse event stream; Pipeline scheduling is implemented between layers of the neural network at the level of fine-grained control units, so that the completion event of the computation of a single fine-grained control unit can independently trigger the computation of its successor layer. During pulse event transmission, multiple pulse events belonging to the same fine-grained control unit and targeting the same neuron state row are bundled and aggregated into a data packet with a shared header for communication. In the processing unit, the event entries within a single data packet are treated as a computation batch. A single read and a single write-back operation are performed on the neuron state rows involved in the batch, and the weights corresponding to each event in the batch are accumulated in parallel.

10. The spiking neural network inference method according to claim 9, characterized in that, The method further includes, in the flow scheduling step: For convolutional layers, the dependency of the output spine on the input spine is dynamically maintained; A mechanism of triggering upon dependency satisfaction is adopted. Once all the dependent input spines of a certain output spine are detected to be ready, the computation and forwarding of that output spine are immediately started.

11. The spiking neural network inference method according to claim 9, characterized in that, The method further includes, in the flow scheduling step: For self-attention operations in Transformer class models, insert synchronization points in the token-level asynchronous pipeline; At the synchronization point, after all relevant tokens involved in the attention calculation have arrived, the pulse softmax calculation is performed and the token pipeline continues.

12. The spiking neural network inference method according to claim 9, characterized in that, The bundled aggregation step further includes: When a fine-grained control unit detects that the event stream has ended at the current time step or calculation phase, if the number of merged event entries is insufficient to form a complete standard data packet, a fixed-length final data packet is generated and sent using a padding method.

Images