A neuromorphic processor based on layer-by-layer impulse event batch processing

Abstract

This invention relates to the fields of artificial intelligence and neuromorphic intelligent chips, specifically to a neuromorphic processor based on layer-by-layer spiking event batch processing, comprising a microprocessor core array, a spiking AER memory, a spiking AER cyclic priority arbiter, an output error AER cyclic priority arbiter, and a global controller. In the forward processing stage, a relative time concept is introduced into the spiking neuron update calculation. During the backward processing stage, when training the spiking neural network of the layer-by-layer spiking event batch processing neuromorphic processor, the EO-DeepTempo learning rule is used to update the weights of each synapse in the fully connected layers of the neural network. This invention improves computational efficiency and reduces computational and storage resources.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and neuromorphic intelligent chip technology, specifically to a neuromorphic processor based on layer-by-layer pulse event batch processing. Background Technology

[0002] In recent years, neuromorphic processors have garnered significant attention due to their unique advantages in running brain-inspired spiking neural network (SNN) models, particularly in processing spatiotemporally sparse binary impulses to address a wide range of edge intelligence scenarios. While many neuromorphic processors employ application-specific integrated circuit (ASIC) designs to improve energy efficiency, field-programmable gate arrays (FPGAs) offer new opportunities for rapid hardware prototyping and extensive exploration of the design space, thanks to their superior flexibility.

[0003] In building edge intelligent systems, adaptive capabilities are crucial for adapting to dynamic and ever-changing open environments. However, current FPGA-based neuromorphic processors generally face a key challenge: how to achieve high-precision on-chip SNN learning capabilities under limited resources. These processors often need to strike a difficult balance between resource constraints, energy efficiency requirements, and response speed. On the one hand, simplifying learning rules to suit shallow SNN designs can reduce resource consumption, but the recognition performance on benchmarks such as MNIST is unsatisfactory. On the other hand, attempting to train deep fully connected SNNs, while achieving high recognition rates through complex neuron models and learning algorithms, significantly increases FPGA resource consumption and prolongs learning latency.

[0004] Therefore, it is necessary to develop a new neuromorphic processor based on layer-by-layer pulse event batch processing. Summary of the Invention

[0005] The purpose of this invention is to provide a neuromorphic processor based on layer-by-layer pulse event batch processing to improve computational efficiency and reduce computational and storage resources.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The neuromorphic processor based on layer-by-layer pulse event batch processing described in this invention includes:

[0008] A microprocessor core array is used to process pulsed AER and update the synaptic weights of the fully connected layers of the neural network using the EO-DeepTempo learning rule;

[0009] The pulse AER memory is used to store the sample input pulse and the pulse AER generated by the microprocessor core.

[0010] The pulse AER cyclic priority arbiter is used to arbitrate the pulse AER generated by each microprocessor core.

[0011] The Output Error AER (AER) cyclic priority arbiter is used to arbitrate the AER errors generated by each microprocessor core.

[0012] Error AER memory is used to store the error AER generated by the microprocessor core.

[0013] The global controller is used for overall processor control, data scheduling, and data initialization.

[0014] The concept of relative time is introduced into the calculation of spiking neuron update during the forward processing stage;

[0015] During the reverse phase, when training the spiking neural network of the layer-by-layer pulse event batch processing neuromorphic processor, the EO-DeepTempo learning rule is used to update the weights of each synapse in the fully connected layer of the neural network to ensure a high recognition rate.

[0016] Optionally, the EO-DeepTempo learning rule includes error backpropagation and weight gradient update, wherein:

[0017] The backpropagation of the error is as follows: the output layer error is calculated and packaged into an error AER. The hidden layer error is calculated by the error AER event and the output layer error is directly projected to each hidden layer through a fixed random matrix using a direct feedback alignment mechanism.

[0018] The weight gradient update is driven by a presynaptic impulse AER event. When a neuron receives a presynaptic impulse AER, it triggers an update of the AER address and the weights of the synapse to which the neuron is connected.

[0019] Optionally, the spiking neural network consists of L fully connected layers, and the neurons in the spiking neural network are IF neurons. In the forward phase, the concept of relative time is introduced into the state update calculation of the IF neurons.

[0020] Optionally, a unified neuron-microprocessor kernel mapping mechanism is adopted, specifically as follows:

[0021] In this fully connected layer, neurons are uniformly mapped to all microprocessor kernels. The input impulse AER (Average Erector Error) or the output error AER activates all neurons in the fully connected layer, ensuring a balanced workload across all microprocessor kernels with no idle cycles. This unified neuron-microprocessor kernel mapping mechanism maximizes computational resource utilization and reduces learning and inference latency. This neuromorphic processor, based on layer-by-layer impulse event batch processing, provides robust technical support for edge intelligence applications.

[0022] Optionally, the microprocessor cores in the microprocessor core array include a neural state update unit, an output error generation unit, an error propagation unit, a weight update unit, and a local data storage unit, wherein the local data storage unit includes multiple memories, all of which operate in an event-driven manner.

[0023] The neural state update unit, error propagation unit, and weight update unit are each connected to the local data storage unit. The output error generation unit and error propagation unit are each connected to the error AER memory. The neural state update unit is connected to the pulse AER memory. The neural state update unit is used to update the neuronal membrane potential state information. The output error generation unit is used to output the neuronal error AER. The error propagation unit is used to propagate the output error to the hidden layer. The weight update unit is used to update the synaptic weights of the layer neurons. The local data storage unit is used to cache neuronal state information and synaptic weights.

[0024] Optionally, the neural state update unit is equipped with a first-in-first-out 64b deep ping-pong buffer to reduce arbitration latency when multiple microprocessor cores simultaneously request an emitter pulse (AER).

[0025] The beneficial effects of this invention are:

[0026] 1. The EO-DeepTempo on-chip learning algorithm proposed in this invention adopts a multi-layer fully connected network structure. The IF neuron model and the introduction of the relative time concept effectively improve computational efficiency and reduce computational and storage resources for subsequent hardware implementation. Furthermore, AER event-driven methods are used for neuron state updates in the forward phase and error propagation and synaptic weight gradient updates in the backward phase, significantly improving processing efficiency.

[0027] 2. The neuron-microprocessor kernel mapping mechanism proposed in this invention enables all microprocessor kernels to have a balanced workload, which can maximize the utilization of computing resources and reduce learning and inference latency.

[0028] 3. The neuromorphic processor architecture proposed in this invention can support on-chip learning. Its low power consumption, high frame rate and high energy efficiency make it suitable for edge intelligent systems with resource constraints, energy efficiency requirements and response speed. Attached Figure Description

[0029] Figure 1 This is a block diagram illustrating the principle of the neuromorphic processor based on layer-by-layer pulse event batch processing described in the embodiments of this application.

[0030] Figure 2 This is a schematic diagram illustrating the relative time concept of different network layers in the embodiments of this application;

[0031] Figure 3 This is a circuit schematic diagram of the microprocessor core in the embodiments of this application. Detailed Implementation

[0032] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0033] In this embodiment, a neuromorphic processor based on layer-by-layer pulse event batch processing includes a microprocessor core array, a pulse AER memory, a pulse AER cyclic priority arbiter, an output error AER cyclic priority arbiter, an error AER memory, and a global controller. Specifically: the microprocessor core array processes pulse AERs and updates the synaptic weights of the fully connected layers of the neural network using the EO-DeepTempo learning rule. The pulse AER memory stores sample input pulses and pulse AERs generated by the microprocessor cores. The pulse AER cyclic priority arbiter arbitrates the output pulse AERs generated by each microprocessor core. The output error AER cyclic priority arbiter arbitrates the output error AERs generated by each microprocessor core. The error AER memory stores the error AERs generated by the microprocessor cores. The global controller is used for overall processor control, data scheduling, and data initialization. A relative time concept is introduced into the pulse neuron update calculation during the forward processing stage. During the backward processing stage, when training the spiking neural network of the layer-by-layer pulse event batch processing neuromorphic processor, the EO-DeepTempo learning rule is used to update the weights of each synapse in the fully connected layers of the neural network. Simultaneously, a unified neuron-microprocessor kernel mapping mechanism is employed. Specifically, neurons in the fully connected layers are uniformly mapped to all microprocessor kernels, and the input impulse AER or output error AER activates all neurons in the fully connected layers. The workload on all microprocessor kernels remains balanced, and there are no idle cycles between microprocessor kernels. This unified neuron-microprocessor kernel mapping mechanism ensures a balanced workload for all microprocessor kernels, maximizing computational resource utilization and reducing learning and inference latency. This neuromorphic processor based on layer-by-layer impulse event batch processing provides robust technical support for edge intelligence applications.

[0034] In this embodiment, a neuromorphic processor based on layer-by-layer spiking event batch processing incorporates the concept of relative time into the spiking neuron update calculation during the forward processing stage, further reducing hardware resource overhead during hardware implementation. Furthermore, during the reverse training stage of the spiking neural network of the layer-by-layer spiking event batch processing neuromorphic processor, the EO-DeepTempo learning rule is used to update the weights of each synapse in the fully connected layers of the neural network, ensuring a high recognition rate. Simultaneously, the neuromorphic processor employs a unified neuron-microprocessor kernel mapping mechanism, ensuring a balanced workload for all microprocessor kernels, maximizing computational resource utilization, and reducing learning and inference latency. This neuromorphic processor based on layer-by-layer spiking event batch processing can provide strong technical support for edge intelligence applications.

[0035] In this embodiment of the application, the EO-DeepTempo learning rule includes error backpropagation and weight gradient update, wherein:

[0036] Error backpropagation is as follows: the output layer error is calculated and packaged into an error AER. The hidden layer error is calculated by the error AER event. The output layer error is directly projected to each hidden layer through a fixed random matrix (i.e., the matrix is ​​randomly generated and fixed after generation).

[0037] The weight gradient update is driven by the presynaptic impulse AER event. When a neuron receives a presynaptic impulse AER, it triggers an update of the AER address and the weights of the synapse to which the neuron is connected.

[0038] In this embodiment, the spiking neural network consists of L fully connected layers, and the neurons in the spiking neural network are IF neurons. In the forward phase, the concept of relative time is introduced into the state update calculation of the IF neurons.

[0039] The following is a detailed description of the neuromorphic processor based on layer-by-layer pulse event batch processing in the embodiments of this application:

[0040] Step 1: Propose the EO-DeepTempo learning algorithm for training a multi-layer fully connected network model.

[0041] Step 1.1: Using ordered encoding, the pulse timing sequences converted from the input image pixels are arranged in descending order according to pixel intensity.

[0042] Step 1.2: The neurons in the spiking neural network are IF (leakage-free integral-ignition) neurons. Specifically, the IF neuron is updated as follows: In the feedforward phase driven by the spiking AER event, neuron j in layer l will only update its membrane potential after receiving the spiking AER event from neuron i in layer (l-1).

[0043]

[0044] in, It is the synaptic weight connecting two neurons, t n This is the time step of the nth presynaptic pulse. Once the membrane potential of the IF neuron... Once the emission threshold θ is reached, a postsynaptic pulse will be emitted, and It remains unchanged until it is zeroed at the start of the next input sample. Additionally, in equation (1), the precise pulse time t is not required. n Since only the pulse sequence n is needed, n can be used instead of t. n We can obtain:

[0045]

[0046] Therefore, the concept of relative time can be introduced. For example, Figure 2 as shown, the relative time axes of each layer are orthogonal to each other. Suppose the 5th presynaptic spike from the first-layer network triggers the 3rd postsynaptic spike of the second-layer network and inputs this spike to the third layer. Although the absolute time steps of these two spikes are the same, their relative times are different. Therefore, the time information of the spikes is contained in the spike order, and there is no need to record the time information t additionally. n Thus, the computational efficiency is improved. Compared with the ordinary precise global timing scheme, this local relative spike timing scheme can calibrate the time of different layers.

[0047] Step 1.3, in the backward phase, propagate the output error in an error AER event-driven manner. Each neuron in the output layer L is pre-assigned to a specific category. When the category of the output neuron k matches the training sample but the neuron does not emit a spike, the output error is a positive event, and when its category does not match but the neuron emits a spike, the output error is a negative event. Then, the output error AER event drives the calculation of the hidden layer neuron error. Specifically, the corresponding binary values in the fixed random matrix B are backpropagated through DFA (Direct Feedback Alignment) and accumulated to each neuron j in all hidden layers l < L: where

[0048]

[0049] is the neuron error. After processing all output error AER events, the update of the synaptic weights of all layers is driven by the spike AER events in the forward phase: where λ represents the learning rate,

[0050]

[0051] is the relative time of the presynaptic spike from the upstream neuron i in the l-th layer, represents the time when reaches the peak. <00001�6>

[0052] Step 2: Propose a neuromorphic hardware architecture for layer-by-layer spike event batch processing of the neuron-microprocessor core mapping mechanism.

[0053] Step 2.1: As Figure 1As shown, the proposed neuromorphic processor based on layer-by-layer pulse event batch processing employs a conventional array parallel multi-core architecture, mainly consisting of a microprocessor core array (e.g., 16 microprocessor cores), a pulse AER memory, an error AER memory, and a global controller coordinating the various modules. Each microprocessor core can process up to m = 64 IF neurons and a total q = 2 14 Each neuron has a maximum of 1024 synapses. The pulse events processed by the microprocessor are encapsulated in AER format, carrying the neuron index information (i.e., the neuron address that emitted the pulse) of the pulse. The relative time information of the pulse AER is implicitly encoded in the storage order of the pulse AER memory, requiring no additional resource overhead to record the time information. Additionally, sparse non-zero output errors are also handled in AER format. The event is encoded into an error AER event. The error AER has no event information but has an extra bit width to represent its polarity ±1. The neuromorphic processor based on layer-by-layer spiking event batching proposed in this application embodies the neuron-microprocessor kernel mapping relationship. Neurons in each SNN layer (spiking neural network) are uniformly distributed across all microprocessor kernels. This cross-layer mapping scheme implements a hierarchical processing mechanism based on event batching, with a consistently balanced workload, thereby maximizing computational resource utilization and reducing learning and inference latency.

[0054] Step 2.2, the layer-by-layer pulse event batch processing based on the neuron-microprocessor kernel mapping mechanism is specifically implemented as follows: In the forward pathway, the microprocessor kernel starts from the first layer (l=1), where the presynaptic pulse of the l=1 layer is the ordered encoded input pulse. For example... Figure 1 As shown, all presynaptic impulses (AERs) in layer l are stored in a contiguous region of the impulse AER memory and arranged in firing order, considered as a batch of impulse AER events. The global controller retrieves a batch of impulse AER events each time and broadcasts them to the microprocessor core array. Upon receiving an impulse AER, all microprocessor cores update the membrane potential of the layer l neurons mapped by the impulse AER event in a fully parallel manner and write the emitted impulse AER back to the impulse AER memory. When multiple microprocessor cores simultaneously request to send impulses, the impulse AER cyclic priority arbiter arbitrates and outputs these impulses. Figure 1As shown, the postsynaptic pulses of layer l (i.e., the presynaptic pulses of layer (l+1)) are also stored sequentially from the memory address next to the last presynaptic pulse AER of the current layer, following the firing order of the pulse AER events. After completing the entire batch of pulse AER events, the microprocessor core will process the neurons of the next layer (l+1). This process terminates after the output layer is completed. For the reverse phase, all microprocessor cores simultaneously scan the output layer neurons they map, calculate the output error AER, and send it to the error AER memory. Similarly, these error AERs are also considered as error event batches, and the same error event batches are repeatedly acquired to complete error propagation according to the layer-by-layer event batch drive. Finally, also driven by pulse AER events, when a neuron receives a presynaptic pulse AER, it triggers an update of the pulse AER address and the weight of the synapse connected to that neuron. Since the neurons in the fully connected layer are uniformly mapped to all microprocessor cores, and the input pulse AER or the output error AER will also activate all neurons in the fully connected layer, the workload on all microprocessor cores is always balanced, and there are no idle cycles between microprocessor cores. Therefore, all microprocessor cores can always operate at peak performance, maximizing computational resource utilization and minimizing learning and inference latency.

[0055] Step 2.3, the circuit design of the microprocessor core is as follows: Figure 3 As shown, it includes a weight update unit, an error propagation unit, an output error generation unit, a neural state update unit, and a local data storage unit. The local data storage unit includes multiple memories (including...). memory, memory, memory, memory, memory and All memory units operate in an event-driven manner. The neural state update unit updates neuronal state information such as membrane potential; the output error generation unit outputs the neuronal error (AER); the error propagation unit propagates the output error to the hidden layers; and the weight update unit updates the synaptic weights of neurons in the layers. When multiple microprocessor cores simultaneously request to provide emission pulses (AERs), a 64-bit wide, deep ping-pong buffer based on First-In-First-Out (FIFO) is added to the neural state update unit to reduce arbitration latency. When one channel of the buffer buffers the pulse AER caused by the current pulse, the pulse AER caused by the previous pulse stored in the other channel is read and written to the pulse AER memory. The local data storage unit caches neuronal state information and synaptic weights. Variables in each memory within the local data storage unit are arranged in order of occupying contiguous memory space. memory, memory, memory and In memory, the data address of the j-th mapped neuron in a layer is the offset of that layer (i.e., the starting address) plus j. Memory, weight Located at layer offset plus j×I l At the address of +i, where I l It is the synapse count of each neuron in layer l.

[0056] This application also proposes an incremental access method that eliminates the need for expensive hardware multipliers. When the presynaptic pulse AER containing neuron index i triggers the weight update of the neurons in the l-th layer, starting from the first neuron j = 0, where... The address is located at layer offset plus i. Once neuron j in that layer is accessed... The address pointer only needs to be incremented by I. l Then you can access the next j+1 neuron. A similar incremental access pattern applies when propagating an output error (AER). Memory. The address offset for each level is configured by registers in the local controller.

[0057] In this embodiment, forward processing refers to the process by which the neural network, when processing input data, passes information layer by layer forward from the input layer until the output layer produces the output result. Backward processing, on the other hand, refers to the process during neural network training where, based on the error signal from the output layer, error information is passed backward layer by layer, and the weights between neurons in each layer are updated.

[0058] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A neuromorphic processor based on layer-by-layer pulse event batch processing, characterized in that, include: A microprocessor core array is used to process pulsed AER and update the synaptic weights of the fully connected layers of the neural network using the EO-DeepTempo learning rule; The pulse AER memory is used to store the sample input pulse and the pulse AER generated by the microprocessor core. The pulse AER cyclic priority arbiter is used to arbitrate the pulse AER generated by each microprocessor core. The Output Error AER (AER) cyclic priority arbiter is used to arbitrate the AER errors generated by each microprocessor core. Error AER memory is used to store the error AER generated by the microprocessor core. The global controller is used for overall processor control, data scheduling, and data initialization. The concept of relative time is introduced into the calculation of spiking neuron update during the forward processing stage; During the reverse phase of training the spiking neural network of the layer-by-layer spiking event batch processing neuromorphic processor, the EO-DeepTempo learning rule is used to update the weights of each synapse in the fully connected layer of the neural network, specifically: Order coding is used to sort the pulse timing sequences converted from the input image pixels in descending order according to pixel intensity; The neurons in a spiking neural network are IF neurons, and the specific updates of an IF neuron are as follows: In the forward phase driven by the pulsed AER event, the first l neurons in a layered network j Only when the first l - Neurons in layer 1 i The membrane potential is updated only after a pulsed AER event. U l j : ； in, w l ji It is the synaptic weight connecting two neurons. n It is the order of the pulses; The EO-DeepTempo learning rule includes error backpropagation and weight gradient update; The backpropagation of the error is as follows: the output layer error is calculated and packaged into an error AER. The hidden layer error is calculated by the error AER event and the output layer error is directly projected to each hidden layer through a fixed random matrix using a direct feedback alignment mechanism. The weight gradient update is driven by a presynaptic impulse AER event. When a neuron receives a presynaptic impulse AER, it triggers an update of the AER address and the weights of the synapse to which the neuron is connected. The specific implementation based on layer-by-layer pulse event batch processing is as follows: In the forward path, the first l All presynaptic impulses (AERs) of a layer are stored in a contiguous region of the impulse AER memory and arranged in the order of emission, considered as a batch of impulse AER events. The global controller retrieves a batch of impulse AER events each time and broadcasts the impulse AER events to the microprocessor core array. When an impulse AER is received, all microprocessor cores update the impulse AER event map in a fully parallel manner. l The microprocessor nucleus records the membrane potential of neurons in the layer and writes the emitted pulses (AER) back to the pulse AER memory; after completing the entire batch of pulse AER events, the microprocessor nucleus will process neurons in the next layer; For the reverse phase, the error AER is also treated as an error event batch, and the same error event batch is repeatedly acquired to complete the error propagation according to the layer-by-layer event batch drive.

2. The neuromorphic processor based on layer-by-layer pulse event batch processing according to claim 1, characterized in that: A unified neuron-microprocessor kernel mapping mechanism is adopted, specifically as follows: The neurons in the fully connected layer are uniformly mapped onto all microprocessor kernels, and the input impulse AER or the output error AER activates all neurons in the fully connected layer. The workload on all microprocessor kernels is always balanced, and there are no idle cycles between microprocessor kernels.

3. The neuromorphic processor based on layer-by-layer pulse event batch processing according to claim 2, characterized in that: The microprocessor core array includes a neural state update unit, an output error generation unit, an error propagation unit, a weight update unit, and a local data storage unit. The local data storage unit includes multiple memories, all of which operate in an event-driven manner. The neural state update unit, error propagation unit, and weight update unit are each connected to the local data storage unit. The output error generation unit and error propagation unit are each connected to the error AER memory. The neural state update unit is connected to the pulse AER memory. The neural state update unit is used to update the neuronal membrane potential state information. The output error generation unit is used to output the neuronal error AER. The error propagation unit is used to propagate the output error to the hidden layer. The weight update unit is used to update the synaptic weights of the layer neurons. The local data storage unit is used to cache neuronal state information and synaptic weights.

4. The neuromorphic processor based on layer-by-layer pulse event batch processing according to claim 3, characterized in that: The neural state update unit is equipped with a 64-bit wide-depth ping-pong buffer based on first-in-first-out (FIFO) to reduce arbitration latency when multiple microprocessor cores simultaneously request an emit pulse (AER).

Images