Neuromorphic processor based on layer-by-layer pulse event batch processing

Abstract

The invention relates to the field of artificial intelligence and the field of brain-like intelligent chips, in particular to a neuromorphic processor based on layer-by-layer pulse event batch processing, which comprises a micro-processing core array, a pulse AER memory, a pulse AER cycle priority arbiter, an output error AER cycle priority arbiter and a global controller, a relative time concept is introduced into spiking neuron updating calculation in a forward processing stage; and when the pulse neural network of the layer-by-layer pulse event batch processing neuromorphic processor is trained in a reverse phase, updating the weight of each synapse of a full connection layer of the neural network by adopting an EO-DeepTempo learning rule. According to the method, the calculation efficiency is improved, and calculation resources and storage resources are reduced.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and brain-like intelligent chip technology, and in particular to a neuromorphic processor based on layer-by-layer pulse event batch processing. Background Art

[0002] In recent years, neuromorphic processors have attracted extensive attention due to their unique advantages in running brain-inspired spiking neural network (SNN) models, especially in processing spatiotemporally sparse binary pulses to cope with various edge intelligence scenarios. Although many neuromorphic processors are designed with application-specific integrated circuits (ASICs) to improve energy efficiency, field-programmable gate arrays (FPGAs) provide new opportunities for rapid development of hardware prototypes and extensive exploration of the design space with their excellent flexibility.

[0003] In the process of building edge intelligent systems, adaptive capabilities are essential to adapt to dynamic and changing open environments. However, current FPGA-based neuromorphic processors generally face a key challenge: how to achieve high-precision on-chip SNN learning capabilities under limited resource conditions. These processors often need to make a difficult balance between resource constraints, energy efficiency requirements, and response speed: on the one hand, simplifying the learning rules to apply to the design of shallow SNNs can reduce resource consumption, but the recognition effect on benchmarks such as MNIST is not satisfactory; on the other hand, attempts to train deep fully connected SNNs, although high recognition rates are achieved by adopting complex neuron models and learning algorithms, significantly increase FPGA resource consumption and extend learning delays.

[0004] Therefore, it is necessary to develop a new neuromorphic processor based on layer-by-layer spike event batch processing. Summary of the invention

[0005] The purpose of the present invention is to provide a neuromorphic processor based on layer-by-layer spike event batch processing to improve computing efficiency and reduce computing resources and storage resources.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A neuromorphic processor based on layer-by-layer spike event batch processing according to the present invention comprises:

[0008] An array of microprocessor cores for processing spike AERs and updating the synaptic weights of the fully connected layers of the neural network using the EO-DeepTempo learning rule;

[0009] A pulse AER memory, used for storing sample input pulses and pulse AERs generated by microprocessor core processing;

[0010] A pulse AER cyclic priority arbiter is used to arbitrate and output the pulse AER generated by each microprocessor core;

[0011] Output error AER circular priority arbiter, used for arbitrating and outputting error AER generated by each microprocessor core processing;

[0012] An error AER memory is used to store the error AER generated by the microprocessor core processing;

[0013] Global controller, used for overall processor control, data scheduling, and data initialization;

[0014] Introducing the concept of relative time into the calculation of spike neuron update in the forward processing stage;

[0015] When training the spiking neural network of the layer-by-layer spike event batch processing neuromorphic processor in the reverse phase, the EO-DeepTempo learning rule is used to update the weights of each synapse in the fully connected layer of the neural network to ensure a high recognition rate.

[0016] Optionally, the EO-DeepTempo learning rule includes error back propagation and weight gradient update, where:

[0017] The error back propagation is as follows: the output layer error is calculated and packaged into an error AER, the hidden layer error calculation is driven by the error AER event, and the output layer error is directly projected to each hidden layer through a fixed random matrix using a direct feedback alignment mechanism;

[0018] The weight gradient is updated by: driven by a pulse AER event, when a neuron receives a presynaptic pulse AER, it triggers the update of the pulse AER address and the weight of the synapse connected to the neuron.

[0019] Optionally, the spiking neural network is composed of L fully connected layers, and the neurons in the spiking neural network are IF neurons. In the forward phase, the concept of relative time is introduced into the state update calculation of the IF neurons.

[0020] Optionally, a unified neuron-microprocessor core mapping mechanism is adopted, specifically:

[0021] The neurons in the fully connected layer are evenly mapped to all microprocessor cores, and the input pulse AER or output error AER activates all neurons in the fully connected layer. The workload on all microprocessor cores is always balanced, and there are no idle cycles between microprocessor cores. The unified neuron-microprocessor mapping mechanism is adopted, so that all microprocessor cores have a balanced workload, which can maximize the utilization of computing resources and reduce learning and reasoning delays. This neuromorphic processor based on layer-by-layer pulse event batch processing can provide strong technical support for edge intelligence applications.

[0022] Optionally, the microprocessor cores in the microprocessor core array include a neural state updating unit, an output error generating unit, an error propagation unit, a weight updating unit, and a local data storage unit, wherein the local data storage unit includes a plurality of memories, all of which work in an event-driven manner;

[0023] The neural state update unit, error propagation unit and weight update unit are respectively connected to the local data storage unit, the output error generation unit and the error propagation unit are respectively connected to the error AER memory, and the neural state update unit is connected to the pulse AER memory; wherein, the neural state update unit is used to update the state information of the neuron membrane potential; the output error generation unit is used to output the neuron error AER; the error propagation unit is used to propagate the output error to the hidden layer; the weight update unit is used to update the synaptic weights of the layer neurons; the local data storage unit is used to cache the neuron state information and synaptic weights.

[0024] Optionally, the neural state update unit is provided with a 64b deep ping-pong buffer based on first-in-first-out, so as to reduce arbitration delay when multiple microprocessor cores simultaneously request to provide an emission pulse AER.

[0025] Beneficial effects of the present invention:

[0026] 1. The EO-DeepTempo on-chip learning algorithm proposed in the present invention adopts a multi-layer fully connected network structure. The IF neuron model adopted and the relative time concept introduced effectively improve the computing efficiency and effectively reduce the computing resources and storage resources for subsequent hardware implementation. In addition, the neuron state update in the forward phase and the error propagation and synaptic weight gradient update in the reverse phase are all driven by AER events, which significantly improves the processing efficiency.

[0027] 2. The neuron-microprocessor core mapping mechanism proposed in the present invention enables all microprocessor cores to have a balanced workload, which can maximize the utilization of computing resources and reduce learning and reasoning delays.

[0028] 3. The neuromorphic processor architecture proposed in the present invention can support on-chip learning. Its low power consumption, high frame rate and high energy efficiency make it suitable for edge intelligent systems with resource constraints, energy efficiency requirements and response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a principle block diagram of a neuromorphic processor based on layer-by-layer spike event batch processing described in an embodiment of the present application;

[0030] Figure 2 This is a conceptual diagram of relative time of different network layers in the embodiment of the present application;

[0031] Figure 3 4 is a circuit diagram of the microprocessor core in the embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0033] In an embodiment of the present application, a neuromorphic processor based on layer-by-layer pulse event batch processing includes a microprocessor core array, a pulse AER memory, a pulse AER cyclic priority arbitrator, an output error AER cyclic priority arbitrator, an error AER memory, and a global controller. Among them: the microprocessor core array is used to process the pulse AER, and the EO-DeepTempo learning rule is used to update the synaptic weights of the fully connected layer of the neural network. The pulse AER memory is used to store sample input pulses and pulse AERs generated by microprocessor core processing. The pulse AER cyclic priority arbitrator is used to arbitrate and output the pulse AERs generated by each microprocessor core processing. The output error AER cyclic priority arbitrator is used to arbitrate and output the error AERs generated by each microprocessor core processing. The error AER memory is used to store the error AERs generated by microprocessor core processing. The global controller is used for overall processor control, data scheduling, and data initialization. The relative time concept is introduced in the pulse neuron update calculation in the forward processing stage. When the pulse neural network of the layer-by-layer pulse event batch processing neuromorphic processor is trained in the reverse stage, the EO-DeepTempo learning rule is used to update the weights of each synapse in the fully connected layer of the neural network. At the same time, a unified neuron-microprocessor mapping mechanism is adopted, specifically: the neurons in the fully connected layer are evenly mapped to all microprocessor cores, and the input pulse AER or output error AER activates all neurons in the fully connected layer. The workload on all microprocessor cores is always balanced, and there are no idle cycles between microprocessor cores. The unified neuron-microprocessor mapping mechanism is adopted, so that all microprocessor cores have a balanced workload, which can maximize the utilization of computing resources and reduce learning and reasoning delays. This neuromorphic processor based on layer-by-layer pulse event batch processing can provide strong technical support for edge intelligence applications.

[0034] In an embodiment of the present application, a neuromorphic processor based on layer-by-layer batch processing of pulse events, on the one hand, incorporates the concept of relative time into the pulse neuron update calculation in the forward processing stage, which can further reduce the hardware resource overhead when implemented in hardware. On the other hand, when training the pulse neural network of the layer-by-layer batch processing of pulse events neuromorphic processor in the reverse stage, the EO-DeepTempo learning rule is used to update the weights of each synapse in the fully connected layer of the neural network to ensure a high recognition rate. At the same time, the neuromorphic processor adopts a unified neuron-microprocessor core mapping mechanism, so that all microprocessor cores have a balanced workload, which can maximize the utilization of computing resources and reduce learning and reasoning delays. The neuromorphic processor based on layer-by-layer batch processing of pulse events can provide strong technical support for edge intelligence applications.

[0035] In the embodiment of the present application, the EO-DeepTempo learning rule includes error back propagation and weight gradient update, wherein:

[0036] Error back propagation is as follows: the output layer error is calculated and packaged into the error AER. The hidden layer error calculation is driven by the error AER event, and the output layer error is directly projected to each hidden layer through a fixed random matrix (that is, the matrix is ​​randomly generated, fixed after generation, and no longer changes) using a direct feedback alignment mechanism.

[0037] The weight gradient is updated as follows: driven by the pulse AER event, when the neuron receives the presynaptic pulse AER, it triggers the update of the pulse AER address and the weight of the synapse connected to the neuron.

[0038] In the embodiment of the present application, the pulse neural network is composed of L fully connected layers, and the neurons in the pulse neural network are IF neurons. In the forward stage, the concept of relative time is introduced into the state update calculation of the IF neurons.

[0039] The following is a detailed description of the neuromorphic processor based on layer-by-layer spike event batch processing in an embodiment of the present application:

[0040] Step 1: Propose EO-DeepTempo learning algorithm training for a multi-layer fully connected network model.

[0041] Step 1.1, using order coding, the pulse timing converted from the input image pixels is arranged in descending order according to the pixel intensity.

[0042] Step 1.2: The neurons in the spiking neural network are IF (i.e. leakless integrate-and-fire) neurons. The IF neurons are updated as follows: In the forward phase driven by the spiking AER event, the neuron j in the lth layer of the network will update its membrane potential only after receiving the spiking AER event of the neuron i in the l-1th layer.

[0043]

[0044] in, is the synaptic weight connecting two neurons, t n is the time step of the nth presynaptic spike. Once the membrane potential of the IF neuron When the firing threshold θ is reached, a postsynaptic spike is fired, and It remains unchanged until the next input sample starts and is reset to zero. In addition, in formula (1), the precise pulse time t is not required. n , and only the order n of the pulses is needed, so n can be used instead of t n , we can get:

[0045]

[0046] Therefore, the concept of relative time can be introduced. For example, Figure 2 As shown, the relative time axes of each layer are orthogonal to each other. Suppose the 5th presynaptic spike from the first-layer network triggers the 3rd postsynaptic spike of the second-layer network and inputs this spike to the third layer. Although the absolute time steps of these two spikes are the same, their relative times are different. Therefore, the time information of the spikes is contained in the spike order, and there is no need to record the time information t additionally. n Thus, the computational efficiency is improved. Compared with the ordinary precise global timing scheme, this local relative spike timing scheme can calibrate the times of different layers.

[0047] Step 1.3, in the backward phase, propagate the output error in an error AER event-driven manner. Each neuron in the output layer L is pre-assigned to a specific category. When the category of the output neuron k matches the training sample but the neuron does not emit a spike, the output error is a positive event, while when it does not match the category but the neuron emits a spike, the output error is a negative event. Then, the output error AER event drives the calculation of the hidden layer neuron errors. Specifically, the corresponding binary values in the fixed random matrix B are backpropagated through DFA (Direct Feedback Alignment) and accumulated to each neuron j in all hidden layers l < L: where

[0048]

[0049] is the neuron error. After processing all output error AER events, the update of the synaptic weights of all layers is driven by the spike AER events in the forward phase: where λ represents the learning rate,

[0050]

[0051] is the relative time of the presynaptic spike from the upstream neuron i in the l-th layer, represents reaching the peak time.

[0052] Step Two: Propose a neuromorphic hardware architecture for layer-by-layer spike event batch processing of the neuron-microprocessor core mapping mechanism.

[0053] Step 2.1: As Figure 1 ​As shown in FIG. 1 , the proposed neuromorphic processor based on layer-by-layer spike event batch processing adopts a common array parallel multi-core architecture, which is mainly composed of a microprocessor core array consisting of multiple microprocessor cores (for example, 16), a spike AER memory, an error AER memory, and a global controller that coordinates various modules. Each microprocessor core can process up to m = 64 IF neurons and a total of q = 2 14 synapses per neuron, and the maximum number of synapses per neuron is limited to 1024. The pulse events processed by the microprocessor are encapsulated in the AER format, carrying the neuron index information of the emitted pulse (i.e., the neuron address of the emitted pulse). The relative time information of the pulse AER is implicitly encoded in the storage order of the pulse AER memory, and no additional resource overhead is required to record the time information. In addition, the sparse non-zero output error is also recorded in the AER format. Encoded into an error AER event. The error AER has no event information, but has an extra bit width to indicate its polarity ±1. The neuromorphic processor based on layer-by-layer pulse event batch processing proposed in the embodiment of the present application embodies the mapping relationship between neurons and microprocessing cores. The neurons of each SNN layer (pulse neural network) are evenly distributed on all microprocessing cores. This cross-layer mapping scheme implements a hierarchical processing mechanism based on event batch processing, with a always balanced workload, thereby maximizing the utilization of computing resources and reducing learning and reasoning delays.

[0054] Step 2.2, layer-by-layer pulse event batch processing based on the neuron-microprocessor mapping mechanism is specifically implemented as follows: In the forward path, the microprocessor starts from the first layer (l=1), where the presynaptic pulses of the l=1 layer are order-coded input pulses. Figure 1 As shown, all presynaptic pulse AERs in the lth layer are stored in a continuous area of ​​the pulse AER memory and arranged in the order of emission, which is regarded as a batch of pulse AER events. The global controller takes a batch of pulse AER events each time and broadcasts the pulse AER events to the microprocessor core array. When a pulse AER is received, all microprocessor cores update the membrane potential of the lth layer of neurons mapped by the pulse AER event in a completely parallel manner, and write the emitted pulse AER back to the pulse AER memory. When multiple microprocessor cores request to send pulses at the same time, the pulse AER cyclic priority arbitrator arbitrates and outputs these pulses. Figure 1As shown in the figure, the postsynaptic pulses of the lth layer (i.e., the presynaptic pulses of the (l+1)th layer) will also be stored continuously starting from the memory address next to the last presynaptic pulse AER of the current layer according to the emission order of the pulse AER events. After completing the entire batch of pulse AER events, the microprocessor core will process the neurons of the next layer (l+1). This process ends after the output layer is completed. For the reverse phase, all microprocessor cores simultaneously scan the output layer neurons mapped to them, calculate the output error AER, and send it to the error AER memory. Similarly, these error AERs are also regarded as error event batches, and the same error event batches are repeatedly obtained to complete the error propagation according to the layer-by-layer event batch drive. Finally, also driven by the pulse AER event, when a neuron receives a presynaptic pulse AER, it triggers the update of the pulse AER address and the weight of the synapse connected to the neuron. Since the neurons of the fully connected layer are evenly mapped to all microprocessor cores, and the input pulse AER or output error AER will also activate all neurons in the fully connected layer, the workload on all microprocessor cores is always balanced, and there are no idle cycles between microprocessor cores. As a result, all microprocessor cores can always run at peak performance, maximizing computing resource utilization and minimizing learning and inference latency.

[0055] Step 2.3, the circuit design of the microprocessor core is as follows Figure 3 As shown, it includes a weight updating unit, an error propagation unit, an output error generating unit, a neural state updating unit and a local data storage unit, and the local data storage unit includes a plurality of memories (including Memory, Memory, Memory, Memory, Memory and memory), all work in an event-driven manner. The neural state update unit is used to update the state information such as the neuron membrane potential, the output error generation unit is used to output the neuron error AER, the error propagation unit is used to propagate the output error to the hidden layer, and the weight update unit is used to update the synaptic weights of the layer neurons. When multiple microprocessor cores request to provide a firing pulse AER at the same time, in order to reduce the arbitration delay, the neural state update unit adds a 64-bit wide deep ping-pong buffer based on first-in-first-out (FIFO). When the pulse AER caused by the current pulse is cached in one channel of the buffer, the pulse AER caused by the previous pulse stored in another channel is read and written to the pulse AER memory. The local data storage unit is used to cache neuron state information and synaptic weights. The variables in each memory in the local data storage unit are arranged in the order of occupying continuous memory space. For Memory, Memory, Memory and The data address of the jth mapping neuron in a layer is the offset (i.e., the starting address) of the layer plus j. Memory, weight Located at layer offset plus j×I l +i address, where I l is the synaptic count of each neuron in layer l.

[0056] The present application also proposes an incremental access method that does not require expensive hardware multipliers. When the presynaptic pulse AER containing neuron index i triggers the update of the weight of the neurons in the lth layer, starting from the first neuron j=0, where Located at the layer offset plus i. Once the neuron j of this layer is accessed The address pointer only needs to increase by I l , you can access the next j+1 neuron A similar incremental access pattern also applies when propagating output errors AER. Each level of memory address offset is configured by registers in the local controller.

[0057] In the embodiment of the present application, forward processing refers to the process in which the neural network processes input data, starting from the input layer, and passing information forward layer by layer until the output layer generates an output result. Backward processing refers to the process in which, during the training of the neural network, error information is passed backward layer by layer according to the error signal of the output layer, and the weights between neurons in each layer are updated.

[0058] The above-mentioned embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above-mentioned embodiments. Any other changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement modes and shall be included in the protection scope of the present invention.

Claims

1. A neuromorphic processor based on layer-by-layer spike event batch processing, characterized in that: include: An array of microprocessor cores for processing spike AERs and updating the synaptic weights of the fully connected layers of the neural network using the EO-DeepTempo learning rule; A pulse AER memory, used for storing sample input pulses and pulse AERs generated by microprocessor core processing; A pulse AER cyclic priority arbiter is used to arbitrate and output the pulse AER generated by each microprocessor core; Output error AER circular priority arbiter, used for arbitrating and outputting error AER generated by each microprocessor core processing; An error AER memory is used to store the error AER generated by the microprocessor core processing; Global controller, used for overall processor control, data scheduling, and data initialization; Introducing the concept of relative time into the calculation of spike neuron update in the forward processing stage; When training the spiking neural network of the layer-by-layer spike event batching neuromorphic processor in the reverse phase, the EO-DeepTempo learning rule is used to update the weights of each synapse in the fully connected layer of the neural network.

2. The neuromorphic processor based on layer-by-layer spike event batch processing according to claim 1, characterized in that: The EO-DeepTempo learning rule includes error back-propagation and weight gradient update, where: The error back propagation is as follows: the output layer error is calculated and packaged into an error AER, the hidden layer error calculation is driven by the error AER event, and the output layer error is directly projected to each hidden layer through a fixed random matrix using a direct feedback alignment mechanism; The weight gradient is updated by: driven by a pulse AER event, when a neuron receives a presynaptic pulse AER, it triggers the update of the pulse AER address and the weight of the synapse connected to the neuron.

3. The neuromorphic processor based on layer-by-layer spike event batch processing according to claim 1, characterized in that: The spiking neural network consists of L The neurons in the spiking neural network are IF neurons. In the forward stage, the concept of relative time is introduced into the state update calculation of IF neurons.

4. The neuromorphic processor based on layer-by-layer spike event batch processing according to claim 1, characterized in that: A unified neuron-microprocessor kernel mapping mechanism is adopted, specifically: The neurons in the fully connected layer are evenly mapped to all microprocessor cores, and the input pulse AER or output error AER activates all neurons in the fully connected layer. The workload on all microprocessor cores is always balanced, and there are no idle cycles between microprocessor cores.

5. The neuromorphic processor based on layer-by-layer spike event batch processing according to claim 2, characterized in that: The microprocessor cores in the microprocessor core array include a neural state updating unit, an output error generating unit, an error propagation unit, a weight updating unit, and a local data storage unit, wherein the local data storage unit includes a plurality of memories, all of which work in an event-driven manner; The neural state update unit, error propagation unit and weight update unit are respectively connected to the local data storage unit, the output error generation unit and the error propagation unit are respectively connected to the error AER memory, and the neural state update unit is connected to the pulse AER memory; wherein, the neural state update unit is used to update the state information of the neuron membrane potential; the output error generation unit is used to output the neuron error AER; the error propagation unit is used to propagate the output error to the hidden layer; the weight update unit is used to update the synaptic weights of the layer neurons; the local data storage unit is used to cache the neuron state information and synaptic weights.

6. The neuromorphic processor based on layer-by-layer spike event batch processing according to claim 5, characterized in that: The neural state update unit is provided with a 64-bit wide deep ping-pong buffer based on first-in-first-out, which is used to reduce arbitration delay when multiple microprocessor cores request to provide an emission pulse AER at the same time.

Images