Sequential neural machine for optimizing inference memory

Abstract

A spiking neural processor is configured to receive one or more input signals and generate one or more inference output signals. A spiking neural processor includes a plurality of neurons interconnected by a plurality of synaptic elements to form a spiking neural network. A portion of the neurons are connected to receive an input signal, and each neuron is configured to generate a neuron output signal. The spiking nerve processor also includes a memory unit connected to receive one or more neuron output signals from a selected subset of the neurons, and one or more enhancement input circuits connected to selected ones of the neurons by synaptic elements. The storage unit is configured to store data indicative of the received neuron output signal, and output at least a portion of the stored data after a predetermined delay. The enhancement input circuit is connected to receive stored data output by the memory cell after a predetermined delay.

Description

Technical Field

[0001] The present invention relates generally to spiking neural processors and, more particularly, to systems and methods for storing and recalling neural network state information in a spiking neural processor. Background Art

[0002] Spiking Neural Processors (SNPs) are signal processing systems whose design is inspired by biological neural networks. Information is encoded as patterns of spikes distributed across a network of neurons and synapses. SNPs can perform signal processing for many types of sensors and applications, such as image recognition, sound recognition, and event detection based on multiple sensor inputs.

[0003] Analog neurons use analog storage elements (such as capacitors) to accumulate spikes and retain temporal neural network state information between spikes. In order to achieve acceptable latency for SNPs, reduce chip area driving costs, and reduce power consumption, these components are designed to retain state information only within a short time window, typically on the order of a few hundred microseconds. Therefore, over time, network state information is gradually lost, for example due to leakage of accumulated charge from the neuron's capacitor, causing the network to return to a neutral state.

[0004] On the other hand, real-world digital sensors sample analog variables at a certain rate and quantize the sampled values. Quantized data values ​​are generated and periodically forwarded to the next processing stage. For example, audio processing engines, RADAR sensors, etc. The interval between the output of sensor data values, for example, in the order of milliseconds or tens of milliseconds, can be the data collection interval, which can be several orders of magnitude larger than the ability of the SNP's neurons to hold a charge. This raises the question of how to connect the SNP to such a sensor when the SNP cannot retain the neural network state information during the time period between the reception of sensor data values.

[0005] One approach to retaining neural network state information is to use a large memory buffer in which the network state information is stored during these cycles. However, this comes at the significant cost of increased chip area and higher power consumption. Streaming and continuous inference operations are also not feasible because the limited buffer size limits the maximum size of the inference window. Summary of the invention

[0006] The present invention provides a method for solving the above-mentioned problems by providing a means for effectively recording the state information of a neural network and providing the recorded state information during the operation of the neural network.

[0007] In one aspect, the present invention provides a spike neural processor configured to receive one or more input signals and generate one or more inference output signals. The spike neural processor includes a plurality of neurons that are interconnected through a plurality of synaptic elements to form a spike neural network (SNN). A portion of the neurons are connected to receive input signals, and each neuron is configured to generate a neuron output signal. The spike neural processor also includes a storage unit that is connected to receive one or more neuron output signals from a selected subset of neurons, and includes one or more enhanced input circuits that are connected to the selected neurons through synaptic elements. The storage unit is configured to store data indicating the received neuron output signal and output at least a portion of the stored data after a predetermined delay. The enhanced input circuit is connected to receive the stored data output by the storage unit after a predetermined delay.

[0008] The neuron output signals of a selected subset of neurons are stored in a storage unit (as a means for recording at least a portion of the neural network state information during the active period of the SNN) and the neuron output signals from the selected subset of neurons are recorded. Therefore, the storage unit can be configured to store data indicating the received neuron output signals during the period when the spiking neural network receives input signals. In addition, the storage unit can be configured not to store data during the period when the spiking neural network does not receive input signals. This corresponds to the inactive period of the SNN, during which the storage unit retains previously recorded data but does not record new data, thereby saving memory.

[0009] The storage unit can be configured to output at least a portion of the stored data during a subsequent cycle in which the spiking neural network receives an input signal. Thus, when the SNN receives another batch of input signals, previously recorded data indicating the state of the neural network during a previous period of network activity is output by the storage unit and received by the augmented input circuit. This provides feedback of previous network state information to create a "context" of the previous period of SNN activity. In addition, the storage unit can be configured not to output the stored data during a cycle in which the spiking neural network does not receive an input signal.

[0010] The data output by the storage unit in a subsequent cycle may include at least a portion of the data stored in a previous cycle in which the spiking neural network received the input signal. The storage unit may be configured to store data encoding the spike time, spike amplitude and / or spike rate of the neuron output signals from the selected subset of neurons.

[0011] The operation of the storage unit can be coordinated with the input buffer circuit so that the storage unit records data indicative of the received neuron output signal during a burst cycle in which the input buffer circuit forwards the input signal to the spiking neural network, and does not record data during a cycle in which the input buffer circuit does not forward the input signal to the spiking neural network.

[0012] The spiking neural processor may further include an input buffer circuit connected to receive one or more signals from an input signal source, wherein the input buffer circuit is configured to accumulate the received signals for a buffer period and output the accumulated signals to the spiking neural network as input signals during a burst period.

[0013] The buffering period of the input buffer circuit can be coordinated with the predetermined delay of the storage unit. The storage unit can be configured to output the stored data during the period in which the input buffer circuit outputs the accumulated signal to the spiking neural network. The burst period can be at least 10 times shorter than the buffering period, and the input buffer circuit is configured to output the accumulated signal at a compressed time scale compared to the signal received from the input signal source.

[0014] In another aspect, a method of operating a spiking neural processor is provided, the spiking neural network being configured to receive one or more input signals and generate one or more inference output signals, the spiking neural processor comprising a plurality of neurons interconnected by a plurality of synaptic elements to form a spiking neural network, each neuron being configured to generate a neuron output signal. The method comprises: connecting one or more enhanced input circuits to selected neurons of the spiking neural network by the synaptic elements; receiving one or more input signals by a portion of the neurons; receiving one or more neuron output signals from a selected subset of the neurons by a storage unit; storing data indicating the received neuron output signals in the storage unit; outputting at least a portion of the stored data from the storage unit after a predetermined delay; and receiving the stored data output by the storage unit after the predetermined delay by the enhanced input circuit.

[0015] The step of storing data indicative of the received neuron output signal may be performed during a period in which the spiking neural network receives an input signal. The step of outputting at least a portion of the stored data from the storage unit may be performed during a subsequent period in which the spiking neural network receives an input signal.

[0016] The method may further include coordinating the operation of the storage unit with the input buffer circuit such that the storage unit stores data indicative of the received neuron output signal during a burst cycle in which the input buffer circuit forwards the input signal to the spiking neural network, and does not record data during a cycle in which the input buffer circuit does not forward the input signal to the spiking neural network.

[0017] The method may also include connecting an input buffer circuit to receive one or more signals from an input signal source, accumulating the received signals during a buffer period, and outputting the accumulated signals to the spiking neural network as input signals during a burst period. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which corresponding reference numerals indicate corresponding parts, and in which:

[0019] Figure 1 is a simplified schematic diagram of a spiking neural processor;

[0020] Figure 2 is a schematic diagram of neuron and synaptic elements implemented using a crossbar array design;

[0021] Figure 3 yes Figure 1 A schematic diagram of a spiking neural processor in which a memory unit and an enhanced input circuit are also included;

[0022] Figure 4 It is a hint Figure 3 A timing diagram of an example of a signal timeline in a spiking neural processor in; and

[0023] Figure 5 Illustration of simulation test results of the spiking neural processor. DETAILED DESCRIPTION

[0024] Certain illustrative embodiments are shown and described in the following description. As will be appreciated by those skilled in the art, these embodiments may be modified in various ways without departing from the scope of the present invention. Therefore, the drawings and descriptions are to be regarded as illustrative in nature, rather than restrictive. The same reference numerals represent the same elements in the specification.

[0025] Figure 1 is a simplified schematic diagram of a spiking neural processor 1 including a spiking neural network (SNN). The SNN in this example includes an input layer 2 of neurons 10 (input neurons), a hidden layer 3 of neurons 11 (hidden neurons), and an output layer 4 of neurons 12 (output neurons). The input neurons are connected to the hidden neurons 11 through synaptic elements 17, and the hidden neurons 11 are connected to the output neurons 12 through synaptic elements 18. The output 9 of the entire system is generated by the last layer of output neurons 12 in the SNN. The output 9 of the output neurons 12 is then passed to a decoding layer, which can pass the information forward for further processing or output to a user.

[0026] For simplicity, Figure 1The spiking neural processor shown is illustrated with only three layers with very few neurons and synaptic elements, but actual processors may have many more layers, neurons, and synaptic elements to achieve satisfactory performance. Actual implementations of spiking neural processors typically include hundreds of thousands or millions of neurons and a correspondingly large number of synapses. A spiking neural processor may be implemented using hardware circuits or a combination of hardware and software or firmware, and may be implemented as a single integrated circuit or as an embedded system. Neurons may be implemented using analog or digital circuits or mixed signal circuits.

[0027] The input neuron 10 receives an input signal from a signal source 6 (such as a sensor) and generates a neuron output signal 14 in the form of a train of spikes. The neurons 11, 12 in the subsequent layers 3, 4 receive the output signals generated by the synapses 17, 18 and generate neuron output signals 15, 16 in the form of a train of spikes. Depending on the configured synaptic connections, each neuron 11, 12 receives a synaptic output signal from one or more of the synapses 17, 18. For example, Figure 1 As shown, each neuron in one layer can be connected to each neuron in the next layer through synapses, or the network can be configured to selectively connect through synapses between selected neurons in adjacent layers. In addition to what is described, many different connections between neurons can be used, and can also include skip connections, highly recursive liquid state machine architectures, etc.

[0028] Each neuron 10, 11, 12 accumulates or integrates the received signals (input signals or synaptic output signals) and generates a neuron output signal 14, 15, 16. A neuron generates spikes at its output depending on the received input signals and its current state.

[0029] When the integrated value (called the membrane potential) reaches a predetermined threshold, the neuron output signal will contain a spike. In an analog implementation of a neuron, the integrated value of the received signal can be stored as a charge stored on a capacitor.

[0030] When the threshold is reached, the neuron fires, generating a spike (i.e., a voltage or current spike) at the output of the neuron. During the discharge, the membrane potential decreases due to the discharge. If the membrane potential subsequently reaches the threshold again, the neuron will fire again, generating a second spike. Each neuron is thus configured to generate a neuron output signal 14, 15, 16 in the form of a spatiotemporal spike sequence. The neuron output signal 14, 15, 16 depends on multiple parameters of the neuron, such as input gain, integration constant, and threshold. If no input signal is received, the membrane potential of each neuron will also "leak", and the potential will gradually decrease over time, resulting in an increase in potential.

[0031] Each synaptic element 17, 18 (also referred to as a synapse) receives an output signal from one of the input circuits 10 or neurons 11. The synapses 17, 18 amplify or attenuate the received output signal by a predetermined factor determined by their weight settings, which are configurable. The weight of a synapse can be positive, so that the synaptic output signal received from the synapse excites the neuron receiving the signal, raising their membrane potential. The weight can be negative, which inhibits the neuron receiving the synaptic output from the synapse, potentially lowering their membrane potential. Alternatively, the weight can be zero, which effectively removes the synaptic connection between the two neurons connected by the synapse. The weight of each synapse is stored in a memory cell associated with the synapse. The values ​​of all weights in a network are called a weight matrix, and the weights are typically determined by a network training process.

[0032] Figure 2 A simplified schematic diagram of synaptic connections implemented as a crossbar array. The crossbar array design is an effective approach to implementing reconfigurable neural networks, especially when fabricated on integrated circuits. Figure 2 The design in includes a rectangular array of synapses 17 for interconnecting two layers of the SNN, such as synapse 17 connecting neuron 10 on one side of the array to neuron 11 on the other side, or synapse 18 connecting neuron 11 on one side of the array to neuron 12 on the other side. Figure 2 In an embodiment, neurons 10 are arranged in a column, and each neuron drives a row of synapses 17. The synapses 17 are connected in columns, and the outputs of all synapses 17 in a column are added together and used as inputs to neurons 11. By programming appropriate weights in the synaptic array and properly configuring the interconnection system, a variety of network topologies can be achieved.

[0033] exist Figure 1 In a spiking neural processor, an input layer neuron 10 receives an input signal 8 from an input signal source 6 (e.g., a sensor). Many such input signal sources 6 generate data at a relatively slow rate or in bursts with long periods of time between bursts. For example, a microphone may generate an analog output signal that is sampled at a certain frequency. The samples may then be converted to digital data by an analog-to-digital converter (ADC), or to a spike train output. For example, multiple outputs may be used, each of which transmits a spike when the sampled value falls within a certain value range for that output. If a sampling rate of 1 kHz is used, for example, a digital output value or corresponding spike output will be generated every millisecond.

[0034] Other types of input signal sources 6 may generate output data rapidly but intermittently, for example due to the nature of the variable or phenomenon being measured or sampled, or the function of the signal source.

[0035] SNP 1 operates at a much faster rate than many input signal sources, and during the intervals between receiving input signals from input signal source 6, neurons in the neural processor will forget their previous states, i.e., their membrane potentials may return to zero or a lower value due to leakage. The loss of neural network state information may occur in a very short time, such as within a fraction of a millisecond.

[0036] Although neurons can be designed to maintain their membrane potential for longer periods of time, this will slow down the operation of the neuron and increase the latency of the SNP, increase the size of the neuron, resulting in a significant increase in the chip area and manufacturing cost of the SNP, and increase the power consumption of the SNP. Alternatively, a memory can be used to store the membrane potential of the neuron during inactivity. However, this will require a large memory, which will also lead to an increase in chip area and manufacturing cost and higher power consumption, and the size of the memory will limit the operation of the SNP in many applications.

[0037] SNP can solve this problem by implementing a "forget and remember" strategy. Imagine that you are reading a book, stop reading on page 39 of the book, and place a bookmark on that page. You continue reading a week later and do not remember all the details of the first 39 pages. Therefore, you read the last few lines of the page where the bookmark is located to recall the context, and then you can continue reading from page 40. The "forget and remember" strategy is a similar strategy. When there is no input signal for a long time, such as due to inactivity of the input signal source or low data rate, the neurons of the SNN forget the temporal network state information. However, part of the neuron output signal generated during the last period in which the input signal was received and the SNN was active can be recorded and stored in memory. These stored data can then be replayed during the subsequent activity period of the SNN to enable the SNN to "remember" part of the network state information present in the previous active period, thereby improving the inference accuracy of the SNP.

[0038] Figure 3 Show Figure 1 Schematic diagram of a spike neural processor in which a storage unit 20 and an enhanced input circuit 25 are added. The enhanced input circuit 25 is connected to the neurons of the SNP via additional synapses. For example, in Figure 3 In the illustrated embodiment, the enhanced input circuit 25 is connected to the neuron 11 in the neural network layer 3 via an additional synapse 17, i.e., the same layer of neurons that receive signals from the input layer neuron 10 via synapse 17. However, other configurations are possible, such that the enhanced input circuit 25 can be connected to any neuron in the SNP via a synapse. The number of enhanced input circuits 25 can be configurable, and the synaptic connection of the enhanced input circuit 25 to the neuron can be set, for example, after training the neural processor.

[0039] The enhanced input circuit 25 may be a neuron similar to the input layer neuron 10 that generates spikes at its output terminal according to the received input signal and the current state, or may be a simple pass-through circuit that passes the received output signal to its output terminal, or other types of circuits suitable for receiving inputs from the storage unit 20 and providing outputs at the synapse. The enhanced input circuit 25 may be treated the same as the neuron 10 of the input layer 2, and the synaptic connections with other neurons in the network may be configured in the same manner as the neuron 10, such as being fully connected to each neuron 11 in the next layer 3 of the SNN through synapses 17, or being sparsely connected to other neurons.

[0040] The number of augmented input circuits 25 can be configured by a network hyperparameter, ie, a parameter whose value is used to control the network learning process during training, and in most cases has a fixed value throughout the training process.

[0041] The storage unit 20 includes a memory 21, a delay circuit 22, and a control unit 23. The storage unit 20 receives neuron output signals from a selected subset of the neurons of the SNN (referred to herein as "memory-neurons"). The output spikes from these memory neurons are used as a way to record the recent spike history of the SNN by recording the outputs of the memory neurons in the storage unit 20. This provides a method of storing information about recent neural network states, which was obtained during the most recent activity of the SNN, which activity was caused by the SNN receiving input signals. By storing this neural network state information in the storage unit 20, the state information can be saved for longer periods of time, such as during inactivity periods when the SNN does not receive any input signals, causing neurons in the SNN to lose their membrane potential due to leakage.

[0042] For example, the SNN can be configured so that the storage unit 20 receives neuron output signals from all output neurons 12 (e.g., all neurons in the last layer of the SNN). Other methods can also be used to select specific neurons from any layer of the SNN to provide neuron outputs to the storage unit 20. For example, neurons that exhibit more dynamic characteristics can be identified and selected as memory neuron candidates, such as neurons that update weights more frequently than other neurons during training, or neurons that exhibit a wider range of spike patterns during inference. Figure 3 In the example shown, the storage unit receives neuron output signals from neurons in layer 3 and neurons in layer 4. The number of neurons selected as memory neurons may be equal to the number of boost input circuits 25, or may be different.

[0043] The storage unit 20 is configured to record and store the memory neuron output signal 26 in the memory 21 under the control of the control unit 23 during the periodic period when the SNP receives the input signal 8. For example, the input neuron 10 may receive the input signal 8 from the input signal source 6 or the input buffer 7 in the form of short bursts, with longer inactive intervals in between (during which the input signal 8 is not received). The control unit 23 is configured to coordinate the recording and storage of the memory neuron output 26 with the bursts of the input signal received by the input neuron 10 of the SNP. The control unit 23 may start recording and storing the memory neuron output 26 during the short bursts when the SNP receives the input signal 8, and may stop recording the memory neuron output 26 during the inactive intervals when the SNP does not receive the input signal 8 (to reduce the required storage capacity) (while retaining the data last recorded during the last burst of the input signal 8 in the memory 21). This coordination may be achieved by communication between the control unit 23 and the input buffer 7, as further described below.

[0044] The storage unit 20 is configured to store data indicating the received memory neuron output signal 26 in the memory 21. For example, the storage unit 20 may store data about one or more parameters of any spike in the memory neuron output signal 26. The amount of information for each recorded spike is configurable, for example, the storage unit 20 may store data about the time when the spike occurs, the amplitude of the spike, and / or the spike rate of the neuron output signal. In one embodiment, the storage unit 20 is configured to store data for each spike generated by each memory neuron, that is, any spike on the input 26 of the storage unit 20. For example, the storage unit 20 may record the identity of the neuron that generated the spike and the time value indicating when the spike occurred. These data are preferably encoded to efficiently store data and reduce the size required for the memory 21. For example, the storage unit 20 may record the neuron ID and relative time of each spike, where the relative time represents the time difference between the start time of the data burst received by the SNN and the time when the spike occurs. By effectively encoding the spike data received by the storage unit 20, the SNP can record the neural network state information in an efficient manner while keeping the size of the memory 21 in the storage unit 20 to a minimum.

[0045] The storage unit 20 is configured to output previously stored data during a period in which the input neuron 10 receives the input signal 8. In this way, the storage unit 20 provides the last recorded neural network state information to the enhanced input circuit 25 during a period in which the input neuron 10 of the SNN receives another input signal. This enables the neural network to incorporate the last state information into the reasoning processing of the currently received input signal.

[0046] For example, the control unit 23 may be configured to control the storage unit 20 to output the stored data during the SNP receiving a burst of the input signal 8, wherein the data includes at least a portion of the data stored during the SNP receiving a previous burst of the input signal 8. During the current burst of the input signal 8, the storage unit 20 may output all the data recorded and stored during the previous burst of the input signal, or only a portion of the data.

[0047] The enhanced input circuit 25 is connected to receive the output of the storage unit 20. The enhanced input circuit 25 receives the storage data output from the storage unit 20, but may additionally receive input signals 8, such as signals from the input buffer 7, and / or inputs of the input neurons 10. The enhanced input circuit 25 can be configured to generate spikes at its output terminal according to the received input signals and their current state. The SNP can be configured to adapt the number of enhanced input circuits 25 to the number of memory neurons (whose neuron outputs 26 are recorded by the storage unit 20). In one embodiment, each enhanced input circuit 25 receives storage data of spikes generated by one memory neuron. For example, each enhanced input circuit 25 can be connected to receive storage data (from the storage unit 20) from a specific one of the memory neurons. Alternatively, the enhanced input circuit 25 can receive storage data of spikes generated by multiple memory neurons, or storage data of spikes generated by multiple memory neurons can be received by a single enhanced input circuit 25.

[0048] The last recorded data may be output by the memory unit 20 in a manner that preserves the timing of the spikes recorded during the last burst of the input signal 8. For example, the relative timing of the spikes within the input signal burst may be preserved to be "replayed" with the same relative timing during the current input signal burst. This enables the enhanced input circuit 25 to generate output spikes within the current input signal burst with the same timing to reproduce the timing of the spikes generated by the memory neuron during the last input signal burst.

[0049] Alternatively, the storage unit 20 may transform the stored data. For example, the spike timing may be reversed, e.g., so that the last spike recorded during the previous input signal burst is output first during the current input signal burst, and so on. This may be achieved, for example, if the last spike generated by the output neuron is considered to be most important for reproducing the state of the neural network during the previous input signal burst.

[0050] The storage unit 20 includes a memory 21, which may be a digital memory such as a DRAM, SRAM, or register memory. A shift register, such as a FIFO (first in, first out) shift register may be used to preserve the spike timing, or a LIFO (last in, first out) shift register may be used to reverse the spike timing. The storage unit 20 also includes a delay circuit 22. This may be a separate memory, or a logic circuit that controls the output of the memory 21, or may be part of the memory 21 (e.g., when the memory 21 is a shift register). The delay circuit 22 (together with the control unit 23) implements a predetermined delay in the data output of the storage unit 20. The predetermined delay is selected so that the timing of the output of the stored data from the storage unit 20 is consistent with the current input signal burst, or before it, as further described below. The storage unit 20 also includes a control unit 23, which controls the operation of the storage unit 20. The control unit 20 may be implemented as a hardware logic circuit, such as an ASIC or FPGA, or as a processor that executes software or firmware, or a combination of these.

[0051] Figure 3 Also shown is an optional input buffer 7 which may be used to buffer the signals generated by the signal source 6 and output the accumulated input signal 8 in bursts to the input layer neurons 10. The input buffer 7 may be included in the input signal source 6, may be included in the spiking neural processor 1, may be implemented as a separate unit, or may be omitted if not required.

[0052] In one embodiment, the input buffer 7 and the storage unit 20 are controlled to coordinate their functions. For example, the storage unit 20 can record the output signal 26 of the memory neuron in each burst cycle in which the input buffer 7 forwards the input signal 8 to the neuron 10. The storage unit 20 can be configured to record in the entire burst cycle, or in a portion of the burst cycle, such as at the end of the burst cycle. In addition, the storage unit 20 can output the previously stored output signal 26 of the memory neuron to the enhanced input circuit 25 in each burst cycle in which the input buffer 7 forwards the input signal 8 to the neuron 10.

[0053] Now refer to Figure 4 The example signal timeline shown in Figure 3 Operation of a spiking neural processor.

[0054] Figure 4The first row in is an example timeline 30 of output data generated by the signal source 6. The output data 30 may be in the form of analog values, digital values, or one or more spike train signals. If the signal source 6 generates analog or digital values, these values ​​are preferably converted to a spike train signal suitable for input to the SNP at some point during the processing of the SNP input signal. For example, the analog or digital values ​​may be converted to a plurality of spike train signals, each of which represents a range of values, and a spike is generated in one of the spike train signals when the analog or digital value falls within the range of values ​​of the spike train signal. For simplicity, the following description assumes that the output data 30 is in the form of spikes, although the conversion of the spikes may be performed at a later stage in the system.

[0055] The output data (e.g., spikes) from the signal source 6 may appear in rapid bursts with relatively long time intervals between the bursts, e.g., 1 ms. However, there may be longer time intervals between each spike, or the spikes may appear intermittently with longer time intervals between some of the spikes, or the spikes may appear in bursts with relatively long time intervals between the spikes during the burst.

[0056] The input buffer 7 can be used to accumulate output data 30 from the signal source 6, thereby generating fast spike bursts at regular periodic intervals with longer periods between bursts. The input buffer 7 accumulates the output data 30 during a buffer period 34 (e.g., 20 ms) and outputs the accumulated output data at regular intervals during a short burst period 35. The buffer period 34 and the burst period 35 can be selected based on the SNP design and the inference application to be performed by the SNP. For example, in a memory-constrained system (e.g., a SNP implemented as an embedded system that uses only kilobytes of system memory for the input buffer 7 and the memory 21), the maximum expected number of spikes per burst can be set to a smaller number to reduce memory requirements, resulting in shorter buffer periods 34 and burst periods 35. However, the expected number of spikes per burst cannot be arbitrarily small because a sufficient number of spikes is required to cause the neuron to exhibit a minimum level of activity (which will be stored for input to the enhanced input circuit).

[0057] Figure 4The second row is an example timeline 31 of the output from the input buffer 7 (which constitutes the input signal 8 of the spiking neural processor). During the burst period 35, the accumulated data (stored during the buffer period 34) is transmitted to the input neuron 10. If the original signal is not in the form of a spike train signal, the output of the input buffer 7 can be converted to a spike train signal. The burst period 35 can be set to a predetermined time period, which can be configurable, as described above. After the burst period 35, the input buffer 7 stops outputting data during period 36 until the next burst period, while buffering data for the next input signal burst.

[0058] During input signal burst 40A, input buffer 7 outputs the accumulated data as input signal 8 to input neuron 10. Input neuron 10 generates neuron output signal 14, which is passed via synapse 17 to hidden neuron 11, which generates neuron output signal 15, which is passed via synapse 18 to output neuron 12, which generates neuron output signal 16 (SNP output 9).

[0059] Figure 4 The third row of FIG. 3 is an example timeline 32 of spiking neuron outputs generated by selected memory neurons in the SNP, which are recorded in the memory unit 20. The memory neuron output 41A is recorded by the memory unit 20 during the burst 40A of the input signal 8. The memory unit 20 stores data indicating the received memory neuron output signal 26 during this period.

[0060] Figure 4 The fourth row of FIG. 3 is an example timeline 33 of outputs from the storage unit 20 that become inputs to the enhanced input circuit 25. During the burst 40A of the input signal 8 and in the process of recording the memory neuron output signal 41A, there are no outputs from the storage unit 20 and no inputs to the enhanced input circuit 25. The next burst 40B of the input signal 8 occurs after another buffering period as shown in the timeline 31. This results in the memory neuron output signal 41B, as shown in the timeline 32. During the input signal burst 40B, the storage unit 20 outputs the stored data 42A of the memory neuron output signal 41A recorded during the earlier input signal burst 40A. As shown in the timeline 33, the stored data 42A is output from the storage unit 20 after the memory neuron output signal 41A generated during the earlier input signal burst 40A is recorded after a predetermined delay 37. Similarly, during the next input signal burst 40C, the storage unit 20 outputs the stored data 42B of the memory neuron output signal 41B recorded during the earlier input signal burst 40B.

[0061] exist Figure 4In the example shown, the timing of the storage data output from the storage unit 20 is consistent with the next input signal burst (e.g., the storage data output 42A coincides with the input signal burst 40B). However, the storage unit 20 can also be configured to output the storage data with a different timing, such as in a time period before the next input signal burst (e.g., the output 42A can be output from the storage unit 20 just before the input signal burst 40B). This configuration can be used to at least partially reproduce the neural network state that existed at the end of the input signal burst 40A just before further input signals are received during the input signal burst 40B.

[0062] Training the spiking neural network of the SNP can be done using the labeled data fed to the SNP in bursts. Prior to training, the enhanced input circuit 25 is configured and the memory neurons are identified. The training data is divided into smaller data sets (corresponding to the input signal bursts) and artificial delays are inserted between the data sets so that the training simulates the reasoning environment (SNNs are primarily trained to process specific data sets in a specific environment). During the training process, the SNN learns the presynaptic weights of both the normal input neurons 10 and the enhanced input circuit 25. This is achieved by training the network by providing the network with a training data set, during which the network "learns" these weights.

[0063] SNNs were benchmarked in simulations using the data storage and feedback approach described in this paper. The publicly available “Spiking Heidelberg Digits” dataset (an audio-based classification dataset consisting of spoken digits from 0 to 9 converted into spike sequences) was used to empirically verify the effectiveness of this technique. The dataset containing 100 spike vectors was used to train a spiking neural network consisting of 700 input neurons, 400 neurons in the second hidden layer, and 20 output neurons using an alternative gradient descent training technique. Training was performed with different burst sizes ranging from no bursts to 20, 10, and 8 spikes per burst. The storage capacity required (for the input signal buffer and the memory neuron storage unit) for each burst size is estimated in the table below, where the storage capacity is measured in terms of the number of spikes to be stored.

[0064]

[0065] Figure 5The performance curves of the spiking neural network derived from the test are shown. The horizontal axis represents the number of training rounds (epoch), and the vertical axis represents the network performance estimate based on inference accuracy. The blue solid line A represents the performance when the data (SpikingHeidelberg Digits) is input to the SNN without being subdivided into bursts, which requires a lot of memory. The dotted lines B1, B2, and B3 represent the network performance of bursts with 8, 10, and 20 spikes per burst, respectively, but without feedback of the recorded memory neuron data. The solid lines C1, C2, and C3 represent the network performance of bursts with 8, 10, and 20 spikes per burst, respectively, and the recorded memory neuron data is fed back to the enhanced input circuit.

[0066] Tests show that when the data set is received by the SNN in bursts, especially smaller bursts (dashed line), the network performance will drop significantly. However, this drop in performance is compensated when feedback of the memory neuron output is provided to the augmented input circuit (solid line).

[0067] The systems and methods described herein are particularly suitable for systems with limited resources (e.g., memory and data bandwidth), such as embedded systems, which need to store large amounts of data from slow or intermittent sensors or other signal sources for processing in fast neuromorphic systems. The systems and methods described herein greatly reduce the amount of memory required to store input signal data, and also reduce the loss of accuracy by allowing the SNN to "forget" the temporal network state information between sensor samples and by providing it with a snapshot of the previous neural network state to help it "remember" the lost information.

Claims

1. A spiking neural processor (1), the spiking neural processor being configured to receive one or more input signals (8) and to generate one or more inference output signals (9), the spiking neural processor comprising: a plurality of neurons (10, 11, 12) interconnected by a plurality of synaptic elements (17, 18) to form a spiking neural network, wherein a portion of the plurality of neurons (10) are connected to receive the one or more input signals (8), and each of the plurality of neurons (10, 11, 12) is configured to generate a neuron output signal; a storage unit (20) connected to receive one or more neuron output signals (26) from a selected subset of the plurality of neurons (10, 11, 12); and one or more reinforcing input circuits (25) connected by the synaptic element (17) to selected neurons among the plurality of neurons (11, 12) of the spiking neural network; wherein the storage unit (20) is configured to store data indicative of the received neuron output signal (26) and to output at least a portion of the stored data after a predetermined delay (37); and The enhanced input circuit (25) is connected to receive the storage data (27) output by the storage unit (20) after the predetermined delay (37).

2. The spiking neural processor of claim 1, wherein: The storage unit (20) is configured to store the data indicating the received neuron output signal (26) during a period in which the spiking neural network receives an input signal (8), and is configured not to store the data during a period in which the spiking neural network does not receive an input signal (8).

3. The spike neural processor according to claim 1 or 2, wherein: The storage unit (20) is configured to output at least a portion of the stored data in a subsequent cycle in which the spiking neural network receives an input signal (8), and is configured not to output the stored data in a cycle in which the spiking neural network does not receive an input signal (8).

4. The spiking neural processor of claim 3, wherein: The data output by the storage unit (20) in the subsequent cycle includes at least a portion of the data stored in the previous cycle in which the spiking neural network received the input signal (8).

5. A spiking neural processor according to any one of the preceding claims, wherein: The storage unit (20) is configured to store data encoding spike times, spike amplitudes and / or spike rates of neuron output signals (26) from a selected subset of the plurality of neurons.

6. A spiking neural processor according to any one of the preceding claims, wherein: Coordinating the operation of the storage unit (20) with the input buffer circuit (7) so that the storage unit (20) records data indicating the received neuron output signal (26) during a burst cycle (35) in which the input buffer circuit (7) forwards the input signal (8) to the spiking neural network, and does not record the data during a cycle (36) in which the input buffer circuit (7) does not forward the input signal (8) to the spiking neural network.

7. The spiking neural processor according to any one of claims 1 to 5, further comprising an input buffer circuit (7) connected to receive one or more signals from an input signal source (6), wherein: The input buffer circuit (7) is configured to accumulate the received signal during a buffer period (34) and output the accumulated signal as the input signal (8) to the spiking neural network during a burst period (35).

8. The spiking neural processor of claim 7, wherein: The buffering period (34) of the input buffer circuit (7) is coordinated with the predetermined delay (37) of the storage unit (20).

9. The spiking neural processor according to claim 7 or 8, wherein: The storage unit (20) is configured to output storage data (27) during a period in which the input buffer circuit (7) outputs the accumulated signal to the spiking neural network.

10. The spiking neural processor according to any one of claims 7 to 9, wherein: The burst period (35) is at least 10 times shorter than the buffer period (34), and the input buffer circuit (7) is configured to output the accumulated signal at a compressed time scale compared to the signal received from the input signal source (6).

11. A method of operating a spiking neural processor (1), the spiking neural processor being configured to receive one or more input signals (8) and to generate one or more inference output signals (9), the spiking neural processor comprising a plurality of neurons (10, 11, 12) interconnected by a plurality of synaptic elements (17, 18) to form a spiking neural network, each of the plurality of neurons (10, 11, 12) being configured to generate a neuron output signal, the method comprising: connecting one or more reinforcing input circuits (25) to selected neurons of the plurality of neurons (11, 12) of the spiking neural network by the synaptic element (17); Receiving one or more input signals (8, 40A) by a portion of the plurality of neurons (10); Receiving, by a storage unit (20), one or more neuron output signals (26, 41A) from a selected subset of the plurality of neurons (10, 11, 12); storing data indicative of the received neuron output signal (26, 41A) in the storage unit (20); outputting at least a portion of the stored data (27, 42A) from the storage unit (20) after a predetermined delay (37); and The storage data (27, 42A) output by the storage unit (20) after the predetermined delay is received by the enhanced input circuit (25).

12. The method according to claim 11, wherein: The step of storing data indicative of received neuron output signals is performed during a period in which the spiking neural network receives an input signal (8).

13. The method according to claim 11 or 12, wherein: The step of outputting at least a portion of the stored data from the storage unit (20) is performed in a subsequent cycle of the spiking neural network receiving an input signal (8).

14. The method according to any one of claims 11-13, further comprising coordinating the operation of the storage unit (20) with the input buffer circuit (7) so that the storage unit (20) stores data indicating the received neuron output signal (26) during a burst cycle (35) in which the input buffer circuit (7) forwards the input signal (8) to the spiking neural network, and does not record the data during a cycle (36) in which the input buffer (7) does not forward the input signal (8) to the spiking neural network.

15. The method according to any one of claims 11 to 13, further comprising: An input buffer circuit (7) is connected to receive one or more signals from an input signal source (6), accumulate the received signals during a buffer period (34), and output the accumulated signals as the input signals (8) to the spiking neural network during a burst period (35).

Images