Sequential Neural Machines for Memory-Optimized Inference

Abstract

A spiking neural processor configured to receive one or more input signals and generate one or more inferred output signals. The spiking neural processor comprises a plurality of neurons interconnected by a plurality of synaptic elements to form a spiking neural network. A portion of the neurons are connected to receive the input signals, and each of the neurons is configured to generate a neuron output signal. The spiking neural processor also comprises a storage unit connected to receive one or more of the neuron output signals from a selected subset of the neurons, and one or more extended input circuits connected by synaptic elements to selected ones of the neurons. The storage unit is configured to store data indicative of the received neuron output signals and to output at least a portion of the stored data after a predetermined delay. The extended input circuit is connected to receive the stored data output by the storage unit after the predetermined delay.

Description

[Technical Field]

[0001]

[0001] This disclosure relates generally to spiking neural processors, and more particularly to systems and methods for storing and recalling neural network state information in spiking neural processors. [Background technology]

[0002]

[0002] A spiking neural processor (SNP) is a signal processing system whose design is inspired by biological neural networks. Information is encoded in patterns of spike signals distributed across a network of neurons and synapses. SNPs can perform signal processing for multiple types of sensors and applications, such as image recognition, sound recognition, event detection based on inputs from multiple sensors, etc.

[0003]

[0003] Analog neurons utilize analog memory elements, such as capacitors, to store spikes and retain temporal neural network state information between spikes. To achieve acceptable latency for SNPs, reduce chip area costs, and reduce power consumption, these components are designed so that they can retain state information only over short time windows, typically on the order of hundreds of microseconds. As a result, network state information leaks over time, for example, due to leakage of stored charge from the neuron's capacitors, thereby returning the network to a neutral state.

[0004]

[0004] On the other hand, real-world digital sensors sample analog variables at a certain rate and quantize the sampled values. The quantized data values are periodically generated and transferred to the next processing stage. Examples are audio processing engines, RADAR sensors, etc. The interval between the output of sensor data values may be, for example, on the order of milliseconds or tens of milliseconds, and the data collection interval may be several orders of magnitude greater than the ability of the SNP's neurons to maintain their charge. The problem arises of how to interface an SNP with such a sensor if the SNP cannot maintain neural network state information during the time period between receipt of sensor data values.

[0005] One way to maintain neural network state information is to use large memory buffers in which the network state information is stored during these periods. However, this incurs significant costs in increased chip area and higher power consumption. Also, the finite buffer size imposes limitations on how large the maximum size of the inference window can be, making it prohibitive for streaming and continuous inference operations. Summary of the Invention

[0006]

[0006] The present invention provides a means for addressing the above-described problems by providing an efficient means for recording neural network state information and providing the recorded state information for use during neural network operation.

[0007] In one aspect, the present invention provides a spiking neural processor configured to receive one or more input signals and generate one or more inference output signals. The spiking neural processor comprises a plurality of neurons interconnected by a plurality of synaptic elements to form a spiking neural network (SNN). A portion of the neurons are connected to receive the input signals, and each of the neurons is configured to generate a neuron output signal. The spiking neural processor also comprises a storage unit connected to receive one or more of the neuron output signals from a selected subset of the neurons, and one or more extended input circuits connected by synaptic elements to selected ones of the neurons. The storage unit is configured to store data indicative of the received neuron output signals and to output at least a portion of the stored data after a predetermined delay. The extended input circuit is connected to receive the stored data output by the storage unit after the predetermined delay.

[0008]

[0008] Neuron output signals from a selected subset of neurons are stored in a storage unit as a means of recording information embodying at least a portion of the neural network state during periods when the SNN is active and neuron output signals from the selected subset of neurons are recorded. Accordingly, the storage unit may be configured to store data indicative of neuron output signals received during periods when input signals are received by the spiking neural network. Additionally, the storage unit may be configured not to store data during periods when input signals are not received by the spiking neural network. This corresponds to periods of inactivity of the SNN, during which the storage unit retains previously recorded data but does not record new data to conserve memory.

[0009]

[0009] The storage unit may be configured to output at least a portion of the stored data during a subsequent period in which the input signal is received by the spiking neural network. Thus, when the SNN receives another burst of the input signal, previously recorded data indicative of the neural network state during the previous network activity period is output by the storage unit and received by the extended input circuit. This provides feedback of previous network state information to recreate the "context" of the previous SNN activity period. Additionally, the storage unit may be configured not to output the stored data during periods in which the input signal is not received by the spiking neural network.

[0010] The data output by the storage unit during the subsequent time period may comprise at least a portion of the data stored during the immediately preceding time period when the input signal was received by the spiking neural network. The storage unit may be configured to store data encoding spike times, spike amplitudes, and / or spiking rates of neuron output signals from a selected subset of neurons.

[0011]

[0011] The operation of the storage unit may be coordinated with the input buffer circuit so that the storage unit records data indicative of received neuron output signals during burst periods when the input buffer circuit transfers input signals to the spiking neural network, and does not record data during periods when the input buffer does not transfer input signals to the spiking neural network.

[0012]

[0012] The spiking neural processor may further include an input buffer circuit connected to receive one or more signals from an input signal source, the input buffer circuit configured to accumulate the received signals over a buffering period and output the accumulated signals during the burst period as input signals to the spiking neural network.

[0013] The buffering period of the input buffer circuit can be coordinated with a predetermined delay of the storage unit. The storage unit can be configured to output the stored data during a period in which the input buffer circuit outputs the accumulated signal to the spiking neural network. The burst period can be at least 10 times shorter than the buffering period, and the input buffer circuit can be configured to output the accumulated signal on a compressed time scale compared to the signal received from the input signal source.

[0014] In another aspect, a method is provided for operating a spiking neural processor for a spiking neural network configured to receive one or more input signals and generate one or more inference output signals, the spiking neural processor comprising a plurality of neurons interconnected by a plurality of synaptic elements to form a spiking neural network, each of the neurons configured to generate a neuron output signal. The method includes connecting one or more augmented input circuits to selected ones of the neurons of the spiking neural network by the synaptic elements, receiving one or more input signals by a portion of the neurons, receiving one or more of the neuron output signals from a selected subset of the neurons by a storage unit, storing data indicative of the received neuron output signals in the storage unit, outputting at least a portion of the stored data from the storage unit after a predetermined delay, and receiving by the augmented input circuit the stored data output by the storage unit after the predetermined delay.

[0015] The storage of data indicative of the received neuron output signals may be performed during a period in which the input signals are received by the spiking neural network, and the output of at least a portion of the stored data from the storage unit may be performed during a subsequent period in which the input signals are received by the spiking neural network.

[0016]

[0016] The method may further comprise coordinating operation of the storage unit with the input buffer circuit, so that the storage unit stores data indicative of neuron output signals received during burst periods when the input buffer circuit forwards input signals to the spiking neural network, and does not record data during periods when the input buffer does not forward input signals to the spiking neural network.

[0017]

[0017] The method may further comprise connecting an input buffer circuit to receive one or more signals from an input signal source, accumulating the received signals over a buffering period, and outputting the accumulated signals during a burst period as input signals to the spiking neural network.

[0018]

[0018] Embodiments will now be described, by way of example only, with reference to the accompanying drawings in which corresponding reference symbols indicate corresponding parts and in which: [Brief explanation of the drawings]

[0019] [Figure 1]

[0019] FIG. 1 is a schematic diagram of a simple spiking neural processor. [Figure 2]

[0020] FIG. 1 is a schematic diagram of neurons and synaptic elements implemented using a crossbar design. [Figure 3]

[0021] FIG. 2 is a schematic diagram of the spiking neural processor of FIG. 1, also including a storage unit and extended input circuitry. [Figure 4]

[0022] FIG. 4 is a timing diagram illustrating an example of a signal timeline in the spiking neural processor of FIG. 3. [Figure 5]

[0023] Simulation test results for a spiking neural processor are presented. DETAILED DESCRIPTION OF THE INVENTION

[0020]

[0024] In the following description, certain exemplary embodiments are illustrated and described. Those skilled in the art will recognize that these embodiments may be modified in a variety of different ways without departing from the scope of the present disclosure. Therefore, the drawings and description should be regarded as illustrative in nature and not restrictive. Like reference numerals refer to like elements throughout the specification.

[0021]

[0025] Figure 1 is a schematic diagram of a simple spiking neural processor 1 comprising a spiking neural network (SNN). The SNN in this example comprises an input layer 2 of neurons 10 (input neurons), a hidden layer 3 of neurons 11 (hidden neurons), and an output layer 4 of neurons 12 (output neurons). The input neurons are connected to the hidden neurons 11 via synaptic elements 17, which are connected to the output neurons 12 via synaptic elements 18. The output 9 of the overall system is generated by the last layer of output neurons 12 in the SNN. The output 9 of the output neurons 12 is then passed to a decoding layer, which can pass the information onward for further processing or output to a user.

[0022]

[0026] Although the spiking neural processor shown in FIG. 1 is illustrated with only three layers with a very small number of neurons and synaptic elements for simplicity, an actual processor may have a much larger number of layers, neurons, and synaptic elements to achieve satisfactory performance. Actual implementations of spiking neural processors typically include hundreds of thousands or millions of neurons and a corresponding large number of synapses. Spiking neural processors may be implemented using hardware circuits or a combination of hardware and software or firmware, as a single integrated circuit, or as an embedded system. Neurons may be implemented using analog or digital circuits, or mixed-signal circuits.

[0023]

[0027] An input neuron 10 receives an input signal from a signal source 6, such as a sensor, and generates a neuron output signal 14 in the form of a spike train. Neurons 11, 12 in subsequent layers 3, 4 receive output signals generated by synapses 17, 18 and generate neuron output signals 15, 16 in the form of spike trains. Each neuron 11, 12 receives a synaptic output signal from one or more of the synapses 17, 18 depending on the configured synaptic connections. For example, as shown in FIG. 1, all neurons in one layer may be connected via synapses to all neurons in the next layer, or the network may be configured to selectively connect via synapses between selected neurons in adjacent layers. Many different connectivity configurations between neurons may be used in addition to those described, including skip connections, highly recurrent liquid state machine architectures, etc.

[0024]

[0028] Each neuron 10, 11, 12 accumulates or integrates a received signal (input signal or synaptic output signal) and generates a neuron output signal 14, 15, 16. Neurons generate spikes at their output as a function of the received input signal and its current state.

[0025]

[0029] The neuron output signal will contain a spike when the integral (called the membrane potential) reaches a certain threshold. In an analog implementation of a neuron, the integral of the received signal may be stored as a charge stored on a capacitor.

[0026]

[0030] When the threshold is reached, the neuron fires, generating a spike (i.e., a voltage or current spike) at the neuron's output. Upon firing, the membrane potential is reduced as a result of the firing. If the membrane potential subsequently reaches the threshold again, the neuron fires again, generating a second spike. Each neuron is configured to generate from this a neuron output signal 14, 15, 16 in the form of a spatiotemporal spike train. The neuron output signal 14, 15, 16 depends on several parameters of the neuron, such as the input gain, integration constant, and threshold. The membrane potential of each neuron also "leaks," meaning the potential gradually decreases over time if no input signal is applied to increase the potential.

[0027]

[0031] Each synaptic element 17, 18 (also called a synapse) receives an output signal from one of the input circuits 10 or neurons 11. The synapses 17, 18 amplify or attenuate the received output signal by a predetermined factor determined by their configurable weight settings. A synaptic weight can be positive, so that the synaptic output signal received from that synapse excites the receiving neuron, increasing its membrane potential. A weight can be negative, so that the neuron receiving the synaptic output from that synapse inhibits the neuron receiving the synaptic output from that synapse, potentially decreasing its membrane potential. Alternatively, the weight can be zero, which effectively removes the synaptic connection between the two neurons connected via the synapse. The weight for each synapse is stored in a memory cell associated with the synapse. The values of all the weights in the network are known as a weight matrix, and the weights are typically determined by a network training process.

[0028]

[0032] FIG. 2 is a simplified schematic diagram of synaptic connections implemented as a crossbar array. Crossbar designs are an efficient way to implement reconfigurable neural networks, especially when fabricated on integrated circuits. The design in FIG. 2 includes a rectangular array of synapses 17 used to interconnect two layers of an SNN; for example, synapses 17 connect neurons 10 on one side of the array to neurons 11 on the other side, or synapses 18 connect neurons 11 on one side of the array to neurons 12 on the other side. In the embodiment of FIG. 2, neurons 10 are arranged in a column, each driving a row of synapses 17. The synapses 17 are connected in a column, and the outputs of all synapses 17 in a column are summed together and serve as inputs to neurons 11. By programming appropriate weights into the synapse array and properly configuring the interconnection system, a wide variety of network topologies can be realized.

[0029]

[0033] In the spiking neural processor of FIG. 1 , input layer neurons 10 receive input signals 8 from input signal sources 6, such as sensors. Many such input signal sources 6 generate data at a relatively slow rate or in bursts with long periods between bursts. For example, a microphone may generate an analog output signal that is sampled at a certain frequency. The samples may then be converted to digital data by an analog-to-digital converter (ADC) or converted to a spike train output. This may be implemented, for example, using multiple outputs, each of which may transmit a spike when the sampled value falls within a certain value range for that output. For example, if a sampling rate of 1 kHz is used, a digital output value or corresponding spike output would be generated every millisecond.

[0030]

[0034] Other types of input signal sources 6 may generate output data at high speed but intermittently, for example, due to the nature of the variable or phenomenon being measured or sampled, or the function of the signal source.

[0031]

[0035] SNP1 operates at a much faster rate than many input signal sources, and during the intervals between receipt of input signals from input signal source 6, neurons in the neural processor may forget their previous state, i.e., their membrane potential may return to zero or a low value due to leakage. This loss of neural network state information may occur over a very short period of time, e.g., a fraction of a millisecond.

[0032]

[0036] Neurons could be designed to retain their membrane potential for longer periods of time, but this would slow down the operation of the neuron, increase the latency of the SNP, increase the size of the neuron, significantly increase the chip area and manufacturing cost of the SNP, and increase the power consumption of the SNP. Alternatively, memory could be used to store the membrane potential of the neuron during periods of inactivity. However, a large amount of memory would be required, which would in turn increase the chip area and manufacturing cost, as well as higher power consumption, and the size of the memory would be a constraint on the operation of the SNP for many applications.

[0033]

[0037] SNPs can address this problem by implementing a "forget and remember" strategy. Imagine reading a book and stopping at page 39 and placing a bookmark on that page. If you resume reading a week later, you won't remember all the details of the first 39 pages. Instead, you can read the last few lines of the bookmarked page to recall the context, and then continue reading from page 40. The "forget and remember" strategy is a similar strategy. Neurons in an SNN forget temporal network state information when there are long periods without input signals, for example, due to inactivity or low data rates of the input signal source. However, input signals are received and portions of neuron output signals generated during the last period the SNN was active can be recorded and stored in memory. This stored data can then be replayed during later periods of SNN activity, allowing the SNN to "remember" some of the network state information present during previous periods of activity, improving SNP inference accuracy.

[0034]

[0038] FIG. 3 shows a schematic diagram of the spiking neural processor of FIG. 1 with the addition of a storage unit 20 and an augmented input circuit 25. The augmented input circuit 25 is connected to the neurons of the SNP via an additional synapse. For example, in the embodiment shown in FIG. 3, the augmented input circuit 25 is connected via an additional synapse 17 to neuron 11 in layer 3 of the neural network, i.e., the same layer of neuron that receives a signal from input layer neuron 10 via synapse 17. However, other configurations are possible, such that the augmented input circuit 25 can be connected via a synapse to any of the neurons in the SNP. The number of augmented input circuits 25 may be configurable, and the synaptic connections of the augmented input circuits 25 to the neurons may be configurable, e.g., they may be configured following training of the neural processor.

[0035]

[0039] The augmented input circuits 25 may be neurons like the input layer neurons 10 that generate spikes at their output as a function of the received input signal and its current state, or may be simple pass-through circuits that pass the received input signal to their output, or other types of circuits suitable for receiving input from the storage unit 20 and providing outputs to synapses. The augmented input circuits 25 may be treated similarly to the neurons 10 of the input layer 2, and may be configured with synaptic connections to other neurons in the network just like the neurons 10, for example, fully connected to all neurons 11 in the next layer 3 of the SNN via synapses 17, or with sparse connections to other neurons.

[0036]

[0040] The number of augmented input circuits 25 is configurable by a network hyperparameter, i.e., a parameter whose value is used to control the learning process of the network during training and which in most cases has a fixed value throughout the training process.

[0037]

[0041] The storage unit 20 includes a memory 21, a delay circuit 22, and a control unit 23. The storage unit 20 receives neuron output signals from a selected subset of the SNN's neurons, referred to herein as "memory neurons." Output spikes from these memory neurons are used as a means of recording the SNN's recent spike history by recording the memory neuron outputs in the storage unit 20. This provides a means of storing information about the recent neural network state obtained during the SNN's recent active periods resulting from the SNN's receipt of input signals. By storing this neural network state information in the storage unit 20, the state information can be preserved for long periods of time, such as during inactive periods when the SNN does not receive any input signals and, as a result, neurons in the SNN lose their membrane potential due to leakage.

[0038]

[0042] For example, the SNN may be configured such that the memory unit 20 receives neuron output signals from all of the output neurons 12, e.g., all of the neurons in the last layer of the SNN. Other approaches may be used to select specific neurons from any layer of the SNN to contribute neuron outputs to the memory unit 20. For example, neurons that exhibit more dynamics, e.g., neurons whose weights are updated more frequently than other neurons during training, or neurons that exhibit a wider range of spiking patterns during inference, may be identified and selected as memory-neuron candidates. In the example shown in FIG. 3 , the memory unit receives neuron output signals from neurons in layer 3 and neurons in layer 4. The number of neurons selected to serve as memory-neurons may be equal to or different from the number of augmented input circuits 25.

[0039]

[0043] The storage unit 20 is configured to record and store the memory-neuron output signals 26 in the memory 21 under the control of the control unit 23 during periods when the input signals 8 are received by the SNPs. For example, the input neurons 10 may receive the input signals 8 in short bursts from the input signal source 6 or the input buffer 7, with longer intervening periods of inactivity during which the input signals 8 are not received. The control unit 23 is configured to coordinate the recording and storage of the memory-neuron outputs 26 with the bursts of input signals received by the SNPs' input neurons 10. The control unit 23 may start recording and storing the memory-neuron outputs 26 during the short bursts when the SNPs are receiving the input signals 8, and may stop recording the memory-neuron outputs 26 (to reduce the required storage capacity) during the intervening periods of inactivity during which the input signals 8 are not received by the SNPs (while maintaining data previously recorded in the memory 21 during the last burst of the input signals 8). This coordination may be performed by communication between the control unit 23 and the input buffer 7, as further described below.

[0040]

[0044] The storage unit 20 is configured to store data indicative of the received memory-neuron output signal 26 in the memory 21. For example, the storage unit 20 may store data regarding one or more parameters of any spike in the memory-neuron output signal 26. The amount of information about each recorded spike is configurable; for example, the storage unit 20 may store data regarding the spike occurrence time, spike amplitude, and / or spiking rate of the neuron output signal. In one embodiment, the storage unit 20 is configured to store data about each spike generated by each of the memory-neurons, i.e., any spike on the input 26 to the storage unit 20. For example, the storage unit 20 may record the identity of the neuron generating the spike and a time value indicating when the spike occurred. This data is preferably encoded to enable efficient storage of data and reduce the required size of the memory 21. For example, the storage unit 20 may record the neuron ID and relative time for each spike, where the relative time indicates the time difference from the start of the burst of data received by the SNN to the time the spike occurred. By efficiently encoding the spike data received by storage unit 20, SNP is able to efficiently record neural network state information while keeping the size of memory 21 in storage unit 20 to a minimum.

[0041]

[0045] The storage unit 20 is configured to output previously stored data during periods when input signals 8 are received by the input neurons 10. In this manner, the storage unit 20 provides previously recorded neural network state information to the extended input circuit 25 during periods when the input neurons 10 of the SNN are receiving further input signals. This allows the neural network to incorporate previous state information into its inferential processing of currently received input signals.

[0042]

[0046] For example, the control unit 23 may be configured to control the storage unit 20 to output data stored during a burst of the input signal 8 received by the SNP, where the data comprises at least a portion of the data stored during a previous burst of the input signal 8 received by the SNP. During a current burst of the input signal 8, the storage unit 20 may output all of the data recorded and stored during a previous burst of the input signal, or only a portion of that data.

[0043]

[0047] The extended input circuits 25 are connected to receive output from the storage unit 20. The extended input circuits 25 receive stored data output from the storage unit 20, but may additionally receive, for example, input signals 8 from the input buffer 7 and / or inputs from the input neurons 10. The extended input circuits 25 may be configured to generate spikes at their output as a function of the received input signals and their current state. The SNP may be configured to match the number of extended input circuits 25 to the number of memory-neurons whose neuron outputs 26 are recorded by the storage unit 20. In one embodiment, each extended input circuit 25 receives stored data of spikes generated by one of the memory-neurons. For example, each extended input circuit 25 may be connected to receive (from the storage unit 20) stored data derived from a particular one of the memory-neurons. Alternatively, the extended input circuits 25 may receive stored data of spikes generated by more than one memory-neuron, or stored data of spikes generated by more than one memory-neuron may be received by a single extended input circuit 25.

[0044]

[0048] Previously recorded data may be output by the storage unit 20 to preserve the timing of spikes recorded during a previous burst of the input signal 8. For example, the relative timing of spikes within an input signal burst may be preserved to be "played back" with the same relative timing during a current input signal burst. This allows the extended input circuit 25 to generate output spikes with the same timing within the current input signal burst to recreate the timing of spikes generated by the memory-neuron during a previous input signal burst.

[0045]

[0049] Alternatively, storage unit 20 may transform the stored data. For example, spike timing may be inverted, e.g., so that the last spike recorded during a previous input signal burst is output first during a current input signal burst. This may be done, for example, if the last spike generated by an output neuron is deemed most important for recreating the neural network state during the previous input signal burst.

[0046]

[0050] The storage unit 20 includes a memory 21, which may be a digital memory such as a DRAM, an SRAM, or a register memory. A shift register may be used, such as a FIFO (first-in, first-out) shift register for storing spike timing or a LIFO (last-in, first-out) shift register for inverting spike timing. The storage unit 20 also includes a delay circuit 22, which may be a separate memory, a logic circuit that controls the output from the memory 21, or may be included as part of the memory 21 (e.g., when the memory 21 is a shift register). The delay circuit 22 (in conjunction with the control unit 23) implements a predetermined delay in the output of data from the storage unit 20. This predetermined delay is selected to time the output of the stored data from the storage unit 20 to coincide with or precede the current input signal burst, as described further below. The storage unit 20 also includes a control unit 23, which controls the operation of the storage unit 20. The control unit 20 may be implemented as hardware logic, such as an ASIC or FPGA, or a processor executing software or firmware, or a combination of these.

[0047]

[0051] 3 also shows an optional input buffer 7, which may be used to buffer signals generated by signal source 6 and output accumulated input signals 8 in bursts to input layer neurons 10. Input buffer 7 may be included in input signal source 6, may be included in spiking neural processor 1, may be implemented as a separate unit, or may be omitted if not needed.

[0048]

[0052] In one embodiment, input buffer 7 and storage unit 20 are controlled to coordinate their functions. For example, storage unit 20 can be controlled to record memory-neuron output signal 26 during each burst period when input buffer 7 transfers input signal 8 to neuron 10. Storage unit 20 can be configured to record during the entire burst period or over a portion of the burst period, e.g., over a period at the end of the burst period. Furthermore, storage unit 20 can be controlled to output a previously stored memory-neuron output signal 26 to expansion input circuit 25 during each burst period when input buffer 7 transfers input signal 8 to neuron 10.

[0049]

[0053] The operation of the spiking neural processor of FIG. 3 will now be explained with reference to the example signal timeline shown in FIG.

[0050]

[0054] The first row of FIG. 4 is an illustrative timeline 30 of output data generated by signal source 6. Output data 30 may be in the form of, for example, analog values, digital values, or a sequence of one or more spike train signals. If signal source 6 generates analog or digital values, these are preferably converted to spike train signals suitable for input to the SNP at some point during the processing of the SNP input signals. For example, the analog or digital values may be converted to multiple spike train signals, where each spike train signal represents a particular range of values, and a spike is generated in one of the spike train signals when the analog or digital value is within the range of values for that spike train signal. In the following description, output data 30 is assumed to be in the form of spikes for simplicity, although the conversion to spikes may be performed at a later stage in the system.

[0051]

[0055] The output data (e.g., spikes) from signal source 6 may occur in fast bursts with relatively long time periods, e.g., 1 ms, between bursts. However, there may be long time periods between each of the spikes, or the spikes may occur intermittently with long time periods between some of them, or the spikes may occur in bursts but with relatively long time periods between spikes during a burst.

[0052]

[0056] The input buffer 7 may be used to accumulate output data 30 from the signal source 6 to generate fast bursts of spikes at regular periodic intervals separated by long periods between bursts. The input buffer 7 accumulates the output data 30 for a buffering period 34, e.g., 20 ms, and outputs the accumulated output data at regular intervals in short burst periods 35. The buffering period 34 and burst period 35 may be selected based on the SNP design and the inference application to be performed by the SNP. For example, in a memory-constrained system (e.g., an SNP implemented as an embedded system operating with only kilobytes of system memory for the input buffer 7 and memory 21), the maximum expected number of spikes per burst may be set to a smaller number to reduce memory requirements, resulting in a shorter buffering period 34 and burst period 35. However, the expected number of spikes per burst cannot be arbitrarily small because a sufficient number of spikes is required for the neuron to indicate a minimum activity level (to be stored for input to the extended input circuitry).

[0053]

[0057] The second row of Figure 4 is an illustrative timeline 31 of the output from the input buffer 7, which forms the input signal 8 to the spiking neural processor. During burst period 35, the accumulated data stored during buffering period 34 is transmitted to input neurons 10. If the raw signal is not in the form of a spike train signal, the output from the input buffer 7 may be converted to a spike train signal. The burst period 35 may be set to a predetermined period of time, which may be configurable, as described above. Following burst period 35, the input buffer 7 stops outputting data during period 36 until the next burst period, while data is buffered for the next input signal burst.

[0054]

[0058] During input signal burst 40A, input buffer 7 outputs data stored as input signal 8 to input neuron 10. Input neuron 10 generates neuron output signal 14 that is passed via synapse 17 to hidden neuron 11, which generates neuron output signal 15 that is passed via synapse 18 to output neuron 12, which generates neuron output signal 16 (SNP output 9).

[0055]

[0059] The third row of Figure 4 is an illustrative timeline 32 of spiking neuron outputs generated by selected memory-neurons in SNP, which outputs are recorded in storage unit 20. Memory-neuron outputs 41A are recorded by storage unit 20 during bursts 40A of input signal 8. Storage unit 20 stores data indicative of memory-neuron output signals 26 received during this period.

[0056]

[0060] The fourth row of Figure 4 is an illustrative timeline 33 of output from storage unit 20, which becomes input to expansion input circuit 25. During burst 40A of input signal 8 and during recording of memory-neuron output signal 41A, there is no output from storage unit 20 and no input to expansion input circuit 25. The next burst 40B of input signal 8 occurs after another buffering period, as shown in timeline 31. This results in memory-neuron output signal 41B, shown in timeline 32. During input signal burst 40B, storage unit 20 outputs stored data 42A of memory-neuron output signal 41A recorded during the earlier input signal burst 40A. As shown in timeline 33, the stored data is output 42A from storage unit 20 after a predetermined delay 37 following recording of memory-neuron output signal 41A generated during the earlier input signal burst 40A. Similarly, during the next input signal burst 40C, the storage unit 20 outputs the stored data 42B of the memory-neuron output signal 41B that was recorded during the earlier input signal burst 40B.

[0057]

[0061] 4, the output of stored data from storage unit 20 is timed to coincide with the next input signal burst (e.g., stored data output 42A coincides with input signal burst 40B). However, storage unit 20 may instead be configured to output stored data at a different timing, for example, during the time period immediately prior to the next input signal burst (e.g., output 42A may be output from storage unit 20 immediately prior to input signal burst 40B). This configuration may be used to at least partially recreate the neural network state present at the end of input signal burst 40A, immediately prior to receiving a further input signal during input signal burst 40B.

[0058]

[0062] Training of the SNP's spiking neural network can be accomplished using labeled data supplied to the SNP in bursts. Prior to training, the augmented input circuit 25 is configured and memory neurons are identified. The training data is divided into smaller data sets (corresponding to input signal bursts), and artificial delays are inserted between the data sets so that the training mimics the inference environment (SNNs are largely trained to work on specific data sets in specific environments). During training, the SNN learns presynaptic weights for both the regular input neurons 10 and the augmented input circuit 25. This is accomplished through training the network by feeding it training data sets; the network "learns" the weights during the training process.

[0059]

[0063] A simulated benchmark test was performed on an SNN using the data storage and feedback method described herein. To experimentally examine the effectiveness of this technique, we used the publicly available Spiking Heidelberg Digits (a speech-based classification dataset of spoken digits 0-9 converted into spike trains). A dataset with 100 spike vectors was used to train a spiking neural network with 700 input neurons, 400 neurons in the second hidden layer, and 20 output neurons using a surrogate gradient descent training technique. Training was performed with various burst sizes ranging from no bursts to 20, 10, and 8 spikes per burst. The memory capacity (for the input signal buffer and memory-neuron storage unit) required for each burst size is estimated in the table below, where memory capacity is measured in terms of the number of spikes to be stored.

[0060] [Table 1]

[0061]

[0064] Figure 5 shows the performance profile of a spiking neural network derived from testing. The horizontal axis indicates the number of training epochs, and the vertical axis indicates an estimate of the network's performance based on inference accuracy. The solid blue line A shows the performance when data (spiking Heidelberg digits) are input to the SNN without being subdivided into memory-intensive bursts. The dashed lines B1, B2, and B3 show the network performance for bursts of 8, 10, and 20 spikes per burst, respectively, but without feedback of recorded memory-neuron data. The solid lines C1, C2, and C3 show the network performance for bursts of 8, 10, and 20 spikes per burst, respectively, with feedback of recorded memory-neuron data provided to the extended input circuit.

[0062]

[0065] This test shows that when the dataset is received by the SNN in bursts, especially small bursts (dashed line), the network performance degrades significantly. However, this performance degradation is compensated when feedback of memory-neuron outputs to the extended input circuit is provided (solid line).

[0063]

[0066] The systems and methods described herein are particularly useful in resource (e.g., memory and data bandwidth) constrained systems, such as embedded systems, that require storing large amounts of data from slow or intermittent sensor or other signal sources for processing in high-speed neuromorphic systems. The systems and methods described herein significantly reduce the amount of memory required to store input signal data and also reduce accuracy loss because the SNN can be enabled to "forget" temporal network state information between sensor samples and is assisted in "remembering" lost information by providing it with a snapshot of the previous neural network state.

Claims

1. A spiking neural processor (1) configured to receive one or more input signals (8) and generate one or more inferential output signals (9), said spiking neural processor comprising: a plurality of neurons (10, 11, 12) interconnected by a plurality of synaptic elements (17, 18) to form a spiking neural network, wherein a portion of the neurons (10) are connected to receive the one or more input signals (8), and each of the neurons (10, 11, 12) is configured to generate a neuron output signal; a storage unit (20) connected to receive one or more of said neuron output signals (26) from a selected subset of said neurons (10, 11, 12); one or more augmented input circuits (25) connected by said synaptic elements (17) to selected ones of said neurons (11, 12) of said spiking neural network; the storage unit (20) is configured to store data indicative of the received neuron output signal (26) and to output at least a portion of the stored data after a predetermined delay (37); The spiking neural processor, wherein the extended input circuit (25) is connected to receive the stored data (27) output by the storage unit (20) after the predetermined delay (37).

2. 2. The spiking neural processor of claim 1, wherein the storage unit is configured to store the data indicative of the received neuron output signal during periods when the input signal is received by the spiking neural network, and not to store the data during periods when the input signal is not received by the spiking neural network.

3. 3. The spiking neural processor of claim 1, wherein the storage unit is configured to output at least a portion of the stored data during a subsequent period in which the input signal is received by the spiking neural network, and to not output the stored data during a period in which the input signal is not received by the spiking neural network.

4. 4. The spiking neural processor of claim 3, wherein the data output by the storage unit (20) during the subsequent period comprises at least a portion of the data stored during the period immediately prior to the time the input signal (8) was received by the spiking neural network.

5. 5. The spiking neural processor of claim 1, wherein the storage unit is configured to store data encoding spike times, spike amplitudes, and / or spiking rates of the neuron output signals from the selected subset of neurons.

6. 6. The spiking neural processor of claim 1, wherein operation of the storage unit is coordinated with an input buffer circuit so that the storage unit records the data indicative of the received neuron output signal during burst periods when the input buffer circuit forwards the input signal to the spiking neural network, and does not record the data during periods when the input buffer does not forward the input signal to the spiking neural network.

7. 6. The spiking neural processor of claim 1, further comprising an input buffer circuit (7) connected to receive one or more signals from an input signal source (6), the input buffer circuit (7) configured to accumulate the received signals over a buffering period (34) and output the accumulated signals as the input signals (8) to the spiking neural network during a burst period (35).

8. 8. The spiking neural processor of claim 7, wherein the buffering period (34) of the input buffer circuit (7) is coordinated with the predetermined delay (37) of the storage unit (20).

9. 9. The spiking neural processor of claim 7, wherein the storage unit is configured to output the stored data during a period when the input buffer circuit outputs the accumulated signal to the spiking neural network.

10. 10. The spiking neural processor of claim 7, wherein the burst period (35) is at least ten times shorter than the buffering period (34), and the input buffer circuit (7) is configured to output the accumulated signal on a compressed time scale compared to the signal received from the input signal source (6).

11. 1. A method of operating a spiking neural processor (1) configured to receive one or more input signals (8) and generate one or more inferred output signals (9), the spiking neural processor comprising a plurality of neurons (10, 11, 12) interconnected by a plurality of synaptic elements (17, 18) to form a spiking neural network, each of the neurons (10, 11, 12) configured to generate a neuron output signal, the method comprising: connecting one or more augmented input circuits (25) to selected ones of the neurons (11, 12) of the spiking neural network by the synapse elements (17); receiving the one or more input signals (8, 40A) by a portion of the neurons (10); receiving, by a storage unit (20), one or more of the neuron output signals (26, 41A) from a selected subset of the neurons (10, 11, 12); storing data indicative of the received neuron output signals (26, 41A) in the storage unit (20); outputting at least a portion of the stored data (27, 42A) from the storage unit (20) after a predetermined delay (37); receiving, by the expansion input circuit (25), the stored data (27, 42A) output by the storage unit (20) after the predetermined delay; A method comprising:

12. 12. The method of claim 11, wherein the storing of the data indicative of the received neuron output signals is performed during the period in which the input signals (8) are received by the spiking neural network.

13. 13. The method of claim 11 or 12, wherein the outputting of at least a portion of the stored data from the storage unit (20) is performed during a subsequent period in which the input signal (8) is received by the spiking neural network.

14. 14. The method of claim 11, further comprising coordinating operation of the storage unit (20) with an input buffer circuit (7), so that the storage unit (20) stores the data indicative of the received neuron output signal (26) during burst periods (35) when the input buffer circuit (7) forwards the input signal (8) to the spiking neural network, and does not record the data during periods (36) when the input buffer (7) does not forward the input signal (8) to the spiking neural network.

15. 14. The method of claim 11, further comprising connecting an input buffer circuit (7) to receive one or more signals from an input signal source (6), accumulating the received signals over a buffering period (34), and outputting the accumulated signals as the input signals (8) to the spiking neural network during a burst period (35).

Images