Neural Devices for Neural Network Systems

Abstract

A neural device for a neural network system may be configured to receive, during a decoding period, one or more input signals, decode, during the decoding period, the one or more input signals resulting in decoded signals, and, upon completion of the decoding period, process the decoded signals using internal neuron dynamics. The processed signals may be used to encode and emit one or more output signals to another neural device of the neural network system in a subsequent decoding period.

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer systems, and more particularly to neural devices for neural network systems. [Background technology]

[0002] A neural network is a computational model used in artificial intelligence systems. A neural network may be based on multiple artificial neurons. Each artificial neuron may be connected to one or more other artificial neurons, and these connections ("links") may reinforce or inhibit the firing state of adjacent neurons. Artificial neurons in a spiking neural network (SNN) may be provided with a firing threshold, which the neuron's membrane potential must exceed to generate a spike. This thresholding may be a component of artificial spiking neurons, enabling energy-efficient sparse communication using spikes. Summary of the Invention

[0003] Various embodiments result in neural devices, methods, and computer program products for neural network systems as described by the independent claims. Advantageous embodiments are described in the dependent claims. The embodiments of the present disclosure may be freely combined with each other if they are not mutually exclusive.

[0004] In one aspect, some embodiments of the present disclosure relate to a neural device ("sending device") for a neural network system. The neural device may be configured to receive one or more input signals during a decoding period, decode the one or more input signals during the decoding period to result in decoded signals, process the decoded signals using a model representing internal neural dynamics when the decoding period ends, and use the processed signals to encode and emit one or more output signals to another neural device ("receiving device") of the neural network system in a subsequent decoding period.

[0005] In other aspects, some embodiments of the present disclosure relate to a method for a neural device, which may include receiving one or more input signals during a decoding period, decoding the one or more input signals during the decoding period to result in decoded signals, processing the decoded signals using a model representing internal neural dynamics upon completion of the decoding period, and encoding and emitting one or more output signals to another neural device of a neural network system in a subsequent decoding period using the processed signals.

[0006] In another aspect, some embodiments of the present disclosure relate to an artificial neural network system having multiple layers, where at least one of the multiple layers may comprise one or more neuron realizations, including a neuron device according to the aforementioned embodiments. For example, each neuron of the artificial neural network system may be a neuron device according to the aforementioned embodiments.

[0007] In another aspect, some embodiments of the present disclosure relate to a computer program product comprising a computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code being configured to perform a method according to the aforementioned embodiments.

[0008] In the following, by way of example only, embodiments of the present disclosure will be described in more detail with reference to the drawings, in which: [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a neuron device consistent with certain embodiments of the present disclosure. [Figure 2] FIG. 10 illustrates an example implementation of an internal neuron dynamics unit, consistent with certain embodiments of the present disclosure. [Figure 3] 1 is a flowchart of a method for processing an input signal by a neural device, consistent with some embodiments of the present disclosure. [Figure 4] FIG. 10 illustrates an example time-dependent function for determining a modulation value, consistent with certain embodiments of the present disclosure. [Figure 5A] FIG. 10 illustrates an example time-dependent function for determining a modulation value, consistent with certain embodiments of the present disclosure. [Figure 5B] FIG. 10 illustrates an example time-dependent function for determining a modulation value, consistent with certain embodiments of the present disclosure. [Figure 6] 1 is a flowchart of a method for generating an output value by an internal neuron dynamics unit, consistent with some embodiments of the present disclosure. [Figure 7] FIG. 1 illustrates a method for decoding and encoding a signal using two neural devices, consistent with some embodiments of the present disclosure. [Figure 8A] 1 is a graph showing the test accuracy of an example SNN, consistent with certain embodiments of the present disclosure. [Figure 8B]1 is a graph illustrating the average number of spikes produced by an example SNN, consistent with certain embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] The description of various embodiments of the present disclosure is presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to help explain the principles of the embodiments, practical applications, or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0011] Some embodiments of neural devices may result in multi-timescale spiking neurons, in which different portions of the neuron operate at different timescales. Some embodiments may enhance the operation of neuromorphic spike-based devices by efficiently controlling the spiking capabilities of the neuromorphic spike-based devices. Specifically, some embodiments may utilize input timing information, which may have theoretically attractive capabilities for transmitting information more efficiently. This may improve accuracy compared to standard spiking neurons and improve efficiency compared to state-of-the-art rate coding. Furthermore, this may enable end-to-end trainable SNNs using approximation-free backpropagation through time (BPTT) techniques. In this way, some embodiments may improve the spiking process of artificial spiking neurons to enhance the sending and receiving of rich and accurate information.

[0012] For example, processing the decoded signal with a model may enable image processing, e.g., for handwritten digit recognition. The model f, which indicates (or represents) internal neuronal dynamics, may be a model of the internal neuronal dynamics of a neural device. The model may be, e.g., a model of somatic computation. The model may be, e.g., a stateful model. The model may use, e.g., one or more activation functions and an internal state, e.g., representing the membrane potential of the neural device. In one example, the model may be an activation function or a combination of multiple activation functions.

[0013] The neural network system may be configured to implement / execute an artificial neural network, such as an SNN. In one example, the neural devices may be implemented as neurons of the artificial neural network. For example, the artificial neural network may include multiple layers, each layer including neurons, and each neuron of the neurons may be a neural device.

[0014] According to some embodiments, each signal of the input and output signals may encode information in the signal's arrival time at the respective device. Each signal of the input and output signals may be a spike. The spike may have a value of, for example, zero or one. Each signal of the input signals may encode information in the signal's arrival time at the sending device. The input signals may be received, for example, from one or more feedforward connections to the neural device, or from one or more recurrent connections to the neural device, or both. The neural device may be connected to other neural devices in the neural network system through feedforward connections, recurrent connections, or both. Each signal of the output signals may encode information in the signal's arrival time at the receiving device. The received signals may be variable values, such as zero-one normalized real values. For example, the values ​​may indicate the content of a pixel in an image.

[0015] According to some embodiments, the device may be configured to decode input signals by: a) upon receiving one of the input signals, determining, for the input signal, a modulation value corresponding to the arrival time of the input signal; b) weighting the received signal by the determined modulation value; c) integrating the weighted signal with a current value of an input state of the device; and d) repeating operations a) to c) for each received input signal during a decoding period, where the decoded signal may be the integrated value. The input state may, for example, be (re)initialized to a given value, e.g., zero, for each decoding period.

[0016] For example, each input signal of the input signals received in operation a) can be a signal that can be weighted by a synaptic weight associated with the signal, e.g., a signal can pass through a synaptic unit having synaptic weight w, and the signal can be weighted by synaptic weight w, resulting in the input signal received in operation a).

[0017] According to some embodiments, the device can be configured to determine the modulation values ​​as values ​​of an input function at the arrival times, where the input function is a time-dependent piecewise linear function. These embodiments can enhance the ability of the input unit to distinguish between signal sequences. For example, if the same set of signals is received multiple times, and the signals are received at different times and in a different order each time, this can result in different decoded signals.

[0018] According to some embodiments, the input function may have a value that decreases with increasing arrival time values.

[0019] According to some embodiments, the device may be further configured to determine the modulation value as the value of an input function at the arrival time, the input function being a time-dependent non-linear function involving a predefined range of values. This embodiment may be advantageous since, compared to a piecewise linear function defined for a given decoding period, the range of values ​​can be increased and the resolution can be maintained through the emission of further pulses with increasingly finer contributions.

[0020] According to some embodiments, the device may be configured to encode and emit one or more output signals using an output function. The output function may be configured to provide a value for each time point within a subsequent decoding period. The device may be further configured to determine one or more values ​​of the output function such that a combination (e.g., a sum) of the determined values ​​represents a value of the processed signal, providing one or more output signals at the time points corresponding to the determined values.

[0021] According to some embodiments, the output function may be a time-dependent piecewise linear function or a time-dependent non-linear function.

[0022] According to some embodiments, the output function may be a linear function, an exponential function, or a step function.

[0023] According to some embodiments, the apparatus may include an input unit, an internal neuron dynamics unit, and an output unit. The input unit may be configured to receive and decode one or more input signals. The internal neuron dynamics unit may be configured to process the decoded signals, and the output unit may be configured to encode and emit one or more output signals. The internal neuron dynamics unit may be configured to implement a model representing the internal dynamics.

[0024] Each of the input units, the internal neuronal dynamics unit, and the output unit may use its kernel or dynamics to perform its respective operations. The input unit may be referred to as a dendritic unit having a dendritic kernel. The output unit may be referred to as an axon unit having an axon kernel. The internal neuronal dynamics unit may process the decoded signal (e.g., for each successive processing operation) according to a model of the internal neural dynamics. Without limiting the scope of the internal neural dynamics, further description utilizes illustrative examples of models that may be nonlinear, time-varying, and involve internal states. The present disclosure may balance the fast-acting characteristics of the dendritic and axon kernels with the slow-acting characteristics of the internal neural dynamics. The fast-acting dendritic and axon kernels may enable information transmission using timing. The slow-acting internal neural dynamics may utilize more precise input and output values. The model f function may be a differentiable function. Differentiable functions may allow end-to-end training with BPTT.

[0025] Some embodiments may be advantageous because they may result in simple neuromorphic hardware implementations. For example, the same simple dendritic and axonal kernels may be implemented as lookup tables (LUTs), oscillators, or capacitors and shared by multiple neurons. The coding may be adapted through LUT reprogramming (e.g., based on hardware limitations, task requirements, desired accuracy, etc.) without changes in the neuron design. For example, longer kernels may result in higher accuracy. Kernels may result in different energy / latency tradeoffs, such as linear or exponential. Kernels may be robust to jittery spike temporal positions, e.g., staircase-shaped.

[0026] According to some embodiments, the input unit may be configured to decode one or more input signals using an input function (or dendritic kernel), and the output unit may be configured to encode and emit one or more output signals using an output function (or axon kernel).

[0027] According to some embodiments, the output function of the sending device and the input function of the receiving device can be different or the same function, which may allow, for example, the axon kernel of the sending device to correspond to the dendritic kernel of the receiving device so that they can communicate values ​​accurately.

[0028] According to some embodiments, the internal neuron dynamics unit may comprise an accumulation block and an output generation block. The internal neuron dynamics unit may have current state variables corresponding to one or more previously received signals. The output generation block may be configured to use an activation function to generate a current output value based on the current state variables. The accumulation block may include: Calculating an adjustment to the current state variable using the current output value and a correction function that describes the decay behavior of the time constant of the internal neuron dynamics unit; Receiving a current signal; updating current state variables using the calculated adjustments and the received signals, where the updated state variables become current state variables; · Have the output generation block generate the current output value based on the current state variables. The method may be configured to repeatedly perform the above steps. This embodiment may result in the desirable implementation of models of internal neural dynamics.

[0029] According to some embodiments, the accumulation block may be configured to perform updates using an activation function that is different from the activation function of the output generation block.

[0030] According to some embodiments, the accumulation block may be configured to receive a reset signal from the output generation block indicating the current output value used for the adjustment calculation.

[0031] According to some embodiments, the output generation block may be configured, upon generating a current output value, to automatically provide a reset signal to the storage block indicating the current output value to be used for the calculation of the adjustment.

[0032] According to some embodiments, the output generation block may be configured to provide a reset signal indicating the current output value used for the adjustment calculation through a reset gate connecting the accumulation block and the output generation block.

[0033] 1 illustrates an example of a neural device 100 consistent with some embodiments of the present disclosure. The neural device 100 may be implemented using a neuromorphic hardware implementation. For example, the neural device may be a memristor-based circuit to implement at least a portion of the present disclosure. The neural device 100 may alternatively be implemented using, for example, analog or digital CMOS circuitry.

[0034] The neural device 100 may include an input unit 102, an internal neuron dynamics unit 103, and an output unit 104. The neural device 100 may receive and process incoming signals or values. The input unit 102 receives a signal or value during a decoding period p k A set of one or more spikes present in

number

number

number

number

number

number

number

number

[0035] Decryption period p k When this is completed, the internal neuronal dynamics unit 103 receives the somatic input

number

number

number

number

number

[0036] The output unit 104 outputs the axon kernel to the somatic output

number

number

number

number

number

number

number

number

[0037] 2 illustrates another example implementation of the internal neuron dynamics unit 103, consistent with some embodiments of the present disclosure. k Signal between

number

number

number

[0038] The internal neuron dynamics unit 103 may comprise an accumulation block 201 and an output generation block 203. The accumulation block 201 may comprise a summing circuit 210, a multiplication circuit 211, and a start-up circuit 212. The multiplication circuit 211 may be, for example, a reset gate. The accumulation block 201 may be configured to output the calculated state variables in parallel to the output generation block 203 and the multiplication logic 211 at a branch point 214. The connection 209 between the branch point 214 and the multiplication logic 211 is shown with a dashed line to indicate that the connection 209 has a time delay. That is, the internal neuron dynamics unit 103 may generate a somatic input signal from the received somatic input signal.

number

number

number

number

[0039] The output generation block 203 may comprise a wake-up logic 215. The output generation block 203 may be configured to receive state variables from the accumulation block 201. Based on the received state variables, the output generation block 203 may generate an output value and provide or output it in parallel to another neural device and to a reset module 207 of the internal neuron dynamics unit 103 at a branch point 217. The reset module 207 may be configured to generate a reset signal from the received output value and provide the reset signal to the multiplication logic 211. For example, a given output value

number

number

number

number

number

number

[0040] State variable values

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0041] 3 is a flowchart of a method for processing an input signal by a neural device, such as 100, consistent with some embodiments of the present disclosure. For ease of explanation, FIG. 3 will be described with reference to the simplified example of FIG. 4, although the present disclosure is not limited to this example.

[0042] The neural device (operation 301) performs a decoding period p k During this time, the input signal

number

number

number

number

number

number

number

[0043] The neural device (at operation 303)k During this time, the input signal

number

number

number

number

number

number

number

[0044] Decryption period p k Once this is complete, the neural device (at operation 305) generates the decoded signal

number

number

number

number

number

number

number

[0045] Processed Signal

number

number

number

number

[0046] FIG. 5A illustrates an example time-dependent function (also named a kernel function) for determining modulation values, consistent with certain embodiments of the present disclosure.

[0047] The time-dependent function 501 is a function of the decoding period p kThe function 501 may be a linear function with a maximum value of 1000 and a minimum value of 100 within the decoding period p. A linear function may provide a simplified implementation and reliable results. As shown in FIG. 5A, ten values ​​of the function 501 may be provided. The ten values ​​may be associated with ten respective time slots. For example, for a received value 545, time slot number 6 may be identified as the time corresponding (closest) to value 545. Therefore, one spike may be emitted within time slot number 6. However, that time slot may be associated with value 500, which may result in an error of 45 with respect to the actual value of the input signal. To improve this, the time-dependent function 510 of FIG. 5B may be used. The time-dependent function 510 is a function of the decoding period p k , may be an exponential kernel with a maximum value of 512 and a minimum value of 1. The exponential kernel is

number

[0048] FIG. 6 illustrates an internal neuron dynamics unit 103 with an accumulation block 201 and an output generation block 203 generating output values, consistent with some embodiments of the present disclosure.

number

number

number

number

number

number

number

[0049] In operation 601, the storage block 201 stores the current state variable

number

number

number

[0050] After calculating the adjustment, the storage block 201 calculates the current signal

number

number

[0051] At operation 605, the current state variable

number

number

number

number

number

[0052] In operation 607, the storage block 201 transmits the current output value to the output generation block 203.

number

number

[0053] Operations 601-607 are performed for each subsequent received signal.

number

number

number

number

number

number

number

[0054] 7 is a diagram illustrating a method for decoding and encoding a signal using two neural devices 701 and 702, consistent with some embodiments of the present disclosure. Neural device 701 may be termed a sending device, and neural device 702 may be termed a receiving device. Neural device 701 receives temporally encoded values

number

number

number

number

number

number

number

number

number

number

number

[0055] Output spike X within decoding period p2 L2 (t) can be received at another neural device 702. As in the case of neural device 701, for example, input spikes X from any neural device connected in a feedforward manner. L2,k (t) together with the input spikes X received from the neural device 701 L2 (t) and input spikes X from any neural device connected in a recurrent manner. L3,l (t) can be decoded by the input unit 706 of the neural device 702 as follows:

number

number

number

number

number

number

number

number

number

[0056] 8A and 8B are graphs showing test accuracy and spike counts of an example SNN consistent with some embodiments of the present disclosure. The SNN in this embodiment may include neurons, each of which may be a neural device according to some embodiments of the present disclosure. Each of the neurons may use a piecewise linear kernel as an input and output function. The SNN may have three layers, with the input layer having 784 inputs and two successive layers having 250 and 10 neural devices, respectively, consistent with some embodiments of the present disclosure. A baseline SNN is used for comparison purposes. The baseline SNN receives inputs encoded using a common-rate coding technique. The MNIST image dataset is used for testing. As shown in FIG. 8A, an SNN consistent with some embodiments may achieve 97.8% accuracy (as shown by curve 801), higher than the 97.68% accuracy of the baseline SNN (as shown by curve 802). Also, the average number of spikes per epoch, per neuron, and per image of an SNN consistent with some embodiments may be significantly smaller than that of a baseline SNN, which may allow for higher code efficiency and fewer spikes.

[0057] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0058] The present disclosure may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform aspects of the present disclosure.

[0059] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved ridge structures having instructions recorded thereon, and any suitable combination thereof. As used herein, computer-readable storage media should not be construed to include ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted over wires.

[0060] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may comprise copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device may receive the computer-readable program instructions from the network and forward the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0061] The computer-readable program instructions for carrying out the operations of the present disclosure may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages ​​such as Smalltalk®, C++®, and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute as a stand-alone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection to an external computer may be made (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the present disclosure.

[0062] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0063] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, can perform the functions / acts / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions that implement aspects of the functions / acts / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0064] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operations on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0065] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified function(s) or action(s), or executes a combination of dedicated hardware and computer instructions.

Claims

1. A neural device for a neural network system, comprising: receiving one or more input signals during a decoding period; decoding the one or more input signals during the decoding period, resulting in a decoded signal; and Once the decoding period is complete, processing the decoded signal with a model representing the internal neural dynamics; using the processed signal to encode and emit one or more output signals to another neural device of the neural network system in a subsequent decoding period; 1. An apparatus configured to:

2. 2. The apparatus of claim 1, wherein each of the input and output signals is encoded with information at the signal's arrival time at the respective apparatus, and wherein the same signal can have different information encoded therein depending on its arrival time.

3. The device comprises: a) upon receiving said one or more input signals, determining, for said input signals, modulation values ​​corresponding to arrival times of said input signals; b) weighting the received signal by the determined modulation value to produce a weighted signal; c) integrating the weighted signal with current values ​​of input states of the device to produce an integrated value; d) repeating operations a) to c) for each received input signal during the decoding period, wherein the decoded signal is the integrated value; and 2. The apparatus of claim 1, configured to decode the input signal by:

4. The apparatus of claim 3 , wherein the apparatus is configured to determine the modulation value as a value of an input function at the arrival time, the input function being a time-dependent piecewise linear function.

5. The apparatus of claim 4 , wherein the input function has a value that decreases with increasing values ​​of the arrival time.

6. 4. The apparatus of claim 3, wherein the apparatus is configured to determine the modulation value as a value of an input function at the arrival time, the input function being a time-dependent non-linear function involving a predefined range of values.

7. the apparatus is configured to encode and emit the one or more output signals using an output function, the output function configured to result in a value for each time point within the subsequent decoding period; the apparatus is configured to determine one or more values ​​of the output function such that a combination of determined values ​​represents a value of the processed signal, resulting in the one or more output signals at times corresponding to the determined values.

10. The apparatus of claim 1.

8. The apparatus of claim 7 , wherein the output function comprises a time-dependent piecewise linear function or a time-dependent nonlinear function.

9. The apparatus of claim 8 , wherein the output function is a linear function, an exponential function, or a step function.

10. 10. The apparatus of claim 1, comprising: an input unit, an internal neuron dynamics unit, and an output unit, the input unit configured to receive and decode the one or more input signals, the internal neuron dynamics unit configured to process the decoded signals, and the output unit configured to encode and emit one or more output signals.

11. 11. The apparatus of claim 10, wherein the input unit is configured to decode the one or more input signals using an input function, and the output unit is configured to encode and emit one or more output signals using an output function.

12. 12. The apparatus of claim 11, wherein the other neural device is configured to operate as the neural device, and the output function of the neural device and the input function of the other neural device are different or the same function.

13. the internal neuron dynamics unit comprises an accumulation block and an output generation block, the internal neuron dynamics unit having current state variables corresponding to one or more previously received signals, the output generation block being configured to use an activation function to generate a current output value based on the current state variables; The storage block comprises: calculating an adjustment to the current state variable using the current output value and a correction function that describes the decay behavior of the time constant of the internal neuron dynamics unit; receiving a current signal; updating the current state variables using the calculated adjustments and the received signals, the updated state variables becoming the current state variables; causing the output generation block to generate the current output value based on the current state variable; The apparatus of claim 10 , configured to repeatedly perform

14. The apparatus of claim 13 , wherein the accumulation block is configured to perform the updating using an activation function that is different from the activation function of the output generation block.

15. 14. The apparatus of claim 13, wherein the storage block is configured to receive a reset signal from the output generation block indicating the current output value used for the calculation of the adjustment.

16. 14. The apparatus of claim 13, wherein the output generation block is configured, upon generating the current output value, to automatically provide a reset signal to the storage block indicating the current output value used for the calculation of the adjustment.

17. 14. The apparatus of claim 13, wherein the output generation block is configured to provide a reset signal indicative of the current output value used for the calculation of the adjustment through a reset gate connecting the accumulation block and the output generation block.

18. 1. A method for a neural device, comprising: receiving one or more input signals during a decoding period; decoding the one or more input signals during the decoding period, resulting in a decoded signal; and Once the decoding period is complete, processing the decoded signal using a model representing internal neuron dynamics; using the processed signal to encode and emit one or more output signals to another neural device of the neural network system during a subsequent decoding period; A method comprising:

19. 20. A computer readable storage medium having computer readable program code embodied thereon, said computer readable program code being configured to implement the method of claim 18.

Images