Reservoir nodes-enabled neuromorphic vision sensing network

Abstract

Event-based cameras are inspired by the sparse and asynchronous spike representation of the biological visual system. However, processing the event data requires either using expensive feature descriptors to transform spikes into frames, or using spiking neural networks that are expensive to train. This disclosure proposes a neural network architecture, Reservoir Nodes-enabled neuromorphic vision sensing Network (RN-Net), based on simple convolution layers integrated with dynamic temporal encoding reservoirs for local and global spatiotemporal feature detection with low hardware and training costs. The RN-Net allows efficient processing of asynchronous temporal features, and achieves the highest accuracy of 99.2% for DVS128 Gesture reported to date, and one of the highest accuracy of 67.5% for DVS Lip dataset at a much smaller network size. By leveraging the internal device and circuit dynamics, asynchronous temporal feature encoding can be implemented at very low hardware cost without preprocessing and dedicated memory and arithmetic units.

Description

RESERVOIR NODES-ENABLED NEUROMORPHIC VISION SENSING NETWORKGOVERNMENT CLAUSE

[0001] This invention was made with government support under 1915550 and 1900675 awarded by the National Science Foundation. The government has certain rights in the invention.CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 527,366, filed on July 18, 2023. The entire disclosure of the above application is incorporated herein by reference.FIELD

[0003] The present disclosure relates to reservoir node enabled network.BACKGROUND

[0004] Event-based cameras are neuromorphic vision sensors that produce visual signals as asynchronous spikes. An event camera produces a spike when and only when a momentary pixel intensity difference exceeds a threshold, which can offer better energy efficiency and latency when compared with conventional cameras that produce data at a constant frame rate even when the scene is stationary during video recording.

[0005] Various visual tasks have been implemented using event-based cameras. Earlier datasets were generated by reproducing conventional image classification datasets such as MNIST, CIFAR10, and Caltech O, where stationary images were placed in front of the camera and moved around to create pixel intensity differences along time. Many networks have been proposed for and performed well on these datasets where the object does not actually change over time. Behaviorally dynamic datasets, such as DVS128 Gesture and DVS Lip, were later created to maximize the event camera’s strength where the temporal evolutions of both the object’s shape and movement are critical.

[0006] To process these behaviorally and morphologically more dynamic event datasets, deep neural networks (DNNs) consisting of convolution and fully connected layers, and spiking neural networks (SNNs) have been adopted. Given that DNNs arespecialized in processing static data, recurrent units, such as long short-term memory (LSTM), gated recurrent unit (GRU) and bidirectional gated recurrent unit (BiGRU) are typically required to process temporally sequential (global) information hidden in the series of momentary features. The features are processed synchronously in temporally local frames which are created either by simple accumulation of spikes within a prefixed time range or using input representation algorithms like Graph construction, 3D point cloud, Event frame, Event spike tensor and voxel grid. Synchronized processing requires storing and analyzing a large number of events as a pre-processing step, which diminishes the advantages of asynchronous and sparse spike generation features of event-based cameras. Recurrent units require storing multiple state data for each node and increase the training cost.

[0007] SNNs store temporal information in the neuron dynamics using models such as leaky integrate-and-fire (LIF) neurons and can be trained using gradient-based approaches such as backpropagation through time (BPTT). To address the nondifferentiability of spikes, surrogate approaches that replace spikes with neuron membrane potential have been developed. However, BPTT requires backpropagation through both the network layers and through time, and is expensive to train.

[0008] To use spikes directly, feature descriptors such as Time Surface (TS) were proposed. TS stores only the last spike event for every pixel, and converts time to the last spike information into an analog value. The analog surface after TS conversion allows the encoded data to be processed with DNN and trained using gradient-based training. However, since TS only stores the last spike, it lacks capability to handle spatiotemporal features whose correlation is beyond its temporal neighbors, which is common in real-world problems. More advanced approaches, such as Leaky Surface, were subsequently developed to encode the temporal information beyond the last spike. However, expensive pre-processing is still required, for example, to store the previous node state and time elapsed from the latest spike, and to calculate the new state for every pixel at every time instant.

[0009] This disclosure introduces a neural network architecture, Reservoir Nodes- enabled neuromorphic vision sensing Network (RN-Net), that uses simple reservoir node (RN) layers along with DNN blocks for efficient processing of asynchronous data with low hardware implementation and training costs.

[0010] This section provides background information related to the present disclosure which is not necessarily prior art.SUMMARY

[0011] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0012] In one aspect, a vision system is presented. The vision system includes: a neural network having an input layer, a plurality of hidden layers and an output layer. The neural network may further include one or more reservoir node layers. Each reservoir node layer is comprised of a plurality of nodes (reservoir nodes), where each node in the plurality of nodes has a magnitude, such that the magnitude of a given node increases in response to a spike in the input and decays over time.

[0013] The vision system may further include an event camera, where the first reservoir node layer is configured to receive the input from the event camera. In some embodiments, the event camera has an n by m pixel arrangement and the plurality of nodes comprising the first reservoir node layer are arranged in an n by m array to match the event camera.

[0014] The node state for each in the plurality of nodes follows: Gt= Pc* Gmax~Gt-i) * <5Spfc(t)* e~T , where Gt is node state at time t, Pcis potentiation factor, Gmax is an upper bound of node state, T is relaxation time constant and 5Spk(t) is a delta function representing a spike. In some implementation, each node of the reservoir node layer is implemented by a memristor.

[0015] The vision system may further include a spike conversion layer configured to receive output from the output layer of the neural network and convert the output to spikes.

[0016] In an example embodiment, each node in the plurality of nodes is a memristor having a device conductance, such that the device conductance of a given node increases in response to a spike in the input and decays over time.

[0017] In other embodiment, at least one hidden layer in the plurality of hidden layers is a reservoir layer. That is, the at least on hidden layer is comprised of a plurality of nodes, where each node in the plurality of nodes has a magnitude, such thatthe magnitude of a given node increases in response to a spike in the input and decays over time.

[0018] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.DRAWINGS

[0019] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

[0020] Figure 1A is a diagram of an example network structure according to this disclosure.

[0021] Figure 1 B is a diagram of an alternative network structure according to this disclosure.

[0022] Figure 2 is an example circuit implementation of a reservoir node.

[0023] Figure 3 is a graph showing the dynamics of a reservoir node (solid) and a time surface node (dashed) under identical spikes (dotted).

[0024] Figures 4A and 4B illustrate temporally consecutive states of the input reservoir nodes, responding to receiving asynchronous events in the (a) DVS128 Gesture dataset and (b) DVS Lip dataset, respectively; each state is retrieved at a constant time interval of 30ms and the deeper red represents higher amplitude value of a RN state.

[0025] Figures 5A and 5B are plots showing the visualization of asynchronous input spikes and spikes generated after the spike conversion layer.

[0026] Figure 6 visualizes the outputs from the 2nd reservoir node layer over the whole 1 .5s clip, along with spikes generated from the spike conversion layer. For Data2 whose input video length is only 0.9s, the reservoir node states will continue to relax and are still used for classification.

[0027] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings.

[0029] In a reservoir computing (RC) system, the reservoir maintains short-term memory and performs nonlinear transformation (encoding) of the temporal input data into the reservoir states, represented by the states of the reservoir nodes (RNs). Reservoir computing systems have been efficiently implemented in hardware using devices (e.g., memristors) for vision (MNIST handwritten digits recognition), speech (NIST TI46) and time-series forecasting (Mackey-Glass time series) tasks. In these systems, the reservoir nodes encode the spikes’ spatiotemporal information naturally following the internal device dynamics, without any external memory or arithmetic and logic units (ALUs). By leveraging the internal device and circuit dynamics to process temporal data, these implementations have shown excellent energy efficiency and performance.

[0030] Generally, due to the short term memory property, reservoir nodes will be more strongly affected by the near history events and weakly affected by far history events, with the extent of the nonlinearity determined by the internal RN time constant. Inspired by this principle, this disclosure hypothesizes that reservoir nodes can be directly used to encode temporal spike data similar to what time surface aims to accomplish, but at a lower cost and performing better spatiotemporal feature extraction beyond the last spike since RNs nonlinearly accumulate all incoming spike information. Additionally, beyond only processing the inputs as in the case of time surface implementations, RN layers can be implemented multiple times within the network, typically with increasing internal RN time constants, to capture higher level (longer sequence) temporal dynamics.

[0031] Figure 1 depicts an example network structure 10 according to this disclosure. The network structure 10 is comprised of a neural network 14 having an input layer, an input reservoir node (RN) layer 12, a plurality of hidden layers and an output layer. The neural network may further include a spike conversion layer 16 and a second RN layer 18. In one example, the hidden layers in neural network 14 are convolutional neural network layers although other types of neural networks are contemplated by this disclosure.

[0032] The network structure 10 includes one or more reservoir node layers, such as an input RN (or spike encoding) layer 12. The input RN layer 12 is configured to receive an input and is interfaced with the input layer of the neural network. The inputRN layer 12 is comprised of a plurality of nodes, where each node in the plurality of nodes has a magnitude, such that the magnitude of a given node increases in response to a spike in the input and decays over time as will be further described below. In one example, the input RN layer 12 is configured to receive input (i.e., an image or series of images) from an event camera. In this example, the input from the event camera has an n by m pixel arrangement and the plurality of nodes in the input RN layer are arranged in an n by m array to match the input from the event camera.

[0033] In one example embodiment, a reservoir node in the input RN layer 12 can be implemented using only a single short term memory (STM) memristor, making hardware implementation very lightweight. In a STM memristor, the node state (i.e. device conductance) is natively excited by the incoming1spikes (for example, implemented as voltage pulses) and relaxes in between the spikes, as described by the following equationwhere Gt is the device (node) state at time t, Pcis a potentiation factor, Gmax is the upper bound of the device state, T is the characteristic relaxation time constant, and 6Spk(t) is a delta function representing a spiking event: = when no spike. sp / cl Jwhen receiving spike. ' '6Spk(t) can represent incoming events from an event-based camera or spikes from preceding layers within the network.

[0034] Figure 2 is an example circuit implementation of a reservoir node. When the node receives a spike, the spike will charge the capacitor and simultaneously turn off the transistor. Without a spike, the transistor is turned on and the charge stored on the capacitor will leak through the resistor with a decay time constant set by the product of RC. Other suitable implementations are contemplated for the reservoir node.

[0035] In an alternative embodiment, the reservoir layer 22 is one of the hidden layers in the neural network14 as seen in Figure 1 B. Likewise, the reservoir layer 22 is comprised of a plurality of nodes, where each node in the plurality of nodes has a magnitude, such that the magnitude of a given node increases in response to a spike in the input and decays over time. Except with respect to the differences discussed herein, the reservoir layer 22 functions substantially the same as the RN layer 12 described in this disclosure.

[0036] Figure 3 shows the comparison of the RN approach implemented with a single STM memristor based on equation (2) versus the conventional Time Surface (TS) approach. Both approaches convert the spiking patterns into an analog state that can be processed by subsequent DNN blocks. Compared to TS encoding that resets the state to 1 after each spike, the RN implementation allows longer term history to be represented in the state in a non-linear fashion. For example, at t=50, the TS output is identical to that at f=30, since both values were reset to one 10 time steps earlier (at t=20 and 40, respectively). However, this representation missed the differences in the two temporal sequences (e.g. an additional preceding spike at t=15). On the other hand, the RN implementation clearly differentiates the two cases as all inputs prior to the current time are accumulated nonlinearly. Combined with the hardware efficiency, e.g., equation (2) is natively implemented in a single device owing to internal device physics (ionic and electronic dynamic processes) without the need of additional dedicated memory (to store t and T(t)) and ALUs to convert the temporal data. Thus, the reservoir nodes approach of this disclosure is a suitable solution for asynchronous, real-time processing of event data. While reference has been made to the single STM memristor implementation, other types of circuit or devices that implement equation (2) can be used as reservoir nodes, including resistor-capacitance circuits.

[0037] The proposed network structure 10 operates by directly taking asynchronous spikes as they are generated, and the memristor-based reservoir nodes transform the temporal spikes into analog values following equation (2) in real time. The conductance values from the reservoir nodes are retrieved only when the network needs to operate a forward-pass, as shown in Figure 1 , analogous to visual inputs encoded by cells in a retina.

[0038] Figures 4A and 4B show examples of five temporally consecutive states of the input RN layer 12 (Rin), responding to asynchronous events from the two datasets, respectively. The states are obtained at a constant time interval (i.e. 30ms). As shown in the figure, frequent input spikes lead to stronger RN states due to the excitation term (the first term of equation (2)), while RN states natively relax for inputs that are temporally too far from the current time instant due to the internal decay term (the second term of equation (2)). These properties of the RNs allow the in layer to capture both the spatial and temporal features of the inputs, as shown in Figures 4A and 4B. For example, in lower plots, both the speaking motion and movement (top to bottom) ofthe lips can be captured by the RNs’ states in the input RN layer Rin. Temporally fast movements are reflected in a single moment (e.g. the 3rd plot in lower plots), while temporally slow movements are reflected in multiple moments captured at the different time instants (e.g. the first two plots).

[0039] With continued reference to Figure 1 A, the example network structure 10 is formed sequentially by in input RN layer 12 ( Rin) as a feature descriptor for temporally local encoding, convolution (Conv) layersa spike conversion (SC) layer 16, a second RN layer 18 ( Rt ) for temporally global encoding, and fully-connected (FC) layers (FN for classification.

[0040] In the example embodiment, the input RN layer 12 Rin has as many reservoir nodes as the output dimension of the event-based camera. Each reservoir node in the input RN layer 12 Rin processes input spikes asynchronously from a corresponding pixel in the event camera in real time following equation (2). Examples of the outputs generated by the input RN layer 12 Rin are shown in Figures 4A and 4B.

[0041] Depending on the application, different temporal resolutions can be chosen by adjusting the potentiation factor Pcand the time constant T of equation (2), and the interval of Rin state acquisitions. A shorter acquisition interval creates more frequent activations for the following layers. A shorter time constant T and more frequent acquisitions produce temporally finer outputs, while a longer time constant and less frequent acquisitions produce spatially more detailed outputs owing to more input spikes and the slower internal decay. The convolution layers (C1-C7 with depths D1-D7 in Figure 1 A) then process the encoded spatiotemporal features in Rin at a constant time interval (i.e. 30ms). Similar to Conv blocks in conventional DNNs, the output Ofrom the conv layers reflects the spatial features existing in the Rin states, and in this case, capturing the spatiotemporal features from the original spikes.

[0042] Output O is then flattened and sent to the spike conversion SC layer. A predetermined threshold is used to convert the outputs Ofrom the Conv layers into spikes in the spike conversion SC layer. Spikes generated after the spike conversion SC layer are shown in Figures 5A and 5B. Thus, the spikes after the spike conversion layer SC represent the local spatiotemporal features captured by the Rin layer and the Conv layers, and become much sparser compared with the original spiking inputs.

[0043] The spikes from the spike conversion SC layer are then supplied to the reservoir nodes in the second RN layer F or global spatiotemporal feature capture andsubsequent classification in the FC layers. Similar to the reservoir nodes in the input RN layer Rin, reservoir nodes (e.g. also implemented with STM memristors) in the second RN layer Rf naturally potentiates / relaxes with the presence / absence of spikes from the spike conversion SC layer. The state of F thus represents the historical longer sequence (e.g. more global) spatiotemporal features by encoding the processed local spatiotemperal features over time, and is then used by the subsequent FC layers (FCi- FC2 in Figure 1 A) to perform final classification functions. Similar to the reservoir nodes in the input RN layer Rin, the reservoir nodes states in Rf are retrieved at a certain time interval (i.e. 0.3s, which is longer than that used in Rin) over the whole video clip and fed to the FC layers.

[0044] The reservoir node network 10 is trained with standard backpropagation, using the output potentials calculated by FC2 at the end of data presentation. Unlike BPTT that calculates error and gradient across each timestep, error and gradient of reservoir node network 10 is calculated only once for one input training video, leading to lower training cost. To address the non-differentiability arising from spike generation in the SC layer, one can adopt surrogate gradient for the SC layer. Since surrogate is used only in one layer of the network, the error introduced by this approach is lower when compared with SNN approaches which require surrogates at all layers.

[0045] Several data augmentation techniques can also be applied, such as random / center crop, horizontal flip and Gaussian noise to minimize overfitting effects. During training, the original input dimension (128 x 128) is center cropped to 96 x 96 and then random cropped to 76 x 76, while the inputs are directly center cropped to 76 x 76 in the testing phase. Horizontal flip with a probability of 0.5 and Gaussian noise with standard deviation of 5e^ were used during training for the same purpose. These techniques helped generalize the input data and reduce overfitting during training. These techniques were used only for the DVS Lip dataset since motions in DVS128 Gesture such as a left / right hand waving are sensitive to the horizontal flip and objects positions.

[0046] Experiments were performed as proof of concept. The DVS128 Gesture dataset consists of repetitions of the same motion of a participant along the clip, while lip motions in the DVS Lip dataset are not repetitive, as shown in Figures 4A and 4B. As a result, a fraction of a clip should be sufficient to classify the action in DVS128 Gesture, while the whole clip is necessary for DVS Lip classification. Based on this understanding, the first 1 .5s was used for all videos in the DVS128 Gesture datasetduring training and inference, corresponding to 10% of the longest video (15.5s). The DVS Lip dataset has variable input lengths due to the irregular length of the words and the unique speaker’s traits, with most video clips (99%) having lengths between 0.75s and 1 .5s. To make the data size regular across the dataset, we chose to keep the captured Rin outputs at 50 by simply applying null (no spike) for clips shorter than 1 .5s and clipping videos longer than 1 .5s during DVS Lip training and inference. X- and Y- dimension were set as 128 and 128, respectively. Pcand T in equation (2) were set to 0.5 and 60ms.

[0047] For both datasets, the same potentiation factor Pc(0.5) and time constant T (60ms) in equation (2) are used for the reservoir nodes in the input RN layer 12. The states of Rin are captured every 30ms and sent to the subsequent Conv layers.

[0048] Due to the different input dimensions and the different number of categories in the two datasets, the convolution layers and FC layers in reservoir node network 10 are configured accordingly. The detailed network configurations for both datasets are summarized in Table 1 below.Table 1 : Network configuration of RN-Net for DVS128 Gesture dataset (left columns) and DVS Lip dataset (right columns).DVS128 Gesture DVS128 LipThe output of a DNN block consists of a convolution and a pooling layer (if exists), following:ON = f (BN (MP (C(IN )))) (4) where ON is the output of N-th layer, f(x) is a ReLU activation function, BN(x) is a batch normalization function, MP(x) / C(x) is a maxpooling / convolution operation, and IN is input of the N-th layer.

[0049] After the last convolution layer, the outputs are converted to spikes by the spike conversion layer 16 for subsequent temporal feature encoding at the second RN layer 18. The spike conversion layer 16 generates spikes by comparing the analogue values in O (Figure 1 ) with a pre-fixed threshold value, set as 0.3 for both datasets.

[0050] Reservoir nodes in the second RN layer 18 Rf encode the global spatiotemporal features. Pcand T for the reservoir nodes in the second RN layer 18 are set as 0.1 , 2s and 0.3, 2s for DVS128 Gesture and DVS Lip dataset, respectively. A longer T is used in the second RN layer 18 than in the first RN layer 12 to process the longer temporal correlations. The RN states in the second RN layer 18 are captured every 0.3s for both datasets (corresponding to 5 Rf acquisitions during a 1 .5s clip). In the case of clips shorter than 1 .5s, let reservoir nodes in the second RN layer 18 continue relaxing after the last meaningful input without padding the input data with new spikes (e.g., through repeating the clip). These relaxed states are still captured following the normal schedule and fed to the classification layers. Examples of RN states in the second RN layer at different time instants are shown in Figure 6, along with the spikes they receive from the SC layer. For better visualization, the data are reshaped in 2D format. As shown in the figure, even without any new inputs (e.g., after 0.9s for Data2), the RN states do not immediately decay to zero due to the slow decay term in equation (2), and the evolution of the reservoir nodes still represents useful information. This approach also simplifies training and inference processes and allows the system to handle inputs with irregular lengths.

[0051] A classification block consists of a FC layer, an activation function, and a batch normalization layer, following:OF= f(BA / (FC( / ))) (5)where OF is the output of the block, f(x) is a ReLU activation function, BN(x) is a batch normalization function, FC(x) is an FC layer, and I is the input to the FC layer. In the proposed RN-Net, two classification blocks are used, and output of the 2nd block is obtained without the activation and the normalization function, and used as the final output of the network.

[0052] For the experiments, the Pytorch framework was used with methods described above. SoftMax function was chosen to calculate the probability of an output neuron in final output, and the neuron with the largest probability was selected as the classification result. During training, the cost was derived by categorical cross-entropy function. ATan was selected as surrogate gradient calculation for the SC layer during training. Also used Adam optimizer with an initial learning rate of 7e^ 17e~5and a weight decay of 0 15e^ as the model optimizer and ReduceLROn Plateau with a factor of 0.9 1 0.9, patience of 2 / 1 , threshold of 1 e5 / 1 e-6, minimum learning rate of 1 e-41 1 e~7as the learning scheduler for DVS128 Gesture I DVS Lip, respectively. Batch size of 32 and 150 epochs were used for both datasets. All experiments were performed on an Intel Xeon Gold 6226R and an NVIDIA A40.

[0053] The reservoir node network 10 shows excellent performance on both datasets; 99.2% for DVS128 Gesture (highest reported to date) and 67.5% for DVS Lip (one of the highest for event-inputs). Comparisons of classification accuracy with existing networks are shown in Table 2 and Table 3. The accuracies of networks listed in Table 2 were obtained from the work of Tan et al in “Multi-grained spatio-temporal features perceived network for event-based lip-readuing” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2022). For DVS128 Gesture, the reservoir node network 10 outperforms all existing networks based on SNN and CNN platforms, while only using the first 10 percent of the longest clip. For DVS Lip, the reservoir node network 10 achieves top 2 accuracy among eventbased models, only behind MSTP which is a multi-branch network with different input channels generating frames in different temporal granularity, and employs Voxel Grid as the feature descriptor and using larger BiGRU and ResNet-18 network architectures. By comparison, the reservoir node network 10 uses a single branch and a much lighter DNN network, without expensive feature description / preprocessing and recurrent units that can lead to significant hardware and latency overheads. The relevant model sizes and their performances are listed in Table 2. Power of a proposed hardware system runningthe reservoir node network 10 for both tasks is estimated based on existing device technology and reported values. When running DVS128 Gesture and DVS Lip tasks, the proposed RN-Net hardware is estimated to consume 10.2mW and 1 1 .4mW, respectively.Table 2: Comparison of existing models and RN-Net on Lip reading dataset including DVS Lip.Model Input ACC Params Base Preprocess Local Global[%] [M] model encoding encodingTANet

[0056] Video 68.7 42.8 ResNet- - TAM Temporal101 AvgPoolACTION- Video 68.8 28.1 ResNet- Random STE / CE / M TemporalNet

[0057] 50 Sampling E AvgPoolDFTN

[0018] Video 63.2 40.5 ResNet- - DFN BiGRU18Feng et Video 63.4 11.2 ResNet- - - BiGRU al

[0019] 18Martinez Video 65.5 11.2 ResNet- Greyscaling - Multi-scale et al

[0020] 18 TCNEvent Event 42.2 8.1 PointNe Random 3D point 3D pointClouds

[0021] t Sampling cloud cloudEV-Gait- Event 32.0 7.2 - OctreeGrid 3D-Graph N / A3DGraph[2 Filtering4] EV-Gait- Event 34.5 64.6 - Event Image-like N / AIMG

[0022] NoiseCancelling EST

[0023] Event 48.7 21 .5 ResNet- Normalized MLP (Grid) N / A 34 TimeStampMSTP

[0017] Event 72.1 38.5 ResNet- Multiple Voxel Grid BiGRU18 Time-ScalingRN-Net Event 67.5 7.5 - - Reservoir ReservoirTable 3: Comparison of existing models and RN-Net for DVS128 Gesture.Table 4: Performance comparison of different Rin / Rf configurations for DVS128 Gesture and DVS Lip tasks.

[0054] The effects of the reservoir nodes in the input RN layer 12 and the second RN layer 18 on model performance are investigated in an ablation study by replacing each RN layer with a TS or Temporal average pooling (TAP, only for Rf ) layer without modifying the rest of the network. See, Table 4. In both tasks, the case with both RN layers shows the best performance, while the case with TS at both places shows the worst performance. However, replacing F with TS degrades the performance more than that of Rf for the DVS128 Gesture task, while the DVS Lip task shows the opposite tendency. This difference is attributed to different characteristics of the datasets. Data in DVS128 Gesture are repetitive along the clip, which can tolerate less precise input encoding because there are opportunities to capture the lost information at a different time instant. On the other hand, in DVS Lip information at different time instants is unique, which makes input encoding more critical for the network performance. Interestingly, although the use of reservoir nodes achieves the best results for both datasets, the use of TAP in the Rf layer achieves better results than the use of TS for the DVS Lip task, while the use of TS is better for the DVS128 Gesture task. This may be caused by the characteristics of TS, which completely discards far history that is critical for the DVS Lip dataset. On the other hand, the repetitive inputs in DVS128 Gesture alleviates this problem for TS implementation, where its capability to capture local temporal dynamics outperforms TAP. As a result, this study illustrates that the RNs’ capability to capture both local and global temporal dynamics in different RN layers allows networks based on them (i.e. RN-Net) to achieve higher accuracy for both tasks.

[0055] The techniques described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions that are stored on a non-transitory tangible computer readable medium. The computer programs may also include stored data. Non-limiting examples of the non-transitory tangible computer readable medium are nonvolatile memory, magnetic storage, and optical storage.

[0056] Some portions of the above description present the techniques described herein in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described functionally or logically, are understood to be implemented by computer programs. Furthermore, it has also provenconvenient at times to refer to these arrangements of operations as modules or by functional names, without loss of generality.

[0057] Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0058] Certain aspects of the described techniques include process steps and instructions described herein in the form of an algorithm. It should be noted that the described process steps and instructions could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.

[0059] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer. Such a computer program may be stored in a tangible computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

[0060] The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatuses to perform the required method steps. The required structure for a variety of these systems will be apparent to those of skill in the art, along with equivalent variations. In addition, the present disclosure is not described with reference to any particular programming language. It is appreciated that a variety ofprogramming languages may be used to implement the teachings of the present disclosure as described herein.

[0061] The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.

Claims

CLAIMSWhat is claimed is:1 . A vision system, comprising: a neural network having an input layer, a plurality of hidden layers and an output layer; and an input reservoir node layer configured to receive an input and interfaced with the input layer of the neural network, where the input reservoir node layer is comprised of a plurality of nodes, each node in the plurality of nodes has a magnitude, such that the magnitude of a given node increases in response to a spike in the input and decays over time.

2. The vision system of claim 1 further comprises an event camera, where the input reservoir node layer is configured to receive the input from the event camera.

3. The vision system of claim 1 wherein the event camera has an n by m pixel arrangement and the plurality of nodes comprising the input reservoir node layer are arranged in an n by m array to match the event camera.

4. The vision system of claim 1 wherein node state for each node in the plurality of nodes follows:where Gt is node state at time t, Pcis potentiation factor, Gmax is an upper bound of node state, T is relaxation time constant and 5Spk(t) is a delta function representing a spike.

5. The vision system of claim 1 wherein each node of the input reservoir node layer is implemented by a memristor.

6. The vision system of claim 1 further comprises a spike conversion layer configured to receive output from a certain hidden layer of the neural network and convert the output to spikes.

7. The vision system of claim 6 wherein at least one hidden layer in the plurality of hidden layers is comprised of a plurality of nodes, each node in the plurality of nodes has a magnitude, such that the magnitude of a given node increases in response to a spike in the input and decays over time.

8. A vision system, comprising: a neural network having an input layer, a plurality of hidden layers and an output layer; and an input reservoir node layer configured to receive an input and interfaced with the input layer of the neural network, where the input reservoir node layer is comprised of a plurality of nodes, each node in the plurality of nodes is a memristor having a device conductance, such that the device conductance of a given node increases in response to a spike in the input and decays over time.

9. The vision system of claim 8 further comprises an event camera, where the input reservoir node layer is configured to receive the input from the event camera.

10. The vision system of claim 8 wherein the event camera has an n by m pixel arrangement and the plurality of nodes comprising the input reservoir node layer are arranged in an n by m array to match the event camera.1 1 . The vision system of claim 8 wherein node state for each node in the plurality of nodes follows:where Gt is node state at time t, Pcis potentiation factor, Gmax is an upper bound of node state, T is relaxation time constant and 6Spk(t) is a delta function representing a spike.

12. The vision system of claim 8 further comprises a spike conversion layer configured to receive output from a certain hidden layer of the neural network and convert the output to spikes.

13. The vision system of claim 8 wherein at least one hidden layer in the plurality of hidden layers is comprised of a plurality of nodes, each node in the pluralityof nodes is a memristor having a device conductance, such that the device conductance of a given node increases in response to a spike in the input and decays over time.

14. A vision system, comprising: a neural network having an input layer, a plurality of hidden layers and an output layer; wherein at least one hidden layer in the plurality of hidden layers is comprised of a plurality of nodes, each node in the plurality of nodes has a magnitude, such that the magnitude of a given node increases in response to a spike in the input and decays over time.

15. The vision system of claim 14 wherein node state for each node in the plurality of nodes follows:where Gt is node state at time t, Pcis potentiation factor, Gmax is an upper bound of node state, T is relaxation time constant and 5Spk(t) is a delta function representing a spike.

16. The vision system of claim 14 wherein each node of the at least one hidden layer is implemented by a memristor.

17. The vision system of claim 14 further comprises a spike conversion layer configured to receive output from a certain hidden layer of the neural network and convert the output to spikes.

Images