Feature extraction and encoding of spiking neural networks using convolutional neural networks and trainable encoders for deployment in neuromorphic chips.

Abstract

A method for operating a neuromorphic processor comprising: a convolutional neural network adapted to receive an input signal and generate a corresponding real value; an encoder connected to receive a real value and generate a corresponding time-encoded binary value using an encoding function; and a spiking neural network connected to receive a time-encoded binary value and generate a corresponding spiking output signal. The method comprises: calculating a loss based on the spiking output signal; calculating a backpropagation gradient based on the loss; transforming the backpropagation gradient using a fully differentiable surrogate function which is a surrogate of the encoding function; and updating one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network based on the transformed gradient.

Description

TECHNICAL FIELD

[0001] The present invention relates to neuromorphic processors, and more particularly, to a system and method for co-training of convolutional neural networks and spiking neural networks for deployment on neuromorphic chips such as neuromorphic processors. BACKGROUND ART

[0002] A convolutional neural network (CNN) is a type of feedforward artificial neural network (ANN) that utilizes convolution operations to repeatedly apply kernels to any type of data. See Chollet, F. (2021), Deep learning with Python, Simon and Schuster. While combining CNNs with training algorithms such as gradient descent (GD) or other learning algorithms, these are efficient methods for identifying local variations in data related to specific tasks, that is, they are powerful data-driven feature extractors. CNNs are often used together with max pooling or average pooling operations. The former takes the maximum value of a kernel of a specified size, and the latter averages all values of the kernel. The combination of CNNs and pooling operations can be used to achieve a highly desirable property, namely translation invariance, that is, a shift in input data will not change the result of the model. Hereinafter, for the sake of simplicity, the combined CNN-pooling operation is referred to as "CNN" or "CNNs". In addition, CNN layers can be stacked hierarchically on top of each other, which makes it possible to detect frequency-related or higher-level features. These advantages establish CNNs as translation-invariant, hierarchical feature extractors, and make CNNs attractive to the embedded AI industry thanks to the development of efficient hardware accelerators.

[0003]

[0003] Drawing inspiration from the brain, a spiking neural network (SNN) is a neural network that can receive binary inputs and produce binary outputs, and each synaptic-neuron complex can be conceptualized as a small processing unit with memory. Synapses scale input events (spikes) by multiplying them by weight values. The synaptic output is then added to the neuron's membrane potential. If a threshold is reached, the information (another spike) propagates to the next one or more synaptic-neuron complexes, if there are any. Otherwise, the information, or a modified version thereof, remains within the neuron's membrane potential until further spikes are received, scaled, and then added to the neuron's membrane potential. Modifications include, for example, leaks, which can be linear or exponential, and / or biases. Neurons have memory as they track previous input spikes (events) that they have received. When the integrated events reach the neuron's threshold, the memory (membrane potential) is reset, and the process restarts immediately or after a certain period of time (refractory period). Neurons in SNNs are also small processing units because their membrane potentials have internal dynamics that can be characterized, for example, by dynamical systems theory. Their membrane potentials follow specific physical principles that perform in-memory computations, i.e., modify the membrane potential. Some examples of in-memory processing include the integration of input events, having some kind of time-dependent membrane potential decay function, how the membrane potential is reset, the refractory period, etc. As a result of these inherent properties, SNNs are recognized as a powerful pattern recognition solution for implantable AI, given the existence of efficient hardware implementations, i.e., neuromorphic chips.

[0004]

[0004] SNNs are energy-efficient when performing event-driven pattern recognition tasks, but they typically rely on preprocessing methods that are not automatically corrected during SNN training. Conventional preprocessing methods are often power-intensive (e.g., spectrogram analysis such as wavelet transform) or limit their use cases (e.g., pass-band filters). There are several reasons why these preprocessing techniques are typically used, for example, that the input data for an SNN must be encoded by an encoder before being fed into the SNN, i.e., converted from real values ​​to time-encoded binary events (i.e., spike sequences) that spread over time. An example of how an encoder can be used to create events from real values ​​is any kind of time coding, such as rate, group, or precise spike time coding, which is commonly used in the SNN domain. (See Auge, Daniel, et al. "A survey of encoding techniques for signal processing in spiking neural networks." Neural Processing Letters 53.6 (2021): 4693-4710), or, for example, binary encoding techniques not commonly used in SNNs, such as straight-through estimators (STEs) (see Bengio, Yoshua, Nicholas Leonard, and Aaron Courville. "Estimating or propagating gradients through stochastic neurons for conditional computation." arXiv preprint arXiv:1308.3432 (2013)), or the Binary Spatter Code (see Kanerva, P. (1996, July)).Binary spatter coding of ordered k-tuples. International conference on artificial neural networks (pp. 869-873). Berlin, Heidelberg: Springer Berlin Heidelberg) or any other. Another reason is that the features to be detected in the sensor data span relatively long time periods relative to the time constant of the internal dynamics of spiking neurons, which is particularly relevant to obtaining good performance in leaky analog or mixed-signal chips, for example.

[0005]

[0005] Conventional data processing techniques work well for a range of applications, but tailoring them to specific applications is a time-consuming task. Additionally, if preprocessing techniques require parameter selection, their performance is usually determined by the results of the parameter search for preprocessing. Once preprocessing parameters are selected, the same preprocessing blocks using those parameters are typically used throughout the entire machine learning (ML) model training process, regardless of the error gradient for the task being performed. In the case of preprocessing an ML model with a time component, for example, specific dynamics of in-memory computation in an SNN, the above preprocessing rarely automatically optimizes to represent the preprocessed data as events with the desired time dynamics.

[0006]

[0006] For example, in optimization techniques such as gradient descent, preprocessing parameters remain static throughout the training process, even when the error gradient changes during each iteration. This fixed preprocessing can hinder performance because the data representation may no longer be optimal for the evolutionary model. Similarly, in models with a time component where backpropagation through time (BPTT) is used to propagate errors over time steps, static preprocessing may fail to capture time dependencies that are important for effective learning. The situation is even more complex in the case of spiking neural networks (SNNs), as these models rely on event-based temporal dynamics that typical preprocessing techniques do not consider, further limiting their performance.

[0007]

[0007] For example, to decompose a frequency-modulated continuous wave (FMCW) radar signal into its frequency components, the standard practice in the industry is to compute its range fast Fourier transform (FFT). This results in an accurate depiction of reflective objects in a room, but finding the best features for a task such as people counting and creating binary events will be different from a task of tracking people. Furthermore, this task becomes more difficult if the network analyzing these events has some internal dynamics, such as a social networking network (SNN).

[0008]

[0008] Automated feature extraction and conversion to events can complement and enhance the pattern recognition capabilities of an SNN. The present invention implements this using a CNN for the aforementioned desirable properties, and the system of the present invention comprises a CNN, an encoder, and a forward path to an SNN. Note that the term “encoder” is defined for the purposes of this description as distinct from a CNN or an SNN. An encoder converts real input values ​​(output by a CNN) into time-encoded binary values ​​(for input to an SNN). Note that, under given definitions of CNN and encoder, an encoder is distinct from an encoder, since an encoder receives real numbers, converts them, and outputs binary values, while a CNN can be abstracted as a function that receives real numbers and outputs real numbers. In the most extreme case, where the activation function of a CNN is binary, for example, even when the activation function is a straight-through estimator (STE), the encoder is the STE itself. Nevertheless, SNNs that receive non-binary input values ​​are not considered encoders in the context of the present invention, since their parameters can be jointly optimized with binary SNNs without issue, even if they output binary numbers. Furthermore, while both CNNs and SNNs comprise a network of interconnected neurons and synapses, the encoders defined herein do not include neurons and synapses.

[0009]

[0009] Cutting-edge technologies already use CNNs together with SNNs in various ways. For example, Q. Xu et al., CSNN: an augmented spiking based framework with perceptron-inception, IJCAI 2018 describes a system in which a CNN is pre-trained with a fully connected artificial neural network (ANN), then the ANN is discarded and replaced with a time encoder and an SNN, and then the SNN is trained. S. Kim et al., C-DNN: A 24.5-85.8 TOPS / W complementary-deep-neural-network processor with heterogeneous CNN / SNN core architecture and forward-gradient-based sparsity generation, ISSCC 2023 describes using forward gradient training for the SNN and backpropagation training for the CNN when the SNN forward gradient exceeds a certain value. Other examples, such as U.S. Patents 11,227,210 and 11,989,645, suggest the use of spiking CNNs rather than combining CNNs with SNNs.

[0010]

[0010] Although CNNs are widely used in the field of AI, in previously known systems they are not trained together with SNNs in the same training flow. [Overview of the project]

[0011]

[0011] This specification describes a system and method for integrating the training of CNNs and SNNs into the same flow and optimizing their deployment to a neuromorphic processor. Unlike conventional methods, the present invention uses a CNN to derive output feature vectors for input to an SNN, so that the entire pipeline is simultaneously optimized for deployment to a neuromorphic processor.

[0012]

[0012] In a first aspect, the present invention provides a method for deploying a pipeline to a neuromorphic processor and then operating the neuromorphic processor. The pipeline and neuromorphic processor comprises a convolutional neural network adapted to receive an input signal and generate a corresponding real value; an encoder connected to receive the convolution result and generate a corresponding time-coded binary value using an encoding function; and a spiking neural network connected to receive the time-coded binary value and generate a corresponding spiking output signal. The pipeline models at least a portion of the neuromorphic processor.

[0013]

[0013] The method comprises: calculating a loss for the pipeline based on an input signal and a spiking output signal; calculating a backpropagation gradient for the pipeline based on the loss; transforming the backpropagation gradient using a fully differentiable surrogate function which is a surrogate of the coding function; updating one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network for the pipeline based on the transformed gradient; and deploying one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network of the pipeline to the spiking neural network and convolutional neural network of the neuromorphic processor, respectively.

[0014]

[0014] In one embodiment of the first aspect, the method may further comprise updating one or more parameters of a pipeline encoder based on a transformed gradient, and then deploying the updated one or more parameters of the pipeline encoder to an encoder of a neuromorphic processor. Note that since the encoder may have a surrogate gradient, it may also have zero parameters that need to be updated according to the present invention.

[0015]

[0015] In one embodiment of the first aspect, the transformed gradient generated by the surrogate function comprises a continuous approximation of the backpropagated gradient.

[0016]

[0016] In one embodiment of the first aspect, the transformed gradient generated by the surrogate function comprises a continuous relaxation of the backpropagated gradient.

[0017]

[0017] In one embodiment of the first aspect, the loss is used to compute a backpropagation gradient for use when updating the parameters of the spiking neural network, convolutional neural network, and / or pipeline encoder using a chain rule, preferably the backpropagation gradient depends on the loss, the spiking output of the forward pass, the previous input, and / or the in-memory computation and dynamics of the neurons of the spiking neural network.

[0018]

[0018] In one embodiment of the first aspect, the encoder is a time encoder, a rate encoder, a group encoder, a precise spike time encoder, a thermometer encoder, and / or a straight-through estimator encoder.

[0019]

[0019] In one embodiment of the first aspect, the parameters of each element of the pipeline are initialized before the input processing begins, preferably by drawing samples from a specific distribution, such as a uniform distribution or a Gaussian distribution, using a genetic algorithm or other initialization method.

[0020]

[0020] In one embodiment of the first aspect, the parameters for a convolutional neural network include one or more of the following: convolutional filter (weights), bias, fully connected layer weights, and / or batch normalization parameters; and / or the parameters for an encoder include the weights and bias of an encoder layer, and / or learnable transformation parameters; and / or the parameters for an SNN include one or more of the following: synaptic weights, membrane potential threshold, STDP parameters (if spike timing learning is used), and / or neuron and synaptic dynamics parameters.

[0021]

[0021] In one embodiment of the first aspect, the method further comprises the steps of: updating one or more parameters of a spiking neural network and one or more parameters of a convolutional neural network of a neuromorphic processor based on a transformed gradient; the convolutional neural network of the neuromorphic processor receiving a sensor input signal and generating a sensor real value corresponding to the sensor input signal; the encoder of the neuromorphic processor receiving a real value corresponding to the sensor input and generating a corresponding time-encoded sensor binary value using an encoding function; the spiking neural network of the neuromorphic processor receiving the time-encoded sensor binary value and generating a corresponding sensor spiking output signal; and the inference module of the neuromorphic processor performing inference on the sensor input signal based on the sensor spiking output signal.

[0022]

[0022] In one embodiment of the first aspect, the sensor input signal is obtained from an image sensor, optical sensor, LiDAR sensor, RADAR sensor, inertial measurement sensor, accelerometer sensor, vibration sensor, gas sensor, proximity sensor, acoustic sensor, electroencephalogram sensor, electromyogram sensor, and / or electrocardiogram sensor.

[0023]

[0023] In a second aspect, the present invention discloses a neuromorphic processor. The neuromorphic processor comprises a convolutional neural network adapted to receive an input signal and generate a corresponding real value; an encoder connected to receive a real value and generate a corresponding time-encoded binary value using an encoding function; and a spiking neural network connected to receive a time-encoded binary value and generate a corresponding spiking output signal.

[0024]

[0024] One or more parameters of the spiking neural network and one or more parameters of the convolutional neural network are set by the encoder and spiking neural network of the neuromorphic processor calculating a loss for the pipeline based on the input signal and the spiking output signal, calculating a backpropagation gradient for the pipeline based on the loss, transforming the backpropagation gradient using a fully differentiable surrogate function which is a surrogate of the coding function, and updating one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network for the pipeline based on the transformed gradient, and deploying one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network of the pipeline to the spiking neural network and convolutional neural network of the neuromorphic processor, respectively.

[0025]

[0025] In one embodiment of the second aspect, one or more parameters of the encoder of the neuromorphic processor may be set by updating one or more parameters of the pipeline encoder based on the transformed backpropagation gradient and then deploying them. Note that since the encoder may have a surrogate gradient, it may also have zero parameters that need to be updated according to the present invention.

[0026]

[0026] In an embodiment of the second aspect, the transformed gradient generated by the surrogate function comprises a continuous approximation of the backpropagation gradient.

[0027]

[0027] In an embodiment of the second aspect, the transformed gradient generated by the surrogate function comprises a continuous relaxation of the backpropagation gradient.

[0028]

[0028] In an embodiment of the second aspect, the loss is used to calculate the backpropagation gradient for use in updating the parameters of the spiking neural network, the convolutional neural network, and / or the encoder of the pipeline using the chain rule, preferably, the backpropagation gradient depends on the loss, the spiking output of the forward pass, the previous input, and / or the in-memory computation and dynamics of the neurons of the spiking neural network.

[0029]

[0029] In an embodiment of the second aspect, the encoder is a temporal encoder, a rate encoder, a group encoder, an exact spike time encoder, a thermometer encoder, and / or a straight-through estimator encoder.

[0030]

[0030] In an embodiment of the second aspect, the parameters of each element of the pipeline are initialized before processing of the input begins, preferably, the initialization is performed by drawing samples from a specific distribution, for example, a uniform distribution or a Gaussian distribution, using a genetic algorithm or other initialization methods.

[0031]

[0031] In one embodiment of the second aspect, the parameters for a convolutional neural network include one or more of the following: convolutional filter (weights), bias, fully connected layer weights, and / or batch normalization parameters; and / or the parameters for an encoder include the weights and bias of an encoder layer, and / or learnable transformation parameters; and / or the parameters for an SNN include one or more of the following: synaptic weights, membrane potential threshold, STDP parameters (if spike timing learning is used), and / or neuron and synaptic dynamics parameters.

[0032]

[0032] In one embodiment of the second aspect, the convolutional neural network of the neuromorphic processor is configured to receive a sensor input signal and generate a sensor real value corresponding to the sensor input signal; the encoder of the neuromorphic processor is configured to receive a real value corresponding to the sensor input and generate a corresponding time-encoded sensor binary value using an encoding function; the spiking neural network of the neuromorphic processor is configured to receive a time-encoded sensor binary value and generate a corresponding sensor spiking output signal; and the neuromorphic processor further comprises an inference module, the inference module is configured to perform inference of the sensor input signal based on the sensor spiking output signal.

[0033]

[0033] In one embodiment of the second aspect, the sensor input signal is obtained from an image sensor, optical sensor, LiDAR sensor, RADAR sensor, inertial measurement sensor, accelerometer sensor, vibration sensor, gas sensor, proximity sensor, acoustic sensor, electroencephalogram sensor, electromyogram sensor, and / or electrocardiogram sensor, and preferably, the neuromorphic processor is electrically or operably connected to the sensor that obtained the sensor input signal.

[0034]

[0034] Since CNNs are common feature extractors, by integrating CNNs and SNNs into the same training flow, the present invention enables: (1) creation of a trained end-to-end pipeline, e.g., from sensor data to SNN output, or interleaving CNNs, encoders, SNN blocks, and other differentiable components in any order; (2) trainable feature extraction for SNNs by CNNs. Features are extracted based on the joint optimization of CNNs, encoders, and SNN blocks, as well as any other differentiable parts in the pipeline; (3) features extracted by CNNs are encoded based on the joint optimization of CNNs, encoders, and SNN blocks, as well as any other differentiable parts in the pipeline; (4) identification of the task, the features extracted by CNNs, how the internal dynamics of spiking neurons constitute the SNN, and the optimized encoding inherent to the system-wide dynamics of the SNN; (5) the ability to train CNNs and encoders to represent time-domain important features. (6) The present invention enables training a CNN for a specific set of features for an application, and allows the power performance of the feature extractor to be optimized or constrained for that application. (7) Parameters, including those of the CNN, encoder, and SNN, can be constrained during the software optimization process to address different hardware components and their specifications. For example, since CNNs and SNNs are typically deployed on different types of hardware accelerators, co-training allows for optimization of different quantization schemes. (8) Co-use of efficient CNN hardware accelerators in combination with neuromorphic hardware, which can reduce chip area, latency, and energy. (9) Reduces embedding time for tuned embedding solutions. (10) In particular, the creation of pipelines including CNN, encoder, and SNN blocks for inference, such as classification, regression, segmentation, or generative tasks. (11) Reduces the number of pre-established parameters and behaviors of conventional fully connected CNN networks for tasks requiring pattern recognition.

[0035]

[0035] Next, embodiments will be described simply by reference to the accompanying drawings, where corresponding reference numerals indicate corresponding parts. [Brief explanation of the drawing]

[0036] [Figure 1]

[0036] Figure 1 is a schematic diagram of a simple neural network. [Figure 2]

[0037] Figure 2 is a simplified schematic diagram of synaptic connections implemented as a crossbar array. [Figure 3]

[0038] Figure 3 is a schematic diagram of the forward and backward paths in a neuromorphic processor with a pipeline containing both CNNs and SNNs, illustrating both the simulation and the pipeline deployment. [Figure 4]

[0039] Figure 4 is a block diagram of the hardware implementation of the neuromorphic device shown in Figure 3. [Figure 5]

[0040] Figure 5 is a block diagram illustrating an example of data flow in the neuromorphic processor shown in Figures 3 and 4. [Figure 6]

[0041] Figure 6 illustrates the data flow for the forward path in a neuromorphic processor. [Modes for carrying out the invention]

[0037]

[0042] Certain embodiments are described in more detail below. However, it should be recognized that these embodiments are merely examples and should not be construed as limiting the scope of protection under this disclosure.

[0038]

[0043] Figure 1 is a schematic diagram of a simple neural network 1 that can be implemented as a convolutional neural network (CNN) or a spiking neural network (SNN). In this example, the CNN or SNN comprises an input layer 2 of neuron 10 (input neuron), a hidden layer 3 of neuron 11 (hidden neuron), and an output layer 4 of neuron 12 (output neuron). The input neuron is connected to the hidden neuron 11 via synaptic element 17, and the hidden neuron 11 is connected to the output neuron 12 via synaptic element 18. The output 9 of the entire system is generated by the last layer of the network, the output neuron 12. The output 9 of the output neuron 12 can then be passed forward for further processing or to a decoder 13 (which can be implemented as the decoding layer of the network) that can output the information to the user.

[0039]

[0044] The CNN / SNN shown in Figure 1 is illustrated with only three layers containing a very small number of neurons and synaptic elements for simplicity; however, actual networks can have a very large number of layers, neurons, and synaptic elements to achieve satisfactory performance. Practical implementations of CNN / SNNs can have any other number of neurons and synapses. Neural networks can be implemented using hardware circuits, or a combination of hardware and software or firmware, as a single integrated circuit, and / or as an embedded system. Neurons and synapses can be implemented using analog or digital circuits, or mixed-signal circuits.

[0040]

[0045] The input neuron 10 receives an input signal from a signal source 6, such as a sensor. The signal from source 6 may first be processed by an encoder 7 to generate a signal suitable for input to the neural network. For example, signal source 6 may generate real values ​​that need to be converted into binary spikes if the neural network is a SNN.

[0041]

[0046] The input neuron 10 generates a neuron output signal 14 in the form of real numbers for a CNN, or in the form of a spike sequence for a SNN. Neurons 11 and 12 in the subsequent layers 3 and 4 receive the output signals generated by synapses 17 and 18 and generate neuron output signals 15 and 16 in the form of real numbers or spikes. Each neuron 11 and 12 receives a synaptic output signal from one or more of the synapses 17 and 18, depending on the configured synaptic connections. For example, as shown in Figure 1, all neurons in one layer may be connected via synapses to all neurons in the next layer, or the network may be configured to make selective connections via synapses between selected neurons in adjacent layers. Many different connectivitys between neurons may be used in addition to those described, and may also include skip connections, highly recurrent liquid state machine architectures, etc.

[0042]

[0047] Each neuron 10, 11, and 12 processes the received signal (input signal or synaptic output signal) and generates neuron output signals 14, 15, and 16 as a function of the received input signal and the neuron's current state. For SNNs, the neuron output signals will contain spikes when the neuron's integral value (called membrane potential) reaches a predetermined threshold. When the threshold is reached, the neuron fires, generating spikes (i.e., voltage or current spikes) at the neuron's output and reducing the membrane potential. Each neuron is thus configured to generate neuron output signals 14, 15, and 16 in the form of spatiotemporal spike trains. The neuron output signals 14, 15, and 16 depend on the neuron's parameters, such as the input gain, integration constant, and threshold.

[0043]

[0048] Each synaptic element 17, 18 (also called a synapse) receives an output signal from one of the neurons 10, 11. Synapses 17, 18 amplify or attenuate the received output signal by a predetermined coefficient determined by their weight parameters, and this is configurable. The weight of a synapse can be positive, so that the synaptic output signal received from that synapse excites the receiving neuron, raising their membrane potential. The weight can be negative, which inhibits the neuron receiving the synaptic output from that synapse, potentially lowering their membrane potential. Alternatively, the weight can be zero, which effectively eliminates the synaptic connection between two neurons connected via the synapse. The weight for each synapse can be stored in a memory cell associated with the synapse. The values ​​of all the weights in the network are known as the weight matrix, and the weights are typically determined by a network training process.

[0044]

[0049] Figure 2 is a simplified schematic diagram of synaptic connectivity implemented as a crossbar array. The crossbar design is an efficient way to implement reconfigurable neural networks, especially when manufactured on integrated circuits. The design in Figure 2 includes a rectangular array of synapses 17 used to interconnect two layers of a neural network, for example, synapses 17 connecting neurons 10 on one side of the array to neurons 11 on the other side, or synapses 18 connecting neurons 11 on one side of the array to neurons 12 on the other side. In the embodiment of Figure 2, neurons 10 are arranged in a column, each driving a row of synapses 17. The synapses 17 are connected in a column, and the outputs of all synapses 17 in the column are added together and serve as inputs to neurons 11. By programming appropriate weights in the synaptic array and correctly configuring the interconnection system, a wide variety of network topologies can be implemented.

[0045]

[0050] The neuromorphic processor of the present invention comprises one or more convolutional neural networks (CNNs) and one or more spiking neural networks (SNNs). Training may be performed using a simulator of neuromorphic hardware, for example, on a PC. The simulator may not need to fully capture the elements and dynamics of the hardware. The network may be deployed on the hardware after training. Training may also be performed on the neuromorphic processor. The present invention provides joint training of CNNs and SNNs as described herein. After training the pipeline, the trained pipeline is deployed on the neuromorphic processor.

[0046]

[0051] A CNN provides task-dependent, hierarchical, translation-invariant feature extraction from an input signal. An encoder is provided to convert the output from the CNN, representing the extracted features, into time-encoded events for input to a SNN. The SNN provides efficient, low-power inference of the extracted features, e.g., classification. Thus, this combination provides an efficient neuromorphic processor for inference of input data, comprising, for example, classification, feature extraction, and / or signal processing.

[0047]

[0052] This invention provides a computing paradigm for a "fully differentiable" flow for creating events based on the output produced by a CNN. In this context, the term fully differentiable means that the derivative or an approximation of the derivative exists at every point in its domain. Thus, being fully differentiable involves finding an approximation or surrogate gradient for the encoder if it is not fully differentiable itself. This requirement is necessary because having a fully differentiable pipeline is a necessary condition for applying chain rules to compute gradients when training using gradient descent, for example.

[0048]

[0053] Joint training of CNNs and SNNs allows the CNN to be trained while taking into account the internal dynamics of the SNN. In conventional methods, CNNs and SNNs are trained independently because they lack differentiable encoders that would allow them to be trained directly together.

[0049]

[0054] The input data typically originates from a sensor and consists of real values, either analog (stepwise) values ​​or digital representations of analog values, such as two's complement or signed-magnitude. While these digital representations are also binary, attempting to encode information using them would not utilize the SNN capabilities, as they do not account for the dynamics of spiking neurons. By creating differentiable event coding of the data, the neuromorphic processor of the present invention can use gradient descent (GD) or other learning algorithms to explain how these real-valued features should be transformed into time-encoded binary events for the SNN.

[0050]

[0055] From a higher-level perspective, by including CNNs, encoders, and SNNs in the same training flow, feature extraction and encoding vary according to the defined loss, the SNN's dynamics, and how well the SNN performs, and vice versa. As a result, this allows the CNN to extract task-dependent, translation-invariant features, and the SNN to find continuity patterns of these features. Additionally, the need to create custom embedding solutions for each of the highly specialized preprocessing techniques and encodings is significantly reduced.

[0051]

[0056] To better illustrate the method, an example using a typical gradient descent (GD) training flow is described below, i.e., an example having a forward pass through the processing pipeline and a backward pass for training. The forward pass generates an inference output based on the entire pipeline and the input. Next is the calculation of the loss, i.e., a function that quantifies the quality of the pipeline model's predictions against ground truth. In most learning algorithms, including GD, the goal is to minimize the loss on average over all the data provided by updating the model's weights or other parameters. In GD, this task is carried out by the backward pass, which calculates the gradient of the parameters in the pipeline against the loss, starts from the output node and returns to the first node by applying a chain rule, and also updates those parameters. A description of how the present invention may be used in this setup is provided below, but the present invention is not limited to the details of this example.

[0052]

[0057] Figure 3 is a schematic diagram of a neuromorphic processor 30 comprising a pipeline having one or more CNNs 31, encoders 32, one or more SNNs 33, and decoders 34. Figure 3 illustrates a forward path through the pipeline for processing input 40 and a backward path for training the elements of the pipeline. The figure shows both a mathematical model or simulated version of pipeline 30A and a hardware implementation 30B that is directly or indirectly coupled to a sensor and used to perform inference on sensor input 36. Exemplary use cases of sensor data are described further below.

[0053]

[0058] Training of the mathematical model of pipeline 30A can be performed in simulation (for example, on a computer that simulates the behavior of different hardware components that form the pipeline within the neuromorphic processor) or using actual hardware, such as hardware on which the pipeline is implemented, including the neuromorphic processor 30. The model of pipeline 30A describes the relevant parts of the neuromorphic processor that form and / or implement the pipeline. The neuromorphic processor may implement the pipeline using various accelerators that perform the functions of different parts of the pipeline, such as CNN accelerators, SNN accelerators, and / or encoder and decoder accelerators.

[0054]

[0059] The parameters of each element in the pipeline are initialized before input processing begins. This initialization can be performed by drawing samples from a specific distribution, such as a normal distribution or a uniform distribution, using a genetic algorithm or any other standard initialization method.

[0055] [Forward Pass]

[0060] The pipeline receives an n-dimensional input 40 and processes the input in a forward pass through the pipeline to identify one or more features of the input 40.

[0056]

[0061] CNN31S performs a convolution operation on the input data 40 to project the input data to a higher or lower dimension and extract features from the input data. CNN31S generates an output representing the extracted features in real domain, and therefore the output is fed to encoder 32S so that it can then be processed by SNN33S.

[0057]

[0062] The encoder 32S takes output values ​​41 (which are real numbers) and converts them into time-binary events, i.e., a series of time-encoded binary spikes 42, which are suitable for input to the SNN33.

[0058]

[0063] The SNN33S integrates the input events 42 and generates a spiking output 43 that is supplied to the decoder 34S. The decoder 34S is an algorithm block that converts the binary spiking output 43 of the SNN33S into a real number 44.

[0059]

[0064] The output 44 generated by the decoder 34S is used to calculate the loss 45 using a loss function (e.g., mean squared error function). The loss 45 measures the quality of inference performed by the pipeline against ground truth (e.g., a target for training with a labeled dataset). The output 44 can also be fed to another CNN or SNN, depending on the application of the neuromorphic processor.

[0060] [Backward Path]

[0065] The trainable parameters of the CNN, encoder, decoder, etc., are trained by running a backward pass of the training data through a pipeline for joint training of the CNN31S and SNN33S, and optionally the encoder32S and / or decoder34S. After the loss45 is calculated, backpropagation begins in this example. Note that the embodiment described in this example uses gradient descent optimization, but other learning algorithms may be used instead.

[0061]

[0066] The loss 45 is used to compute the backpropagation gradient 46 for use when updating the parameters of the SNN33S and CNN31S (and optionally the encoder 32S and / or decoder 34S) using a chain rule. The backpropagation gradient may depend on the loss 45, the forward pass spiking output 44, the previous input 40, the in-memory computation of the neuron, and the neuron dynamics of the SNN33S, among other variables.

[0062]

[0067] One or more parameters of SNN33S can be updated based on the backpropagation gradient 46. The backpropagation gradient 46 is also used to update one or more parameters of CNN31S, and optionally to update encoder32S as well. However, in order to use the backpropagation gradient 46 from SNN33S to update CNN31S and encoder32S, the backpropagation gradient 46 is first transformed. This requires that the encoding function of encoder32S is fully differentiable, i.e., the encoder is a pipeline block that takes real values ​​and transforms them into their binary representations, and at the same time has or approximates a derivative over all of its domain. Note that surrogate gradient functions can also be applied to SNNs.

[0063]

[0068] To implement this requirement, a surrogate function can be defined that is completely differentiable, i.e., has derivatives over its entire domain. The surrogate function is used to transform the backpropagation gradient 46 to derive the transformed gradients 47, 48. The transformed gradients 47, 48 can then be used to update one or more parameters of the CNN31S and optionally the encoder32S. The transformed gradients 47, 48 can also be used to update one or more parameters of the SNN33S. The transformed gradients 47, 48 produced by the surrogate function may have a continuous approximation of the backpropagation gradient 46, and the transformed gradients may be generated by the continuous relaxation of the backpropagation gradient.

[0064]

[0069] A simple example of a differentiable encoder is the straight-through estimator (STE). In the forward path, the STE characterizes the signal m(t) as given by the following equation:

[0065]

number

[0066]

number

[0067]

[0070] Therefore, for encoder 32 using the STE coding function, the backpropagation gradient 46 can be transformed for the backward path using a surrogate function.

[0068]

number

[0069]

[0071] Another example of simple thermometer coding is described below. For example, a thermometer generates a positive integer value x that changes over a time period, as described by the following equation.

[0070]

number

[0071]

[0072] The function f(x,t) = Thermometer(x,t) is not differentiable because, especially considering that the Dirac delta function is discrete and not continuous, there is no set of differentiable operations that can be applied to the value x, resulting in Thermometer(x,t). However, the approximation Thermometer(x,t) can be used as a surrogate function to provide a completely differentiable function.

[0072]

number

[0073]

[0073] It should be noted that the present invention is not limited to the two examples described above, and includes the use of other surrogate functions that provide a fully differentiable approximation of the encoding function of encoder 32S in order to transform the backpropagation gradient 46 and derive the transformed gradients 47, 48. These examples demonstrate how jointly training a CNN and an SNN can benefit hardware optimization, for example, if the time dynamics are given by an analog SNN, this joint optimization can be used to fit the CNN output to them.

[0074]

[0074] The use of surrogate functions enables joint training of a pipeline including CNN31S and SNN33S, and optionally encoder32S. Training CNN31S, for example, updating the synaptic weights of CNN31S with transformed gradients48 computed using chain rules, provides improved feature extraction from input40 according to the computed loss45 and all previous steps.

[0075]

[0075] It should be noted that training a CNN and / or encoder may result in a CNN or encoder that produces outputs outside the data value boundaries supported by the neuromorphic processor, microcontroller, and / or system-on-a-chip hardware. The trained pipeline allows for the co-optimization of multiple networks for embedding, including cases where one or more networks are embedded on hardware with different constraints than others. This can be addressed, for example, by modifying the encoder's surrogate function to constrain the optimization by taking the constraints into account throughout the co-training process. Hardware specifications such as data type, quantization, data range, and data throughput may be taken into account during training. This results in an end-to-end optimized pipeline that also conforms to the hardware specifications while making the most of the temporal dynamics of the SNN.

[0076]

[0076] Returning to the RADAR example, due to its hierarchical characteristics, the CNN can replace the FFT to extract frequency-dependent features depending on the task being addressed. A differentiable encoder creates events from the output of the CNN, which are then fed to the SNN. Since the CNN and SNN are trained simultaneously, the pipeline of the present invention optimizes not only feature extraction but also feature coding, and the SNN then temporally integrates these sequences of coded features to perform an inference task, such as classification. For example, if the frequency of events exceeds a boundary supported by the neuromorphic hardware, the training optimization can be constrained to bring the events within the boundary or to dynamically scale these events to match the hardware specifications by changing the encoder's surrogate function. Thus, the same CNN-encoder-SNN pipeline can detect counting tasks or tracking tasks in a fully data-driven manner, taking into account the temporal dynamics of the SNN. The same pipeline can be used to train end-to-end for a specific purpose without requiring the time-consuming manual adjustment of conventional non-trainable stages. For example, people counting and people tracking applications may seem similar, but they require different pipelines to produce the correct results. This can be achieved very easily with end-to-end training.

[0077]

[0077] The CNN weights are updated using gradients calculated using chain rules to improve loss and feature extraction according to all previous steps.

[0078] [Hardware Royale]

[0078] The CNN weights, encoders, and SNN weights are deployed on the chip. The SNN33 will have certain physical parameters that are modeled before the simulation, for example if it is analog, and the values ​​will be supplied by encoders fitted to those parameters. An example of how they are fitted may be by constraining the input / output in the encoder by modifying its surrogates, as well as by a CNN trained to take those constraints and the physical modeling of the SNN33 into account.

[0079]

[0079] The hardware pipeline comprises one or more sensors, one or more CNNs 31, one or more encoders 32, one or more SNNs 33, one or more decoders 34, and inference 38, e.g., classification, results. The inference process begins with data acquisition of sensor input data 36, ​​either directly from the sensor or from sensor input data stored in memory. The sensor may be one or more cameras, radars, microphones, or other devices that collect input data. Depending on the type of sensor, this data may take various forms, such as images, audio signals, or physical measurements. Sensory input may or may not be preprocessed and contains a wide range of features, many of which may not be relevant to inference. To address this, the data first undergoes a feature extraction process using one or more CNNs.

[0080]

[0080] The CNN31 processes the sensor data by applying convolutional filters, which identify and extract important patterns and features. These filters can detect low-level features such as edges in an image or frequency components in audio. The CNN outputs a set of feature vectors or feature maps that represent the extracted features in the real domain, which summarizes the important characteristics of the original input data in a more compact form.

[0081]

[0081] Once important features are extracted, they are passed to the encoder 32. The encoder converts the feature vectors into a format suitable for the next processing stage by the SNN 33. After processing by the SNN, the resulting spike pattern is passed to the decoder 34, which converts the spiking signal back into a more interpretable format, often reconstructing a feature representation that is more understandable to downstream processes. The decoder essentially reverses the encoding process, converting the sparse spike-based information back into a denser feature representation or activation that can be further analyzed.

[0082]

[0082] Finally, the processed data is passed to the inference module 38, which interprets the decoded signals and, for example, assigns the input data to one of several predetermined classes. For other types of tasks, the inference module may output, for example, scalar values ​​for a regression task. The decisions of the inference module are based on the output from the SNN. The final output is the inference result, for example, the classification result, which may be a label, category, or probability distribution depending on the specific application. This end-to-end pipeline enables efficient processing of sensor data and robust inference of complex inputs.

[0083]

[0083] Note that this is just one example, and any number of CNN31, encoder32, or SNN33 can be used in the deployment. Also, their order can be interchangeable, which allows for more complex architectures, such as autoencoders or tasks, such as regression.

[0084] [Hardware architecture for efficiently deploying CNN-SNN architectures]

[0084] The described flow may primarily involve starting with modeling the physical properties of the SNN33, which are usually given by differential equations or, in the case of analog SNNs, by direct measurement. Furthermore, quantization that will be present in the synaptic portion of the synaptic-neuron complex may be taken into consideration.

[0085]

[0085] Input / output constraints within the encoder can be modeled, but training can converge to a solution that does not hit these constraints in such a way that these constraints only exist in the backward path.

[0086]

[0086] Assuming hardware constraints, training is performed using the previously presented method, enabling the advantages already mentioned and other advantages, for example, enabling optimization of sparser coding of CNN features, and thus reducing the overall dynamic power of the system, making it inherently more efficient. Training is used in software, but can be used to optimize the hardware forward pass.

[0087]

[0087] Figure 4 is a block diagram of a possible hardware implementation of the neuromorphic processor 30 that enables joint training of the CNN 31 and SNN 33 as described above. The architecture presented below includes a hardware accelerator that implements the aforementioned computing blocks, namely the CNN 31, encoder 32, SNN 33, and decoder 34. These communicate with a general-purpose microprocessor 53, memory 54, peripheral interface 55, direct memory access (DMA) 56, and hardware barrier 57 via an interconnect 52. Input data (e.g., sensor data) is provided to the peripheral interface 55 via an input / output 58.

[0088]

[0088] The microprocessor 53 acts as an orchestrator for data movement between the internal computing elements 31-34 and external data acquisition or communication, and for operations that are not supported or are not optimal for other accelerators present on the chip. This provides flexibility and efficiency.

[0089]

[0089] Flexibility is made possible by allowing arbitrary data movement through the main memory 54. For example, the output from CNN31 may be directly converted by encoder32 on the way to SNN33, or the output may be sent to main memory 54, or buffered or otherwise processed by microprocessor 53 and then sent to SNN33 as input.

[0090]

[0090] Efficiency is provided by several means. Firstly, the output of CNN31 is a relatively compressed data representation, which represents information at a higher level of abstraction and is more separable than that produced by earlier layers. Secondly, SNN33 natively consumes and produces events that are sparse in time and space.

[0091]

[0091] For these reasons, the advantages of this design lie in the interconnection 52 and the data manipulation operations performed between the CNN 31 and the SNN 33. Various implementations of individual blocks (whether digital, mixed analog signals, in-material computation, memristor, or other novel methods), or variations in the physical configuration (system-on-chip, system-in-package, system-on-module, etc.) can be adapted without affecting the operation of the system. Systems of different scales can benefit from physically separating computation blocks on different systems connected through network-level interconnections. Such systems (or nodes) may have distributed CNNs locally organized by one or more microprocessors, which then exchange information with each other and with one or more SNN systems.

[0092]

[0092] Input (sensor) data is received via peripheral interface 55 and written to main memory 54 by DMA 56. The CNN 31 is configured either statically or dynamically by the microprocessor 53 using parameters stored in memory, either on-chip or off-chip. This is either supplied with input data from main memory 54, whose output is sent to encoder 32 and consequently converted into events, or sent to memory 54 and processed by the microprocessor 53 (for example, in series or parallel) to generate events, which are then sent to SNN 31.

[0093]

[0093] A hardware barrier 57 may be used to coordinate data transfer from DMA 56 to encoder 32. This arbitration is based on the availability of data in DMA 56 and whether encoder 32 needs to terminate an existing process. The interaction of hardware barrier 57, encoder 32, CNN 31, and DMA 56 is illustrated in Figure 5 as a feedback loop, as DMA 56 may return data to CNN 31's SRAM if encoder 32 is not ready.

[0094]

[0094] The SNN33 is similarly configured either statically or dynamically by the microprocessor 53 using parameters stored in memory, either on-chip or off-chip. Upon receiving an encoded input event, the SNN33 performs efficient temporal integration of spatial features extracted from the input signal by the CNN31. The output of the SNN33 is decoded either by a dedicated decoder 34 or in software by the microprocessor 53.

[0095]

[0095] Figure 5 is a block diagram illustrating an example of how data can flow in the neuromorphic processor 30 of the embodiments shown in Figures 3 and 4.

[0096]

[0096] The figure is divided into two parts: the hardware side 60B and the simulation side 60A. The simulation side shows a schematic diagram of the forward and backward paths in a simulated neuromorphic processor with a pipeline that has both a CNN and an SNN.

[0097]

[0097] As described above, in the forward pass, the pipeline receives an n-dimensional input 40 and processes the input in the forward pass through the pipeline to identify one or more features of the input 40. The CNN31S performs a convolution operation on the input data 40 to project the input data to a higher or lower dimension and extract features from the input data. The CNN31S generates an output 41 representing the extracted features in the real domain, which is then fed to the encoder 32S so that it can be processed by the SNN33S. The encoder 32S takes the output values ​​41 (which are real values) and converts them into temporally binary events, i.e., a series of time-encoded binary spikes 42, which are suitable for input to the SNN33S. The SNN33S integrates the input events 42 and generates a spiking output 43 which is fed to the decoder 34S. The decoder 34S is an algorithmic block that converts the binary spiking output 43 of the SNN33S into real values ​​44. The output 44 generated by the decoder 34 is used to calculate the loss 45 using a loss function (e.g., mean squared error function).

[0098]

[0098] In the backward pass, the pipeline is trained by running a backward pass of the training data through the pipeline, with the CNN31S and SNN33S, and optionally the encoder 32S, also for joint training. After the loss 45 is calculated, in this example, backpropagation begins. The loss 45 is used to calculate the backpropagation gradient 46 for use when updating the parameters of the SNN33S and CNN31S (and optionally the encoder 32S and / or decoder) using the chain rule described above.

[0099]

[0099] The acquired parameters of the SNN, CNN, and the encoder and other parts of the pipeline can be exported to memory 54 provided in the hardware implementation of the neuromorphic processor 30, as also shown exemplarily above in relation to Figure 5 (see arrow indicated by A).

[0100]

[0100] Parameters for a CNN may include, for example, convolutional filter (weights), bias, fully connected layer weights, and / or batch normalization parameters. Parameters for an encoder may include, for example, encoder layer weights and bias, and / or learnable transformation parameters. Parameters for a SNN may include synaptic weights, membrane potential threshold, STDP parameters (if using spike timing learning), and / or neuron and synaptic dynamics parameters.

[0101]

[0101] The parameters of the SNN in the simulated pipeline can be set based on the modeling of the physical dynamics of the SNN provided in the hardware implementation. Similarly, the input constraints of the simulated encoder can be modeled based on the encoder accelerator present in the hardware implementation.

[0102]

[0102] The hardware pipeline is the same as that described in Figure 3, and is given in more detail for exemplary embodiments.

[0103]

[0103] First, the hardware is configured and initialized by the microprocessor 53 by obtaining parameters from memory 54, which are obtained through forward and backward passes via simulated pipeline memory from memory, and by initializing, for example, the CNN 31, SNN 33, encoder 32, and decoder 34 based on the obtained parameters (see box B). For example, in the SNN 33, the synaptic weights are set according to the weights obtained during the forward and backward passes.

[0104]

[0104] The hardware pipeline may comprise a sensor, one or more CNNs, one or more encoders, one or more SNNs, one or more decoders, and an inference module 38. The inference process begins with the acquisition of sensor input data 36, ​​either directly from the sensor or from sensor input data stored in memory 54. The data first undergoes a feature extraction process using one or more CNNs.

[0105]

[0105] CNN31 processes the raw sensor data by applying convolutional filters, which identify and extract important patterns and features. The CNN outputs a set of feature vectors or feature maps that represent the extracted features in the real domain, which summarizes the important characteristics of the original input data in a more compact form.

[0106]

[0106] Once important features are extracted, they can be passed to the encoder 32. The encoder converts the feature vectors into a format suitable for the next processing stage by the SNN 33. After processing by the SNN, the resulting spike pattern is passed to the decoder 34. The decoder essentially reverses the encoding process, converting the sparse spike-based information back into a dense feature representation or activation that can be further analyzed.

[0107]

[0107] Finally, the decoded data is interpreted according to the type of task being performed, e.g., classification, regression, or other task. This may be done in the inference module 38, which receives the decoded data and performs inference on the decoded data.

[0108]

[0108] Direct memory access (DMA) 56 can play a role in efficient data transfer within a pipeline involving sensors, CNNs, SNNs, and encoders and decoders. DMA 56 can optimize how data moves between components. Without DMA, a microprocessor (e.g., CPU or GPU) typically manages the movement of data between the sensor (or any peripheral) and memory. This requires microprocessor intervention, which consumes processing cycles and can create bottlenecks, especially with large datasets such as images or sensor streams. With DMA, DMA 56 allows input sensor data 36 to be transferred directly from the sensor to memory 54 (e.g., RAM or GPU memory) without involving the microprocessor for each transfer. This allows the microprocessor to focus on more critical tasks, such as performing CNN or SNN computations.

[0109]

[0109] As described above, hardware barriers 57 can be used in hardware implementations. Hardware barriers are mechanisms that help synchronize and control the execution of tasks, especially when different hardware components are operating simultaneously. In particular, hardware barriers are synchronization points that ensure the proper ordering and timing of operations across different hardware units or components. They can prevent memory inconsistencies, for example, by making components wait until certain conditions are met before proceeding with their tasks, in order to ensure that memory access and execution order are respected. In a multiprocessing environment, where data flows between components (e.g., DMA, CPU, GPU, and neural network accelerator), hardware barriers can also prevent inconsistencies by ensuring that data flows through the pipeline in the correct order.

[0110]

[0110] After the CNN31 processes the data and passes the feature map to the encoder or SNN, it may use DMA56 to move the data between memory54 and the accelerator. If the next stage of the pipeline accesses memory before the data is fully written or transferred, it can lead to a memory hazard (such as reading stale or incomplete data). The hardware barrier57 ensures that the feature map is fully written to or transferred to memory before the next component (e.g., the encoder or SNN) begins to read or process its data. This can prevent race conditions and ensure that each stage of the pipeline receives the correct input. The DMA involved may inform the hardware barrier that it has data available to the SNN and / or encoder, and the SNN and / or encoder may inform the hardware barrier that it is ready to receive the data. In each of these checks, the hardware barrier then determines whether the data is available for transfer to the next step in the pipeline (e.g., the SNN or encoder) (see checks C and D) (if so, follow Y). These two hardware barrier loops are illustrated in Figure 5.

[0111]

[0111] Hardware barriers 57 can also be used in other ways within the hardware implementation, which are not explicitly shown in Figure 5.

[0112]

[0112] For example, after sensor data has been transferred to memory via DMA, the CNN must wait until the transfer is complete before it can begin processing the data. A hardware barrier in this situation can ensure that the CNN does not begin processing the data until the DMA transfer is completely finished. This prevents a situation where the CNN attempts to process incomplete or corrupted data due to premature access.

[0113]

[0113] As another example, neural networks are often executed in parallel across multiple hardware units (e.g., CPUs, GPUs, or neural accelerators). Intermediate data (such as weights, feature maps, or spike patterns) may be shared across these units. Without proper synchronization, one processing unit may modify data while another unit is still reading or writing to the same location. Hardware barriers can ensure that memory updates by one hardware unit are visible and complete before other hardware units can access the data. This can be important in ensuring memory consistency in parallel processing. This prevents data corruption and ensures the reliable and accurate execution of parallel tasks.

[0114]

[0114] Furthermore, in real-time sensor-driven systems (such as robotics, medical devices, or autonomous vehicles), minimizing latency is crucial. For example, while an SNN is processing the current batch of data, a CNN may be working on the next batch while a DMA is handling the input data from the sensor. In this case, hardware barriers can ensure that different stages in the pipeline (CNN, SNN, decoder, etc.) are precisely synchronized without causing data collisions or unnecessary delays, ensuring optimal performance in real-time operation. This enables a smooth and predictable data flow and real-time processing.

[0115]

[0115] As can be seen from the above examples, hardware barriers can be implemented at different levels of hardware implementation depending on the complexity of the architecture and pipeline. For example, a hardware barrier active in memory can ensure that all previous memory operations (read / write) are completed before any subsequent memory operations are performed. This can be important, for example, when managing DMA transfers, where data must be completely written to or completely read from memory before moving to the next stage. Hardware barriers can also be active at the execution / pipeline level and can ensure that the current stage of computation is completely finished before the next stage begins. In a neural network pipeline, it prevents stages (such as CNNs, SNNs, or decoders) from overlapping in a way that causes errors or data corruption.

[0116] [Problems and limitations to be solved or overcome by the present invention]

[0116] Combining CNNs and SNNs (and optionally encoders and / or decoders) in the same training flow yields the advantages already presented in previous sections. This solves problems including: (1) the lack of task specificity and associated optimizations in standard SNN encoders. Feature representation as events can require a large number of operations in the encoding stage. (2) Traditional fully connected CNN or CNN-only architectures are suboptimal, for example, requiring too many computations, making methods for co-training with different network types such as SNNs beneficial. (3) Coordinated preprocessing, dimensionality reduction, and feature extraction embeddings are costly in terms of implementation time, latency, and power efficiency. (4) Embedding different encoders to convert real values ​​to binary spikes is not optimal for extracted features, as well as for the hardware neuron model and SNN dynamics. (5) Training networks within hardware and / or use case constraints is likely to be suboptimal when pipeline blocks are trained independently. (6) Searching for the optimal combination of parameters for different pipeline elements is time- and computationally intensive, and identifying the optimal value is more difficult without using a loss function that spans multiple pipeline elements. (7) Unconstrained input event frequencies do not fit hardware constraints. (8) Even if features are encoded as events, events do not necessarily produce the desired spiking behavior in an SNN, and the time series of encoded events may not take into account SNN dynamics, for example, they may decay or be membrane potential resets. (9) Translation-invariant features are key for SNNs for the task to be solved. (10) Time integration of high-level features in a CNN is important for the task to be solved.

[0117] [Uses of the invention]

[0117] From a general standpoint, the applications arising from these inventions are chips containing neuromorphic processors, because they enable a data-driven method of extracting features and encoding data in a specific way that only SNNs can benefit from.

[0118]

[0118] Due to how common CNNs and SNNs can be, there is a wide range of applications in which they can be used together. Common applications include industrial maintenance, the Internet of Things, wearable devices, object or person tracking, scene classification, pattern generation, gesture recognition, simulation of biological systems, navigation tasks, object or person detection or classification, object or person segmentation, keyword spotting, and signal or image processing. Specific applications include image and video recognition, image classification, image segmentation, image and language generation, medical image analysis, biomedical signal processing, LiDAR signal processing, RADAR signal processing, RADAR human presence detection, RADAR gesture recognition, audio scene classification, audio signal processing, natural language processing, brain-computer interfaces, autonomous driving, human activity recognition by inertial measurement devices, and bearing failure diagnosis.

[0119]

[0119] The following are some examples of applications for the neuromorphic processor 30 described above.

[0120] [Human activity recognition using inertial measurement units (IMUs)]

[0120] An IMU is a combination of an accelerator, a gyroscope, and sometimes a magnetometer sensor, which can measure acceleration, angular velocity, and orientation, respectively. When worn by a person using some kind of wearable (such as a smartwatch), an IMU can provide sufficient insight to detect or classify different tasks performed routinely by a person. IMU data is highly temporal, meaning that its information is contained not only by the events that occur, but also by the relative timing of those events to other events. As previously established, this is a scenario in which SNNs are good.

[0121]

[0121] A neuromorphic processor 30 having a pipeline comprising a CNN, a straight-through estimator (STE) as a differentiable encoder, and a SNN can be used to detect classes of human activity from IMU input data. The data used is the dataset provided in V. Fra et al., Human activity recognition: suitability of a neuromorphic approach for on-edge AIoT applications, Neuromorphic Computing and Engineering, 2(1), 014006 (2022). The data contains six channels (three accelerometers and three gyroscopes) each with 40 samples. This dataset contains seven classes, which are listed in Table 1.

[0122] [Table 1]

[0123]

[0122] The pipeline input consists of 40 data samples from six sensors (three accelerometers and three gyroscopes). The data is quantized to int8 and then fed into the CNN, which is used as a preprocessing step for the SNN. The CNN may consist of three layers, with rectified linear unit (ReLU) activations between them and different kernel sizes. Zero-padding is used so that the 40x6 data format remains intact, but the output depth of the CNN is 10, i.e., the output size of the CNN is 40x6x10. The spatial features of the CNN are then reshaped along the last axis to produce a 40x60 data shape. The data is then binarized using the STE coding function as described above, with a softmax layer added between the CNN and the encoder. The binary input spikes from the encoder are fed into the SNN, and the output spikes of the SNN are then fed into the decoder, which in this case is simply the sum of the output spikes and a post-transformation to probabilities. Depending on the decoder's result, one of the seven classes contained in the dataset (Table 1) is predicted.

[0124]

[0123] The data flow of the forward path through pipeline 60 is illustrated in Figure 6. This pipeline can be automatically implemented in neuromorphic hardware as described herein, accelerating the embedding process and eliminating the need for intensive preprocessing.

[0125]

[0124] The IMU sensor 61 provides sensor data 62, which is fed to the CNN 63. During training the pipeline, the loss function is minimized using GD. Weight updates are performed through the decoder 66 and SNN 65 based on the backpropagation loss. This reaches the STE encoder 64, which propagates the gradient that the SNN 65 had within a certain range as described above. Conceptually, the time gradient created in the SNN 65 is propagated back through the STE to the softmax function and later to the CNN 63, giving them a sense of their dynamics. Weight updates continue until they reach the first CNN layer. Weight updates and the entire process are controlled by an optimizer, which in this case was Adam. All of the former steps are performed iteratively in batches for several epochs, as is often done in normal machine learning (ML). The input size I is shown for each part of the pipeline.

[0126] [Audio Scene Classification]

[0125] In this example, the audio is 1 second long with a sampling frequency of 8 kHz, producing 8000 float 32 values. The dataset is a low-complexity acoustic scene classification using multiple devices, using only device A from the dataset. The 10 seconds of audio data is split into 10 × 1 second long audio as part of the pipeline design selection.

[0127]

[0126] The audio is resampled to 2kHz (2000 samples) by selecting every Nth value (N=8000 / 2000=4). The maximum value from among the 2000 is selected through an O(N) search, and the value is then normalized by performing a float division by the previously found maximum value. All values ​​are then rescaled to [-127, 127]int8 values ​​(multiply the float 32 by 128 and cast to int8).

[0128]

[0127] 2000 int8 values ​​are fed here to a 1D CNN (actually a 2D CNN where one dimension is always equal to 1). The CNN output is an int8 with an 8×1×8 shape. The CNN has 2,056 parameters and requires approximately 65k MACs. The CNN consists of convolution followed by batch normalization, two pairs of ReLU+convolution, batch normalization, convolution, and batch normalization.

[0129]

[0128] The next step in the pipeline is encoding, where the 8×1×8 int8 values ​​are reshaped into 64 int8 values ​​and then encoded into spikes by using a thermometer encoder. After encoding, there is an analog SNN simulated in hardware for 32 time steps. The CNN int8 output values ​​are cast back to float32, multiplied by a float32 constant, and then converted into 32×64 SNN input spikes (stored in a 32×2 unit 32 array) by performing some float32 addition, subtraction, and multiplication.

[0130]

[0129] The thermometer encoder encodes values ​​into spikes having a range of values ​​and a number of time steps. The higher the input value is relative to the configured range of values, the more consecutive spikes will be generated by the encoder, just as with a real thermometer. These input spikes are then fed into the SNN and inference is performed. The encoder is differentiable as previously mentioned, enabling simultaneous training of the CNN and the analog SNN, and thus enabling better feature coding for the analog SNN since its physical properties are taken into account in the process.

[0131]

[0130] SNN output spikes are decoded by the sum of output spikes. The number of spikes per output class is calculated, and the predicted class is the one with the most spikes (and, if the number is the same between classes, the lower index).

[0132]

[0131] In training, the ultimate goal is to minimize the loss function, which is performed using gradient descent. The backward pass of the gradient starts with decoding, goes through the SNN using surrogate gradient descent, and finally reaches the encoder. The differentiable encoder allows the gradient to reach the CNN from the SNN, while extracting the temporal information and dynamics produced by the SNN. This process continues through the CNN until it reaches the very first layer of the forward pass. The entire training process is monitored by an optimizer, namely Adamas in this case.

[0133]

[0132] Note that any feature of the embodiments disclosed herein can be appropriately combined.

[0134] [Clause]

[0133] As clause 1, a method for operating a neuromorphic processor, comprising: a convolutional neural network adapted to receive an input signal and generate a corresponding real value; an encoder connected to receive a real value and generate a corresponding time-coded binary value using an encoding function; and a spiking neural network connected to receive a time-coded binary value and generate a corresponding spiking output signal, the method comprising: calculating a loss based on the spiking output signal; calculating a backpropagation gradient based on the loss; transforming the backpropagation gradient using a fully differentiable surrogate function which is a surrogate of the encoding function; and updating one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network based on the transformed gradient.

[0135]

[0134] The method according to clause 1, further comprising, as clause 2, updating one or more parameters of the encoder based on the transformed gradient.

[0136]

[0135] The method according to clause 1 or 2, wherein the transformed gradient generated by the surrogate function has a continuous approximation of the backpropagated gradient.

[0137]

[0136] Clause 4 is the method of any one of Clauses 1 to 3, wherein the transformed gradient generated by the surrogate function is a continuous relaxation of the backpropagation gradient.

[0138]

[0137] As per clause 5, a neuromorphic processor comprises a convolutional neural network adapted to receive an input signal and generate a corresponding real value; an encoder connected to receive a real value and generate a corresponding time-encoded binary value using an encoding function; and a spiking neural network connected to receive a time-encoded binary value and generate a corresponding spiking output signal, wherein one or more processors are configured to calculate a loss based on the spiking output signal, calculate a backpropagation gradient based on the loss, transform the backpropagation gradient using a fully differentiable surrogate function which is a surrogate of the encoding function, and update one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network based on the transformed gradient.

[0139]

[0138] The neuromorphic processor according to Clause 5, wherein one or more processors are configured to update one or more parameters of an encoder based on the converted backpropagation gradient.

[0140]

[0139] The neuromorphic processor according to clause 5 or 6, wherein the transformed gradient generated by the surrogate function has a continuous approximation of the backpropagated gradient.

[0141]

[0140] A neuromorphic processor according to any one of clauses 5 to 7, wherein the transformed gradient generated by the surrogate function is a continuous relaxation of the backpropagated gradient.

Claims

1. A method for deploying a pipeline to a neuromorphic processor and then operating the neuromorphic processor, wherein the pipeline and the neuromorphic processor are A convolutional neural network adapted to receive an input signal and generate a corresponding real value, An encoder connected to receive the aforementioned real value and generate a corresponding time-encoded binary value using an encoding function, A spiking neural network connected to receive the time-encoded binary value and generate a corresponding spiking output signal, Equipped with, Herein, the pipeline models at least a portion of the neuromorphic processor, The aforementioned method, Based on the input signal and the spiking output signal, the loss for the pipeline is calculated, Based on the aforementioned loss, the backpropagation gradient for the pipeline is calculated, The backpropagation gradient is transformed using a fully differentiable surrogate function which is a surrogate of the aforementioned coding function, Based on the transformed gradient, update one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network for the pipeline. The one or more parameters of the spiking neural network and the one or more parameters of the convolutional neural network in the pipeline are deployed to the spiking neural network and the convolutional neural network of the neuromorphic processor, respectively. A method that includes [a certain feature].

2. The method according to claim 1, further comprising updating one or more parameters of the encoder in the pipeline based on the transformed gradient, and then deploying the updated one or more parameters of the encoder in the pipeline to the encoder in the neuromorphic processor.

3. The method according to claim 1 or 2, wherein the transformed gradient generated by the surrogate function provides a continuous approximation of the backpropagated gradient.

4. The method according to any one of claims 1 to 3, wherein the transformed gradient generated by the surrogate function comprises a continuous relaxation of the backpropagation gradient.

5. The method according to any one of claims 1 to 4, wherein the loss is used to compute the backpropagation gradient for use when updating the parameters of the spiking neural network, the convolutional neural network, and / or the encoder of the pipeline using the chain rule, preferably the backpropagation gradient depends on the loss, the spiking output of the forward pass, the previous input, and / or the in-memory computation and dynamics of the neurons of the spiking neural network.

6. The method according to any one of claims 1 to 5, wherein the encoder is a time encoder, a rate encoder, a group encoder, a precise spike time encoder, a thermometer encoder, and / or a straight-through estimator encoder.

7. The method according to any one of claims 1 to 6, wherein the parameters of each element of the pipeline are initialized before the processing of the input begins, preferably by drawing samples from a specific distribution, such as the uniform distribution or the Gaussian distribution, using a genetic algorithm or other initialization method.

8. The parameters for the convolutional neural network include one or more of the following: convolutional filters (weights), biases, fully connected layer weights, and / or batch normalization parameters, and / or Herein, the parameters for the encoder include the weights and biases of the encoder layer, and / or the parameters of the learnable transformation, and / or Herein, the parameters for the SNN include one or more of the following: synaptic weights, membrane potential threshold, STDP parameters (when using spike timing learning), and / or neuron and synaptic dynamics parameters. The method according to any one of claims 1 to 7.

9. The method, after the step of updating one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network of the neuromorphic processor based on the transformed gradient, The convolutional neural network of the neuromorphic processor receives a sensor input signal and generates a real sensor value corresponding to the sensor input signal. The encoder of the neuromorphic processor receives the real value corresponding to the sensor input and generates the corresponding time-encoded sensor binary value using the encoding function, The spiking neural network of the neuromorphic processor receives the time-encoded sensor binary value and generates a corresponding sensor spiking output signal. The inference module of the neuromorphic processor performs inference on the sensor input signal based on the sensor spiking output signal, The method according to any one of claims 1 to 8, further comprising:

10. The method according to claim 9, wherein the sensor input signal is obtained from an image sensor, an optical sensor, a LiDAR sensor, a RADAR sensor, an inertial measurement sensor, an accelerometer sensor, a vibration sensor, a gas sensor, a proximity sensor, an acoustic sensor, an electroencephalogram sensor, an electromyogram sensor, and / or an electrocardiogram sensor.

11. It is a neuromorphic processor, A convolutional neural network adapted to receive an input signal and generate a corresponding real value, An encoder connected to receive the aforementioned real value and generate a corresponding time-encoded binary value using an encoding function, A spiking neural network connected to receive the time-encoded binary value and generate a corresponding spiking output signal, Equipped with, Herein, one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network are such that, at least for the pipeline that models the convolutional neural network, the encoder of the neuromorphic processor and the spiking neural network are such that Based on the input signal and the spiking output signal, the loss for the pipeline is calculated, Based on the aforementioned loss, the backpropagation gradient for the pipeline is calculated, Transforming the backpropagation gradient using a fully differentiable surrogate function which is a surrogate of the aforementioned coding function, and Based on the transformed gradient, update one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network for the pipeline. Deploying one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network of the pipeline to the spiking neural network and the convolutional neural network of the neuromorphic processor, respectively. A neuromorphic processor configured by [a specific system / tool].

12. The neuromorphic processor according to claim 11, wherein one or more parameters of the encoder of the neuromorphic processor are set by updating and then deploying one or more parameters of the encoder of the pipeline based on the transformed backpropagation gradient.

13. The neuromorphic processor according to claim 11 or 12, wherein the transformed gradient generated by the surrogate function comprises a continuous approximation of the backpropagation gradient.

14. The neuromorphic processor according to any one of claims 11 to 13, wherein the transformed gradient generated by the surrogate function comprises a continuous relaxation of the backpropagation gradient.

15. The loss is used to compute the backpropagation gradient for use when updating the parameters of the spiking neural network, the convolutional neural network, and / or the encoder of the pipeline using the chain rule, preferably the backpropagation gradient depends on the loss, the spiking output of the forward pass, the previous input, and / or the in-memory computation and dynamics of the neurons of the spiking neural network, according to any one of claims 11 to 14.

16. The neuromorphic processor according to any one of claims 11 to 15, wherein the encoder is a time encoder, a rate encoder, a group encoder, a precise spike time encoder, a thermometer encoder, and / or a straight-through estimator encoder.

17. The neuromorphic processor according to any one of claims 11 to 16, wherein the parameters of each element of the pipeline are initialized before the processing of the input begins, preferably by drawing samples from a specific distribution, such as the uniform distribution or the Gaussian distribution, using a genetic algorithm or other initialization method.

18. The parameters for the convolutional neural network include one or more of the following: convolutional filters (weights), biases, fully connected layer weights, and / or batch normalization parameters, and / or Herein, the parameters for the encoder include the weights and biases of the encoder layer, and / or the parameters of the learnable transformation, and / or Herein, the parameters for the SNN include one or more of the following: synaptic weights, membrane potential threshold, STDP parameters (when using spike timing learning), and / or neuron and synaptic dynamics parameters. A neuromorphic processor according to any one of claims 11 to 17.

19. The convolutional neural network of the neuromorphic processor is configured to receive a sensor input signal and generate a sensor real value corresponding to the sensor input signal. The encoder of the neuromorphic processor is configured to receive the real value corresponding to the sensor input and to generate the corresponding time-encoded sensor binary value using the encoding function, and The spiking neural network of the neuromorphic processor is configured to receive the time-encoded sensor binary value and generate a corresponding sensor spiking output signal, and the neuromorphic processor further comprises an inference module. The inference module is configured to perform inference of the sensor input signal based on the sensor spiking output signal. A neuromorphic processor according to any one of claims 11 to 18.

20. The neuromorphic processor according to claim 19, wherein the sensor input signal is obtained from an image sensor, optical sensor, LiDAR sensor, RADAR sensor, inertial measurement sensor, accelerometer sensor, vibration sensor, gas sensor, proximity sensor, acoustic sensor, electroencephalogram sensor, electromyogram sensor, and / or electrocardiogram sensor, and preferably the neuromorphic processor is electrically or operably connected to the sensor that obtained the sensor input signal.

Images