Feature extraction and encoding of spiking neural networks using convolutional neural networks and trainable encoders for deployment in neuromorphic chips

Abstract

A method of operating a neuromorphic processor, the neuromorphic processor includes a convolutional neural network adapted to receive an input signal and to generate a corresponding real value, an encoder connected to receive the real value and to generate a corresponding time-domain encoded binary value using an encoding function, and a spiking neural network connected to receive the time-domain encoded binary value and to generate a corresponding pulsed output signal. The method includes calculating a loss based on the pulsed output signal, calculating a back-propagation gradient based on the loss, transforming the back-propagation gradient using a fully differentiable proxy function that is a proxy of the encoding function, and updating one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network based on the transformed gradient.

Description

Technical Field

[0001] The present invention relates to a neuromorphic processor, and more specifically, to a system and method for jointly training convolutional neural networks and spiking neural networks for deployment in a neuromorphic chip (e.g., in a neuromorphic processor). Background Technology

[0002] Convolutional Neural Networks (CNNs) are a type of feedforward artificial neural network (ANN) that repeatedly applies convolutional kernels to various data using convolution operations. See Chollet, F. (2021), Deep learning with Python, Simon and Schuster. Combining CNNs with training algorithms such as gradient descent (GD) or other learning algorithms can effectively identify task-specific local variations in data; that is, they are powerful data-driven feature extractors. CNNs are often used in conjunction with max pooling or average pooling operations. The former takes the maximum value of a convolutional kernel of a specified size, while the latter takes the average of all values ​​of the convolutional kernel. The combination of CNNs and pooling operations achieves a highly desirable property: translation invariance, meaning that translation of the input data does not change the model's output. For simplicity, we will refer to neural networks combining CNNs and pooling operations as "CNNs" or "CNNs" below. Furthermore, CNN layers can be stacked hierarchically, enabling them to detect frequency-dependent or higher-level features. These advantages make CNNs translation-invariant, hierarchical feature extractors, and with the development of efficient hardware accelerators, they are highly attractive to the embedded artificial intelligence industry.

[0003] Inspired by the brain, spiking neural networks (SNNs) are neural networks capable of receiving binary inputs and producing binary outputs, where each synapse-neuron complex can be conceptualized as a small processing unit with memory. The synapse scales the input event (pulse) by multiplying it by a weight value. The synaptic output is then added to the neuron's membrane potential. If a threshold is reached, the information (another pulse) propagates to the next or more synapse-neuron complexes (if any). If not, the information, or its modified version, remains in the neuron's membrane potential until more pulses are received, which are then scaled and finally accumulated to the neuron's membrane potential. This modification may involve leakage (e.g., linear or exponential leakage) and / or bias. Because the neuron tracks previously received input pulses (events), it possesses a memory function. Once the integrated event reaches the neuron's threshold, the memory (membrane potential) is reset, and the process restarts immediately or after a period of time (refractory period). The neurons in a spiking neural network (SNN) are also small processing units because their membrane potentials have internal dynamic properties, which can be described, for example, using dynamical systems theory. Their membrane potentials follow certain physical principles that allow for in-memory computation, i.e., modifying the membrane potential. Examples of in-memory processing include: integration of input events, the decay of membrane potential as a function of time, methods for resetting membrane potentials, refractory periods, and so on. Based on these inherent characteristics, spiking neural networks (SNNs) are considered a powerful pattern recognition solution in embedded artificial intelligence because efficient hardware implementations, namely neuromorphic chips, exist.

[0004] While spiking neural networks (SNNs) are highly energy efficient in performing event-driven pattern recognition tasks, they often rely on preprocessing methods that are not automatically modified during SNN training. Traditional preprocessing methods are often very energy-intensive (e.g., spectral analysis like wavelet transform) or limit their application scenarios (e.g., bandpass filters). There are many reasons why these preprocessing techniques are often used; for example, the input data to the SNN must be encoded by an encoder, that is, converted from real numbers into temporally encoded binary events (i.e., pulse sequences) with a time span, before it can be input into the SNN. Examples of encoders that can be used to create events from real values ​​include various temporal codes, such as rate coding, group coding, or exact pulse temporal coding, which are commonly used in the spiking neural network (SNN) field (see Auge, Daniel, et al., "A survey of encoding techniques for signal processing in spikingneural networks." Neural Processing Letters 53.6 (2021):4693-4710), or binarization techniques, which are not commonly used in the SNN field. Straight-through estimators (STE) (see Bengio, Yoshua, Nicholas Léonard, and Aaron Courville's paper "Estimating or propagating gradients through stochastic neurons for conditional computation." arXivpreprint arXiv:1308.3432 (2013)), binary spatter-coding (see Kanerva, P.'s paper "Binary spatter-coding of ordered k-tuples." In International conference on artificial neural networks (pp. 869-873). Berlin, Heidelberg: Springer Berlin Heidelberg), or other methods. Another reason is that the features to be detected in the sensor data span a long time period relative to the time constant of the dynamic characteristics within the spiking neurons, which is particularly important for achieving good performance in, for example, leakage current analog chips or mixed-signal chips.

[0005] While traditional data processing techniques perform well across a range of applications, customizing them for specific applications is a time-consuming task. Furthermore, if a preprocessing technique requires parameter selection, its performance is often limited by the results of the preprocessor parameter search. Once preprocessing parameters are selected, the same preprocessing block using these parameters is typically employed throughout the entire machine learning model training process, regardless of the error gradient of the task being performed. For preprocessing of ML models with a temporal component (e.g., the specific dynamics of in-memory computation in SNNs), the aforementioned preprocessing rarely achieves automatic optimization to represent the preprocessed data as events with ideal temporal dynamics.

[0006] For example, in optimization techniques like gradient descent, preprocessing parameters remain constant throughout training, even if the error gradient changes in each iteration. This fixed preprocessing can hinder performance because the data representation may no longer be suitable for the evolving model. Similarly, in models with temporal components, static preprocessing may fail to capture the temporal dependencies crucial for effective learning if backpropagation (BPTT) is used to propagate the error across time steps. The situation is even more complex for spiking neural networks (SNNs), as these models rely on event-based temporal dynamics that typical preprocessing techniques cannot account for, further limiting their performance.

[0007] For example, to decompose a frequency modulated continuous wave (FMCW) radar signal into its frequency components, the industry standard practice is to calculate its range Fast Fourier Transform (FFT). While this can accurately depict reflecting objects in a room, finding optimal features and creating binary events for tasks such as people counting differs from finding optimal features and binary events for people tracking tasks. Furthermore, the task becomes more difficult if the network analyzing these events has some internal dynamic characteristics (e.g., a subnet mask network).

[0008] Automatic feature extraction and conversion into events can complement and enhance the pattern recognition capabilities of SNNs. This invention utilizes Convolutional Neural Networks (CNNs) to achieve the above objectives because CNNs possess the aforementioned superior properties. The system of this invention comprises a forward path consisting of a CNN, an encoder, and a Spiking Neural Network (SNN). It should be noted that, for ease of explanation, the definition of "encoder" in this paper differs from that of a CNN or SNN. The encoder converts real-number input values ​​(the output of the CNN) into time-encoded binary values ​​(which serve as the input to the SNN). It is important to note that, according to the definitions of CNN and encoder, a CNN differs from an encoder because a CNN can be abstracted as a function that takes real numbers and outputs real numbers, while an encoder takes real numbers, converts them, and then outputs binary values. Even in the most extreme case where the activation function of a CNN is binary, such as when the activation function is a pass-through estimator (STE), the encoder itself is also an STE. Nevertheless, even if the output is a binary number, an SNN that takes non-binary input values ​​is not considered an encoder in this invention because its parameters can be jointly optimized with a binary SNN without problems. Furthermore, although both CNN and SNN contain interconnected networks of neurons and synapses, the encoder defined in this paper does not contain neurons and synapses.

[0009] Currently, there are several methods for combining Convolutional Neural Networks (CNNs) with Spiking Neural Networks (SNNs). For example, Q. Xu et al.'s paper "CSNN: an augmented spiking based framework with perceptron-inception" published at IJCAI 2018 describes a system that first pre-trains a CNN using a fully connected artificial neural network (ANN), then discards the ANN and replaces it with a temporal encoder and an SNN, and finally trains the SNN. S. Kim et al.'s paper "C-DNN: A 24.5-85.8 TOPS / W complementary-deep-neural-network processor with heterogeneous CNN / SNN core architecture and forward-gradient-based sparsity generation" published at ISSCC 2023 describes training the SNN using the forward gradient when its forward gradient is higher than a certain value, and then training the CNN using backpropagation. Other examples, such as U.S. Patent Nos. 11,227,210 and 11,989,645, propose using a Spiking CNN instead of combining a Convolutional Neural Network (CNN) with a Spiking Neural Network (SNN).

[0010] Although CNNs have been widely used in the field of artificial intelligence, in previously known systems they were not trained together with SNNs in the same training process. Summary of the Invention

[0011] This invention describes a system and method for integrating the training of convolutional neural networks (CNNs) and spiking neural networks (SNNs) in the same process and optimizing their deployment on neuromorphic processors. The difference between this invention and existing methods lies in using the output feature vectors of the SNN derived from the CNN as input, while simultaneously optimizing the entire process for deployment on neuromorphic processors.

[0012] In a first aspect, the present invention includes a method of deploying a pipeline to a neuromorphic processor and subsequently running the neuromorphic processor. The pipeline and the neuromorphic processor include: a convolutional neural network adapted to receive an input signal and generate a corresponding real value; an encoder connected to: receive the real value and generate a corresponding time-domain encoded binary value using an encoding function; and a spiking neural network connected to: receive the time-domain encoded binary value and generate a corresponding pulse output signal; wherein the pipeline models at least a portion of the neuromorphic processor.

[0013] The method includes: calculating a loss for the pipeline based on an input signal and a pulse output signal; calculating a backpropagation gradient for the pipeline based on the loss; transforming the backpropagation gradient using a fully differentiable surrogate function, wherein the fully differentiable surrogate function is a surrogate for the encoding function; updating one or more parameters of the pipeline's spiking neural network and one or more parameters of the convolutional neural network based on the transformed gradient; and deploying one or more parameters of the pipeline's spiking neural network and one or more parameters of the convolutional neural network to the spiking neural network and the convolutional neural network of the neuromorphic processor, respectively.

[0014] In an embodiment of the first aspect, the method may further include: updating one or more parameters of the pipeline encoder based on the transformed gradient, and subsequently deploying the updated one or more parameters of the pipeline encoder to the encoder of the neuromorphic processor. It should be noted that, according to the invention, the encoder may also have zero parameters that need to be updated, because the encoder may have surrogate gradients.

[0015] In an embodiment of the first aspect, the transformed gradient generated by the surrogate function includes a continuous approximation of the backpropagation gradient.

[0016] In an embodiment of the first aspect, the transformed gradient generated by the surrogate function includes continuous relaxation of the backpropagation gradient.

[0017] In an embodiment of the first aspect, the loss is used to compute the backpropagation gradient to update the parameters of the pipeline's spiking neural network, convolutional neural network, and / or encoder using the chain rule. Preferably, the backpropagation gradient depends on the loss, the forward-passing pulse output, the previous input, and / or the memory computation and dynamic characteristics of the neurons of the spiking neural network.

[0018] In embodiments of the first aspect, the encoder may be a time encoder, a rate encoder, a group encoder, a precise pulse time encoder, a thermometer encoder, and / or a through-pass estimator encoder.

[0019] In an embodiment of the first aspect, the parameters of each element of the pipeline are initialized before the input begins processing. Preferably, the initialization is performed by sampling from a specific distribution, such as a uniform distribution or a Gaussian distribution, using a genetic algorithm or other initialization method.

[0020] In an embodiment of the first aspect, the parameters of the convolutional neural network include one or more of convolutional filters (weights), biases, fully connected layer weights, and / or batch normalization parameters; and / or the parameters of the encoder include the weights and biases of the encoder layer, and / or parameters of the learnable transformation; and / or the parameters of the SNN include one or more of synaptic weights, membrane potential thresholds, STDP parameters (if pulse-time learning is used), and / or neuron and synaptic dynamic characteristic parameters.

[0021] In an embodiment of the first aspect, after the step of updating one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network of the neuromorphic processor based on the transformed gradient, the method further includes: the convolutional neural network of the neuromorphic processor receiving a sensor input signal and generating a sensor real value corresponding to the sensor input signal; the encoder of the neuromorphic processor receiving the real value corresponding to the sensor input and generating a corresponding time-domain encoded sensor binary value using an encoding function; and the spiking neural network of the neuromorphic processor receiving the time-domain encoded sensor binary value and generating a corresponding sensor pulse output signal; and the inference module of the neuromorphic processor performing inference on the sensor input signal based on the sensor pulse output signal.

[0022] In embodiments of the first aspect, the sensor input signal is obtained from an image sensor, an optical sensor, a lidar sensor, a radar sensor, an inertial measurement sensor, an accelerometer sensor, a vibration sensor, a gas sensor, a proximity sensor, an acoustic sensor, an electroencephalogram (EEG) sensor, an electromyogram (EMG) sensor, and / or an electrocardiogram (ECG) sensor.

[0023] In a second aspect, the present invention discloses a neuromorphic processor. The neuromorphic processor includes: a convolutional neural network adapted to receive an input signal and generate a corresponding real value; an encoder connected to receive the real value and generate a corresponding time-domain encoded binary value using an encoding function; and a spiking neural network connected to receive the time-domain encoded binary value and generate a corresponding pulse output signal.

[0024] For a pipeline that models at least the convolutional neural network, encoder, and spiking neural network of the neuromorphic processor, one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network are set as follows: a loss is calculated for the pipeline based on the input signal and the spiking output signal; a backpropagation gradient is calculated for the pipeline based on the loss; the backpropagation gradient is transformed using a fully differentiable surrogate function, which is a surrogate for the encoder function; one or more parameters of the pipeline's spiking neural network and one or more parameters of the convolutional neural network are updated based on the transformed gradient; and one or more parameters of the pipeline's spiking neural network and one or more parameters of the convolutional neural network are deployed to the neuromorphic processor's spiking neural network and convolutional neural network, respectively.

[0025] In an embodiment of the second aspect, one or more parameters of the neuromorphic processor encoder can be set by updating and subsequently deploying one or more parameters of the pipeline encoder based on the transformed backpropagation gradient. It should be noted that, according to the invention, the encoder can also have zero parameters that need to be updated, because the encoder can have surrogate gradients.

[0026] In an embodiment of the second aspect, the transformed gradient generated by the surrogate function includes a continuous approximation of the backpropagation gradient.

[0027] In an embodiment of the second aspect, the transformed gradient generated by the surrogate function includes continuous relaxation of the backpropagation gradient.

[0028] In a second aspect embodiment, the loss is used to compute the backpropagation gradient to update the parameters of the pipeline's spiking neural network, convolutional neural network, and / or encoder using a chain rule. Preferably, the backpropagation gradient depends on the loss, the forward-passing pulse output, the previous input, and / or the memory computation and dynamic characteristics of the neurons in the spiking neural network.

[0029] In embodiments of the second aspect, the encoder is a time-domain encoder, a rate encoder, a volume encoder, a precise pulse-time encoder, a thermometer encoder, and / or a through-pass estimator encoder.

[0030] In an embodiment of the second aspect, the parameters of each element of the pipeline are initialized before the input begins processing. Preferably, the initialization is performed by sampling from a specific distribution, such as a uniform distribution or a Gaussian distribution, using a genetic algorithm or other initialization method.

[0031] In embodiments of the second aspect, the parameters of the convolutional neural network include one or more of convolutional filters (weights), biases, fully connected layer weights, and / or batch normalization parameters; and / or the parameters of the encoder include the weights and biases of the encoder layer, and / or parameters of the learnable transformation; and / or the parameters of the SNN include one or more of synaptic weights, membrane potential thresholds, STDP parameters (if pulse-time learning is used), and / or neuron and synaptic dynamic characteristic parameters.

[0032] In a second aspect embodiment, the convolutional neural network of the neuromorphic processor is configured to: receive a sensor input signal and generate a sensor real value corresponding to the sensor input signal; wherein, the encoder of the neuromorphic processor is configured to: receive a real value corresponding to the sensor input and generate a corresponding time-domain encoded sensor binary value using the encoding function; and wherein, the spiking neural network of the neuromorphic processor is configured to: receive the time-domain encoded sensor binary value and generate a corresponding sensor pulse output signal; and wherein the neuromorphic processor further includes an inference module, wherein the inference module is configured to: perform inference on the sensor input signal based on the sensor pulse output signal.

[0033] In embodiments of the second aspect, the sensor input signal is obtained from an image sensor, an optical sensor, a lidar sensor, a radar sensor, an inertial measurement sensor, an accelerometer sensor, a vibration sensor, a gas sensor, a proximity sensor, an acoustic sensor, an electroencephalogram (EEG) sensor, an electromyogram (EMG) sensor, and / or an electrocardiogram (ECG) sensor; preferably, the neuromorphic processor is electrically or operatively connected to the sensor that acquires the sensor input signal.

[0034] Since Convolutional Neural Networks (CNNs) are a general-purpose feature extractor, this invention enables the following objectives by integrating CNNs and Spiking Neural Networks (SNNs) in the same training pipeline: (1) Creating a learnable end-to-end pipeline, such as from sensor data to SNN output, or interleaving CNNs, encoders, SNN modules, and other differentiable components in any order. (2) Extracting trainable features for SNNs using CNNs. Features are extracted based on joint optimization of CNNs, encoders, SNN modules, and any other differentiable parts in the pipeline. (3) Encoding the features extracted by CNNs based on joint optimization of CNNs, encoders, SNN modules, and any other differentiable parts in the pipeline. (4) Identifying optimized encodings for task-specific features extracted by CNNs, the internal dynamics of the spiking neurons constituting the SNN, and the overall system dynamics of the SNN. (5) Being able to train CNNs and encoders to represent features critical to the time domain. (6) This invention allows training CNNs on feature sets specific to an application, thereby optimizing or limiting the power consumption performance of the feature extractor for that application. (7) During software optimization, parameters (including those of CNN, encoder, and SNN) can be constrained to adapt to different hardware components and their specifications. For example, since CNN and SNN are typically deployed on different types of hardware accelerators, joint training can optimize different quantization schemes. (8) Combining efficient CNN hardware accelerators with neuromorphic hardware can reduce chip area, latency, and power consumption. (9) Shorten embedding time to achieve customized embedded solutions. (10) Create pipelines containing CNN, encoder, and SNN modules for inference tasks such as classification, regression, segmentation, or generation. (11) Reduce the number of parameters and computations in pre-built CNN-fully connected traditional networks for tasks requiring pattern recognition. Attached Figure Description

[0035] The embodiments will be described below by way of example only, with reference to the accompanying drawings, wherein corresponding reference numerals denote corresponding parts, and:

[0036] Figure 1 This is a schematic diagram of a simple neural network;

[0037] Figure 2 This is a simplified diagram of synaptic connections implemented in a cross array configuration;

[0038] Figure 3 This is a schematic diagram of forward and backward propagation in a neuromorphic processor pipeline that includes CNN and SNN, showing the simulation and deployment of the pipeline;

[0039] Figure 4 yes Figure 3 The diagram shows the hardware implementation block diagram of the neuromorphic processor.

[0040] Figure 5 It shows Figure 3 and Figure 4 A block diagram illustrating an example of data flow in a neuromorphic processor; and

[0041] Figure 6 This is a diagram showing the data flow of forward propagation in a neuromorphic processor. Detailed Implementation

[0042] The following describes some embodiments in more detail. However, it should be understood that these embodiments are merely examples and should not be construed as limiting the scope of protection of the present invention.

[0043] Figure 1 This is a schematic diagram of a simple neural network 1, which can be implemented as a convolutional neural network (CNN) or a spiking neural network (SNN). The CNN or SNN in this example includes an input layer 2 (composed of neurons 10), a hidden layer 3 (composed of neurons 11), and an output layer 4 (composed of neurons 12). The input neurons are connected to the hidden neurons 11 via synaptic elements 17, and the hidden neurons 11 are connected to the output neurons 12 via synaptic elements 18. The output 9 of the entire system is generated by the output neurons 12 in the last layer of the network. The output 9 of the output neurons 12 can then be passed to a decoder 13 (which can be implemented as the decoding layer of the network), which can then pass the information for further processing or output it to the user.

[0044] Figure 1 The CNN / SNN shown here, for simplicity, contains only three layers and a small number of neurons and synapses. However, real-world networks may contain a large number of layers, neurons, and synapses to achieve satisfactory performance. Practical implementations of CNN / SNNs can contain any number of neurons and synapses. Neural networks can be implemented using hardware circuitry or a combination of hardware / software or firmware, and can be implemented as a single integrated circuit and / or embedded system. Neurons and synapses can be implemented using analog circuitry, digital circuitry, or mixed-signal circuitry.

[0045] Input neuron 10 receives input signals from signal source 6 (e.g., a sensor). The signals from signal source 6 can first be processed by encoder 7 to generate signals suitable for neural network input. For example, signal source 6 can generate real numbers, which need to be converted into binary pulses when the neural network is a spiking neural network (SNN).

[0046] Input neuron 10 generates neuron output signal 14. For a convolutional neural network (CNN), the output signal is a real number; for a sub-neural network (SNN), the output signal is a sequence of impulses. Neurons 11 and 12 in subsequent layers 3 and 4 receive the output signals generated by synapses 17 and 18, and generate neuron output signals 15 and 16, which can be real numbers or impulses. Each neuron 11 and 12 receives synaptic output signals from one or more synapses 17 and 18, depending on the configured synaptic connections. For example, as... Figure 1 As shown, each neuron in a layer can be connected via synapses to every neuron in the next layer, or the network can be configured to establish selective connections between selected neurons in adjacent layers. Besides the above connection methods, many different neuron connection methods can be used, such as skip connections, highly recurrent liquid state machine architectures, etc.

[0047] Each neuron 10, 11, and 12 processes the received signals (input signals or synaptic output signals) and generates neuronal output signals 14, 15, and 16 based on the received input signals and the neuron's current state. For a spiking neural network (SNN), the neuron's output signal will contain a pulse when the neuron's integral value (called the membrane potential) reaches a predetermined threshold. When the threshold is reached, the neuron fires, generating a pulse (i.e., a voltage or current pulse) at its output and lowering the membrane potential. Therefore, each neuron is configured to generate neuronal output signals 14, 15, and 16 in the form of a spatiotemporal pulse sequence. The neuron output signals 14, 15, and 16 depend on neuron parameters such as input gain, integral constant, and threshold.

[0048] Each synaptic element 17, 18 (also called a synapse) receives an output signal from one of neurons 10, 11. Synapses 17, 18 amplify or attenuate the received output signal by a predetermined factor according to their configurable weight parameters. The weights of a synapse can be positive, such that the synaptic output signal received from that synapse can excite the neuron receiving that signal, thereby increasing their membrane potential. The weights can also be negative, thereby inhibiting the neuron receiving the synaptic output from that synapse and potentially decreasing their membrane potential. Alternatively, the weights can be zero, which effectively removes the synaptic connection between the two neurons connected by the synapse. The weights of each synapse can be stored in the memory unit associated with that synapse. The values ​​of all weights in the network are called the weight matrix, and these weights are typically determined during the network training process.

[0049] Figure 2This is a simplified diagram of a synaptic connection implemented as a cross array. Cross array design is an effective method for implementing reconfigurable neural networks, especially in integrated circuit manufacturing. Figure 2 The design includes a rectangular synaptic array 17 for interconnecting two layers of a neural network. For example, synapse 17 connects neurons 10 on one side of the array to neurons 11 on the other side, or synapse 18 connects neurons 11 on one side of the array to neurons 12 on the other side. Figure 2 In this embodiment, neurons 10 are arranged in a column, with each neuron driving a row of synapses 17. The synapses 17 are connected in columns, and the outputs of all synapses 17 in the same column are summed as the input to neuron 11. Various network topologies can be achieved by setting appropriate weights in the synapse array and correctly configuring the interconnection system.

[0050] The neuromorphic processor of this invention comprises one or more convolutional neural networks (CNNs) and one or more spiking neural networks (SNNs). Training can be performed, for example, on a PC using a simulator of neuromorphic hardware. This simulator does not need to fully capture the components and dynamic characteristics of the hardware. After training, the network can be deployed to the hardware. Training can also be performed on the neuromorphic processor. This invention provides a joint training method for CNNs and SNNs as described herein. After training, the trained pipeline is deployed to the neuromorphic processor.

[0051] CNNs extract task-relevant, hierarchical, translation-invariant features from input signals. An encoder converts the CNN's output (representing the extracted features) into time-coded events, which serve as input to a SNN. The SNN performs efficient, low-power inference on the extracted features, such as classification. Therefore, this combination provides a highly efficient neuromorphic processor for inference on input data, such as classification, feature extraction, and / or signal processing.

[0052] This invention provides a computational paradigm for a "fully differentiable" pipeline for creating events based on the output of a convolutional neural network (CNN). In this context, "fully differentiable" means that the gradient or its approximation exists at every point within its domain. Therefore, if the encoder itself is not fully differentiable, full differentiability involves finding approximate or alternative gradients for it. This requirement is necessary because having a fully differentiable pipeline is a prerequisite for applying the chain rule to compute gradients, for example, when training using gradient descent.

[0053] Joint training of CNNs and SNNs allows CNN training to take into account the internal dynamic characteristics of SNNs. In previous methods, due to the lack of differentiable encoders, CNNs and SNNs needed to be trained independently, making it impossible to directly train them together.

[0054] Input data typically comes from sensors and usually contains real numbers, which can be analog (hierarchical) values ​​or digital representations of analog values, such as two's complement, signed amplitude, etc. While these digital representations are also binary, they do not take into account the dynamic characteristics of spiking neurons, so attempting to encode information using them fails to fully utilize the capabilities of SNNs. By creating differentiable event codes for the data, the neuromorphic processor of this invention can use gradient descent (GD) or other learning algorithms to determine how to convert these real-valued features into time-coded binary events for the SNN.

[0055] From a higher-level perspective, by incorporating convolutional neural networks (CNNs), encoders, and spiking neural networks (SNNs) into the same training pipeline, feature extraction and encoding vary depending on the performance of the SNN under a specific loss function and its dynamic characteristics, and vice versa. This allows CNNs to extract task-relevant, translation-invariant features, while SNNs can find sequence patterns of these features. Furthermore, the need to create custom embedded solutions for each highly specific preprocessing technique and encoding method is significantly reduced.

[0056] To better illustrate this method, an example of a typical gradient descent (GD) training process is described below, which includes a forward propagation and a backpropagation. The forward propagation generates the inference output based on the entire process and the input. The loss is then calculated, which is a function quantifying the quality of the model's predictions relative to the true values. In most learning algorithms (including gradient boosting), the goal is to minimize the average loss across all given data by updating weights or other parameters.

[0057] Figure 3 This is a schematic diagram of a neuromorphic processor 30, which includes a pipeline comprising one or more convolutional neural networks (CNNs) 31, an encoder 32, one or more spiking neural networks (SNNs) 33, and a decoder 34. Figure 3 The diagram illustrates the forward propagation process of the pipeline (for processing input 40) and the backward propagation process for training the pipeline elements. The figure also shows a mathematical model or simulation version of pipeline 30A, and a hardware implementation 30B directly or indirectly connected to the sensor for inference on sensor input 36. Exemplary use cases for sensor data will be further described below.

[0058] The mathematical model training for pipeline 30A can be performed in a simulation environment (e.g., on a computer simulating the behavior of different hardware components constituting the pipeline in a neuromorphic processor) or using actual hardware, such as the hardware implementing the pipeline, like neuromorphic processor 30. Pipeline model 30A describes the relevant parts of the neuromorphic processor that constitute and / or implement the pipeline. The neuromorphic processor can be implemented using various accelerators that perform the functions of different parts of the pipeline, such as CNN accelerators, SNN accelerators, and / or encoder / decoder accelerators.

[0059] Before processing the input, the parameters of each element in the pipeline are initialized. This initialization can be achieved by drawing samples from a specific distribution (such as a normal or uniform distribution), using a genetic algorithm, or any other standard initialization method.

[0060] Forward propagation

[0061] The pipeline receives an n-dimensional input 40 and performs forward propagation processing to identify one or more features of the input 40.

[0062] The Convolutional Neural Network (CNN) 31S performs convolution operations on the input data 40, projecting the input data to higher or lower dimensions and extracting features from it. The CNN 31S generates a real-domain output representing the extracted features, which is then fed into the encoder 32S for processing by the Spiking Neural Network (SNN) 33S.

[0063] The encoder 32S acquires the output value 41 (a real number) and converts it into a time-coded binary event, namely a series of time-coded binary pulses 42, which are suitable as inputs to the SNN 33.

[0064] The SNN 33S integrates the input event 42 and generates a pulse output 43, which is then fed into the decoder 34S. The decoder 34S is an algorithm module that converts the binary pulse output 43 of the SNN 33S into a real value 44.

[0065] The output 44 generated by decoder 34S is used to calculate loss 45 using a loss function (e.g., mean squared error function). Loss 45 is used to measure the inference quality of the pipeline relative to the true values ​​(e.g., target values ​​trained using a labeled dataset). Depending on the application of the neuromorphic processor, output 44 can also be fed into another CNN or SNN.

[0066] Backpropagation

[0067] The trainable parameters of the CNN, encoder, decoder, etc., are trained by backpropagating the training data into the pipeline to jointly train the CNN 31S and SNN 33S, and optionally the encoder 32S and / or decoder 34S. In this example, backpropagation begins after calculating the loss 45. Note that the embodiment described in this example uses the gradient descent optimization method, but other learning algorithms can also be used.

[0068] Using the chain rule, the backpropagation gradient 46 is calculated with loss 45 to update the parameters of SNN 33S and CNN 31S (and optional encoder 32S and / or decoder 34S). The backpropagation gradient may depend on variables such as the loss function 45, the forward propagation spike output 44, the previous input 40, the in-memory computation of neurons, and the neuronal dynamics of SNN 33S.

[0069] One or more parameters of SNN 33S can be updated based on backpropagation gradient 46. Backpropagation gradient 46 is also used to update one or more parameters of CNN 31S and optionally to update encoder 32S. However, in order to use backpropagation gradient 46 from SNN 33S to update CNN 31S and encoder 32S, the backpropagation gradient 46 needs to be transformed first. This requires that the encoding function of encoder 32S is fully differentiable, i.e., the encoder is a pipeline module that receives real values ​​and converts them to binary representations, while having or approximately having derivatives within its domain. Note that surrogate gradient functions can also be applied to the SNN.

[0070] To achieve this, a fully differentiable surrogate function can be defined, i.e., one that has a derivative in its domain. This surrogate function is used to transform the backpropagation gradient 46, thereby deriving the transformed gradients 47 and 48. The transformed gradients 47 and 48 can be used to update one or more parameters of CNN 31S, and selectively update one or more parameters of encoder 32S. The transformed gradients 47 and 48 can also be used to update one or more parameters of SNN 33S. The transformed gradients 47 and 48 generated by the surrogate function can include continuous approximations of the backpropagation gradient 46, and the transformed gradients can be generated by continuous relaxation of the backpropagation gradients.

[0071] A simple example of a differentiable encoder is the pass-through estimator (STE). During forward propagation, the STE characterizes the signal m(t) as:

[0072] m(t) = U(m(t))

[0073] Where U is the Heaviside function (a discontinuous function whose value is zero at negative numbers and one at positive numbers). Since the Heaviside function is not differentiable at m(t) = 0, and its derivative is zero in the rest of its domain, the hardtanh function can be used as a surrogate function to approximate the backpropagation of STE(m(t)), as defined below:

[0074]

[0075]

[0076] Therefore, for encoder 32 using the STE encoding function, the backpropagation gradient 46 can be transformed using a surrogate function:

[0077] STE(m(t))' m(t))'

[0078] STE(m(t))'

[0079] Below is another simple example of thermometer encoding. For instance, this thermometer generates a positive integer value x that changes over time, as shown in the following equation:

[0080] Thermometer(x, t)

[0081] Where δ is the Dirac delta function, and T is the predefined maximum time determined by the maximum possible value of x.

[0082] The function f(x, t) = Thermometer(x, t) is nondifferentiable because there is no set of differentiable operations that can be applied to the value x to make it Thermometer(x, t), especially considering that the Dirac delta function is discrete rather than continuous. However, an approximation of Thermometer(x, t) can be used as a surrogate function to provide a fully differentiable function:

[0083] Thermometer(x)

[0084] Here, b is a scalar value used to adjust the overall gradient; a(x, t) is a function that scales the gradient based on hardware constraints (such as minimum or maximum impulse intervals, interactions between digital and / or analog components, the CNN's output values, quantization constraints, or other factors). Essentially, this approximation compresses the gradient values ​​from a time-based representation to a timeless representation, since CNNs typically lack a concept of time. However, since the surrogate function is computed based on the time dynamics of the SNN, the SNN 33S can be optimized based on values ​​from the CNN 31S.

[0085] It should be noted that the present invention is not limited to the two examples described above, but also includes the use of other surrogate functions that can provide a fully differentiable approximation of the encoder 32S encoding function, thereby transforming the backpropagation gradient 46 to derive the transformed gradients 47 and 48. These examples demonstrate how jointly training CNNs and SNNs can benefit hardware optimization; for example, if the temporal dynamics are provided by a simulated SNN, such joint optimization can be used to adapt the CNN outputs to them.

[0086] The use of surrogate functions enables the joint training pipeline, including CNN 31S and SNN 33S, as well as an optional encoder 32S. For example, the training of CNN 31S updates the synaptic weights of CNN 31S using transformed gradients 48 computed via the chain rule, providing improved feature extraction from input 40 based on the computed loss 45 and all previous steps.

[0087] Note that training CNNs and / or encoders may cause their outputs to exceed the data value boundaries supported by the neuromorphic processor, microcontroller, and / or system-on-a-chip hardware. The post-training pipeline supports joint optimization of multiple networks for embedding, including cases where one or more networks are embedded into hardware with different constraints than other networks. This can be addressed, for example, by constraining optimization by changing the encoder's surrogate function, thus taking these constraints into account throughout the joint training process. Hardware specifications such as data type, quantization, data range, and data throughput can be considered during training. This forms an end-to-end optimization pipeline that fully leverages the temporal dynamics of SNNs while meeting hardware specifications.

[0088] Returning to the RADAR example, due to its hierarchical nature, a CNN can replace an FFT to extract task-dependent frequency-related features. A differentiable encoder creates events from the CNN's output, which are then fed into an SNN. Since the CNN and SNN are trained simultaneously, the pipeline of this invention optimizes not only feature extraction but also feature encoding, which the SNN then integrates temporally to perform inference, such as classification tasks. For example, if the frequency of events exceeds the boundaries supported by the neuromorphic hardware, the training optimization can be constrained to keep it within those boundaries, or the events can be dynamically scaled by changing the encoder's surrogate function so that they match the hardware specifications. Thus, the same CNN-encoder-SNN pipeline enables detection of counting or tracking tasks in a fully data-driven manner, while taking into account the temporal dynamics of the SNN. The same pipeline can be used and trained end-to-end for a specific purpose without the need for time-consuming manual tuning of traditional non-trainable stages. For example, people counting and people tracking applications may seem similar but require different pipelines to produce correct results. This is easily achieved through end-to-end training.

[0089] The weights of the CNN are updated using gradients calculated via the chain rule, improving feature extraction based on the loss and all previous steps.

[0090] Hardware deployment

[0091] CNN weights, encoder weights, and SNN weights are deployed on the chip. The SNN 33 (e.g., if simulated) has certain physical parameters that have been modeled before simulation, and the encoder feeds it values ​​that match these parameters. For example, the encoder's input / output can be constrained by modifying the encoder's agent, and the fit can be achieved by the CNN taking these constraints and the physical modeling of the SNN 33 into account during training.

[0092] The hardware pipeline includes one or more sensors, one or more CNNs 31, one or more encoders 32, one or more SNNs 33, one or more decoders 34, and inference results 38 (e.g., classification results). The inference process begins with the acquisition of sensor input data 36, ​​which can come directly from the sensors or from sensor input data stored in memory. Sensors can be one or more cameras, radars, microphones, or other devices used to acquire input data. Depending on the sensor type, this data can take various forms, such as images, audio signals, or physical measurement data. Sensor inputs may or may not be preprocessed and contain a large number of features, many of which may be irrelevant to inference. To address this, feature extraction is first performed on the data using one or more convolutional neural networks (CNNs).

[0093] CNN 31 processes sensor data by applying convolutional filters that can identify and extract important patterns and features. These filters can detect low-level features such as edges in images or frequency components in audio. CNN outputs a set of feature maps or feature vectors that represent the extracted features in the real number domain, summarizing the key features of the original input data in a more concise form.

[0094] After extracting the key features, they are fed into encoder 32. The encoder converts the feature vectors into a format suitable for the next stage of processing by the SNN 33. After SNN processing, the generated pulse patterns are fed into decoder 34, which converts the pulse signals back into a more interpretable format, typically reconstructing feature representations that are easier for downstream processes to understand. The decoder essentially reverses the encoding process, converting sparse, pulse-based information back into dense feature representations or activation values ​​for further analysis.

[0095] Finally, the processed data is fed into inference module 38, which interprets the decoded signal, for example, assigning the input data to one of several predefined categories. For other types of tasks, the inference module can output scalar values, such as for regression tasks. The inference module's decisions are based on the SNN's output. The final output is the inference result, such as a classification result, which, depending on the application, can be a label, category, or probability distribution. This end-to-end pipeline efficiently processes sensor data, enabling robust inference on complex inputs.

[0096] Please note that this is just an example; any number of convolutional neural networks (CNNs) 31, encoders 32, or spiking neural networks (SNNs) 33 can be used in the deployment. Furthermore, their order can be interchanged, allowing for the construction of more complex architectures, such as autoencoders or tasks like regression.

[0097] Hardware architecture for efficient deployment of CNN-SNN architecture

[0098] The described process may begin by modeling the physical properties of the SNN, typically given by differential equations, or, in the case of simulating the SNN, by direct measurement. Furthermore, quantification present in the synaptic portion of the synaptic-neuron complex can be considered.

[0099] Input / output constraints in the encoder can be modeled, but these constraints only exist during backpropagation, thus enabling training to converge to a solution that does not violate these constraints.

[0100] Considering hardware constraints, training using the proposed method can achieve the aforementioned advantages, as well as others. For example, it can optimize the sparse encoding of CNN features, thereby reducing the dynamic power consumption of the entire system and essentially improving efficiency. Although training is performed in software, it can be used to optimize the hardware's forward propagation.

[0101] Figure 4 This is a block diagram of a possible hardware implementation of a neuromorphic processor 30, which is capable of jointly training the CNN 31 and SNN 33 as described above. The architecture described below includes the computational modules discussed earlier, namely the hardware accelerators that implement the CNN 31, encoder 32, SNN 33, and decoder 34. These computational modules communicate with a general-purpose microprocessor 53, memory 54, peripheral interface 55, direct memory access (DMA) 56, and hardware barriers 57 via interconnect 52. Input data (e.g., sensor data) is provided to the peripheral interface 55 via input / output 58.

[0102] The microprocessor 53 acts as a coordinator for data movement, responsible for data transfer between the internal computing units 31-34 and external data acquisition or communication, as well as handling data operations that are not supported or applicable by other accelerators on the chip. This provides flexibility and efficiency.

[0103] Flexibility is achieved by allowing data to move arbitrarily within main memory 54. For example, the output of CNN 31 can be directly transformed by encoder 32 on the path to SNN 33, or the output can be sent to main memory 54, buffered or otherwise processed by microprocessor 53, and then sent as input to SNN 33.

[0104] Efficiency is provided in several ways. First, the output of CNN31 is a relatively compressed data representation—it represents information at a higher level of abstraction, making it more separable than the information produced by earlier layers. Second, SNN33 inherently consumes and generates events that are sparse in both time and space.

[0105] For these reasons, the advantage of this design lies in the data processing operations performed between the interconnect 52 and the CNN 31 and SNN 33. Variations in the implementation of the individual modules (whether digital, analog mixed-signal, material computation, memristor, or other novel methods) or in their physical structure (on-chip, packaged, modular, etc.) do not affect the system's operation. Systems of different sizes can benefit from physically separating the computational modules across different systems connected via a network-level interconnect. Such a system (or node) can contain distributed CNNs coordinated locally by one or more microprocessors, which can then exchange information with each other and with one or more SNN systems.

[0106] Input (sensor) data is received via peripheral interface 55 and written to main memory 54 by DMA 56. The configuration of CNN 31 can be static or dynamic, and its parameters are stored by microprocessor 53 in on-chip or off-chip memory. CNN 31 obtains input data from main memory 54, and its output is either sent to encoder 32 to be converted into events, or sent to memory 54 for processing by microprocessor 53 (e.g., serial or parallel) to generate events, which are then sent to SNN 31.

[0107] Hardware barriers 57 can be used to regulate data transfer from DMA 56 to encoder 32. This coordination is based on the availability of data in DMA 56 and whether encoder 32 needs to complete a pre-existing process. Figure 5 The interaction between hardware barrier 57, encoder 32, CNN 31 and DMA 56 is depicted as a feedback loop, because if encoder 32 is not ready, DMA 56 can put data back into the SRAM of CNN 31.

[0108] The configuration of SNN 33 is similar; it can be static or dynamic, and its parameters are stored by microprocessor 53 in on-chip or off-chip memory. After receiving the encoded input event, SNN 33 performs efficient time integration on the spatial features extracted from the input signal by CNN 31. The output of SNN 33 can be decoded by a dedicated decoder 34 or by microprocessor 53 via software.

[0109] Figure 5 It shows Figure 3 and Figure 4 A block diagram illustrating an example of data flow in the neuromorphic processor 30.

[0110] The diagram is divided into two parts: hardware section 60B and simulation section 60A. The simulation section shows a schematic diagram of the forward and backward propagation of a simulated neuromorphic processor in a pipeline containing convolutional neural networks (CNNs) and spiking neural networks (SNNs).

[0111] As previously described, in the forward propagation, the pipeline receives an n-dimensional input 40 and processes it through the pipeline to identify one or more features of the input 40. A convolutional neural network 31S performs a convolution operation on the input data 40, projecting the input data to a higher or lower dimension and extracting features from it. The convolutional neural network 31S generates an output 41 representing the extracted features in the real domain, which is then fed into an encoder 32S for processing by a spiking neural network 33S. The encoder 32S receives the output value 41 (a real number) and converts it into a series of time-coded binary events, i.e., a series of time-coded binary pulses 42, suitable as input to the SNN 33S. The SNN 33S integrates the input events 42, generating a pulse output 43, which is then fed into a decoder 34S. The decoder 34S is an algorithmic module that converts the binary pulse output 43 of the SNN 33S into a real number 44. The output 44 generated by the decoder 34 is used to calculate the loss 45 using a loss function (e.g., a mean squared error function).

[0112] During backpropagation, the pipeline is trained by backpropagating the training data in the pipeline to jointly train CNN 31S and SNN 33S, as well as the optional encoder 32S. In this example, backpropagation begins after calculating loss 45. Loss 45 is used to calculate backpropagation gradient 46 to update the parameters of SNN 33S and CNN 31S (and the optional encoder 32S and / or decoder) using the chain rule described above.

[0113] The parameters obtained from the SNN, CNN, encoder, and other parts of the pipeline can be exported (see arrow A) to memory 54 included in the hardware implementation of the neuromorphic processor 30, as described above. Figure 5 As shown.

[0114] CNN parameters may include, for example, convolutional filters (weights), biases, fully connected layer weights, and / or batch normalization parameters. Encoder parameters may include, for example, the weights and biases of encoder layers and / or parameters of learnable transformations. SNN parameters may include synaptic weights, membrane potential thresholds, STDP parameters (if peak-time learning is used), and / or neuron and synaptic dynamics parameters.

[0115] The parameters of the SNN in the simulated pipeline can be set based on the physical dynamics modeling of the SNN included in the hardware implementation. Similarly, the input constraints of the simulated encoder can also be modeled based on the encoder accelerator present in the hardware implementation.

[0116] Hardware pipeline and Figure 3 The same is discussed in the previous section, but more details of an exemplary implementation are given.

[0117] First, microprocessor 53 configures and initializes the hardware by fetching parameters from memory 54 (see box B). These parameters are fetched from memory via the forward and backward propagation of a simulated pipeline, and are used to initialize, for example, CNN 31, SNN 33, encoder 32, and decoder 34. For example, in SNN 33, the weights of the synapses are set based on the weights obtained during forward and backward propagation.

[0118] The hardware pipeline may include a sensor, one or more convolutional neural networks (CNNs), one or more encoders, one or more spiking neural networks (SNNs), one or more decoders, and an inference module 38. The inference process begins with the acquisition of sensor input data 36, ​​which may come directly from the sensor or from sensor input data stored in memory 54. The data is first used for feature extraction using one or more CNNs.

[0119] CNN31 processes raw sensor data by applying convolutional filters to identify and extract important patterns and features. The CNN outputs a set of feature maps or feature vectors, which represent the extracted features in the real number domain, summarizing the key features of the original input data in a more compact form.

[0120] Once the basic features are extracted, they are passed to encoder 32. The encoder converts the feature vectors into a format suitable for the next stage of processing by SNN 33. After processing by SNN, the generated impulse patterns are passed to decoder 34. Decoder essentially reverses the encoding process, converting sparse, impulse-based information back into dense feature representations or activation values ​​for further analysis.

[0121] Finally, the decoded data is interpreted based on the type of task being performed (e.g., classification, regression, or other tasks). This can be done in inference module 38, which receives the decoded data and performs inference on it.

[0122] Direct Memory Access (DMA) 56 can play a role in efficient data transfer in pipelines involving sensors, convolutional neural networks (CNNs), spiking neural networks (SNNs), and encoders and decoders. DMA 56 optimizes how data is transferred between components. Without DMA, a microprocessor (such as a CPU or GPU) typically manages data transfer between the sensor (or any peripheral device) and memory. This requires microprocessor intervention, consumes processing cycles, and can create bottlenecks, especially when processing large datasets such as images or sensor data streams. With DMA, DMA 56 allows input sensor data 36 to be transferred directly from the sensor to memory 54 (e.g., RAM or GPU memory) without microprocessor involvement in each transfer. This allows the microprocessor to focus on more important tasks, such as running CNN or SNN computations.

[0123] As previously mentioned, hardware barriers 57 can be implemented in hardware. Hardware barriers are mechanisms that help synchronize and control task execution, especially when different hardware components are working concurrently. Specifically, hardware barriers are synchronization points that ensure the correct order and timing of operations between different hardware units or components. For example, they can prevent memory inconsistencies by forcing components to wait until certain conditions are met before continuing their tasks, thereby ensuring that memory access and execution order are followed. In multiprocessing environments, where data flows between various components (such as DMA, CPU, GPU, and neural network accelerators), hardware barriers can prevent data inconsistencies by ensuring that data flows through the pipeline in the correct order.

[0124] After the CNN 31 processes the data and passes the feature map to the encoder or SNN, DMA 56 can be used to move the data between memory 54 and the accelerator. If the next stage of the pipeline accesses memory before the data is fully written or transferred, it can lead to memory hazards (e.g., reading stale or incomplete data). Hardware barriers 57 ensure that the feature map is fully written to memory or transferred before the next component (e.g., the encoder or SNN) begins reading or processing the data. This prevents race conditions and ensures that each stage in the pipeline receives the correct input. The associated DMA notifies the hardware barrier that they have data available for the SNN and / or encoder, and the SNN and / or encoder can also notify the hardware barrier that they are ready to receive data. In each check, the hardware barrier then determines (see checks C and D) whether the data is available for transfer to the next step of the pipeline (e.g., the SNN or encoder) (if so, Y is performed). Figure 5 These two hardware barrier loops are illustrated as an example.

[0125] Hardware barrier 57 can also be used in other ways in hardware implementation, and these ways are... Figure 5 It is not explicitly shown in the text.

[0126] For example, after sensor data is transferred to memory via DMA, the CNN must wait for the transfer to complete before it can begin processing the data. In this case, a hardware barrier can ensure that the CNN does not begin processing the data until the DMA transfer is fully complete. This prevents the CNN from attempting to process incomplete or corrupted data due to premature access.

[0127] For example, neural networks typically run in parallel across multiple hardware units, such as CPUs, GPUs, or neural accelerators. Intermediate data (such as weights, feature maps, or spiking patterns) may be shared among these units. Without proper synchronization, one processing unit might modify data while another unit is still performing read and write operations on the same location. Hardware barriers ensure that a hardware unit's memory updates are visible and completed before other hardware units access the data. This is crucial for ensuring memory consistency in parallel processing. It prevents data corruption and ensures that parallel tasks execute reliably and correctly.

[0128] Furthermore, minimizing latency is crucial in real-time sensor-driven systems such as robots, medical devices, or autonomous vehicles. For example, while an SNN is processing the current batch of data, a CNN may already be processing the next batch, and a DMA may be processing input data from sensors. In such cases, hardware barriers can ensure proper synchronization between different stages in the pipeline (CNN, SNN, decoder, etc.), avoiding data conflicts or unnecessary delays, thereby guaranteeing optimal performance for real-time operations. This enables smooth, predictable data flow and real-time processing.

[0129] As illustrated in the example above, hardware barriers can be implemented at different levels of hardware implementation, depending on the architecture and pipeline complexity. For instance, a memory-level hardware barrier ensures that all previous memory operations (reads / writes) are completed before any subsequent memory operations are performed. This is particularly important when managing, for example, DMA transfers, where data needs to be fully written to or read from memory before proceeding to the next stage. Hardware barriers can also be implemented at the execution / pipeline level, ensuring that the current computational stage is fully completed before the next stage begins. In neural network pipelines, this prevents overlap between stages (e.g., CNNs, SNNs, or decoders), thus avoiding errors or data corruption.

[0130] Problems and limitations solved or overcome by the present invention

[0131] Combining CNNs and SNNs (and optional encoders and / or decoders) in the same training pipeline yields the advantages described in the previous section. This overcomes the following problems: (1) Standard SNN encoders lack task specificity and their associated optimizations. Representing features as events may require a large amount of computation during the encoding phase. (2) Traditional fully connected CNN or pure CNN architectures are not optimal, for example, requiring too much computation, so it is beneficial to use methods that train together with different types of networks such as SNNs. (3) Customized preprocessing, dimensionality reduction, and feature extraction embeddings are costly in terms of implementation time, latency, and energy efficiency. (4) Embedding different encoders to convert real values ​​into binary impulses is not optimal for the extracted features, the hardware's neuron model, and the dynamics of SNNs. (5) Training the network under hardware and / or use case constraints is likely not optimal when pipeline modules are trained independently. (6) Finding the optimal combination of parameters for different pipeline elements is time-consuming and labor-intensive, and it is even more difficult to determine the optimal value without using a loss function across multiple pipeline elements. (7) Unconstrained input event frequencies do not conform to hardware constraints. (8) Even if features are encoded as events, these events may not necessarily produce ideal impulse behavior in SNNs; the encoded event time series may not be able to explain the dynamic characteristics of SNNs, for example, they may decay or the membrane potential may reset. (9) Translation-invariant features are crucial for SNNs to solve the expected tasks. (10) The time integration of high-level features in CNNs is crucial for solving the expected tasks.

[0132] Application of the present invention

[0133] Overall, the application derived from this invention is a chip containing a neuromorphic processor, because it allows for the extraction of features in a data-driven manner and the encoding of data in a specific way, which only SNNs can benefit from.

[0134] Because of their strong versatility, CNNs and SNNs can be combined for applications across a wide range of fields. General applications include industrial maintenance, the Internet of Things (IoT), wearable devices, object or person tracking, scene classification, pattern generation, gesture recognition, biological system simulation, navigation tasks, object or person detection or classification, object or person segmentation, keyword recognition, and signal or image processing. Specific applications include image and video recognition, image classification, image segmentation, image and speech generation, medical image analysis, biomedical signal processing, LiDAR signal processing, radar signal processing, radar human presence detection, radar gesture recognition, audio scene classification, audio signal processing, natural language processing, brain-computer interfaces, autonomous driving, inertial measurement unit (IMU) human activity recognition, and bearing fault diagnosis.

[0135] The following are some application examples of the aforementioned neuromorphic processor 30.

[0136] Human Activity Recognition Based on Inertial Measurement Unit (IMU)

[0137] An IMU (Integrated Measurement Unit) is a complex of sensors, including an accelerometer, a gyroscope, and sometimes a magnetometer, used to measure acceleration, angular velocity, and direction, respectively. If a person wears an IMU device (such as a smartwatch), the IMU can provide enough information to detect or classify the different tasks a person performs daily. IMU data is highly temporal, meaning its information includes not only the event itself but also its relative time to other events. As mentioned earlier, SNNs (Short-Range Neural Networks) are well-suited for handling this situation.

[0138] A neuromorphic processor 30, whose pipeline includes a convolutional neural network (CNN), a pass-through estimator (STE) (as a differentiable encoder), and a sub-NN, can be used to detect human activity categories from IMU input data. The dataset used in this paper comes from the paper "Human activity recognition: suitability of a neuromorphic approach for on-edge AIoT applications" by V. Fra et al. (published in Neuromorphic Computing and Engineering 2(1), 014006(2022)). The dataset contains six channels (three accelerometers and three gyroscopes), with 40 samples per channel. The dataset contains seven categories, which are described in Table 1.

[0139]

[0140] Table 1. Categories of human activity recognition datasets.

[0141] The pipeline's input consists of 40 data samples from six sensors (three accelerometers and three gyroscopes). The data is quantized to int8 type and then fed into a convolutional neural network (CNN), which serves as a preprocessing step for a spiking neural network (SNN). The CNN can contain three layers, using Rectified Linear Unit (ReLU) activation functions between layers and employing different kernel sizes. Zero padding is used to maintain the 40x6 data format, but the CNN's output depth is 10, resulting in an output size of 40x6x10. The CNN's spatial features are then reshaped along the last axis to achieve a 40x60 data shape. Next, the data is binarized using the previously described STE encoding function, and a softmax layer is added between the CNN and the encoder. The binary input pulses from the encoder are fed into the SNN, and the SNN's output pulses are then fed into the decoder, in this case, where only the output pulses are summed, and the posterior probabilities are converted into probabilities. Based on the decoder's results, one of the seven classes contained in the dataset (Table 1) is predicted.

[0142] Figure 6 illustrates the forward propagation data flow of pipeline 60. As described in this invention, this pipeline can be automatically implemented in neuromorphic hardware, thereby accelerating the embedding process and eliminating the need for intensive preprocessing.

[0143] IMU sensor 61 provides sensor data 62, which is input into CNN 63. During the training pipeline, gradient descent (GD) is used to minimize the loss function. Weight updates are based on the backpropagation loss, performed through decoder 66 and SNN 65. It reaches STE encoder 64, which propagates the gradients generated by SNN 65 within a certain range, as previously described. Conceptually, the temporal gradients generated in SNN 65 are backpropagated. STE is applied to the softmax function, and subsequently to CNN 63, enabling them to perceive their dynamic characteristics. Weight updates continue until the first CNN layer is reached. Weight updates and the entire process are controlled by an optimizer, in this case, the Adam optimizer. All the aforementioned processes are executed iteratively in batches (epochs), lasting for multiple epochs, as is typically done in general machine learning. The input size I for each part of the pipeline is listed.

[0144] Audio scene classification

[0145] In this example, the audio duration is 1 second, the sampling frequency is 8kHz, and 8000 float32 values ​​are generated. The dataset is "Multi-device Low-Complexity Acoustic Scene Classification," using only device A from the dataset. Following the pipeline design, the 10 seconds of audio data is segmented into 10 1-second audio clips.

[0146] The audio is resampled to 2kHz (2000 samples) by selecting every N values ​​(N=8000 / 2000=4). The maximum value is selected from these 2000 values ​​using an O(N) search, and these values ​​are then normalized by dividing the float value by the previously found maximum value. Finally, all values ​​are rescaled to the int8 value range of [-127, 127] (float32 multiplied by 128, then cast to int8).

[0147] We now input 2000 int8 values ​​into a one-dimensional convolutional neural network (CNN) (actually a two-dimensional CNN, where one dimension is always equal to 1). The CNN output is of type int8, with a shape of 8x1x8. This CNN has 2056 parameters and requires approximately 65,000 MACs. The CNN consists of one convolutional layer, one batch normalization layer, two pairs of ReLU+ convolutional layers, another batch normalization layer, another convolutional layer, and one more batch normalization layer.

[0148] The next step in the pipeline is encoding, where 8x1x8 int8 values ​​are reshaped into 64 int8 values, which are then encoded into pulses using a thermometer encoder. After encoding, a simulated SNN is simulated in hardware, running for 32 time steps. The CNN's int8 output values ​​are converted back to float32, multiplied by a float32 constant, and then converted into 32x64 SNN input pulses (stored in a 32x2 uint32 array) through some float32 addition, subtraction, and multiplication operations.

[0149] The thermometer encoder encodes values ​​as pulses, with a value range and multiple time steps. The higher the input value is relative to the configured value range, the more consecutive pulses the encoder generates, analogous to a real-world thermometer. These input pulses are then fed into the SNN for inference. The encoder is differentiable, as previously mentioned, allowing simultaneous training of the CNN and the simulated SNN, thus enabling better feature encoding for the simulated SNN because its physical properties are taken into account during the process.

[0150] The output pulses of the SNN are decoded by summing the output pulses. The number of pulses for each output category is calculated, and the predicted category is the one with the most pulses (if the number of categories is the same, the category with the smallest index value is selected).

[0151] During training, the ultimate goal is to minimize the loss function, which is achieved through gradient descent. Gradient backpropagation begins with decoding, proceeds through a surrogate gradient descent method via a spiking neural network (SNN), and finally reaches the encoder. The differentiable encoder allows gradients to propagate from the SNN to a convolutional neural network (CNN), while simultaneously extracting the temporal information and dynamic characteristics generated by the SNN. This process continues through the CNN until the first layer of forward propagation is reached. The entire training process is supervised by an optimizer, in this case, the Adamax optimizer.

[0152] Please note that features of any of the embodiments disclosed herein may be combined in an appropriate manner.

[0153] Terms and Conditions

[0154] As described in Clause 1, a method of operating a neuromorphic processor, the processor comprising: a convolutional neural network adapted to receive an input signal and generate a corresponding real value; an encoder connected to receive the real value and generate a corresponding time-domain encoded binary value using an encoding function; and a spiking neural network connected to receive the time-domain encoded binary value and generate a corresponding spiking output signal; the method comprising:

[0155] Calculate the loss based on the pulse output signal; calculate the backpropagation gradient based on the loss; transform the backpropagation gradient using a fully differentiable surrogate function, which is a surrogate for the encoding function; and update one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network based on the transformed gradient.

[0156] As described in Clause 2, the method described in Clause 1 further includes updating one or more parameters of the encoder based on the transformed gradient.

[0157] As described in Clause 3, the method described in Clause 1 or Clause 2 is employed, wherein the transformed gradient generated by the surrogate function comprises a continuous approximation of the backpropagation gradient.

[0158] As described in Clause 4, the method described in any of the preceding clauses, wherein the transformed gradient generated by the surrogate function includes continuous relaxation of the backpropagation gradient.

[0159] As described in Clause 5, a neuromorphic processor includes: a convolutional neural network adapted to receive an input signal and generate a corresponding real value; an encoder connected to receive the real value and generate a corresponding time-domain encoded binary value using an encoding function; a spiking neural network connected to receive the time-domain encoded binary value and generate a corresponding spiking output signal; and one or more processors configured to:

[0160] Calculate the loss based on the pulse output signal; calculate the backpropagation gradient based on the loss; transform the backpropagation gradient using a fully differentiable surrogate function (which is a surrogate for the encoding function); and update one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network based on the transformed gradient.

[0161] As described in Clause 6, the neuromorphic processor described in Clause 5, wherein the one or more processors are configured to update one or more parameters of the encoder based on the transformed backpropagation gradient.

[0162] As described in Clause 7, the neuromorphic processor described in Clause 5 or Clause 6, wherein the transformed gradient generated by the surrogate function contains a continuous approximation of the backpropagation gradient.

[0163] As described in Clause 8, the neuromorphic processor described in any of Clauses 5 to 7, wherein the transformed gradient generated by the surrogate function contains a continuous relaxation of the backpropagation gradient.

Claims

1. A method of deploying a pipeline to a neuromorphic processor and subsequently operating the neuromorphic processor, the pipeline and the neuromorphic processor comprising: Convolutional neural networks are suitable for receiving input signals and generating corresponding real values. The encoder is connected to receive the real value and use an encoding function to generate the corresponding time-domain encoded binary value. as well as A spiking neural network is connected to receive the time-domain encoded binary value and generate a corresponding pulse output signal. The pipeline models at least a portion of the neuromorphic processor. The method includes the following steps: The loss is calculated for the pipeline based on the input signal and the pulse output signal; The backpropagation gradient is calculated for the pipeline based on the loss; The backpropagation gradient is transformed using a fully differentiable surrogate function, which is a surrogate for the encoding function; The pipeline's spiking neural network and convolutional neural network are updated based on the transformed gradient. One or more parameters of the pipeline's spiking neural network and one or more parameters of the convolutional neural network are respectively deployed to the spiking neural network and the convolutional neural network of the neuromorphic processor.

2. The method according to claim 1, further comprising the following step: The pipeline encoder updates one or more parameters based on the transformed gradient, and then deploys the updated pipeline encoder one or more parameters to the encoder of the neuromorphic processor.

3. The method according to claim 1 or 2, wherein the transformed gradient generated by the surrogate function comprises a continuous approximation of the backpropagation gradient.

4. The method according to any one of the preceding claims, wherein the transformed gradient generated by the surrogate function includes a continuous relaxation of the backpropagation gradient.

5. The method according to any one of the preceding claims, wherein the loss is used to calculate the backpropagation gradient to update the parameters of the pipeline's spiking neural network, convolutional neural network, and / or encoder using a chain rule, preferably wherein the backpropagation gradient depends on the loss, the forward-passing pulse output, the previous input, and / or the memory computation and dynamic characteristics of the neurons of the spiking neural network.

6. The method according to any one of the preceding claims, wherein the encoder is a time-domain encoder, a rate encoder, a group encoder, a precise pulse-time encoder, a thermometer encoder, and / or a through-pass estimator encoder.

7. The method according to any one of the preceding claims, wherein, The parameters of each component in the pipeline are initialized before the input begins processing. Preferably, the initialization is performed by sampling from a specific distribution, such as a uniform distribution or a Gaussian distribution, using a genetic algorithm or other initialization method.

8. The method according to any one of the preceding claims, wherein the parameters of the convolutional neural network include one or more of convolutional filters (weights), biases, fully connected layer weights, and / or batch normalization parameters; and / or The encoder parameters include the weights and biases of the encoder layer, and / or the parameters of the learnable transform; and / or The parameters of the spiking neural network include one or more of the following: synaptic weights, membrane potential thresholds, STDP parameters (if pulse timing learning is used), and / or neuron and synaptic dynamic characteristic parameters.

9. The method according to any one of the preceding claims, wherein, The method, after the step of updating one or more parameters of the neuromorphic processor's spiking neural network and one or more parameters of the convolutional neural network based on the transformed gradient, further includes: The convolutional neural network of the neuromorphic processor receives sensor input signals and generates sensor real values ​​corresponding to the sensor input signals. The encoder of the neuromorphic processor receives real values ​​corresponding to the sensor input and uses the encoding function to generate corresponding time-domain encoded sensor binary values; and The neuromorphic processor's spiking neural network receives the time-domain encoded sensor binary value and generates a corresponding sensor pulse output signal. The inference module of the neuromorphic processor performs inference on the sensor input signal based on the sensor pulse output signal.

10. The method according to claim 9, wherein the sensor input signal is obtained from an image sensor, an optical sensor, a lidar sensor, a radar sensor, an inertial measurement sensor, an accelerometer sensor, a vibration sensor, a gas sensor, a proximity sensor, an acoustic sensor, an electroencephalogram (EEG) sensor, an electromyogram (EMG) sensor, and / or an electrocardiogram (ECG) sensor.

11. A neuromorphic processor, comprising: Convolutional neural networks are suitable for receiving input signals and generating corresponding real values. The encoder is connected to receive the real value and use an encoding function to generate the corresponding time-domain encoded binary value. A spiking neural network is connected to receive the time-domain encoded binary value and generate a corresponding pulse output signal. as well as In a pipeline that models at least the convolutional neural network, encoder, and spiking neural network of the neuromorphic processor, one or more parameters of the spiking neural network and one or more parameters of the convolutional neural network are set in the following manner: The loss is calculated for the pipeline based on the input signal and the pulse output signal; The backpropagation gradient is calculated for the pipeline based on the loss; The backpropagation gradient is transformed using a fully differentiable surrogate function, which is a surrogate for the encoding function; The pipeline's spiking neural network and convolutional neural network are updated based on the transformed gradient. One or more parameters of the pipeline's spiking neural network and one or more parameters of the convolutional neural network are respectively deployed to the spiking neural network and the convolutional neural network of the neuromorphic processor.

12. The neuromorphic processor of claim 11, wherein, One or more parameters of the encoder of the neuromorphic processor are set by updating one or more parameters of the encoder based on the transformed backpropagation gradient and subsequently deploying the pipeline.

13. The neuromorphic processor of claim 11 or 12, wherein the transformed gradient generated by the surrogate function comprises a continuous approximation of the backpropagation gradient.

14. The neuromorphic processor according to any one of claims 11-13, wherein the transformed gradient generated by the surrogate function includes a continuous relaxation of the backpropagation gradient.

15. The neuromorphic processor according to any one of claims 11-14, wherein the loss is used to compute the backpropagation gradient to update the parameters of the pipelined spiking neural network, convolutional neural network, and / or encoder using a chain rule, preferably wherein the backpropagation gradient depends on the loss, the forward-passing pulse output, the previous input, and / or the memory computation and dynamic characteristics of the neurons of the spiking neural network.

16. The neuromorphic processor according to any one of claims 11-15, wherein the encoder is a time-domain encoder, a rate encoder, a volume encoder, a precise pulse-time encoder, a thermometer encoder, and / or a pass-through estimator encoder.

17. The neuromorphic processor according to any one of claims 11-16, wherein the parameters of each element of the pipeline are initialized before the input begins processing, preferably, the initialization is performed by sampling from a specific distribution, such as a uniform distribution or a Gaussian distribution, using a genetic algorithm or other initialization method.

18. The neuromorphic processor according to any one of claims 11-17, wherein the parameters of the convolutional neural network include one or more of convolutional filters (weights), biases, fully connected layer weights, and / or batch normalization parameters; and / or The encoder parameters include the weights and biases of the encoder layer, and / or the parameters of the learnable transform; and / or The parameters of the SNN include one or more of the following: synaptic weights, membrane potential thresholds, STDP parameters (if pulse-time learning is used), and / or neuron and synaptic dynamic characteristic parameters.

19. The neuromorphic processor according to any one of claims 11-18, in, The convolutional neural network of the neuromorphic processor is configured to: receive sensor input signals and generate sensor real values ​​corresponding to the sensor input signals; The encoder of the neuromorphic processor is configured to: receive a real value corresponding to the sensor input, and use the encoding function to generate a corresponding time-domain encoded sensor binary value; and The neuromorphic processor's spiking neural network is configured to: receive the time-domain encoded binary values ​​of the sensor and generate corresponding sensor pulse output signals; and The neuromorphic processor further includes an inference module, which is configured to perform inference on the sensor input signal based on the sensor pulse output signal.

20. The neuromorphic processor of claim 19, wherein the sensor input signal is obtained from an image sensor, an optical sensor, a lidar sensor, a radar sensor, an inertial measurement sensor, an accelerometer sensor, a vibration sensor, a gas sensor, a proximity sensor, an acoustic sensor, an electroencephalogram (EEG) sensor, an electromyogram (EMG) sensor, and / or an electrocardiogram (ECG) sensor; preferably, wherein the neuromorphic processor is electrically connected or operatively connected to the sensor that acquires the sensor input signal.

Images