Method to build and deploy spiking neural networks on hardware device

Abstract

The present invention relates to a method for deploying a spiking neural network to a hardware device. The method comprises providing the spiking neural network, training the spiking neural network to obtain a trained spiking neural network, mapping the neurons and synapses of the trained spiking neural network to corresponding components of the hardware device, simulating the deployment of the trained spiking neural network to the hardware device using the obtained mapping, deploying the trained spiking neural network to the hardware device using the mapping and the simulation. The training, mapping, simulating and deploying steps are performed using hardware information of the hardware device, wherein the hardware information comprises at least one of hardware resource constraints, hardware connectivity constraints, dynamic ranges of hardware design parameters, neuron and synapse characterization or statistical data, reconfigurability, programmability, yield, computational resources, temporal characteristics, constraints on preprocessing, interfaces and peripherals, and available encoders and / or decoders.

Description

Method to build and deploy spiking neural networks on hardware deviceTECHNICAL FIELD

[0001] The present invention relates to a method for creating and deploying a spiking neural network to a target device, in particular in a hardware-aware or in a hardware-agnostic manner, and a hardware device obtained using this method.BACKGROUND

[0002] A Spiking Neural Network (SNN) is a type of neural network which has a few benefits over other types of neural networks. Two of the most important benefits are sparse computation and asynchronous processing, which will lead to a low-power and low-latency solution. However, as an emerging technology, there is a lack of software and hardware support in the market to enable users to develop and deploy an SNN-based application.

[0003] In particular, SNNs deployed to a particular target device may perform sub-optimally or may even fail when using the current support available, due to mismatches between the network's computational requirements, application and user requirements, and the hardware's capabilities, resulting in issues such as excessive latency, insufficient memory, inadequate processing power, poor classification results, inaccurate spike timing, degraded learning effectiveness, and overall system instability.SUMMARY OF INVENTION

[0004] In order to solve the aforementioned problems, the subj ect-matter of the present claims is proposed. The present invention provides a method to design, test and deploy an SNN-based application onto an analog mixed-signal accelerator.

[0005] According to a first aspect of the disclosure, a method for deploying a spiking neural network to a hardware device is disclosed. The method may comprise: providing the spiking neural network, the spiking neural network comprising neurons that communicate via spike train signals, and synapses that interconnect the neurons and modulate the spike train signals; training the spiking neural network to obtain a trained spiking neural network; mapping the neurons and synapses of the trained spiking neural network to corresponding components of the hardware device, wherein the components emulate the neurons and synapses of the trained spiking neural network in the hardware device; simulating the deployment of the trained spiking neural network to the hardware device using the obtained mapping, by simulating thedeployment to the hardware device; deploying the trained spiking neural network to the hardware device using the mapping and the simulation, such that the corresponding components of the hardware device emulate the neurons and synapses of the trained spiking neural network.

[0006] The training, mapping, simulating and deploying steps may be performed using hardware information of the hardware device, wherein the hardware information comprises at least one of hardware resource constraints, hardware connectivity constraints, dynamic ranges of hardware design parameters, neuron and synapse characterization or statistical data, reconfigurability, programmability, yield, computational resources, temporal characteristics, constraints on preprocessing, interfaces and peripherals, and available encoders and / or decoders.

[0007] According to an embodiment of the first aspect, the step of providing comprises creating the spiking neural network by selecting neurons, synapses and topology models from a repository, wherein each neuron, synapse and topology model is selected based on hardware information. The hardware information may comprises: a presence of feedforward and feedback channels on your chip, which limits the selection of feedforward and feedback loops in the network topology; a number of resources, which limits the selection of neurons and synapses; a temporal characteristic of a synapse, which limits the filtering characteristics of synapses; availability of preprocessors, which limits the numbers of inputs.

[0008] According to an embodiment of the first aspect, the simulation may be done by simulating the deploying of the spiking neural network to one or more simulated or physical instances of the hardware device.

[0009] According to an embodiment of the first aspect, the training step may be performed using hardware information by using the neuron and synapse characterization or statistical data to reproduce the behaviour of the hardware device; and / or by using the neuron and synapse characterization or statistical data and the dynamic ranges of hardware design parameters to apply a quantization during the training process.

[0010] According to an embodiment of the first aspect, the mapping step may be performed using hardware information by finding a mapping for the spiking neural network while taking into account the hardware resource and connectivity constraints, the non-uniformity of analog devices and / or an optimization target, preferably wherein the optimization target comprises power consumption, accuracy, latency and / or hardware utilization rate.

[0011] According to an embodiment of the first aspect, the mapping step may accept input data from an application as an input argument, analyse the properties of the data, and subsequently generate a mapping which is optimized for the specific application.

[0012] According to an embodiment of the first aspect, the mapping step may generate a hardware configuration file that configures one or more components within the target hardware device, preferably wherein the hardware configuration file is a binary file.

[0013] According to an embodiment of the first aspect, the simulating step may be performed using hardware information by modelling one or more internal digital and analog components of the hardware device, taking into account the constraints and characteristics of the hardware device.

[0014] According to an embodiment of the first aspect, the simulation in the simulating step may be either cycle accurate or semi-cycle accurate. When the simulation is semi-cycle accurate the timing of events at interface level as well as the analog components such as the one or more synapses and neurons may be simulated accurately, while the details inside the digital components are simulated to a particular degree as a trade-off between model accuracy and runtime performance.

[0015] According to an embodiment of the first aspect, the simulating step may model the statistical property of the non-uniformity of the hardware device, so that behaviour of a large set of hardware devices can be simulated.

[0016] According to an embodiment of the first aspect, the simulating step may comprise the analysis of the performance of the spiking neural network on the hardware device, using a power, accuracy and latency characterization of the hardware device in the simulation.

[0017] According to an embodiment of the first aspect, the mapping step and the simulating step may utilize the same architecture definition and characterization or statistical data. The architecture definition and characterization or statistical data may be changeable in such a way that the mapping step and the simulating step are retargetable to a different hardware device.

[0018] According to an embodiment of the first aspect, the deploying step may be performed using hardware information by generating a binary file that configures the components within the hardware device by utilizing a target device specific template which ensures that the correct on-chip components of the device are configured with the correct parameters.

[0019] According to a second aspect of the disclosure, a method for determining hardware information of a hardware device to which a spiking neural network for an application is suitable to be deployed is disclosed. The method may comprise: providing the spiking neuralnetwork, the spiking neural network comprising neurons, that communicate via spike train signals, and synapses that connect the neurons and modulate the spike train signals; training the spiking neural network for the application to obtain a trained spiking neural network; creating hardware information of the hardware device, such that the hardware device has corresponding components that are configured to emulate the neurons and synapses of the trained spiking neural network; mapping the trained spiking neural network to the dedicated components of the hardware device; simulating the deployment of the trained spiking neural network to the hardware device using the obtained mapping, by simulating the deployment to one or more hardware device instances, and wherein the hardware information of the hardware device is changeable depending on the outcome of the simulation; repeating the steps of providing, training, creating, mapping and simulating until a pre-defined performance threshold is reached and, when the pre-defined performance threshold is reached outputting the obtained hardware information of the hardware device, as well as the obtained spiking neural network. The hardware information may comprise at least one of hardware resource constraints, hardware connectivity constraints, dynamic ranges of hardware design parameters, neuron and synapse characterization or statistical data, reconfigurability, programmability, yield, computational resources, temporal characteristics, constraints on preprocessing, interfaces and peripherals, and available encoders and / or decoders.

[0020] According to an embodiment of the second aspect, the hardware information of the hardware device may be initialised by selecting pre-determined values for the hardware information or the hardware information of the hardware device may be initialized using the mapper, by selecting values that depend on the trained spiking neural network. More preferably, the properties of the trained spiking neural network may determine the values with which the hardware information of the hardware device is initialized.

[0021] According to an embodiment of the second aspect, in subsequent steps of the mapping, the hardware information of the hardware device may be changed based on updates to the trained spiking neural network.

[0022] According to a third aspect of the disclosure, a hardware device, comprising a spiking neural network deployed thereto using the method of the first aspect of the disclosure is disclosed.

[0023] According to a fourth aspect of the disclosure, a hardware device, configured according to hardware information obtained using the method of the second aspect of the disclosure is disclosed.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Embodiments will now be described, by way of example only, with reference to the accompanying drawings in which corresponding reference symbols indicate corresponding parts, and in which:

[0025] FIG. 1 shows a schematic representation of a method according to the invention to create a spiking neural network, train the spiking neural network, map the spiking neural network to a target device, simulate the spiking neural network on a simulated instance of the target device, finally and deploy the spiking neural network to a target hardware device, using hardware information; and

[0026] FIG. 2 shows a schematic representation of a method according to the invention to create a spiking neural network, train the spiking neural network, map the spiking neural network to a target device, simulate the spiking neural network on a simulated instance of the target device, in order to obtain hardware information for a potential deployment to a target hardware device complying with the hardware information.DESCRIPTION OF EMBODIMENTS

[0027] Hereinafter, certain embodiments will be described in further detail. It should be appreciated, however, that these embodiments should not be construed as limiting the scope of protection for the present disclosure.

[0028] FIG. 1 shows a schematic representation of a method 100 according to the invention to create 101 a spiking neural network, train 102 the spiking neural network, map 103 the spiking neural network to a target device, simulate 106 the spiking neural network on a simulated instance of the target device, finally and deploy 104,105 the spiking neural network to a target hardware device, using hardware information 108.

[0029] The method to deploy an SNN-based application may include five major steps: network creation 101, training 102, simulation 106, analysis 107 and deployment 104,105.

[0030] In a first embodiment of the invention, hardware information about the target hardware device is sourced into at least one, at least two, at least three, at least four or each of these steps, enabling hardware-aware development and mapping of SNNs onto a target hardware device, without requiring user knowledge of the internal architecture of the device for at least one, two three, four or each of these steps respectively. The hardware information comprises connectivity constraints, resource constraints, parametric ranges, componentcharacterization data, supported encoder and decoder types, reconfigurability, programmability, yield, computational resources, temporal characteristics, constraints on preprocessing, interfaces and peripherals, among others.

[0031] Thus, hardware information may refer to the specific characteristics and constraints of a hardware device, including for example its resource capacities, connectivity, dynamic parameter ranges, neuron and synapse behaviour, and available encoders and decoders, essential for optimizing the performance and deployment of the SNN.

[0032] For example, hardware resource constraints refer to the physical and computational limits of the hardware on which the SNN will be deployed. These may include: the amount of RAM and / or memory storage available for the network's data and weights; the computational capacity, measured e.g. in FLOPS (floating-point operations per second), that determines how many operations the hardware can perform in a given time; and / or the amount of electrical power the hardware consumes, which is critical for energy-efficient and portable devices.

[0033] For example, hardware connectivity constraints refer to the ways in which different components of the hardware communicate and transfer data. These may include: the data transfer rate between different hardware components, such as between processors or between memory and processors; the time delay in data transmission between hardware components; the design and topology of the connections between processors, memory units, and other components (e.g., bus systems, crossbars, and network-on-chip architectures).

[0034] For example, (dynamic) ranges of hardware design parameters may refer to the range of values that various hardware parameters can take and how they affect the performance and behaviour of the SNN. These may include: the range of voltage levels that can be used for signalling and power, which affects signal integrity and power consumption; the range of clock frequencies that the hardware can operate at, impacting the speed and timing of computations; the operating temperature range, which can affect hardware performance and reliability.

[0035] For example, the neuron and synapse characterization data may describe the specific behaviour and capabilities of the neurons and synapses implemented in hardware. This may include: the precision and variability of spike timing that the hardware neurons can achieve; the ability of hardware synapses to change their weights dynamically in response to learning rules; and / or the characteristics of leakage currents in hardware neurons and synapses, as well as the noise levels inherent in the hardware.

[0036] For example, encoders and decoders may be used to convert information between the format used by the SNN and other formats in the input and output chain. This may includee:input encoders, which are devices or algorithms that convert e.g. sensory data (e.g., images, sounds) into spike trains that the SNN can process; output decoders, which are devices or algorithms that convert the output spike trains from the SNN into a human-readable or further- processable format (e.g., classifications, control signals); and / or hardware interfaces (e.g., ADCs, DACs) that facilitate the conversion between analog signals and digital spike trains. Information about these encoders and decoders, such as characterization data of their electronic circuits can be comprised in the hardware information 108 as well.

[0037] For example, reconfigurability comprises the ability of hardware to be reprogrammed or reconfigured post-manufacturing to adapt to different tasks or applications. Highly reconfigurable hardware allows for flexible adaptation to different SNN models and architectures without needing new hardware for each specific network.

[0038] For example, programmability comprises the extent to which hardware can be programmed, e.g. via software, to perform different functions. Programmability enables the implementation of various neuron and synapse models, allowing for custom SNN configurations tailored to specific applications.

[0039] For example, yield can be the proportion of manufactured hardware devices that meet the required specifications and are functional.

[0040] For example, computational resources can be the available processing power, memory, and other computational capacities of the hardware. Adequate computational resources may be necessary to handle the parallel processing and memory-intensive tasks associated with simulating large-scale SNNs.

[0041] For example, temporal characteristics comprises the timing and synchronization aspects of hardware, including clock speeds and latency. Temporal characteristics can be important for maintaining accurate timing of spikes and synchronizing neuron activities.

[0042] For example, constraints on preprocessing can be limitations imposed by hardware on the preprocessing of input data before it is fed into the SNN. Understanding these constraints helps ensure that preprocessing steps are compatible with hardware capabilities, optimizing the overall performance and efficiency of the SNN.

[0043] For example, interfaces and peripherals comprise the connectivity options and additional components (e.g., sensors) that can be integrated with the hardware. Robust interfaces and peripherals facilitate seamless integration of SNNs with external devices and systems, enhancing their applicability in real-world scenarios.

[0044] The first method in which hardware information can be used in each step is described in more detail below.

[0045] The first step of network creation 101 involves creating a spiking neural network architecture that is optimized for the specific use case.

[0046] The network creation step creates a spiking neural network 101a, which may be considered “empty” and which is initialized by setting the parameters of the network 101a. The parameters may include, among others: a network topology, a neuron model and parameters, a synapse model and parameters, and initial weights of the synapses.

[0047] A network topology refers to the arrangement of neurons and the connections between them within the network 101a. It defines the structure and flow of information through the network 101a. Topologies can be simple, like feedforward networks where connections go in one direction from input to output, or more complex, like recurrent networks where connections can form loops. Other examples include Convolutional Neural Networks, which processes grid-like data with local receptive fields; Small-World Networks which have high clustering, with short path lengths; Scale-Free Networks which have hubs with many connections, and a power-law distribution; Modular Networks which have dense intra-module connections, and sparse inter-module connections; Lattice Networks, which have grid-like, nearest neighbour connections; and Random Networks, which have connections established randomly, lacking a regular structure.

[0048] A neuron model describes the mathematical representation of the neuron's behaviour. In SNNs, neurons generate spikes (action potentials) in response to incoming stimuli. One commonly used model is the Integrate-and-Fire model, which integrates incoming synaptic currents without accounting for natural membrane potential decay. The Leaky Integrate-and- Fire model is an extension of the Integrate-and-Fire model that includes a leak term. This term accounts for the natural decay of the membrane potential over time, making the model more biologically realistic. Some of the parameters of a neuron model can include: membrane potential, which is the electrical potential difference across the neuron's membrane; resting potential, which is the baseline membrane potential when the neuron is not active; threshold potential, which is the membrane potential at which the neuron fires a spike; reset potential, which is the membrane potential to which the neuron returns after firing a spike; membrane resistance, which is the resistance of the neuron's membrane; membrane capacitance, which is the ability of the neuron's membrane to store charge; refractory period, which is the time aftera spike during which the neuron cannot fire again; membrane time constant, which determines the speed at which the membrane potential changes in response to a current.

[0049] Other neuron models may be for example the Hodgkin-Huxley model, which is detailed and biologically realistic model that describes how action potentials in neurons are initiated and propagated. The model uses four differential equations to model ionic currents through sodium (Na+) and potassium (K+) channels. The neuron parameters of the Hodgkin- Huxley model may comprise the membrane potential, ionic currents and conductances, as well as channel kinetics parameters (e.g., activation and inactivation variables). Another example is the FitzHugh-Nagumo model, which is a simplified version of the Hodgkin-Huxley model, focusing on the qualitative behaviour of spikes. Its parameters include membrane potential, recovery variable, and threshold and timescale parameters. Another example of neuron model is the Izhikevich model, which is a more simplified model that captures various spiking and bursting behaviours observed in real neurons, which results. The neuron model’s parameters in this case are denoted as a, b, c, d which are parameters that define the recovery variable dynamics and reset conditions. Another example is the adaptive Exponential Integrate-and-Fire model, which is an extension of the Integrate-and-Fire model that includes adaptation mechanisms. This model captures more complex neuronal behaviours like spike frequency adaptation and uses an exponential term for spike initiation and an adaptation variable. Its parameters include among others membrane potential, adaptation variable, and parameters for spike threshold, adaptation, and membrane properties.

[0050] A synapse model defines how spikes are transmitted from one neuron to another and how they affect the receiving neuron's membrane potential. Synapses can be static, with fixed properties, or dynamic, with properties that change over time based on learning rules.

[0051] Parameters of a static synapse model may include: a weight, which is the strength of the connection between two neurons, determining the influence of a presynaptic spike on the postsynaptic neuron; and a delay, which is the time it takes for a spike to travel from the presynaptic neuron to the postsynaptic neuron.

[0052] Dynamic synapse models account for changes in synaptic strength over time, influenced by the history of presynaptic activity. These models can capture various short-term plasticity effects such as facilitation and depression, and long-term plasticity mechanisms like Spike-Timing-Dependent Plasticity (STDP). Examples include: Short-Term Plasticity (STP) models, wherein short-term plasticity encompasses transient changes in synaptic strength due to recent activity, with common effects being facilitation and depression; Spike-Timing-Dependent Plasticity (STDP) models, which modify synaptic strength based on the relative timing of presynaptic and postsynaptic spikes; and Conductance-Based Synapse models, which account for the conductance change in the postsynaptic neuron rather than a simple current injection.

[0053] Parameters for these dynamic models may include one or more of: Facilitation Time Constant, which is the time constant over which facilitation decays; Facilitation Increment, which is the increase in synaptic strength with each presynaptic spike; Depression Time Constant, which is the time constant over which depression decays; Utilization of Synaptic Efficacy, which is the fraction of synaptic resources used by each spike; Recovery Time Constant, which is the time constant for the recovery of synaptic resources; Learning Rate, which is the rate at which synaptic weights are updated; Time Constant for Potentiation, which is the time constant for the exponential decay of potentiation effects; Time Constant for Depression, which is the time constant for the exponential decay of depression effects; Maximum Weight, which is the upper limit for synaptic weights; Minimum Weight, which is the lower limit for synaptic weights; Synaptic Conductance, which is the conductance change induced by a presynaptic spike; Reversal Potential, which is the equilibrium potential for the synaptic conductance; Synaptic Time Constant, which is the time constant for the decay of synaptic conductance.

[0054] Initial weights refer to the initial values assigned to the synaptic connections when the network is first created. These weights can be set randomly or based on a specific distribution to ensure diversity in the network's response to stimuli.

[0055] Common methods for initializing weights include random initialization where weights are drawn from a random distribution, such as e.g. a normal (Gaussian) distribution with a specified mean and standard deviation, or a fixed initialization where weights are set to specific values, often small, to start with uniform influence across synapses.

[0056] The user needs to choose the most suitable network topology, as well as the optimal parameters for the neuron and synapse models. In order to be deployed on the hardware, the models and its parameters should be compatible with the characteristics of the target hardware.

[0057] Typically, this step would require expertise in SNNs and knowledge of the specific problem domain. A suitable library of software tools may be used to abstract most of the parameters away and leave a few high-level parameters for users to tune (e.g. the software tools wrap the SNN details inside a set of high-level APIs, so that the user can create the SNN without needing to know all the details). For example, one can provide simple functions todefine common network topologies, one can allow selection of neuron models with basic configurations already preset, one can include standardized options for synapse types and basic plasticity settings.

[0058] The low-level details can be abstracted by e.g. internally managing detailed parameters with sensible defaults and exposing only essential ones for tuning (e.g., learning rate, time constants). Furthermore, one can implement hardware-specific optimizations within the library to handle performance tuning and resource management automatically. One can create reusable modules for neurons, synapses, and network layers that can be easily combined and reused and allowing users to compose networks by connecting predefined modules.

[0059] Such a high-level API provides flexibility to tune the performance of the SNN with a few high-level parameters while relieving the user of dealing with all the details of the SNN.

[0060] The step of network creation 101 allows the network 101a to be created according to the user’s specifications. This step may be implemented in Python, and may utilize a standard machine learning framework, e.g. PyTorch or Tensorflow. To provide e.g. the PyTorch framework with additional functions to allow SNN-specific constructs - e.g. time-domain processing, temporal leak parameters, et cetera - custom extensions can be created and sourced into the PyTorch environment or any other software environment used.

[0061] In the proposed method, this extension consists of models of spiking neurons and synapses implemented by the target hardware and allowed parametric ranges for them. In addition, the extension incorporates different types of neural network layers, as well as gradient-based parameter optimization mechanisms, e.g. but not limited to, the computation of different types of surrogate gradients to overcome the discontinuity at spike creation or directly utilising other internal state parameters of the neuro- synaptic components.

[0062] The SNN structures like neurons and synapses can be implemented in the target hardware in multiple ways.

[0063] For example with respect to the different neuron model implementations: capacitors to store and integrate the membrane potential; resistors to model the leaky integration over time; comparators to detect when the membrane potential exceeds the threshold; circuits to reset the membrane potential after a spike; digital and / or analog circuits to adjust parameters like membrane time constant, threshold voltage, and reset voltage; capacitors or digital memory cells to store membrane potential and recovery variables; analog circuits or digital arithmetic units to compute the nonlinear equations governing the dynamics; logic circuits to generate spikes and reset the state variables; configurable elements to set parameters; analog circuits tomodel ionic currents with components like transistors for sodium and potassium channels; analog or digital circuits to solve the differential equations representing the ion channel kinetics; fine-tunable elements to set parameters like conductances and membrane capacitance.

[0064] For example with respect to the different synapse model implementations: digital multipliers or fixed-gain amplifiers to apply synaptic weights to incoming spikes; transmission gates or buffers to forward the weighted spikes to the postsynaptic neuron; memory cells or resistive elements to store and adjust synaptic weights; counters or clocks to measure the timing difference between pre- and postsynaptic spikes; digital logic or analog circuits to implement STDP rules and update synaptic weights based on spike timing; adjustable elements to set the learning rate; non-volatile memory or capacitors to store the updated weights; variable conductors (transistors) to simulate changes in synaptic conductance; voltage sources or DACs to set the reversal potential; circuits to modulate the conductance based on incoming spikes; configurable components to adjust synaptic parameters

[0065] Hence, the hardware information 108 in this step 101 is used to determine the models of spiking neurons and synapses that can be implemented on the target hardware, and the allowed parametric ranges for them. Furthermore, the hardware information 108 in step 101 is used to determine the network topology and the gradient-based parameter optimization mechanisms which can be implemented on the target hardware. On the basis of this, the network 101a is defined.

[0066] These functions within the extension enable the SNN to be specified in e.g. PyTorch to create complex architectures including e.g. multi-layer perceptrons, convolutional networks, and recurrent networks.

[0067] The next step is to train 102 the SNN 101a using an application-specific dataset. This dataset is a labelled training set 101b which consists of input data paired with corresponding target outputs (labels). During training, each input is fed into the network, and the network's output is compared to the target output. The difference (error) is used to adjust the network's parameters to improve performance.

[0068] Suitable training methods for SNNs include for example Spike-Timing-Dependent Plasticity (STDP), Supervised Learning with Surrogate Gradients or Reinforced Learning can be used. STDP adjusts synaptic weights based on the relative timing of pre- and postsynaptic spikes. If a presynaptic spike precedes a postsynaptic spike, the synapse is potentiated (strengthened). If the order is reversed, the synapse is depressed (weakened). STDP can be combined with reward signals to guide learning towards desired outcomes, though STDP aloneis usually unsupervised. Supervised Learning with Surrogate Gradients uses surrogate gradients which approximate the non-differentiable spiking activity using smooth surrogate functions. This allows the use of gradient-based optimization methods (like backpropagation). In Supervised Learning with Surrogate Gradients the network is trained by backpropagating the error through the surrogate gradient, adjusting the synaptic weights to minimize the error. In Reinforcement Learning, SNNs are trained using reinforcement signals (rewards or penalties) based on the network's performance on given tasks. Methods like reward-modulated STDP or policy gradient techniques are used to optimize network behaviour.

[0069] In addition to that, the selection of a gradient function will also affect the final training performance. The gradient function is the mathematical tool used to compute the derivatives of the loss function with respect to the network's parameters (e.g., weights and biases). These derivatives, known as gradients, are essential for optimizing the network through gradientbased learning algorithms like backpropagation. Examples include the Sigmoid Activation Function Gradient, the ReLU Activation Function Gradient, Sigmoid Surrogate Gradient, the Fast Sigmoid Surrogate Gradient, or the Exponential Surrogate Gradient.

[0070] In the proposed method, selecting the gradient function may be taken care of based on the network properties (e.g. network topology, neuron and synapse models inside the network, et cetera). In the case of training networks for analog mixed-signal chips, the training method is also able to utilize the hardware characterization information during the training process to reduce the performance gap between ideal case and actual hardware with analog device nonuniformity.

[0071] The step of training may use a set of training methods specialized for SNNs. The training method supports hybrid training mode, i.e. it can involve the actual hardware or a simulator that simulates a simulated instance of a target device and uses this simulated instance for the training process. In the case that a simulator is involved in the training process, the simulator can read synapse and neuron characterization data comprised within the hardware information 108 to better reproduce the hardware behaviour and provide more accurate info to the training feedback loop, so that the trained model can have better performance after deployment.

[0072] To further boost the training performance for processors that don’t support floating point computation, a few quantization methods are provided which utilize the hardware characterization data and available dynamic range available in the hardware information 108. Quantization refers to the process of mapping continuous-valued parameters, such as synapticweights or membrane potentials, to a discrete set of values. This is important for efficient hardware implementation, particularly on neuromorphic chips where memory and computational resources are limited. Benefits may include needing less memory to store e.g. quantized weights and other quantized parameters, quantized operations can be faster and more power-efficient and quantization may ensure that the network parameters can be represented using the fixed precision of the target hardware.

[0073] Users can do quantization-aware training by applying the quantization during the training process. As a result, the trained model can be easily deployed onto hardware with limited performance loss. The type of training is also configurable, ranging from supervised and semi-supervised to unsupervised or reinforcement learning.

[0074] For example, one can use the hardware characterization data to find the dynamic range of parameters (e.g., membrane potentials, synaptic weights), next one can select an appropriate bit width for the fixed-point representation based on the available precision, next one can scale the parameters to fit within the chosen bit width while preserving their dynamic range, and finally one can convert the scaled parameters to the fixed-point representation. More generally, one can for example: review the dynamic range, precision, power consumption, and memory constraints provided by the hardware characterization data; choose a quantization method (fixed-point, logarithmic, or floating-point) that aligns with the hardware capabilities and the nature of the parameters; identify the range of each parameter (e.g., membrane potential, synaptic weight) based on the SNN models used; quantize the parameters according to the chosen method, ensuring they fit within the hardware's dynamic range and precision limits.

[0075] One difficulty of using an SNN is that the users don’t have spike data, and they are not familiar with the generation of spike data. To solve this, the machine learning framework extension may also include a set of encoders and decoders which can convert existing data sets for conventional neural networks to spike data and vice versa. Users can also dump the encoded spikes to a predefined spike dataset format for future reuse. If users are not sure which one is the best encoding method for their application, the SNN extension is preferably also able to analyse an example input data and suggest the most suitable encoding method and model parameters to the users.

[0076] Furthermore, the extension may also support popular machine learning model exchange formats such as Open Neural Network exchange (ONNX) and other proprietary network formats, and may be able to convert existing trained models in ONNX format to SNN models. So, the users can train a network with the original data in any framework they arefamiliar with, then load the trained network into the SNN extension and convert it to an SNN model with limited performance loss. The conversion process involves replacing the standard neural network modules with their SNN equivalent ones, which preserve the same functionality but operate using spiking neurons and synaptic connections. The converted SNN model can be saved as ONNX format as well for exchanging purposes. Other components within the proposed flow, e.g. mapper 103 and simulator 106, are able to consume ONNX format networks, so no more format conversion is required. It allows developers to explore the benefits of SNNs without needing to learn SNN knowledge and a new programming paradigm.

[0077] To help users to analyse the network behaviour and the training metrics, the extension may also integrate support for Tensorboard for visualizing the results to network training runs, by tracking and visualizing metrics such as loss and accuracy, visualizing the model graph, viewing histograms, displaying images etc. In addition, it implements a wide range of standard or SNN-specific network metrics to visualize. The SNN extension is not limited to PyTorch. The same concept could be applied to other popular machine learning frameworks as well, such as TensorFlow, MXNet, JAX and others.

[0078] Once the SNN is trained, the trained network 102a is sent to a mapper 103. The mapping step (mapper) 103, that maps the trained spiking neural network 102a onto the target hardware, is responsible for translating the trained spiking neural network model into a hardware configuration file 103a that configures the components within the target hardware device.

[0079] The hardware configuration file 103a which is the output of the mapping step 103 can be for example a network binary comprising among others: hardware connectivity configuration, hardware resource allocation, neuron configuration, synapse configuration, and / or offset configuration. Indeed, To ensure that the spiking neural network (SNN) is efficiently mapped onto the target hardware, the hardware configuration file generated from the mapping step may encapsulate various aspects. This file may include: the hardware connectivity configuration, which defines how neurons and synapses are connected on the hardware; hardware resource allocation, which specifies how hardware resources like memory and computational units are distributed for the network; neuron configuration, which details the parameters and settings for each neuron model used, synapse configuration, which includes parameters and settings for each synapse model employed, offset configuration, which contains timing offsets or delays to synchronize operations across the network.

[0080] For example, the hardware connectivity configuration may comprise: a connection matrix, which is a binary matrix representing the connectivity between neurons, for example each entry (i, j) of the matrix indicates whether neuron i is connected to neuron j ; and / or routing information, which specifies paths for data flow between neurons, especially for large-scale networks spread across multiple chips or cores.

[0081] For example, the hardware resource allocation may comprise: memory allocation, which is information on memory blocks assigned to store neuron states, synapse weights, and network parameters; computational resources, which comprises a distribution of computational tasks across available processing units (e.g., cores, DSPs); and / or power budgeting, which comprises the allocation of power resources to ensure efficient power usage and to prevent overheating.

[0082] For example, the neuron configuration may comprise: specifications of the neuron models (e.g., LIF, Izhikevich) used in the network; values for parameters such as membrane time constant, threshold voltage, reset voltage, et cetera; and / or initial conditions for the membrane potential and other state variables.

[0083] For example, the synapse configuration may comprise: types of synapse models (e.g., static, STDP, conductance-based) used; initial synaptic weights and their distribution; and / or parameters governing synaptic plasticity rules (e.g., learning rate, time constants for STDP).

[0084] For example, the offset configuration may comprise: timing offsets which are delays to ensure that spikes and signals are synchronized across the network; and / or phase offsets for oscillatory neurons or synapses to align their activities.

[0085] The mapper 103 takes two inputs: one is the high-level description 102a of the neural network in ONNX, JSON or other format, which includes information about the neuron and synapse parameters, as well as the topology of the network. And the other one is the hardware information 108 comprising the architecture definition and characterization data of the target hardware device, including the available hardware resources, the connectivity constraints between on-chip resources and other architectural constraints of the device. The mapper then finds the optimal mapping for the given neural network while taking into account the hardware resource and connectivity constraints, the non-uniformity of analog devices and also the optimization target set by users such as power consumption, latency, hardware utilization rate et cetera.

[0086] Thus, the mapper 103 receives the neural network description 102a, hardware information (e.g., connectivity constraints, resource limits) 108, and optionally user-definedoptimization objectives (e.g., low power, minimal latency). Next the mapper 103 utilizes a mapping process, for example by determining which neurons connect to which others based on specified connectivity rules and constraints, assigning neurons and synapses to available hardware resources (e.g., cores, memory banks) while respecting capacity limits, minimizing communication paths and delays, and optimizing for low latency and efficient data flow.

[0087] Optimization strategies used by the mapper 103 may include: distributing computational tasks evenly across hardware to minimize power consumption, considering dynamic and static power requirements; configuring routing paths and adjusts processing priorities to reduce the overall latency of the network; and / or balancing workload across hardware resources to maximize utilization rates and prevent bottlenecks.

[0088] The mapper can accept example input data from an application as an optional input argument, and subsequently analyse the property of the data and generate a mapping which is optimized for the specific application, the benefit could be more efficient usage of system I / O bandwidth, better parallelism et cetera.

[0089] By changing the high-level description of hardware, the mapper can be retargeted to a different target device architecture, which makes the mapper generic enough to explore the ability to run the same application on different target hardware.

[0090] The next step of the method involves (hardware-aware) simulation 106 of the target hardware device architecture, configured with the binary 103a generated by the mapper. The simulator is able to read the hardware characterization file in order to reproduce a more accurate hardware behaviour in the software environment. The simulation may produce a runtime trace 106a of the network and generate a list of performance metrics such as accuracy, power consumption, latency et cetera. This helps the user to understand the impact on performance due to other hardware factors such as I / O bandwidth, interconnect latency and so on.

[0091] The runtime trace 106a may comprise at least one of input and output spikes, neuron states, weight distribution, and / or gradients.

[0092] The simulation models the internal architecture of the target hardware device. Specifically, it models the behaviour of the internal digital and analog components of the target hardware device, taking into account the unique constraints and characteristics of the device architecture as comprised in the hardware information 108.

[0093] The simulation may be cycle-accurate or to improve runtime, semi-cycle accurate, wherein the model may accurately simulate the timing of the events at interface level, but abstracts away the details happening inside the digital components as a trade-off betweenmodel accuracy and runtime performance. For the analog part, the simulation step 106 may model all key components such as the neuron, synapse and presynapse et cetera in a continuous time domain. Simulation parameters can further be altered to trade off speed for accuracy of simulation by modifying the time step. As a result, the simulation of large networks can be done very fast, and runtime can be reduced further by manipulating simulation accuracy. The simulation step allows designers and developers to test and optimize the performance of their spiking neural network before deploying it onto the actual hardware.

[0094] The simulation 106 shares the same architecture definition and characterization data as the mapper 103 so it can be easily retargeted to simulate a different target device as well. The sharing also guarantees the consistency between mapper and simulation model so that users can run a trained SNN model on different devices and compare their performance with minimum code changes.

[0095] The simulation also models the statistical property of the analog device nonuniformity, allowing statistical parameters to be changed to completely reproduce the behaviour of different analog target devices. Further, characterization data of on-chip neuron and synapse components can also be read in to configure the simulation. This feature enables the performance of models to be boosted when training using hybrid methods. By using a different random seed, it can also mimic the behaviour of a large set of different chips. In this way, the simulation step can also be used in the training process for mass deployment. See for example PCT / EP2024 / 069527, which is incorporated by reference in its entirety herein.

[0096] . The training process can utilize the simulated hardware non-uniformity, to generate a robust model that can tolerate the statistical mismatch between different chips. Then it can overcome the behaviour inconsistency of running the same SNN model on different chips.

[0097] Finally, the simulation step can be used to profile and analyse 107 the performance of an SNN on a target device, using the power and latency characterization of a specific target device. In this way, the simulation step allows hardware-level issues to be uncovered and debugged early in the SNN development process.

[0098] With the runtime traces 106a generated by the simulation 106, it is possible to gain insights and analyse 107 the runtime behaviour of the network and any possible performance drop. The performance of the SNN and the outcome of the analysis can be visualized to aid the deployment of the SNN.

[0099] When visualizing and analyzing simulation run traces 106a in step 107, tools like TensorBoard and statistical utilities can be used to provide insights into network behaviour,performance metrics, and optimization opportunities. For example, TensorBoard offers visualization capabilities for understanding SNN simulations by: displaying the structure of the SNN, including neuron layers, connectivity patterns, and synaptic weights; illustrating how operations are sequenced during simulation, aiding in debugging and optimization. Furthermore, training and evaluation metrics can be determined such as loss curves, accuracy metrics, weight distributions, activation histograms. Furthermore, one can project highdimensional neuron or synapse representations into lower-dimensional spaces, facilitating cluster analysis and visualization.

[0100] Beyond visualization, statistical utilities provide deeper insights into simulation data. For example, statistical analysis of neuron firing rates, synaptic weights, and other network parameters can be used to identify patterns and anomalies. Furthermore, relationships between variables can be examined (e.g., neuron firing rates and input stimuli), revealing dependencies and causal links. One can also analyse e.g. spikes and activations over time, detecting oscillatory behaviour or response dynamics to stimuli or constructs connectivity matrices to visualize synaptic strengths and identify influential pathways within the network.

[0101] The outcome 107a of the analysis 107 can be used to adapt the network and training process to improve the performance. For the example the outcome 107a of the analysis 107 can be used to redo the network creation step 101 and the subsequent steps, in order to obtain a better mapping of the network on the hardware suitable for the application. The steps 101-103 and the simulation 106 and analysis 107 can be repeated until a pre-defined performance threshold is reached.

[0102] The final step is to deploy the SNN-based application onto the target system. This involves integrating the SNN with other software or hardware components and configuring the system to run the application efficiently. The application-level integration 104 will have access to information about the available hardware resources 103b such as encoders, decoders et cetera, and can generate the configuration 104a of the complete application pipeline in this manner. This step can be made easily reproducible and scalable to a large amount of chips.

[0103] The information about the available hardware resources 103b may comprise application building blocks. For example, the information 103b may comprise a preprocessing algorithm library, an encoder library, a decoder library, and a postprocessing algorithm library.

[0104] SNNs are typically used alongside typical microprocessors or data paths which are programmed using a variety of compilers or tools. Step 104 of the flow integrates the requisite computational steps within a single application description. This allows complete end-to-endapplications to be deployed using the proposed flow onto a target hardware device. The application-level integration step 104 therefore encapsulates all the data preprocessing, encoding, neural network inference, post-processing, and output generation functions into a single description.

[0105] Effectively, the application-level integration step 104 defines the dataflow between each constituent step in an application and orchestrates the different components of the target hardware device that are involved in the implementation of these steps. Necessary compatibility checks or data conversions are automatically performed between steps. Caching of intermediate results between steps enables the flow to execute faster and reduces memory requirements. The application integration step effectively enables custom application pipelines to be built, incorporating SNNs, conventional logic, and software.

[0106] The final stage of application integration is the generation 105 of a binary file that configures the components within one or multiple target hardware devices. The generation of the binary utilizes a target-device specific template that ensures that the correct on-chip components of the device are configured with the correct parameters. By changing the template, different target devices can be supported during deployment.

[0107] In addition, each of these steps plays an important role in different stages of the application development cycle. They can communicate with each other seamlessly with predefined interface and data-exchange format and deliver a robust and complete solution for users to explore SNN and neuromorphic hardware.

[0108] The proposed method thus allows SNN-based applications to be described in Python and implemented on a specialized hardware device without the user requiring any knowledge of the underlying hardware or intervention in the process of hardware implementation. This may be done by propagating hardware information to some or all stages of the application development flow.

[0109] The method may abstract most of the SNN-specific knowledge away and may only expose the key, high-level parameters for users to fine-tune the performance for different applications. The method may utilize hardware information that describes the specialized hardware device, to customize how the neural network is built and implemented at every step of the method.

[0110] The method may utilize a mapper that models the hardware as a graph according to all the resource and connectivity constraints, and then may map the neural network graph onto the hardware graph according to the constraints and optimization objectives. The mapper maybe able to generate optimal mapping of a neural network for either a single device or for compatibility across a large batch of devices. Furthermore, the mapper may be easily retargeted to different device architecture variations using a configuration file.

[0111] The method may utilize a simulator that can simulate the behaviour of a specific device by reading the hardware characterization data, or may simulate the behaviour of any arbitrary device by using the built-in statistical model for analog non -uniformity.

[0112] The method may utilize a simulator that can be retargeted to different device architecture variations easily through the usage of a configuration file.

[0113] The method may include a training method for a SNN which utilizes a mapper, a simulator and hardware characterization data to optimize the performance of a model on hardware.

[0114] The method may involve an application-level integration step where the application program binary is generated according to the application definition and the hardware resource availability. The method may involve an application-level integration step which generates an application program binary that can be deployed across a large batch of devices, to reproduce the same result.

[0115] FIG. 2 shows a schematic representation of a method according to a second embodiment of the invention to create 201 a spiking neural network 201a, train 202 the spiking neural network, map 203 the trained spiking neural network 202a to a target device, simulate 204 the mapped spiking neural network 203a on a simulated (either in software or hardware) instance of the target device, performing an analysis 205 on the simulated run trace 204a, perhaps repeating steps 201-205 until a certain performance threshold is reached, in order to determine hardware information 206 for a potential deployment to a target hardware device complying with the hardware information 206.

[0116] In order to create the spiking neural network without knowing any of the hardware information which would normally constrain the choice for spiking neural network parameters, the parameters of the spiking neural network can be bound by some design constraints of a potential hardware device. For example, the number of input neurons can be limited, the number of resources can be limited, the temporal characteristics can be within certain bounds, et cetera. In this way, the method is sure to create a network which can be deployed to a hardware device that is feasible to be constructed.

[0117] In the present embodiment, one or more of the steps of network creation 201, training 202, mapping 203, simulation 204 and analysis 205 described above in relation to the firstembodiment can provide input to determine the hardware information 206. In this way, the most optimal hardware device can be determined (given design constraints) which would be able to run the spiking neural network for the given application. Based on the feedback from the network creation 201, training 202, mapping 203, simulation 204 and analysis 205 steps, the hardware information 206 may be newly created or adapted.

[0118] The hardware information 206 may be newly created based on the one or more steps of network creation 201, training 202, mapping 203, simulation 204 and analysis 205 in such a manner that the steps are executed in a hardware-agnostic manner.

[0119] For example, the hardware information of the hardware device that is going to be used for the deployment of the spiking neural network thereto can be created or initialised after having trained the spiking neural network, such that the hardware device has corresponding components that are configured to emulate the neurons and synapses of the trained spiking neural network. In this way, an initialisation of the hardware information is made, which can be used by the mapper to map the trained spiking neural network to the corresponding hardware device which is defined by the hardware information. After the mapping is obtained, a simulation can be performed. On the basis of the simulation, the hardware information can be updated and the simulation can be run again. On the basis of the simulation, the spiking neural network can be updated and retrained. The hardware information may be updated based on the update to the spiking neural network.

[0120] After having performed one or more of these steps, the method may create hardware information 206, comprising information for hardware that is best suited for the created SNN, which may or may not be trained. The simulation step can be performed in a hardware-agnostic manner by performing a simulation of the SNN network on a number of different potential hardware target devices, each of the different potential hardware target devices describable by different set of hardware information parameters.

[0121] One or more of the steps of network creation 201, training 202, mapping 203, simulation 204 and analysis 205 may be performed again using the updated hardware information. For example, the method of the first embodiment may be performed after the hardware information is obtained to perform fine-tuning, and to perform simulations of the hardware device perhaps using stochastical information of the different components of the potential hardware device, in order to determine network parameters that would work optimally when deploying the network to a multitude of hardware devices which suffer from stochastic variability due to e.g. inherent manufacturing constraints.

[0122] Note that features of any of the embodiments disclosed herein may be combined in an appropriate manner.

Claims

CLAIMS1. A method for deploying a spiking neural network to a hardware device, the method comprising: providing the spiking neural network, the spiking neural network comprising neurons that communicate via spike train signals, and synapses that interconnect the neurons and modulate the spike train signals; training the spiking neural network to obtain a trained spiking neural network; mapping the neurons and synapses of the trained spiking neural network to corresponding components of the hardware device, wherein the components emulate the neurons and synapses of the trained spiking neural network in the hardware device; simulating the deployment of the trained spiking neural network to the hardware device using the obtained mapping, by simulating the deployment to the hardware device; deploying the trained spiking neural network to the hardware device using the mapping and the simulation, such that the corresponding components of the hardware device emulate the neurons and synapses of the trained spiking neural network; wherein the training, mapping, simulating and deploying steps are performed using hardware information of the hardware device, wherein the hardware information comprises at least one of hardware resource constraints, hardware connectivity constraints, dynamic ranges of hardware design parameters, neuron and synapse characterization or statistical data, reconfigurability, programmability, yield, computational resources, temporal characteristics, constraints on preprocessing, interfaces and peripherals, and available encoders and / or decoders.

2. The method of claim 1, wherein the step of providing comprises creating the spiking neural network by selecting neurons, synapses and topology models from a repository, wherein each neuron, synapse and topology model is selected based on hardware information; wherein the hardware information comprises: a presence of feedforward and feedback channels on your chip, which limits the selection of feedforward and feedback loops in the network topology; a number of resources, which limits the selection of neurons and synapses; a temporal characteristic of a synapse, which limits the filtering characteristics of synapses; availability of preprocessors, which limits the numbers of inputs.

3. The method of claim 1 or 2, wherein the simulation is done by simulating the deploying of the spiking neural network to one or more simulated or physical instances of the hardware device.

4. The method of any one of the preceding claims, wherein the training step is performed using hardware information by using the neuron and synapse characterization or statistical data to reproduce the behaviour of the hardware device; and / or by using the neuron and synapse characterization or statistical data and the dynamic ranges of hardware design parameters to apply a quantization during the training process.

5. The method of any one of the preceding claims, wherein the mapping step is performed using hardware information by finding a mapping for the spiking neural network while taking into account the hardware resource and connectivity constraints, the non-uniformity of analog devices and / or an optimization target, preferably wherein the optimization target comprises power consumption, accuracy, latency and / or hardware utilization rate.

6. The method of any one of the preceding claims, wherein the mapping step accepts input data from an application as an input argument, analyses the properties of the data, and subsequently generates a mapping which is optimized for the specific application.

7. The method of any one of the preceding step claims, wherein the mapping step generates a hardware configuration file that configures one or more components within the target hardware device, preferably wherein the hardware configuration file is a binary file.

8. The method of any one of the preceding claims, wherein the simulating step is performed using hardware information by modelling one or more internal digital and analog components of the hardware device, taking into account the constraints and characteristics of the hardware device.

9. The method of any one of the preceding claims, wherein the simulation in the simulating step is either cycle accurate or semi-cycle accurate, wherein when the simulation is semi-cycle accurate the timing of events at interface level as well as the analog componentssuch as the one or more synapses and neurons are simulated accurately, while the details inside the digital components are simulated to a particular degree as a trade-off between model accuracy and runtime performance.

10. The method of any one of the preceding claims, wherein the simulating step models the statistical property of the non-uniformity of the hardware device, so that behaviour of a large set of hardware devices can be simulated.

11. The method of any one of the preceding claims, wherein the simulating step comprises the analysis of the performance of the spiking neural network on the hardware device, using a power, accuracy and latency characterization of the hardware device in the simulation.

12. The method of any one of the preceding claims, wherein the mapping step and the simulating step utilize the same architecture definition and characterization data, and wherein the architecture definition and characterization data is changeable in such a way that the mapping step and the simulating step are retargetable to a different hardware device.

13. The method of any one of the preceding claims, wherein the deploying step is performed using hardware information by generating a binary file that configures the components within the hardware device by utilizing a target device specific template which ensures that the correct on-chip components of the device are configured with the correct parameters.

14. A method for determining hardware information of a hardware device to which a spiking neural network for an application is suitable to be deployed, the method comprising: providing the spiking neural network, the spiking neural network comprising neurons, that communicate via spike train signals, and synapses that connect the neurons and modulate the spike train signals; training the spiking neural network for the application to obtain a trained spiking neural network; creating hardware information of the hardware device, such that the hardware device has corresponding components that are configured to emulate the neurons and synapses of the trained spiking neural network;mapping the trained spiking neural network to the dedicated components of the hardware device; simulating the deployment of the trained spiking neural network to the hardware device using the obtained mapping, by simulating the deployment to one or more hardware device instances, and wherein the hardware information of the hardware device is changeable depending on the outcome of the simulation; repeating the steps of providing, training, creating, mapping and simulating until a predefined performance threshold is reached and, when the pre-defined performance threshold is reached outputting the obtained hardware information of the hardware device, as well as the obtained spiking neural network; wherein the hardware information comprises at least one of hardware resource constraints, hardware connectivity constraints, dynamic ranges of hardware design parameters, neuron and synapse characterization or statistical data, reconfigurability, programmability, yield, computational resources, temporal characteristics, constraints on preprocessing, interfaces and peripherals, and available encoders and / or decoders.

15. The method of claim 14, wherein the hardware information of the hardware device is initialised by selecting pre-determined values for the hardware information or wherein the hardware information of the hardware device is initialized using the mapper, by selecting values that depend on the trained spiking neural network, more preferably wherein the properties of the trained spiking neural network determine the values with which the hardware information of the hardware device is initialized.

16. The method of claim 14 or 15, wherein in subsequent steps of the mapping, the hardware information of the hardware device is changed based on updates to the trained spiking neural network.

17. A hardware device, comprising a spiking neural network deployed thereto using the method of any one of claims 1-13.

18. A hardware device, configured according to hardware information obtained using the method of any one of claims 14-16.

Images