Method for establishing and deploying spiking neural network on hardware device

Abstract

The invention relates to a method for deploying a spiking neural network to a hardware device. The method includes providing the spiking neural network, training the spiking neural network to obtain a trained spiking neural network, mapping neurons and synapses in the trained spiking neural network to corresponding components of the hardware device, simulating deployment of the trained spiking neural network on a hardware device using the obtained mapping, and deploying the trained spiking neural network to the hardware device using the mapping and the simulation. The training, mapping, simulation and deployment steps are executed by using hardware information of the hardware device; wherein the hardware information comprises at least one of hardware resource constraints, hardware connection constraints, dynamic range of hardware design parameters, characterization or statistics of neurons and synapses, reconfigurability, programmability, yield, computing resources, temporal characteristics, constraints on pre-processing, interfaces and peripheral devices, and available encoders and / or decoders.

Description

Technical Field

[0001] The present invention relates to a method for generating a spike neural network and deploying it to a target device (particularly in a hardware-aware or hardware-independent manner), and a hardware device obtained using this method. Background Technology

[0002] Spiral Neural Networks (SNNs) are a type of neural network that offers several advantages over other types. Two of the most significant advantages are sparse computation and asynchronous processing, which result in low-power and low-latency solutions. However, as an emerging technology, the market lacks the software and hardware support necessary for users to develop and deploy SNN-based applications.

[0003] Specifically, SNNs deployed on specific target devices may perform poorly or even fail when using currently available support. This is due to a mismatch between the network's computing power requirements, application and user requirements and hardware capabilities, resulting in problems such as excessive latency, insufficient memory, insufficient processing power, poor classification results, inaccurate peak timing, reduced learning effectiveness and overall system instability. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes the following technical solution: A method for designing, testing, and deploying SNN-based applications on an analog mixed-signal accelerator.

[0005] According to a first aspect of the present invention, a method for deploying a spike neural network to a hardware device is disclosed. The method may include: providing the spike neural network, the spike neural network including neurons communicating via spike sequence signals, and synapses interconnecting and modulating the spike sequence signals; training the spike neural network to obtain a trained spike neural network; mapping the neurons and synapses of the trained spike neural network to corresponding components of the hardware device, wherein the components in the hardware device emulate the neurons and synapses of the trained spike neural network; simulating the deployment of the trained spike neural network to the hardware device using the obtained mapping by simulating the deployment to the hardware device; and deploying the trained spike neural network to the hardware device using the mapping and the simulation, such that the corresponding components of the hardware device emulate the neurons and synapses of the trained spike neural network.

[0006] These training, mapping, simulation, and deployment steps can be performed using hardware information of the hardware device, which includes at least one of the following: hardware resource constraints, hardware connectivity constraints, dynamic range of hardware design parameters, characterization or statistical data of neurons and synapses, reconfigurability, programmability, yield, computational resources, timing characteristics, constraints on preprocessing, interfaces and peripherals, and available encoders and / or decoders.

[0007] According to one embodiment of the first aspect, the provided steps include: generating the spike neural network by selecting neurons, synapses, and topology models from a repository, wherein each neuron, synapse, and topology model is selected based on hardware information. The hardware information may include: the presence of feedforward and feedback channels on the chip, which limits the selection of feedforward and feedback loops in the network topology; the number of resources, which limits the selection of neurons and synapses; the temporal characteristics of the synapses, which limits the filtering characteristics of the synapses; and the availability of a preprocessor, which limits the number of inputs.

[0008] According to one embodiment of the first aspect, the simulation is accomplished by simulating one or more simulated or physical instances of the spike neural network deployed to the hardware device.

[0009] According to one embodiment of the first aspect, the training step can be performed using hardware information by reproducing the behavior of the hardware device using the characterization or statistical data of the neurons and synapses; and / or the training step can be performed by applying quantization with the characterization or statistical data of the neurons and synapses and the dynamic range of the hardware design parameters during the training procedure.

[0010] According to one embodiment of the first aspect, the mapping step can be performed using hardware information by seeking a mapping of the spike neural network while taking into account the hardware resources and connectivity constraints, the non-uniformity of the simulation device and / or optimization objectives, preferably wherein the optimization objectives include power consumption, accuracy, latency and / or hardware utilization.

[0011] According to one embodiment of the first aspect, the mapping step may accept input data from an application as input parameters, analyze the properties of the data, and subsequently generate a mapping optimized for that specific application.

[0012] According to one embodiment of the first aspect, the mapping step can generate a hardware configuration file that configures one or more components within a target hardware device, preferably wherein the hardware configuration file is a binary file.

[0013] According to one embodiment of the first aspect, the simulation step is performed by modeling one or more internal digital and analog components of the hardware device and using hardware information, while taking into account the constraints and characteristics of the hardware device.

[0014] According to one embodiment of the first aspect, the simulation in the simulation step is periodically accurate or semi-periodically accurate, wherein when the simulation is semi-periodically accurate, the timing of interface-level events and the simulation components (e.g., one or more synapses and neurons) are accurately simulated, while the details inside the digital components are simulated to a certain extent as a trade-off between model accuracy and runtime performance.

[0015] According to one embodiment of the first aspect, the simulation step can model the statistical characteristics of the non-uniformity of the hardware device, enabling the simulation of the behavior of a large set of hardware devices.

[0016] According to one embodiment of the first aspect, the simulation step includes analyzing the performance of the spike neural network on the hardware device using the characterization of the power consumption, accuracy, and latency of the hardware device in the simulation.

[0017] According to one embodiment of the first aspect, the mapping step and the simulation step may utilize the same architecture definition and characterization data, wherein the architecture definition and characterization data enable the mapping step and the simulation step to be redirected to different hardware devices in a different manner.

[0018] According to one embodiment of the first aspect, the deployment step can be performed by generating a binary file and using hardware information, the binary file configuring components within the hardware device using a target device-specific template, the target device-specific template ensuring that the correct on-chip components of the device are configured with the correct parameters.

[0019] According to a second aspect of the present invention, a method is disclosed for determining hardware information of a hardware device suitable for deploying an application using a spike neural network. The method may include the following steps: providing the spike neural network, which includes neurons communicating via spike sequence signals and synapses connecting these neurons and modulating these spike sequence signals; training the spike neural network for the application to obtain a trained spike neural network; generating hardware information of the hardware device such that the hardware device has corresponding components configured to mimic the neurons and synapses of the trained spike neural network; mapping the trained spike neural network to dedicated components of the hardware device; simulating the deployment of the trained spike neural network to the hardware device using the obtained mapping by simulating the deployment to one or more instances of the hardware device, wherein the hardware information of the hardware device can be changed according to the results of the simulation; repeating the steps of providing, training, generating, mapping, and simulating until a predefined performance threshold is reached, and when the predefined performance threshold is reached, outputting the hardware information obtained by the hardware device and the obtained spike neural network. The hardware information may include at least one of the following: hardware resource constraints, hardware connectivity constraints, dynamic range of hardware design parameters, characterization or statistical data of neurons and synapses, reconfigurability, programmability, yield, computational resources, timing characteristics, constraints on preprocessing, interfaces and peripherals, and available encoders and / or decoders.

[0020] According to one embodiment of the second aspect, the hardware information of the hardware device is initialized by selecting a predetermined value for the hardware information, or wherein the hardware information of the hardware device is initialized by selecting a value that depends on the trained spike neural network using a mapper, more preferably wherein the properties of the trained spike neural network determine the value used to initialize the hardware information of the hardware device.

[0021] According to one embodiment of the second aspect, in subsequent steps of the mapping, the hardware information of the hardware device may be changed based on updates to the trained spike neural network.

[0022] According to a third aspect of the invention, a hardware device is disclosed, comprising a spike neural network deployed on the hardware device using the method according to the first aspect of the invention.

[0023] According to a fourth aspect of the present invention, a hardware device is disclosed that is configured based on hardware information obtained using the method of the second aspect of the present invention. Attached Figure Description

[0024] The embodiments will now be described by way of example only with reference to the accompanying drawings, in which corresponding reference numerals indicate corresponding parts, and in the drawings:

[0025] Figure 1 The diagram illustrates a method according to the present invention for creating a spike neural network, training the spike neural network, mapping the spike neural network to a target device, simulating the spike neural network on a simulated instance of the target device, and finally deploying the spike neural network to the target hardware device using hardware information; and

[0026] Figure 2 A schematic diagram of a method according to the present invention is shown, which is used to create a spike neural network, train the spike neural network, map the spike neural network to a target device, and simulate the spike neural network on a simulated instance of the target device in order to obtain hardware information for potential deployment to a target hardware device that conforms to the hardware information. Detailed Implementation

[0027] Certain embodiments will be described in more detail below. However, it should be understood that these embodiments should not be construed as limiting the scope of this disclosure.

[0028] Figure 1 The diagram illustrates a method 100 according to the present invention, which is used to create a spike neural network 101 using hardware information 108, train the spike neural network 102, map the spike neural network 103 to a target device, simulate the spike neural network 106 on a simulated instance of the target device, and finally deploy the spike neural network 104 and 105 to the target hardware device.

[0029] The method of deploying SNN-based applications may include five main steps: network creation 101, training 102, simulation 106, analysis 107, and deployment 104, 105.

[0030] In a first embodiment of the present invention, hardware information about the target hardware device is incorporated into at least one, at least two, at least three, at least four, or every step of these steps, thereby enabling hardware-aware development and mapping of the SNN onto the target hardware device without requiring the user to understand the device's internal architecture for each of these steps individually. This hardware information includes connectivity constraints, resource constraints, parameter ranges, component characterization data, supported encoder and decoder types, reconfigurability, programmability, yield, computational resources, timing characteristics, constraints on preprocessing, interfaces, and peripheral devices, etc.

[0031] Therefore, hardware information can refer to the specific characteristics and constraints of a hardware device, including, for example, its resource capacity, connectivity, dynamic parameter range, neuron and synaptic behavior, and available encoders and decoders, which are crucial for optimizing the performance and deployment of SNNs.

[0032] For example, hardware resource constraints refer to the physical and computational limitations of the hardware on which an SNN will be deployed. These may include: the amount of RAM and / or memory storage available for network data and weights; computing power, such as measured in FLOPS (floating-point operations per second), which determines how many operations the hardware can perform in a given time; and / or the power consumption of the hardware, which is crucial for energy efficiency and portability.

[0033] For example, hardware connectivity constraints refer to the ways in which different hardware components communicate and transfer data. These may include: the data transfer rate between different hardware components (e.g., between processors or between memory and processors); the time delay of data transfer between hardware components; and the design and topology of connections between processors, memory units, and other components (e.g., bus systems, crossbar switches, and on-chip network architectures).

[0034] For example, the (dynamic) range of hardware design parameters can refer to the range of values ​​that various hardware parameters can take and how they affect the performance and behavior of the SNN. These might include: the range of voltage levels available for signals and power supplies, which affects signal integrity and power consumption; the range of clock frequencies the hardware can operate on, affecting computation speed and timing; and the operating temperature range, which affects hardware performance and reliability.

[0035] For example, characterized data of neurons and synapses can describe the specific behaviors and capabilities of neurons and synapses implemented in hardware. This may include: the precision and variability of spike timing that hardware neurons can achieve; the ability of hardware synapses to dynamically change their weights in response to learning rules; and / or the characteristics of leakage currents in hardware neurons and synapses, as well as the noise level inherent in the hardware.

[0036] For example, encoders and decoders can be used to convert information between the formats used by the SNN and other formats in the input and output chains. This may include: input encoders, which are devices or algorithms that convert, for example, sensory data (e.g., images, sounds) into spike sequences that the SNN can process; output decoders, which are devices or algorithms that convert the output spike sequences from the SNN into human-readable or further processable formats (e.g., classification, control signals); and / or hardware interfaces (e.g., ADCs, DACs) that facilitate the conversion between analog signals and digital spike sequences. Information about these encoders and decoders, such as characteristic data of their electronic circuitry, may also be included in hardware information 108.

[0037] For example, reconfigurability includes the ability of hardware to be reprogrammed or reconfigured after manufacturing to suit different tasks or applications. Highly reconfigurable hardware allows for flexible adaptation to different SNN models and architectures without requiring new hardware for each specific network.

[0038] For example, programmability includes the degree to which hardware can be programmed (e.g., via software) to perform different functions. Programmability enables various neuron and synapse models, allowing SNN configurations to be customized for specific applications.

[0039] For example, yield can be the proportion of manufactured hardware devices that meet the required specifications and function properly.

[0040] For example, computing resources can be the available processing power, memory, and other computing capabilities of the hardware. Sufficient computing resources may be needed to handle the parallel processing and memory-intensive tasks associated with simulating large-scale SNNs.

[0041] For example, timing characteristics include the timing and synchronization aspects of the hardware, including clock speed and latency. These timing characteristics can be important for maintaining accurate timing of spikes and synchronizing neuronal activity.

[0042] For example, constraints on preprocessing can be limitations imposed by the hardware on the input data before it is fed into the SNN. Understanding these constraints helps ensure that the preprocessing steps are compatible with the hardware capabilities, thereby optimizing the overall performance and efficiency of the SNN.

[0043] For example, interfaces and peripherals include connectivity options and add-ons (e.g., sensors) that can be integrated with hardware. Robust interfaces and peripherals facilitate seamless integration of SNNs with external devices and systems, enhancing their applicability in real-world scenarios.

[0044] The first method is described in more detail below, where hardware information can be used in each step.

[0045] The first step in Network Creation 101 involves creating a spike neural network architecture optimized for specific use cases.

[0046] The network creation step creates a spike neural network 101a, which can be considered "empty," and initializes it by setting the parameters of network 101a. These parameters may include, but are not limited to: network topology, neuron model and parameters, synapse model and parameters, and initial weights of synapses.

[0047] Network topology refers to the arrangement of neurons within a network 101a and the connections between them. It defines the structure and flow of information through the network 101a. Topologies can be simple, such as feedforward networks, where connections proceed unidirectionally from input to output; or more complex, such as recurrent networks, where connections can form loops. Other examples include: convolutional neural networks, which use local receptive fields to process grid-like data; small-world networks, which have high clustering coefficients and short path lengths; scale-free networks, which have many connected hub nodes and a power-law distribution; modular networks, which have dense intra-module connections and sparse inter-module connections; grid networks, which have grid-like nearest-neighbor connections; and random networks, where connections are randomly established and lack a regular structure.

[0048] Neuron models describe the mathematical representation of neuronal behavior. In SNNs, neurons generate spikes (action potentials) in response to incoming stimuli. A commonly used model is the integration-fire model, which integrates incoming synaptic currents without considering the natural decay of membrane potential. The leaky integration-fire model is an extension of the integration-fire model, incorporating a leak term. This term considers the natural decay of membrane potential over time, making the model more biologically realistic. Some parameters of neuron models can include: membrane potential, the potential difference across the neuronal membrane; resting potential, the baseline membrane potential when the neuron is inactive; threshold potential, the membrane potential at which the neuron fires a spike; reset potential, the membrane potential returning after the neuron fires a spike; membrane resistance, the resistance of the neuronal membrane; membrane capacitance, the neuronal membrane's ability to store charge; refractory period, the time after a spike when the neuron cannot fire again; and membrane time constant, which determines the rate at which the membrane potential changes in response to current.

[0049] Other neuronal models include, for example, the Hodgkin-Huxley model, a detailed and biologically realistic model that describes how action potentials are initiated and propagated in neurons. This model uses four differential equations to simulate ionic currents through sodium (Na+) and potassium (K+) channels. Neuronal parameters in the Hodgkin-Huxley model can include membrane potential, ionic current, and conductance, as well as channel dynamics parameters (e.g., activation and inactivation variables). Another example is the FitzHugh-Nagumo model, a simplified version of the Hodgkin-Huxley model that focuses on the qualitative behavior of spikes. Its parameters include membrane potential, recovery variables, and threshold and time-scale parameters. Yet another example of a neuronal model is the Izhikevich model, a more simplified model that captures the various spike and burst firing behaviors observed in real neurons. In this case, the neuronal model's parameters are denoted as a, b, c, and d, which define the dynamic properties of the recovery variables and the reset conditions. Another example is the adaptive exponential integration-fire model, an extension of the integration-fire model that incorporates adaptation mechanisms. This model captures more complex neuronal behaviors, such as spike frequency adaptation, and uses exponential terms for spike initiation and adaptation variables. Its parameters include, but are not limited to, membrane potential, adaptation variables, and parameters related to spike threshold, adaptation, and membrane properties.

[0050] Synaptic models define how spikes are transmitted from one neuron to another and how they affect the membrane potential of the receiving neuron. Synapses can be static, with fixed properties, or dynamic, with properties that change over time based on learning rules.

[0051] The parameters of a static synaptic model can include: weights, which are the strength of the connection between two neurons and determine the influence of the presynaptic spike on the postsynaptic neuron; and delay, which is the time required for the spike to propagate from the presynaptic neuron to the postsynaptic neuron.

[0052] Dynamic synaptic models consider changes in synaptic strength over time, influenced by the history of presynaptic activity. These models can capture various short-term plasticity effects, such as facilitation and inhibition, as well as long-term plasticity mechanisms, such as peak-time dependent plasticity (STDP). Examples include: short-term plasticity (STP) models, where short-term plasticity encompasses transient changes in synaptic strength due to recent activity, with common effects being facilitation and inhibition; peak-time dependent plasticity (STDP) models, which modify synaptic strength based on the relative timing of presynaptic and postsynaptic peaks; and conductance-based synaptic models, which consider changes in conductance in the postsynaptic neuron rather than simple current injection.

[0053] The parameters of these dynamic models may include one or more of the following: facilitation time constant (the time constant of facilitation decay); facilitation increment (the increase in synaptic strength caused by each presynaptic spike); inhibition time constant (the time constant of inhibition decay); synaptic efficiency utilization (the proportion of synaptic resources used by each spike); recovery time constant (the time constant of synaptic resource recovery); learning rate (the rate at which synaptic weights are updated); enhancement time constant (the time constant of exponential decay of the enhancement effect); inhibition time constant (the time constant of exponential decay of the inhibition effect); maximum weight (the upper limit of synaptic weights); minimum weight (the lower limit of synaptic weights); synaptic conductance (the change in conductance caused by presynaptic spikes); reversal potential (the equilibrium potential of synaptic conductance); and synaptic time constant (the time constant of synaptic conductance decay).

[0054] Initial weights refer to the initial values ​​assigned to synaptic connections when the network is first created. These weights can be set randomly or based on a specific distribution to ensure diversity in the network's response to stimuli.

[0055] Common methods for initializing weights include random initialization, where weights are drawn from a random distribution, such as a normal (Gaussian) distribution with a specified mean and standard deviation; or fixed initialization, where weights are set to specific values ​​(usually very small) so that the effects across synapses are uniform at the beginning.

[0056] Users need to select the most suitable network topology, as well as the optimal parameters for the neuron and synapse models. For deployment on hardware, the model and its parameters should be compatible with the characteristics of the target hardware.

[0057] Typically, this step requires expertise in SNNs and knowledge of the specific problem domain. Appropriate software toolkits can be used to abstract away most parameters, leaving only a few high-order parameters for users to adjust (for example, software tools can wrap the details of SNNs within a set of high-level APIs, so users don't need to understand all the details to create an SNN). For instance, simple functions can be provided to define common network topologies; pre-configured neuron models with preset basic configurations can be selected; and standardized options for synapse type and basic plasticity settings can be included.

[0058] Low-level details can be abstracted, for example, by internally managing detailed parameters with reasonable default values ​​and exposing only the basic parameters used for tuning (e.g., learning rate, time constant). Furthermore, hardware-specific optimizations can be implemented within the library to automate performance tuning and resource management. Reusable neuron, synapse, and network layer modules can be created, which can be easily combined and reused, allowing users to assemble networks by connecting predefined modules.

[0059] Such a high-level API provides the flexibility to tune SNN performance through a few advanced parameters, while relieving users of the burden of handling all the details of SNN.

[0060] The steps for creating network 101 allow for the creation of network 101a according to user specifications. This step can be implemented in Python and can utilize standard machine learning frameworks such as PyTorch or Tensorflow. To provide additional functionality to frameworks such as PyTorch to allow SNN-specific constructs (e.g., time-domain processing, time leakage parameters, etc.), custom extensions can be created and introduced into the PyTorch environment or any other software environment used.

[0061] In the proposed method, this extension includes models of spike neurons and synapses implemented by the target hardware, along with their allowed parameter ranges. Furthermore, the extension integrates different types of neural network layers and gradient-based parameter optimization mechanisms, such as, but not limited to, computing different types of alternative gradients to overcome discontinuities during spike generation, or directly utilizing other internal state parameters of the neural synaptic components.

[0062] SNN structures such as neurons and synapses can be implemented in target hardware in a variety of ways.

[0063] For example, regarding the implementation of different neuron models: capacitors are used to store and integrate membrane potentials; resistors are used to simulate leakage integration over time; comparators are used to detect when the membrane potential exceeds a threshold; circuitry is used to reset the membrane potential after a spike; digital and / or analog circuitry is used to adjust parameters such as membrane time constant, threshold voltage, and reset voltage; capacitors or digital storage units are used to store membrane potentials and recovery variables; analog circuitry or digital arithmetic units are used to calculate nonlinear equations controlling the dynamics; logic circuitry is used to generate spikes and reset state variables; configurable elements are used to set parameters; analog circuitry uses components (such as transistors for sodium and potassium channels) to simulate ion currents; analog or digital circuitry is used to solve differential equations representing ion channel dynamics; and fine-tunable elements are used to set parameters such as conductance and membrane capacitance.

[0064] For example, regarding the implementation of different synaptic models: digital multipliers or fixed-gain amplifiers are used to apply synaptic weights to incoming spikes; transmission gates or buffers are used to forward weighted spikes to postsynaptic neurons; memory cells or resistive elements are used to store and adjust synaptic weights; counters or clocks are used to measure the timing difference between presynaptic and postsynaptic spikes; digital logic or analog circuits are used to implement the STDP rule and update synaptic weights based on spike timing; adjustable elements are used to set the learning rate; non-volatile memory or capacitors are used to store the updated weights; variable conductors (transistors) are used to simulate changes in synaptic conductance; voltage sources or DACs are used to set the inversion potential; circuitry is used to modulate conductance according to incoming spikes; and configurable components are used to adjust synaptic parameters.

[0065] Therefore, in step 101, hardware information 108 is used to determine the models of spike neurons and synapses that can be implemented on the target hardware, as well as their allowed parameter ranges. Furthermore, the hardware information 108 in step 101 is used to determine the network topology and gradient-based parameter optimization mechanism that can be implemented on the target hardware. Based on this, network 101a is defined.

[0066] These features within the extension enable SNNs to be specified in, for example, PyTorch, to create complex architectures, including, for example, multilayer perceptrons, convolutional networks, and recurrent networks.

[0067] The next step is to train the 102 SNN 101a using an application-specific dataset. This dataset is a labeled training set 101b, consisting of input data paired with corresponding target outputs (labels). During training, each input is fed into the network, and the network's output is compared to the target output. The difference (error) is used to tune the network's parameters to improve performance.

[0068] Suitable training methods for SNNs include, for example, Spike Temporally Dependent Plasticity (STDP), supervised learning with alternative gradients, or reinforcement learning. STDP adjusts synaptic weights based on the relative temporal order of presynaptic and postsynaptic spikes. If the presynaptic spike precedes the postsynaptic spike, the synapse is reinforced (strengthened). If the order is reversed, the synapse is suppressed (weakened). STDP can be combined with reward signals to guide learning toward desired outcomes, although STDP alone is typically unsupervised. Supervised learning with alternative gradients uses alternative gradients, which approximate non-differentiable spike activity using a smoothed alternative function. This allows the use of gradient-based optimization methods such as backpropagation. In supervised learning with alternative gradients, the network is trained by adjusting synaptic weights to minimize the error through backpropagation of the error via alternative gradients. In reinforcement learning, SNNs are trained using reinforcement signals (rewards or penalties) based on the network's performance on a given task. Methods such as reward-modulated STDP or policy gradient techniques are used to optimize network behavior.

[0069] In addition, the choice of gradient function will also affect the final training performance. The gradient function is a mathematical tool used to calculate the derivative of the loss function with respect to network parameters (e.g., weights and biases). These derivatives are called gradients and are crucial for optimizing the network using gradient-based learning algorithms such as backpropagation. Examples include the sigmoid activation function gradient, the ReLU activation function gradient, the sigmoid substitution gradient, the fast sigmoid substitution gradient, or the exponential substitution gradient.

[0070] In the proposed method, the gradient function can be selected based on network properties (e.g., network topology, neuron and synapse models within the network, etc.). In the case of training networks for analog mixed-signal chips, the training method can also utilize hardware characterization information during training to reduce the performance gap between ideal conditions and real hardware with the non-uniformity of analog devices.

[0071] The training steps can utilize a set of training methods specifically designed for SNNs. This training method supports a hybrid training mode, meaning it can involve either actual hardware or a simulator simulating an instance of the target device, and use that simulator for the training process. When a simulator is involved in the training process, the simulator can read the synaptic and neuron characterization data contained in the hardware information 108 to better reproduce hardware behavior and provide more accurate information to the training feedback loop, thereby enabling the trained model to achieve better performance after deployment.

[0072] To further improve the training performance of processors that do not support floating-point calculations, several quantization methods are provided that utilize the dynamic range available in hardware-characterized data and hardware information108. Quantization refers to the process of mapping continuous-valued parameters (such as synaptic weights or membrane potentials) to a set of discrete values. This is important for efficient hardware implementation, especially on neuromorphic chips with limited memory and computational resources. Benefits may include: requiring less memory to store, for example, quantized weights and other quantized parameters; faster and more energy-efficient quantization operations; and ensuring that network parameters can be represented with the fixed precision of the target hardware.

[0073] Users can perform quantization-aware training by applying quantization during the training process. Therefore, the trained model can be easily deployed to hardware with limited performance loss. Training types are also configurable, ranging from supervised and semi-supervised to unsupervised or reinforcement learning.

[0074] For example, the dynamic range of parameters (e.g., membrane potential, synaptic weights) can be found using hardware characterization data. Next, an appropriate bit width can be selected for a fixed-point representation based on available precision. The parameters can then be scaled to fit the selected bit width while preserving their dynamic range. Finally, the scaled parameters can be converted back to a fixed-point representation. More generally, one can, for example: examine the dynamic range, precision, power consumption, and memory constraints provided by the hardware characterization data; select a quantization method (fixed-point, logarithmic, or floating-point) that matches the hardware capabilities and parameter properties; determine the range of each parameter (e.g., membrane potential, synaptic weights) based on the SNN model used; and quantize the parameters according to the selected method, ensuring they conform to the hardware's dynamic range and precision limitations.

[0075] One challenge in using SNNs is that users often lack peak data and are unfamiliar with peak data generation. To address this, machine learning framework extensions can also include a set of encoders and decoders that can transform existing datasets from traditional neural networks into peak data and vice versa. Users can also dump encoded peaks into predefined peak dataset formats for future reuse. Ideally, if users are unsure which encoding method is best suited for their application, SNN extensions should also analyze example input data and suggest the most appropriate encoding method and model parameters.

[0076] Furthermore, this extension supports popular machine learning model exchange formats, such as Open Neural Network Exchange (ONNX) and other proprietary network formats, and may be able to convert existing trained models in ONNX format to SNN models. Therefore, users can train networks using raw data in any framework they are familiar with, then load the trained network into the SNN extension and convert it to an SNN model with limited performance loss. The conversion process involves replacing standard neural network modules with their equivalent SNN modules, which retain the same functionality but operate using spike neurons and synaptic connections. The converted SNN model can also be saved in ONNX format for exchange purposes. Other components in the proposed process (e.g., mapper 103 and simulator 106) are able to use networks in ONNX format, thus eliminating the need for further format conversion. It allows developers to explore the advantages of SNNs without needing to learn SNN knowledge and a new programming paradigm.

[0077] To help users analyze network behavior and training metrics, this extension also integrates support for TensorBoard to visualize results during network training runs by tracking and visualizing metrics such as loss and accuracy, visualizing model graphs, viewing histograms, and displaying images. Furthermore, it implements visualization of a large number of standard or SNN-specific network metrics. The SNN extension is not limited to PyTorch. The same concept can be applied to other popular machine learning frameworks such as TensorFlow, MXNet, and JAX.

[0078] Once the SNN training is complete, the trained network 102a is sent to the mapper 103. The mapping step (mapper) 103 maps the trained spike neural network 102a to the target hardware, and is responsible for converting the trained spike neural network model into a hardware configuration file 103a, which configures the components within the target hardware device.

[0079] The hardware configuration file 103a, output from mapping step 103, can be, for example, a network binary file, including, among other things, hardware connection configuration, hardware resource allocation, neuron configuration, synapse configuration, and / or offset configuration. Indeed, to ensure that the Spike Neural Network (SNN) is effectively mapped to the target hardware, the hardware configuration file generated from the mapping step can encapsulate various aspects. This file may include: hardware connection configuration, defining how neurons and synapses are connected on the hardware; hardware resource allocation, specifying how hardware resources (such as memory and computing units) are allocated to the network; neuron configuration, detailing the parameters and settings of each neuron model used; synapse configuration, containing the parameters and settings of each synapse model used; and offset configuration, containing timing offsets or delays to synchronize operations on the network.

[0080] For example, hardware connectivity configuration may include: a connectivity matrix, which is a binary matrix representing the connectivity between neurons, such that each entry (i, j) of the matrix indicates whether neuron i is connected to neuron j; and / or routing information, specifying the path of data flow between neurons, especially for large-scale networks distributed across multiple chips or cores.

[0081] For example, hardware resource allocation may include: memory allocation, i.e., information allocated to memory blocks for storing neuron states, synaptic weights, and network parameters; computing resources, i.e., the distribution of computing tasks among available processing units (e.g., cores, DSPs); and / or power budget, i.e., allocating power resources to ensure efficient power consumption and prevent overheating.

[0082] For example, neuron configuration may include: the specifications of the neuron model used in the network (e.g., LIF, Izhikevich); parameter values, such as membrane time constant, threshold voltage, reset voltage, etc.; and / or initial conditions of membrane potential and other state variables.

[0083] For example, synaptic configuration may include: the type of synaptic model used (e.g., static, STDP, conductance-based); initial synaptic weights and their distribution; and / or parameters governing synaptic plasticity rules (e.g., learning rate, STDP time constant).

[0084] For example, offset configurations may include: timing offsets, which are delays set to ensure that spikes and signals are synchronized throughout the network; and / or phase offsets of oscillating neurons or synapses to align their activity.

[0085] Mapper 103 accepts two inputs: one is a high-level description 102a of the neural network in ONNX, JSON, or other format, which includes information about neuron and synapse parameters and the network topology. The other is hardware information 108, containing the architecture definition and characterization data of the target hardware device, including available hardware resources, connection constraints between on-chip resources, and other architectural constraints of the device. The mapper then takes into account hardware resources and connection constraints, the non-uniformity of the simulated device, and user-defined optimization objectives such as power consumption, latency, and hardware utilization when finding the optimal mapping for a given neural network.

[0086] Therefore, mapper 103 receives neural network description 102a, hardware information (e.g., connection constraints, resource limitations) 108, and optional user-defined optimization objectives (e.g., low power consumption, minimum latency). Next, mapper 103 utilizes a mapping process, for example, to determine which neurons connect to which other neurons based on specified connection rules and constraints, allocates neurons and synapses to available hardware resources (e.g., cores, memory banks), while respecting capacity limitations, minimizing communication paths and latency, and optimizing low-latency and efficient data flow.

[0087] The optimization strategies used by mapper 103 may include: balancing the distribution of computing tasks on the hardware to minimize power consumption, while taking into account both dynamic and static power requirements; configuring routing paths and adjusting processing priorities to reduce overall network latency; and / or balancing the workload across hardware resources to maximize utilization and prevent bottlenecks.

[0088] The mapper can accept sample input data from the application as an optional input parameter, then analyze the properties of the data and generate a mapping optimized for that specific application. Its benefits may include more efficient use of system I / O bandwidth, better parallelism, etc.

[0089] By changing the high-level description of the hardware, the mapper can be redirected to different target device architectures, making the mapper versatile enough to explore the ability to run the same application on different target hardware.

[0090] The next step of this method involves a (hardware-aware) simulation 106 of the target hardware device architecture, configured with a mapper-generated binary file 103a. The simulator is able to read the hardware profiling file to reproduce more accurate hardware behavior in a software environment. The simulation may generate a runtime trace 106a of the network and a list of performance metrics such as accuracy, power consumption, latency, etc. This helps users understand the impact of other hardware factors, such as I / O bandwidth, interconnect latency, etc., on performance.

[0091] Runtime tracking 106a may include at least one of the following: input and output spikes, neuron states, weight distributions, and / or gradients.

[0092] The simulation models the internal architecture of the target hardware device. Specifically, it models the behavior of the internal digital and analog components of the target hardware device, while taking into account the unique constraints and characteristics of the device architecture contained in the hardware information 108.

[0093] Simulations can be periodically accurate, or, to improve speed, semi-periodic accurate, where the model precisely simulates the timing of interface-level events but abstracts away the details occurring within the digital components, serving as a trade-off between model accuracy and runtime performance. For the simulation portion, simulation step 106 can model all key components (such as neurons, synapses, and presynapses) in the continuous time domain. By modifying the time step, simulation parameters can be further altered to balance speed and simulation accuracy. Therefore, simulations of large networks can be completed very quickly, and runtime can be further reduced by manipulating simulation accuracy. The simulation step allows designers and developers to test and optimize the performance of sharp neural networks before deploying them to actual hardware.

[0094] Simulator 106 shares the same architecture definition and feature data as mapper 103, so it can be easily redirected to simulate different target devices. This sharing also ensures consistency between the mapper and the simulation model, allowing users to run trained SNN models on different devices and compare their performance with minimal code changes.

[0095] The simulation also models the statistical properties of non-uniformity in the simulation device, allowing for changes in statistical parameters to fully reproduce the behavior of different simulated target devices. Furthermore, the simulation can be configured by reading in characteristic data of on-chip neurons and synaptic components. This feature can improve model performance when training using hybrid methods. By using different random seeds, it can also simulate the behavior of a large group of different chips. Thus, the simulation step can also be used for training processes in large-scale deployments. See, for example, PCT / EP2024 / 069527, the entire contents of which are incorporated herein by reference.

[0096] The training process can leverage simulated hardware non-uniformity to generate a robust model that can tolerate statistical mismatch between different chips. This allows it to overcome behavioral inconsistencies when running the same SNN model on different chips.

[0097] Finally, the simulation step can be used to analyze and profile the performance of the 107 SNN on the target device, using power consumption and latency characteristics specific to the target device. In this way, the simulation step allows for the discovery and debugging of hardware-level issues early in the SNN development process.

[0098] By simulating runtime trace 106a generated from 106, we can gain in-depth understanding and analysis of the runtime behavior of the 107 network and any potential performance degradation. The performance and analysis results of the SNN can be visualized to assist in the deployment of the SNN.

[0099] When visualizing the simulation run trace 106a in step 107, TensorBoard and statistical tools can be used to provide insights into network behavior, performance metrics, and optimization opportunities. For example, TensorBoard provides visualization capabilities for understanding SNN simulations by displaying the structure of the SNN, including neuron layers, connection patterns, and synaptic weights; illustrating the sequence of operations during the simulation, which aids in debugging and optimization. Furthermore, training and evaluation metrics such as loss curves, accuracy metrics, weight distributions, and activation histograms can be identified. Additionally, high-dimensional neuron or synaptic representations can be projected into a low-dimensional space to facilitate clustering analysis and visualization.

[0100] Beyond visualization, statistical tools offer deeper insights into simulated data. For example, statistical analysis can be performed on neuronal firing rates, synaptic weights, and other network parameters to identify patterns and anomalies. Furthermore, relationships between variables (e.g., neuronal firing rates and input stimuli) can be examined to reveal dependencies and causal connections. Analyzing, for example, spikes and activations over time, detecting oscillatory behavior or response dynamics to stimuli, or constructing connectivity matrices to visualize synaptic strength and identify influential paths within the network are also possible.

[0101] The results 107a of analysis 107 can be used to tune the network and training process to improve performance. For example, the results 107a of analysis 107 can be used to redo network creation step 101 and subsequent steps to obtain a better mapping of the network on the hardware more suitable for the application. Steps 101-103, as well as simulation 106 and analysis 107, can be repeated until a predefined performance threshold is reached.

[0102] The final step is to deploy the SNN-based application onto the target system. This involves integrating the SNN with other software or hardware components and configuring the system to run the application efficiently. Application-level integration 104 will enable access to information about available hardware resources 103b, such as encoders, decoders, etc., and can thus generate a configuration 104a for the complete application pipeline. This step is easily repeatable and scalable to a large number of chips.

[0103] Information about available hardware resources 103b may include application building blocks. For example, information 103b may include preprocessing algorithm libraries, encoder libraries, decoder libraries, and postprocessing algorithm libraries.

[0104] SNNs are typically used with typical microprocessors or data paths, which are programmed using various compilers or tools. Step 104 of the process integrates the necessary computational steps into a single application description. This allows a complete end-to-end application to be deployed to the target hardware device using the proposed process. Therefore, application-level integration step 104 encapsulates all data preprocessing, encoding, neural network inference, post-processing, and output generation functions into a single description.

[0105] In practice, application-level integration step 104 defines the data flow between each component step in the application and orchestrates the different components in the target hardware device involved in implementing these steps. Necessary compatibility checks or data transformations are automatically performed between steps. Caching intermediate results between steps can speed up process execution and reduce memory requirements. The application integration step effectively enables the construction of custom application pipelines, integrating SNN, traditional logic, and software.

[0106] The final stage of application integration is generating a 105 binary file that configures components within one or more target hardware devices. This binary generation utilizes a target device-specific template to ensure that the correct on-chip components are configured with the correct parameters. By changing the template, different target devices can be supported during deployment.

[0107] Furthermore, each of these steps plays a crucial role at a different stage of the application development cycle. They can communicate seamlessly with each other using predefined interfaces and data exchange formats, providing users with a robust and complete solution for exploring SNNs and neuromorphic hardware.

[0108] Therefore, the method proposed in this invention allows SNN-based applications to be described in Python and implemented on dedicated hardware devices, without requiring users to have any knowledge of the underlying hardware or to intervene in the hardware implementation process. This can be achieved by propagating hardware information to some or all stages of the application development process.

[0109] This method can abstract away most of the SNN-specific knowledge and may only expose key high-level parameters to the user to fine-tune performance for different applications. It can also leverage hardware information describing dedicated hardware devices to customize how the neural network is built and implemented at each step of the method.

[0110] This method utilizes a mapper that models the hardware as a graph based on all resource and connectivity constraints, and then maps the neural network graph onto the hardware graph according to the constraints and optimization objectives. This mapper can generate optimal mappings for neural networks for a single device or for compatibility across a large number of devices. Furthermore, the mapper can be easily redirected to different device architecture variants using configuration files.

[0111] This method can utilize a simulator that can simulate the behavior of a specific device by reading hardware characterization data, or it can simulate the behavior of any device by using a built-in simulation non-uniformity statistical model.

[0112] This method can utilize an emulator that can be easily redirected to different device architecture variants using a configuration file.

[0113] This method may include a training approach for SNNs that leverages mappers, simulators, and hardware-specific data to optimize the model's performance on hardware.

[0114] This method may involve an application-level integration step, in which an application binary is generated based on the application definition and hardware resource availability. This method may also involve an application-level integration step that generates an application binary that can be deployed across a large number of devices to reproduce the same results.

[0115] Figure 2 A schematic diagram of a method according to a second embodiment of the present invention is shown, for creating 201 a spike neural network 201a, training 202 the spike neural network, mapping 203 the trained spike neural network 202a to a target device, simulating 204 the mapped spike neural network 203a on a simulated instance of the target device (whether in software or hardware), performing analysis 205 on the simulated operation tracking 204a, possibly repeating steps 201-205 until a certain performance threshold is reached to determine hardware information 206 for potential deployment to a target hardware device that conforms to the hardware information 206.

[0116] To create a spike neural network without knowing any hardware information that would typically constrain the choice of spike neural network parameters, the parameters of the spike neural network can be limited by certain design constraints of the potential hardware device. For example, the number of input neurons can be limited, the amount of resources can be limited, the time characteristics can be within certain limits, and so on. In this way, the method ensures that the created network can be deployed on the hardware devices that can be built.

[0117] In this embodiment, one or more of the steps described above in conjunction with the first embodiment, such as network creation 201, training 202, mapping 203, simulation 204, and analysis 205, can provide input to determine hardware information 206. In this way, the optimal hardware device capable of running a spike neural network for a given application can be determined (under given design constraints). Based on feedback from the network creation 201, training 202, mapping 203, simulation 204, and analysis 205 steps, the hardware information 206 can be newly created or adjusted.

[0118] Hardware information 206 can be newly created based on one or more steps of network creation 201, training 202, mapping 203, simulation 204, and analysis 205, in a form in which these steps are performed in a hardware-independent manner.

[0119] For example, hardware information for the hardware device used to deploy a spike neural network can be created or initialized after the spike neural network has been trained, so that the hardware device has corresponding components configured to mimic the neurons and synapses of the trained spike neural network. This completes the initialization of the hardware information, which the mapper can use to map the trained spike neural network to the corresponding hardware device defined by that hardware information. After obtaining the mapping, a simulation can be performed. Based on the simulation, the hardware information can be updated and the simulation can be run again. Based on the simulation, the spike neural network can be updated and retrained. The hardware information can be updated based on updates to the spike neural network.

[0120] After performing one or more of these steps, the method can create hardware information 206, containing hardware information best suited to the created SNN (whether or not it has been trained). The simulation step can be performed in a hardware-independent manner by simulating the SNN network on multiple different potential hardware target devices, each of which can be described with a different set of hardware information parameters.

[0121] One or more of the steps 201 (network creation), 202 (training), 203 (mapping), 204 (simulation), and 205 (analysis) can be performed again using the updated hardware information. For example, the method of the first embodiment can be performed after obtaining the hardware information to fine-tune and simulate the hardware device, possibly using random information about different components of the potential hardware device, to determine the network parameters that will perform best when the network is deployed to numerous hardware devices that suffer random variability due to, for example, inherent manufacturing limitations.

[0122] It should be noted that features of any of the embodiments disclosed herein can be combined in a suitable manner.

Claims

1. A method for deploying a spike neural network to a hardware device, the method comprising the steps of: The spike neural network is provided, which includes neurons that communicate via spike sequence signals and synapses that interconnect the neurons and modulate the spike sequence signals; Train the spike neural network to obtain the trained spike neural network; The neurons and synapses in the trained spike neural network are mapped to corresponding components of the hardware device, wherein the components in the hardware device mimic the neurons and synapses of the trained spike neural network. By simulating the deployment of the hardware device, the obtained mapping is used to simulate the deployment of the trained spike neural network to the hardware device; The trained spike neural network is deployed to the hardware device using the mapping and the simulation, such that corresponding components of the hardware device mimic the neurons and synapses of the trained spike neural network. The training, mapping, simulation, and deployment steps are performed using hardware information of the hardware device, wherein the hardware information includes at least one of the following: hardware resource constraints, hardware connectivity constraints, dynamic range of hardware design parameters, characterization or statistical data of neurons and synapses, reconfigurability, programmability, yield, computational resources, timing characteristics, constraints on preprocessing, interfaces and peripherals, and available encoders and / or decoders.

2. The method according to claim 1, wherein, The providing step includes: generating the spike neural network by selecting neurons, synapses, and topology models from a repository, wherein each neuron, synapse, and topology model is selected based on hardware information; This hardware information includes: The presence of feedforward and feedback channels on the chip limits the selection of feedforward and feedback loops in the network topology. The number of resources limits the selection of neurons and synapses; The temporal characteristics of synapses limit their filtering properties; The availability of preprocessors, which limits the number of inputs.

3. The method according to claim 1 or 2, wherein, The simulation is accomplished by simulating one or more simulated or entity instances of deploying the spike neural network onto the hardware device.

4. The method according to any one of the preceding claims, wherein, The training steps are performed using hardware information by reproducing the behavior of the hardware device through the characterization or statistical data of the neurons and synapses. And / or quantization is applied during the training process by using the characterization or statistical data of the neurons and synapses, as well as the dynamic range of the hardware design parameters.

5. The method according to any of the preceding claims, wherein the mapping step is performed using hardware information by seeking a mapping of the spike neural network in consideration of the hardware resources and connectivity constraints, the non-uniformity of the simulation device and / or optimization objectives, preferably wherein the optimization objectives include power consumption, accuracy, latency and / or hardware utilization.

6. The method according to any one of the preceding claims, wherein the mapping step accepts input data from the application as input parameters, analyzes the properties of the data, and subsequently generates a mapping optimized for the specific application.

7. The method according to any one of the preceding claims, wherein, The mapping step generates a hardware configuration file for configuring one or more components within the target hardware device, preferably wherein the hardware configuration file is a binary file.

8. The method according to any one of the preceding claims, wherein, The simulation step is performed by modeling one or more internal digital and analog components of the hardware device and using hardware information, while taking into account the constraints and characteristics of the hardware device.

9. The method according to any one of the preceding claims, wherein, The simulation in the simulation step is either periodically accurate or semi-periodically accurate. When the simulation is semi-periodically accurate, the timing of interface-level events and simulation components such as one or more synapses and neurons are accurately simulated, while the details inside the digital components are simulated to a certain extent as a trade-off between model accuracy and runtime performance.

10. The method according to any one of the preceding claims, wherein, The simulation step models the statistical characteristics of the non-uniformity of the hardware device, enabling the simulation of the behavior of a large group of hardware devices.

11. The method according to any one of the preceding claims, wherein, The simulation step includes analyzing the performance of the spike neural network on the hardware device using the characteristics of power consumption, accuracy, and latency of the hardware device in the simulation.

12. The method according to any one of the preceding claims, wherein, The mapping step and the simulation step utilize the same architecture definition and characterization data, wherein the architecture definition and characterization data enable the mapping step and the simulation step to be redirected to different hardware devices in different ways.

13. The method according to any one of the preceding claims, wherein, The deployment step is performed by generating a binary file and using hardware information, which configures the components within the hardware device using a target device-specific template that ensures the correct on-chip components of the device are configured with the correct parameters.

14. A method for determining hardware information of a hardware device suitable for deploying an application's spike neural network, the method comprising the steps of: The spike neural network is provided, the spike neural network including neurons that communicate via spike sequence signals, and synapses that connect the neurons and modulate the spike sequence signals; Train the application's spike neural network to obtain a trained spike neural network; The hardware information of the hardware device is generated such that the hardware device has corresponding components configured to mimic the neurons and synapses of the trained spike neural network. Mapping the trained spike neural network to a dedicated component of the hardware device; By simulating the deployment of one or more hardware device instances, the obtained mapping is used to simulate the deployment of the trained spike neural network to the hardware device, and wherein the hardware information of the hardware device can be changed according to the results of the simulation. The steps of providing, training, generating, mapping, and simulating are repeated until a predefined performance threshold is reached, and when the predefined performance threshold is reached, the hardware information obtained by the hardware device and the obtained spike neural network are output. The hardware information mentioned therein includes at least one of the following: hardware resource constraints, hardware connectivity constraints, dynamic range of hardware design parameters, characterization or statistical data of neurons and synapses, reconfigurability, programmability, yield, computing resources, timing characteristics, constraints on preprocessing, interfaces and peripheral devices, and available encoders and / or decoders.

15. The method according to claim 14, wherein, The hardware information of the hardware device is initialized by selecting a predetermined value for the hardware information, or wherein the hardware information of the hardware device is initialized by selecting a value that depends on the trained spike neural network using a mapper, more preferably wherein the properties of the trained spike neural network determine the value used to initialize the hardware information of the hardware device.

16. The method according to claim 14 or 15, wherein, In subsequent steps of the mapping, the hardware information of the hardware device is changed based on updates to the trained spike neural network.

17. A hardware device comprising a spike neural network deployed on the hardware device using the method according to any one of claims 1-13.

18. A hardware device configured based on hardware information obtained using the method according to any one of claims 14 to 16.

Images