A spiking neural network with reduced constant memory overhead and its training method

Abstract

This invention discloses a spiking neural network (SNN) with reduced constant memory overhead and its training method. The SNN is trained using a multi-parallel implicit flow architecture. During the forward pass, the multi-parallel implicit flow architecture drives multiple fused parallel implicit flows to simultaneously reach equilibrium. During the backward pass, the multi-parallel implicit flow architecture completes backpropagation through a single time-step simulation. An activation function for calculating the firing rate of the SNN during single-step substitution is derived, avoiding the time accumulation error introduced by gradient substitution-based propagation methods. Since the backward pass only needs to store the activation values ​​from a single time-step simulation, constant memory overhead SNN training is achieved. This neural network possesses excellent characteristics such as low latency, low memory overhead, low firing rate, and fast convergence speed, which will contribute to the practical application of neuromorphic computing.

Description

Technical Field

[0001] This invention relates to a neural network and a training method, specifically to a spiking neural network and a training method that reduces constant memory overhead. Background Technology

[0002] Spiking neural networks (SNNs) are a type of biologically plausible and energy-efficient neural network. Due to their pulse coding mechanism, which mimics the biological nervous system, they possess many excellent characteristics, such as low power consumption and fast response, and are considered an important direction for the development of next-generation neural networks. Unlike traditional artificial intelligence networks composed of static, continuous neuron models, spiking neural networks are composed of more biologically-oriented neuron models that can handle discrete pulse events. These characteristics give spiking neural networks enormous potential in terms of both computational and power efficiency.

[0003] However, since SNNs transmit information via impulse signals and their neuron models are non-differentiable, supervised training using the mature backpropagation algorithm from Artificial Neural Networks (ANNs) is not feasible. One approach is to construct a differentiable approximation of the SNN using a surrogate gradient algorithm to propagate over time. However, this method increases memory consumption with each training simulation step, and may even fail due to hardware limitations. Furthermore, the surrogate gradient method accumulates errors over time, degrading model performance. Therefore, it is necessary to consider a training method that can address the memory bottleneck in SNN training and avoid the accumulation of errors caused by the surrogate gradient method. Another approach, using a weight-sharing neural network with stacked weights, converges to a fixed point and has been extended to SNNs. However, this training method requires a sufficient number of time steps for the model to converge to an equilibrium state, and the network's expressive power depends on the depth of the weight-sharing blocks. These two issues significantly limit the training and inference speeds of the model, hindering the practical application of SNNs. Summary of the Invention

[0004] The purpose of this invention is to address the problems of non-differentiable SNNs, increased memory consumption, hardware limitations, and errors accumulating over time in surrogate gradient algorithms, and the limitation of training and inference speeds in weight sharing methods. Therefore, this invention proposes a spiking neural network with reduced constant memory overhead based on a multi-parallel implicit flow architecture, and designs a corresponding training method based on this network, which can achieve high efficiency in terms of memory, time, and energy while ensuring processing accuracy.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A spiking neural network (SNN) with reduced constant memory overhead is proposed. The network employs a spiking-based integrated firing model (IF) and a leaky integrated firing model (LIF), combining the general theory of equilibrium models to construct a multi-parallel architecture implicit flow computation model to solve for fixed points. The simulation time of SNNs per time step is shortened by decomposing the vertical complexity of the SNNs, and the convergence speed of the model is accelerated by fusing feature maps from various implicit flows, reducing the number of iterations required to solve for the equilibrium fixed points. The network's fixed-point solution process includes an input encoding layer, a state transition function layer, and a fixed-point output layer.

[0007] The input encoding layer is used to transform neural network information. The spiking neurons in SNNs are all dynamic models of real nerve cells. The spiking neurons receive binary pulse input and characterize the received information through membrane potential and pulse output. The input image is encoded into pulse form and the pulse signal is transmitted to the next layer of the network.

[0008] The state transition function layer uses a multi-parallel implicit flow architecture to compute the fixed point of the network equilibrium model, and the neurons employ both the integrated firing model and the leaky integrated firing model.

[0009] The fixed-point output layer generates the fixed point of the equilibrium model based on the iterative calculation of the state transition function, which is used for backpropagation error calculation.

[0010] Furthermore, during forward propagation to calculate fixed points, the network initializes its input parameters. Static images are used for fixed-point approximation calculations at specific time steps, and these images are input into the network via an image encoding module. Dynamic images are used for fixed-point approximation calculations at time steps matching the number of frames in the dynamic image, and these images are also input into the network via an image encoding module. The encoding method of the image encoding module can be designed according to actual needs.

[0011] The intermediate layers of the network employ a neural network balancing approach to calculate fixed points, reducing the number of error calculations during backpropagation. The core unit for fixed-point approximation is the state transition function layer, which contains multiple parallel single implicit flows used for input feature extraction and iterative calculation of approximate fixed points. A single FEB does not represent a single traditional network layer, but rather a block composed of a richly structured multi-layered network, which enhances the model's representational capabilities. In the forward propagation fixed-point calculation, the single implicit flow model includes an input impulse, a feature extraction module, an output impulse, a feedback module, an upsampling module, and a downsampling module, which respectively perform the functions of impulse input, feature extraction, impulse output, feedback of the output signal to the input impulse, downsampling to other parallel implicit flows, and upsampling of the output of the upper-layer parallel implicit flows.

[0012] The core unit of the single implicit flow is the feature extraction module, which contains multiple feature extraction layers. Each feature extraction layer performs convolution, batch normalization, and variational dropout, and finally passes the data through IF / LIF neurons, using DBReLU to approximate the balanced firing rate. The feature extraction module consists of l convolutional blocks, each of which sequentially contains a convolutional layer, a batch normalization layer, a variational dropout layer, and a spiking neuron layer, but each convolutional block has different parameter settings. The Trans module transforms the feature map output from the l-th layer of the FEB into a shape and dimension that matches the l-th layer. The Trans module allows the FEB to have a more flexible structure; otherwise, the input and output dimensions of the FEB must be the same.

[0013] To extract features at different scales, the number of implicit flows is at least two, and different implicit flows adopt a parallel architecture. The lower-level implicit flow can obtain the pulse information of its upper-level implicit flow through upsampling, and the upper-level implicit flow can obtain the pulse information of its lower-level implicit flow through downsampling.

[0014] The fixed-point output layer calculates an approximate stable point through state transitions, which serves as the weight-shared fixed point for the spiking neural network, allowing the network to enter the backpropagation process. The backpropagation stochastic gradient error calculation module updates the network parameters using the stochastic gradient descent algorithm by calculating the derivative of the loss function with respect to the approximate equilibrium firing rate.

[0015] Furthermore, based on the above spiking neural network, a spiking neural network training method with reduced constant memory overhead is designed, including the following steps:

[0016] Step 1: Preset the neuron model, initialize the neuron and network parameters. The neuron model adopts the integrated firing model (IF model) or the leaky integrated firing model (LIF model). In the calculation, the general dynamics of the neuron adopts the weighted membrane potential calculation form with discrete time.

[0017] Step 2: Depending on the form of the input image, the input network is encoded through the inject module. The time step for fixed point calculation is defined. For static images, fixed point approximation is performed at a specific time step. For dynamic images, fixed point approximation is performed at a time step that matches the number of frames in the dynamic image.

[0018] Step 3: Perform single implicit flow loop feature extraction. The intermediate layer of the network uses a neural network balancing method to calculate the fixed point in order to reduce the number of error calculations in the backpropagation process. An important idea of ​​neural network balancing is weight sharing. Based on the input-output relationship, a fixed point equation is constructed. In order to solve the differential of the fixed point with respect to any parameter, the differentials of both sides of the fixed point equation are calculated simultaneously. Combining the chain rule and IFT, the target differential is finally obtained, which is transformed into a fixed point iterative problem for solution.

[0019] Step 4: Improve model complexity by using a multi-parallel implicit flow approach. Input information is passed to the lower-level implicit flow module through downsampling to extract smaller and more complex image information. The computation process of each implicit flow is parallel, and information is passed between them through upsampling and downsampling.

[0020] Step 5: Estimate the firing rate at the approximate fixed point based on the results within the expected time step. When calculating the approximate firing rate at the fixed point, the equilibrium firing rate is approximated using the DBReLU method.

[0021] The formula for calculating DBReLU is:

[0022]

[0023] in M represents the firing rate of the i-th neuron in the l-th layer. l-1 This represents the total number of neurons in layer (l-1). V is the connection weight between the j-th neuron in layer l-1 and the i-th neuron in layer l. th This represents the threshold of the neuron.

[0024] Step 6: Calculate the derivative of the network based on the approximate fixed-point loss function using the fixed-point launch rate, and let L be the derivative of the network with respect to r. * Given the loss function, when the neural network converges to equilibrium, the derivative of the loss with respect to the parameter θ can be calculated:

[0025]

[0026] in:

[0027] and It can be calculated by automatic differentiation along the equivalent path;

[0028] It can be calculated by solving a linear system;

[0029] Step 7: Perform parameter updates based on backpropagation, which only requires one time step;

[0030] Step 8: Repeat steps 2 to 7 until the preset number of training rounds is met, then the training is complete.

[0031] We compared the time-based backpropagation (BPTT) method with the multi-parallel implicit flow (MPFL)-based computational method on the static dataset Fashion-MNIST and the neuromorphic dataset N-MNIST. The results show that the MPFL spiking neural network achieves significantly higher performance than directly trained spiking neural networks with less simulation time. The memory overhead of the MPFL computational framework is a constant independent of simulation duration, while the memory overhead of the BPTT training method increases with the simulation step size and eventually becomes unacceptable. The firing rate is a crucial indicator of the energy consumption of SNNs; the MPFL spiking neural network exhibits a significantly lower firing rate than the time-based backpropagation training method, resulting in lower energy consumption. In terms of accuracy, the MPFL architecture achieves higher accuracy with fewer simulation time steps. In summary, the network achieves energy, time, and memory efficiency in spiking neural network computation while maintaining higher accuracy with the same computational resources. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating the overall calculation process of the present invention.

[0033] Figure 2 This is a diagram illustrating the network architecture for the input, multi-parallel implicit flow state transition calculation, and fixed-point output of this invention.

[0034] Figure 3 This is a diagram illustrating the architecture of the Feature Extraction Module (FEB) of this invention;

[0035] Figure 4 This is a comparison chart of the convergence results of the neural network and training method of the present invention on different datasets. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0037] Reference Figure 1 As shown, the network's forward-to-backward computation is divided into two stages. In the forward propagation stage, the neuron parameters are initialized, the image is encoded as input, implicit flow features are extracted, multiple parallel implicit flow synchronous computations are performed, and a sampling method with sliding connections between different layers is adopted. The firing rate stability point is calculated through the output of the last layer, and the error is calculated according to the loss function using stochastic gradient descent based on this stability point. The network parameters are updated in one step.

[0038] Reference Figure 2 As shown, the feature extraction layer consists of m convolutional blocks, each of which sequentially contains a convolutional layer, a batch normalization layer, a variational dropout layer, and a spiking neuron layer, but each convolutional block has different parameter settings.

[0039] Reference Figure 3 As shown, the implicit flows together form the state transition function f mentioned above. θ The feature map size of each implicit flow output gradually decreases from top to bottom. Each implicit flow branch receives injections from other branches, and the emission rate of each implicit flow needs to reach its own equilibrium state, that is, solve for its corresponding fixed point. Finally, equilibrium states of various sizes can be obtained, and the equilibrium state corresponding to the smallest implicit flow will be sent to the classifier.

[0040] Reference Figure 4 As shown, the convergence curves indicate that the computational architecture of two implicit flows converges to equilibrium faster, and smaller implicit flow branches converge faster.

[0041] The computation of a network from one forward pass to the other is divided into two stages, such as... Figure 1 As shown, the forward propagation stage initializes neuron parameters, encodes image input, extracts implicit flow features, performs synchronous computation of multiple parallel implicit flows, and adopts a sampling method with sliding connections between different layers. The firing rate stability point is calculated through the output of the last layer, and the error is calculated based on the loss function using stochastic gradient descent based on this stability point. The network parameters are then updated in one step.

[0042] A spiking neural network with reduced constant memory overhead and its learning method, comprising the following steps:

[0043] Step 1, Forward Propagation - Input Encoding, corresponds to Steps 1 and 2 in the training method. The input image is encoded. For static images, the encoded impulses remain consistent throughout the fixed-point computation process T. For dynamic images, the input impulses change with the image at time T. The network uses an IF (Integrated and Fired Model) neuron model. After encoding, the input image is passed to the fixed-point computation layer of the state transition function, which is the first implicit flow layer of the multi-parallel implicit flow layer.

[0044] The neuron model is preset, and the neuron and network parameters are initialized. The neuron model adopts either the integrated firing model (IF model) or the leaky integrated firing model (LIF model). In the calculation, the general dynamics of the neuron are calculated in the form of weighted membrane potential with discrete time.

[0045] The IF model and LIF model are as follows:

[0046]

[0047]

[0048] in:

[0049] τ is the membrane time constant of the neuron;

[0050] u(t) represents the membrane voltage of the neuron at the current time t;

[0051] u res This represents the neuron's potential at rest, used for potential reset after a neuronal impulse.

[0052] R is the membrane resistance of the neuron, used to calculate the voltage change that external input brings to the current neuron;

[0053] I(t) is the sum of the input currents upstream of the current neuron at time t.

[0054] In practical computation, the general dynamic description of a neuron includes the following discrete-time equations:

[0055]

[0056] in,

[0057] u i [t] represents the membrane potential of the i-th neuron at a discrete time step t, reflecting the distribution of charge inside the neuron;

[0058] u i [t-1] is the membrane potential of the i-th neuron at the previous time step t-1. It is used to calculate the time change of the neuron's state and represents the lag state in the dynamic model.

[0059] This represents the synaptic weighted connection from neuron j to neuron i;

[0060] s j [t] represents the pulse state of neuron j at time t. If neuron j pulses at time t, then s j The value of [t] is 1 if it is not 0 otherwise;

[0061] s i [t] represents the pulse state of neuron i at time t. If neuron i pulses at time t, then s i The value of [t] is 1 if it is not 0 otherwise;

[0062] V t h is the threshold voltage at which the neuron fires;

[0063] M represents the number of neurons in layer l.

[0064] Depending on the form of the input image, the input network is encoded through the inject module, and the time step for fixed point calculation is defined. For static images, fixed point approximation is performed at a specific time step, while for dynamic images, fixed point approximation is performed at a time step that matches the number of frames in the dynamic image.

[0065] Step 2, Forward Propagation - Fixed Point Calculation, corresponds to steps 3, 4, 5, and 6 in the training method. A multi-parallel implicit flow computation method is used. To calculate the fixed point, some numerical analysis methods for accelerating fixed point iteration can be employed, or the fixed point solution problem can be transformed into a root-finding problem. However, to better reflect the time characteristics of SNNs, naive forward iteration is still used here to calculate the fixed point. Let f... θ If (s; x) represents the state transition equations for the entire implicit flow, then the forward process can be expressed as an iterative formula:

[0066]

[0067] Where t is the t-th simulation time step of the SNN, and s t This represents the output pulse. As the simulation duration increases, the contribution of each new simulation step size decreases until the SNNs reach equilibrium; the firing rate at this point is the firing rate equilibrium point r*. Activation values ​​are not stored during the forward pass to solve for the fixed points. The equivalent path of the forward pass is: r * The firing rate of neurons is calculated as a new input to the equation and then processed through a single-step simulation.

[0068] like Figure 2 As shown, the feature extraction layer consists of m convolutional blocks. Each convolutional block sequentially contains a convolutional layer, a batch normalization layer, a variational dropout layer, and a spiking neuron layer, but each convolutional block has different parameter settings. The function of the Trans module is to reconstruct the feature map output by the m-th layer of the feature extraction layer into a shape and dimension that matches the first layer. The Trans module allows the feature extraction layer to have a more flexible structure; otherwise, the input and output dimensions of the feature extraction layer must be the same.

[0069] The intermediate layers of the network use a neural network balancing approach to calculate fixed points, reducing the number of error calculations during backpropagation. A key idea behind neural network balancing is weight sharing. Consider an N-layer weight-sharing model with parameter θ:

[0070] z [i+1] =f θ (z [i] ;x), i=0,…,N-1

[0071] in:

[0072] z [i+1] The output pulse matrix of the current layer;

[0073] X represents the input impulse matrix of the current layer;

[0074] z [i] It is the hidden state of the previous layer;

[0075] f θ It is the state transition function corresponding to this weight-sharing model.

[0076] If we continuously increase the number of layers in the model (i.e., N→∞), the relationship between the input and output will satisfy the following fixed-point equation:

[0077] z * =f θ (z * ;x)

[0078] To solve for the differential of a fixed point with respect to arbitrary parameters, we simultaneously differentiate both sides of the fixed-point equation, and by combining the chain rule and the International Fourier Transform (IFT), we finally obtain the target differential:

[0079]

[0080] in:

[0081] z ★ (·) represents an implicit function of the objective differential parameter.

[0082] It is the intermediate process of chain rule differentiation.

[0083]

[0084] It can be transformed into a fixed-point iterative problem for solution.

[0085] The model complexity is improved by using a multi-parallel implicit flow approach. The input information is passed to the lower-level implicit flow module through downsampling to extract smaller and more complex image information. The computation process of each implicit flow is parallel, and information is passed between them through upsampling and downsampling.

[0086] By decomposing the longitudinal complexity of the spiking neural network, the simulation time of the spiking neural network for one time step is shortened. The convergence speed of the model is accelerated by fusing the feature maps of each implicit flow, and the number of iterations required to solve the fixed point is reduced, thereby shortening the forward propagation process time.

[0087] The multi-parallel implicit flow training method consists of a state transition function f composed of all implicit flows. θThe feature map size of each implicit flow output gradually decreases from top to bottom. Each implicit flow branch receives injections from other branches, and the emission rate of each implicit flow needs to reach its own equilibrium state, i.e., solve for its corresponding fixed point. (Where n represents the index of the implicit flow, n∈[1,2,…,N]). Ultimately, equilibrium states of various sizes can be obtained, and the equilibrium state corresponding to the smallest implicit flow will be fed into the classifier. Each implicit flow can be viewed as a separate state transition function, consisting of a FEB and a transition module (Trans). The input is... (Initial input) (A tensor whose shape and dimensions are paired with FEB values ​​that are all zero).

[0088] The proposed shallower single implicit flow energy reduces the time cost of the forward process. To pursue the highest model representation capability under the current low time cost and improve model performance, a parallel computing approach is adopted to increase model complexity.

[0089] like Figure 3 As shown, the implicit flows together form the state transition function f mentioned above. θ The feature map size of each implicit flow output gradually decreases from top to bottom. Each implicit flow branch receives injections from other branches, and the emission rate of each implicit flow needs to reach its own equilibrium state, i.e., solve for its corresponding fixed point. (Where n represents the index of the implicit flow, n∈[1,2,…,N]). Ultimately, equilibrium states of various sizes can be obtained, and the equilibrium state corresponding to the smallest implicit flow will be fed into the classifier.

[0090] To link the dynamic evolution of the model to the input, feature information of the input data needs to be injected into the FEB. The input is injected only into the first implicit flow, and the initial input of each implicit flow is zero.

[0091] Initially, the input information exists only in the first implicit stream, and the other implicit streams do not receive any features about the input. The balance state of each implicit stream depends on the injection from the higher-level implicit stream (injected through the downsampling module), and at the same time, the balance state of the higher-level implicit stream is also modulated by the other implicit streams.

[0092] As the input information gradually propagates to the last implicit flow during the convergence of all implicit flows to the equilibrium point, this process enables the model to have sufficient representational power.

[0093] To improve model flexibility, the upsampling and downsampling modules responsible for fusing the implicit flows are sliding, meaning they can be fused at different layers of the FEB. Figure 1 The diagram shows fusion occurring in the last layer of the FEB.

[0094] When the model runs, each implicit flow branch executes in parallel and receives the sampled output of other parallel implicit flows as the input for the next iteration. The iterative process of the i-th implicit flow can be represented as:

[0095]

[0096] in:

[0097] i represents the index of the implicit stream, i∈[1,2,...,N];

[0098] l represents the number of layers in the feature extraction block, l∈[1,2,...,L];

[0099] The a to b layers represent the feature extraction block of the i-th implicit flow;

[0100] D d The downsampling module for the d-th implicit stream

[0101] U u The upsampling module for the u-th implicit stream

[0102] L represents the layer number where the fusion module is located.

[0103] Based on the results within the expected time step, the firing rate at the approximate fixed point is estimated, and the output pulse of the current implicit flow at time t is obtained through FEB.

[0104]

[0105] in:

[0106] t is the t-th simulation time step of the SNN.

[0107] s t For the output pulse

[0108] As the simulation duration increases, the contribution of each new simulation step size decreases until the SNNs reach equilibrium. The firing rate at this point is the equilibrium point r of the firing rate. * The output at time t is transformed and then used as the input at time t+1, which is then fed back into the FEB for feature extraction.

[0109] Furthermore, the calculation process of the state transition function layer is as follows:

[0110] First, the input pulse signal is encoded and enters the first implicit flow layer;

[0111] Secondly, the pulse matrix of the current layer is calculated by summing the input of the FEB layer with the upsampled input of the other implicit flow layers;

[0112] Furthermore, the FEB input of the first implicit flow layer is input to the remaining implicit flow layers through downsampling, and the feature size of each layer decreases layer by layer;

[0113] Finally, after t iterations, the fixed point of the current network is calculated using the pulse output of the last layer, where t is the approximate fixed point convergence time.

[0114] The fixed-point output layer calculates the approximate stable point of the state transition as the weight-shared fixed point of the spiking neural network, and the network enters the backpropagation process.

[0115] The derivative of the network based on the approximate fixed-point loss function is calculated using the fixed-point launch rate, and L is the derivative of the network with respect to r. * Given the loss function, when the neural network converges to equilibrium, the derivative of the loss with respect to the parameter θ can be calculated:

[0116]

[0117] in:

[0118] and It can be calculated by automatic differentiation along the equivalent path;

[0119] It can be calculated by solving a linear system;

[0120] For step 2, when the feature extraction module has multiple layers of neurons, even if one time step is used to calculate the gradient (equivalent to T time steps), the differential cannot be directly calculated within the feature extraction module due to the presence of the step function (neural impulse).

[0121] Inspired by the conversion of ANNs to SNNs and the implicit differentiation of balanced SNNs, DBReLU is derived as the firing rate calculation function when SNNs reach equilibrium. In the study of ANNs-SNNs, the performance of the converted SNNs is positively correlated with the number of simulation time steps. However, due to hardware limitations, the simulation steps for SNNs cannot be increased indefinitely. In MPIS-SNNs, the final output is the fixed point corresponding to the model, which is equivalent to the firing rate of neurons after an infinite number of simulation time steps. At this point, the integrated firing (IF) neuron is equivalent to an unbiased estimator of the linear rectifier unit in time, but since the firing rate of the neuron cannot exceed 1, 1 needs to be specified as an upper bound. Its formal calculation formula is:

[0122]

[0123] in M represents the firing rate of the i-th neuron in the l-th layer. l-1This represents the total number of neurons in layer (l-1). V is the connection weight between the j-th neuron in layer l-1 and the i-th neuron in layer l. t h is the threshold of the neuron.

[0124] make Let represent the membrane potential of the i-th neuron in the l-th layer at time t. Then, the formula for the membrane potential transition of a neuron is:

[0125]

[0126] in Let be the pulse emitted by the i-th neuron in the l-th layer at time t, with a value of 1 or 0. For SNNs with a simulation step size of T, the following equation holds:

[0127]

[0128] Calculate the average firing rate of the i-th neuron in the l-th layer within time period T:

[0129]

[0130] Let the sum of all inputs to the previous layer at time t be... The membrane voltage difference at the beginning and end of neuron i When the SNNs reach equilibrium, a single time step simulation is equivalent to an SNN with an infinite simulation time length, i.e., the number of simulations T→∞. The distribution rate calculation formula becomes:

[0131]

[0132] Due to V Δ It is a constant, therefore Furthermore, we directly input the standardized floating-point values ​​of the image data into the MPIS-SNNs as the firing rate of the previous layer. Therefore, for neuron i in the first layer, the sum of all upstream inputs is a constant. We let this constant be... Then we can obtain:

[0133]

[0134] By deriving the formula layer by layer to higher levels, the fixed points of the network can be determined:

[0135]

[0136] in,

[0137] Step 3: Calculate the loss function and its derivative for parameter updates. This step corresponds to step 7 in the training method. Let L be the loss function with respect to r. *Given the loss function, when the neural network converges to equilibrium, the derivative of the loss with respect to the parameter θ can be calculated:

[0138]

[0139] in, and Both can be calculated using automatic differentiation along the equivalent path, and This can be calculated by solving a linear system. The parameters are then updated using the stochastic gradient descent (SGD) algorithm.

[0140]

[0141] Where, θ old and θ new These are the parameters before and after the update, respectively, with α being the set learning rate.

[0142] In practical applications, we ultimately use the r obtained from finite-step simulation. T To be used as r * The approximate solution, when the approximation error is small enough, has a negligible impact on the model performance due to the difference between the approximate value and the true value.

[0143] Step 4: Iterative calculation. Input the next set of images and repeat the forward propagation process and backward gradient calculation to update the network parameters based on the multi-parallel implicit flow architecture until the preset number of training rounds is met. Then the training is completed. This step corresponds to step 8 in the training method.

[0144] Compared to ANNs, SNNs are a more biologically plausible type of neural network. However, directly training SNNs has always faced problems such as high memory overhead and non-differentiability, and the reliability of SNNs depends on sufficient simulation time steps, leading to high latency. This paper proposes an implicit training method for SNNs based on balanced neural network theory. In the forward pass, multiple parallel and fused shallower implicit flows are driven to simultaneously reach equilibrium, accelerating the computation and convergence speed of the SNNs. In the backward pass, a single time step is equivalent to an infinite number of time steps of SNNs, achieving constant memory overhead independent of simulation time. Simultaneously, to address the non-differentiability issue within the feature extraction block, a double-bounded linear rectified unit is derived as the firing rate estimation function for spiking neurons, avoiding the time-accumulated error caused by gradient substitution methods.

[0145] In a specific embodiment of this invention, the performance differences between the BPTT training method and our method were compared on a static dataset and a neuromorphic dataset, respectively, including runtime, memory usage, accuracy, and firing rate (energy consumption). The differences in time efficiency and accuracy between conventional SNNs and MPIS-SNNs based on the balanced model theory were compared. The impact of multi-parallel branch implicit flow on the convergence of balanced SNNs was discussed. A comparison was made with some recently proposed efficient training methods for SNNs. The specific steps are as follows:

[0146] We compared the time-based backpropagation (BPTT) method with our method on the static dataset Fashion-MNIST and the neuromorphic dataset N-MNIST. The results show that MPIS-SNNs significantly outperform directly trained SNNs with significantly less simulation time. The memory overhead of MPIS-SNNs is a constant independent of simulation duration, while the memory overhead of the BPTT training method increases with the simulation step size and eventually becomes unacceptable. The firing rate is an important indicator of the energy consumption of SNNs; MPIS-SNNs exhibit a much lower firing rate than the BPTT training method, indicating lower energy consumption.

[0147] MPIS-SNNs, with the same number of parameters and similar structure to standard balanced SNNs, were constructed and compared on the more complex CIFAR-10 and CIFAR-100 datasets. To explore the performance of MPIS-SNNs, a model with an even larger number of parameters was also built. In terms of accuracy, MPIS-SNNs achieved higher accuracy with fewer simulation time steps; in terms of speed, MPIS-SNNs were still faster than IDE-Net even with more parameters. Specifically, MPIS-SNNs outperformed IDE-Nets (100 time steps) on both datasets with only 30 time steps, and the training time for MPIS-SNNs was less than half that of IDE-Nets. Even with more than twice the number of parameters of IDE-Nets, MPIS-SNNs achieved accuracy improvements of 2.76% and 4.14% on CIFAR-10 and CIFAR-100, respectively, while still maintaining faster training speed. This demonstrates the potential flexibility of MPIS-SNNs.

[0148] The convergence curve of the model changes as follows Figure 4As shown, since neurons may not fire impulses at the beginning of the network simulation, the difference may be zero in the initial stage of the simulation. The convergence curves show that MPIS-SNNs with two implicit flows converge to the equilibrium state faster, and the smaller the implicit flow branch, the faster the convergence speed. This result further explains why MPIS-SNNs require fewer simulation time steps to achieve better results and utilize faster convergence to reduce the latency of SNNs. Simulations of MPIS-SNNs with a single implicit flow and two implicit flows were performed for 100 time steps respectively. The convergence curves show that the computational architecture with two implicit flows converges to the equilibrium state faster, and the smaller the implicit flow branch, the faster the convergence speed.

Claims

1. A spiking neural network with reduced constant memory overhead, characterized in that: The forward and backward computations of the network are performed by two separate modules. The forward computation uses a forward fixed-point computation module, which performs fixed-point computation differently for static and dynamic images. The forward fixed-point computation module contains at least two implicit flow computation units. Each implicit flow unit consists of a single implicit flow model, including an input impulse, a feature extraction module, an output impulse, a feedback module, an upsampling module, and a downsampling module. Different implicit flows adopt a parallel architecture and are responsible for the approximate calculation of the equilibrium fixed point based on the general theory of equilibrium models. The backward computation module is used for backward stochastic gradient error calculation, which uses the approximate fixed point to calculate the direction error and update the network weights. It includes the following steps: Step 1: Preset neuron model, initialize neuron and network parameters. The neuron model adopts either the Integrated Firing Model (IF model) or the Leaked Integrated Firing Model (LIF model). In the calculation, the general dynamics of the neuron adopts the weighted membrane potential calculation form that includes discrete time. Step 2: Depending on the form of the input image, the input network is encoded through the inject module. The time step for fixed point calculation is defined. For static images, fixed point approximation is performed at a specific time step. For dynamic images, fixed point approximation is performed at a time step that matches the number of frames in the dynamic image. Step 3: Perform single implicit flow loop feature extraction. The intermediate layer of the network uses a neural network balancing method to calculate the fixed point in order to reduce the number of error calculations in the backpropagation process. The core of neural network balancing is weight sharing. Based on the input-output relationship, a fixed point equation is constructed, and the differential of the fixed point with respect to any parameter is solved. The differentials of both sides of the fixed point equation are calculated simultaneously. Combining the chain rule and IFT, the target differential is finally obtained, which is converted into a fixed point iterative problem for solution. Step 4: Improve model complexity by using a multi-parallel implicit flow approach. Input information is passed to the lower-level implicit flow module through downsampling to extract smaller and more complex image information. The computation process of each implicit flow is parallel, and information is passed between them through upsampling and downsampling. Step 5: Estimate the equilibrium distribution rate at the approximate fixed point based on the results within the expected time step; Step 6: Calculate the derivative of the network based on the approximate fixed-point loss function using the fixed-point launch rate, and let... For the equilibrium point regarding the distribution rate If the loss function is such that the loss is calculated for the parameters when the neural network converges to an equilibrium state, then the loss can be obtained. Differential: in: and Calculations are performed along the equivalent path using automatic differentiation; Calculate by solving a linear system; Step 7: Perform parameter updates based on backpropagation, which only requires one time step; Step 8: Repeat steps 2 to 7 until the preset number of training rounds is met, then the training is complete.

2. The spiking neural network with reduced constant memory overhead according to claim 1, characterized in that, When calculating the fixed point during forward propagation, the network initializes the input parameters, and the input static image is used to perform fixed point approximation calculation at time steps. The input image is input into the network through the image encoding module.

3. A spiking neural network with reduced constant memory overhead according to claim 1, characterized in that, When calculating the fixed point during forward propagation, the network initializes the input parameters and uses a dynamic image as the input to perform fixed point approximation calculations at time steps matching the number of frames in the dynamic image. The input image is input into the network through an image encoding module.

4. A spiking neural network with reduced constant memory overhead according to claim 1, characterized in that, The core unit for fixed-point approximation calculation is the state transition function layer, which contains multiple parallel structures of a single implicit flow used for input feature extraction and iterative calculation of approximate fixed points.

5. A spiking neural network with reduced constant memory overhead according to claim 4, characterized in that, In forward propagation fixed-point computation, a single implicit flow model includes an input pulse, a feature extraction module, an output pulse, a feedback module, an upsampling module, and a downsampling module, which respectively perform the functions of pulse input, feature extraction, pulse output, feedback to the input pulse according to the output signal, downsampling to other parallel implicit flows, and upsampling the output of the upper-layer parallel implicit flow.

6. A spiking neural network with reduced constant memory overhead according to claim 5, characterized in that, In forward propagation fixed-point computation, the core unit of a single implicit flow is the feature extraction module, which contains multiple feature extraction layers. Each feature extraction layer performs convolution, batch normalization, and variational dropout processes, and finally passes through IF / LIF neurons and uses DBReLU to approximate the balanced firing rate.

7. A spiking neural network with reduced constant memory overhead according to claim 4, characterized in that, In the forward propagation fixed-point computation process, in order to extract features at different scales, the number of implicit flows is at least two, and different implicit flows adopt a parallel architecture. The lower-level implicit flow obtains the pulse information of its upper-level implicit flow through upsampling, and the upper-level implicit flow obtains the pulse information of its lower-level implicit flow through downsampling.

8. A spiking neural network with reduced constant memory overhead according to claim 1, characterized in that, The inverse stochastic gradient error calculation module updates the network parameters using the stochastic gradient descent algorithm by calculating the derivative of the loss function with respect to the approximate equilibrium firing rate with respect to the network parameters.

9. The method for training a spiking neural network with reduced constant memory overhead according to claim 1, characterized in that, In step 5, when calculating the approximate firing rate of the fixed point, the equilibrium firing rate is approximated using the DBReLU method. The formula for calculating DBReLU is: in Let be the firing rate of the i-th neuron in the l-th layer. This represents the total number of neurons in layer (l-1). It is the connection weight between the j-th neuron in layer (l-1) and the i-th neuron in layer l. This represents the threshold of the neuron.

Images