Nonlinear pruning-based spiking neural network lightweight method

Abstract

The invention discloses a pulse neural network lightweight method based on nonlinear pruning, and relates to the technical field of artificial intelligence and neural networks. The method comprises the following steps: S1, constructing an NDI-LIF spiking neural network model, and introducing a bilinear product term into an input current in a dynamic updating process of a neuron membrane potential; s2, constructing a nonlinear synaptic pruning mechanism, representing a connection weight as a re-parameterization function composed of a re-parameterization weight and a conversion gain coefficient, and pruning the connection weight based on the re-parameterization function; and S3, training the NSPDI-SNN model to obtain a lightweight pulse neural network. By introducing a nonlinear dendritic integration mechanism and a state-adjustable synaptic pruning mechanism, balance between network sparsity and high performance is realized while the expression ability of the model is enhanced, and a lightweight spiking neural network model with high spatial-temporal expression ability, highly sparse structure and reasonable biological mechanism is constructed. And the generalization ability and the efficient training performance under various tasks are ensured.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, neural networks, and other technologies, and in particular to a technical solution for a lightweight spiking neural network based on nonlinear pruning. Background Technology

[0002] Spiking Neural Network (SNN) is an artificial neural network model inspired by biological neurons. It uses sparse event-driven pulses for information transmission and processing, achieving excellent spatiotemporal dynamic characteristics and high energy efficiency.

[0003] In recent years, SNNs have gained widespread attention in fields such as vision, control, and signal processing. However, existing SNN methods still face several challenges: on the one hand, simply transplanting deep learning (Artificial Neural Network) structures directly into SNNs can easily lead to massive model sizes, a sharp increase in computational complexity and hardware resource consumption, making it difficult to meet real-time and low-power requirements; on the other hand, most traditional SNNs employ linear impulse integral mechanisms, which are insufficient for modeling the complex nonlinear integration capabilities of dendrites, limiting their ability to express spatiotemporal features. Furthermore, because impulse functions are nondifferentiable, SNN training requires special techniques (such as alternative gradient methods), making the training process complex and difficult to converge. Therefore, how to design an efficient SNN architecture with high sparsity and low computational cost while ensuring high accuracy has become a key research issue. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a lightweight spiking neural network method based on nonlinear pruning. The aim is to enhance the model's expressive power while achieving a balance between network sparsity and high performance by introducing a nonlinear dendritic integration (NDI) mechanism and a state-tunable synaptic pruning (NSP) mechanism. This results in a lightweight spiking neural network model with high spatiotemporal expressive power, a highly sparse structure, and a biologically sound mechanism, while ensuring its generalization ability and efficient training performance across various tasks.

[0005] The technical solution adopted in this invention is as follows:

[0006] A lightweight method for spiking neural networks based on nonlinear pruning includes the following steps:

[0007] S1. Construct the NDI-LIF spiking neural network model;

[0008] The NDI-LIF (Nonlinear Dendritic Integration Leaky Integrate-and-Fire) spiking neural network model consists of several NDI-LIF neurons;

[0009] The NDI-LIF neuron, during the dynamic update of membrane potential, introduces a bilinear product term into the input current to simulate the nonlinear integration process of excitatory and inhibitory inputs in biological neurons, thereby enhancing the neuron's ability to represent complex spatiotemporal patterns.

[0010] After introducing a bilinear product term into the input current I, the expression is as follows:

[0011] I = Wx + b + (Wx) ⊙ (W n x)

[0012] Where ⊙ represents the Hadamard product, x represents the input data, and W is the linear synaptic weight matrix. n is a non-linear integrated weight matrix, and b is a bias term.

[0013] S2. Construct a nonlinear synaptic pruning mechanism;

[0014] A nonlinear synaptic pruning mechanism was introduced to prune the connection weights in the constructed NDI-LIF spiking neural network model, resulting in the NSPDI-SNN (Nonlinear Synaptic Pruning And Nonlinear Dendritic Integration SNN) model.

[0015] The pruning process of the nonlinear synaptic pruning mechanism includes:

[0016] The linear synaptic weight matrix W and the nonlinear integrated weight matrix W n Each connection weight w is represented as a reparameterized function consisting of the reparameterized weight θ and the transformation gain coefficient a:

[0017] w = sign(θ)·[a·(|θ|-d1)] +

[0018] in,[] + This represents the ReLU function, where d1 is the preset state transition threshold.

[0019] The connection weights w are determined and pruned based on the reparameterization function: when the θ value is lower than the preset state transition threshold d1, it indicates that the connection is inactive (corresponding to the "filamentous pseudopodia" state of dendritic spines in organisms), and the corresponding connection weights w are pruned; when the θ value is higher than the preset state transition threshold d1, it indicates that the connection is active (corresponding to the "mature" state of dendritic spines in organisms), and the corresponding connection weights w are retained.

[0020] Furthermore, threshold pruning is introduced into the connection weights w in the constructed NSPDI-SNN model. When the absolute value of the connection weight w is less than the preset weight threshold d2, the corresponding connection weight w is pruned. Threshold pruning can prune connections that contribute very little to the inference process. Through the dual pruning strategy, a high sparse network structure is constructed.

[0021] S3. Train the NSPDI-SNN model;

[0022] The samples in the training set are converted into pulse sequences, and the constructed NSPDI-SNN model is trained to obtain a lightweight spiking neural network.

[0023] Furthermore, in step A3, the training process of the NSPDI-SNN model includes:

[0024] Set the loss function to calculate the loss value;

[0025] The gradient information of the NSPDI-SNN model network parameters is calculated using the loss value, and then the gradient information of the reparameterized weights θ and the transformation gain coefficient a is calculated and simultaneously optimized and updated.

[0026] Furthermore, the mean squared error is used as the loss function to calculate the loss value L:

[0027]

[0028] Among them, y n Let s be the true label of the nth sample. t,n Let n be the output pulse of the nth sample at time t, where n = 1, 2, ..., N, N is the sample batch size, and t = 1, 2, ..., T, T is the time window size.

[0029] The non-differentiable step function g(u) t It is approximated by a differentiable substitution function:

[0030]

[0031] The gradient information of the NSPDI-SNN model network parameters is calculated based on the differentiable substitution function; then, the gradient information of the reparameterized weights θ and the transformation gain coefficient a is calculated based on the gradient information of the connection weights contained in the gradient information of the network parameters.

[0032]

[0033] in, Let a, θ, and w represent the magnitudes of the gradients of a, θ, and w with respect to the loss value L, respectively.

[0034] Furthermore, in step S1, the bilinear product terms introduced by the NDI-LIF neurons support deployment at the synaptic, channel, or hierarchical level.

[0035] Furthermore, in step S2, the nonlinear synaptic pruning mechanism supports deployment at the synaptic level, channel level, or hierarchical structure.

[0036] Furthermore, in step S2, the state transition threshold d1 and weight threshold d2 are both adjusted through a progressive threshold scheduling strategy to achieve a smooth transition from a dense structure to a sparse structure, thereby improving the stability and performance retention of the final model.

[0037] This invention has the following significant advantages over the prior art:

[0038] 1) The present invention constructs an NSPDI-SNN model through the NDI mechanism, which greatly enhances the nonlinear modeling capability of spatiotemporal inputs, enabling the network to capture richer spatiotemporal dependencies, thereby significantly improving the expressive power of spatiotemporal features.

[0039] 2) This invention can achieve an ultra-high sparsity rate without sacrificing classification accuracy: Experiments show that the NSPDI-SNN model can support a connection sparsity rate of up to 98% or more. Even if the sparsity rate reaches an extremely high level, the classification accuracy of the pruned model is still comparable to that of the original dense model, with minimal performance loss.

[0040] 3) This invention can significantly reduce computational and storage overhead: After pruning, most parameters in the model are removed, resulting in a significant reduction in network computation, storage requirements, and energy consumption. On neuromorphic hardware, the power consumption of the pruned network can be reduced to below the microwatt level, making this lightweight SNN particularly suitable for deployment in edge devices and other resource-constrained scenarios. Attached Figure Description

[0041] Figure 1 It is a lightweight method framework for spiking neural networks based on nonlinear pruning.

[0042] Figure 2 This is a demonstration of the results of a lightweight spiking neural network method based on nonlinear pruning. Detailed Implementation

[0043] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples, so that those skilled in the art can better understand the present invention.

[0044] This embodiment provides a lightweight method for spiking neural networks based on nonlinear pruning, such as... Figure 1 As shown, the specific steps include the following:

[0045] S1. Construct the NDI-LIF spiking neural network model;

[0046] The NDI-LIF (Nonlinear Dendritic Integration Leaky Integrate-and-Fire) spiking neural network model consists of several NDI-LIF neurons. This invention, based on traditional LIF neurons, introduces a bilinear product term into the dynamic membrane potential update process to construct NDI-LIF neurons, thereby simulating the nonlinear integration process of excitatory and inhibitory inputs in biological neurons and enhancing the neuron's ability to represent complex spatiotemporal patterns, such as... Figure 1 As shown in (a).

[0047] Specifically, the dynamic update process of membrane potential in NDI-LIF neurons is as follows:

[0048]

[0049] Among them, g(u t ) represents the step function, u t u t-1 Let s represent the membrane potentials at time t and t-1, respectively. t s t-1 These represent the pulse outputs at time t and t-1, respectively, where I represents the input current and τ represents the pulse output at time t-1. m U is the membrane potential time constant. th This is the threshold for issuance.

[0050] To enhance the neuron's ability to integrate complex information, this invention introduces a nonlinear dendritic integration mechanism, namely, introducing a bilinear product term (Wx)⊙(Wx) into the linear term (Wx+b) of the input current. n x), extending the input current from linear integration to bilinear integration, is expressed as follows:

[0051] I = Wx + b + (Wx) ⊙ (W n x)

[0052] Where ⊙ represents the Hadamard product, x represents the input data, and W is the linear synaptic weight. n denoted by , b represents the non-linear integration weights, and b is the bias term. This embodiment introduces a non-linear dendritic integration mechanism at the channel level. This formula is applicable to both fully connected and convolutional neural networks.

[0053] S2. Construct a nonlinear synaptic pruning mechanism;

[0054] To improve network sparsity and reduce computational burden, this embodiment employs a two-level pruning strategy at the channel level, such as... Figure 1 As shown in (b), the NSPDI-SNN model is obtained after pruning.

[0055] The first level of pruning is non-linear synaptic pruning, specifically:

[0056] The linear synaptic weight matrix W and the nonlinear integrated weight matrix W n Each connection weight w is represented as a reparameterized function consisting of the reparameterized weight θ and the transformation gain coefficient a:

[0057] w = sign(θ)·[a·(|θ|-d1)] +

[0058] in,[] + This represents the ReLU function, where d1 is the preset state transition threshold; the reparameterized weights θ and the transition gain coefficient a are randomly assigned initial values ​​and optimized through backpropagation during training.

[0059] The connection weights w are determined and pruned based on the reparameterization function: when the θ value is lower than the preset state transition threshold d1, it indicates that the connection is inactive (corresponding to the "filamentous pseudopodia" state of dendritic spines in organisms), and the corresponding connection weights w are pruned; when the θ value is higher than the preset state transition threshold d1, it indicates that the connection is active (corresponding to the "mature" state of dendritic spines in organisms), and the corresponding connection weights w are retained.

[0060] The second level of pruning is threshold pruning, specifically:

[0061] The connection weights w are pruned using a preset weight threshold d2:

[0062]

[0063] S3. Train the NSPDI-SNN model;

[0064] Samples from speech datasets, event stream datasets, or action decision task datasets are converted into pulse sequences and used to train the constructed NSPDI-SNN model, resulting in a lightweight spiking neural network.

[0065] During training, a loss function is set to calculate the loss value;

[0066] The gradient information of the NSPDI-SNN model network parameters, as well as the gradient information of the reparameterized weights θ and the transformation gain coefficient a, are calculated using the loss value and then simultaneously optimized and updated.

[0067] Specifically, in this embodiment, the mean squared error (MSE) is used as the loss function to calculate the loss value L:

[0068]

[0069] Among them, y m Let s be the true label of the Lth sample. t,n Let n be the output pulse of the nth sample at time t, where n = 1, 2, ..., N, N is the sample batch size, and t = 1, 2, ..., T, T is the time window size.

[0070] The non-differentiable step function g(u) t It is approximated by a differentiable substitution function:

[0071]

[0072] The gradient information of the NSPDI-SNN model network parameters is calculated based on the differentiable substitution function; then, the gradient information of the reparameterized weights θ and the transformation gain coefficient a is calculated based on the gradient information of the connection weights contained in the gradient information of the network parameters.

[0073]

[0074] in, Let a, θ, and w represent the magnitudes of the gradients of a, θ, and w with respect to the loss value L, respectively.

[0075] In this embodiment, the Adam optimizer is used to simultaneously adjust the synaptic pruning state parameters (i.e., reparameterized weights θ and transformation gain coefficients a) in neurons, so as to compress the model structure to the maximum extent while preserving important connections, thereby obtaining a lightweight neural network model with both high accuracy and high sparsity.

[0076] Experimental verification and effect description:

[0077] This invention employs the above method on multiple datasets, including DVS128 Gesture, CIFAR10-DVS, SHD, and Maze2D, to obtain the NSPDI-SNN model, which improves the model sparsity to over 98% while maintaining accuracy. Figure 2As shown in (a), on the DVS128 Gesture dataset, setting only d1 to 0.98 already achieves a high sparsity of 96.88% while maintaining an accuracy of 96.15%, indicating that the state transition mechanism can effectively compress the parameter size without sacrificing performance. Further introducing a weight threshold d2 and gradually increasing its value (0.00, 0.02, 0.04, 0.06, 0.08, 0.10) continuously increases the sparsity from 96.88% to 98.83%, but the accuracy gradually decreases. The results show that d2 can further compress the model size, balancing sparsity and performance as needed. Figure 2 As shown in (b), on the CIFAR10-DVS dataset, when only d2 is set to 0.98, the sparsity of the model reaches 94.98% and the recognition accuracy is 75.60%; when d2 is gradually increased (from 0.00 to 0.05), the sparsity can be further improved to 98.95%.

[0078] Therefore, the state transition threshold d1 is used to achieve basic compression with high sparsity at the neuron level, while the weight threshold accuracy serves as an additional adjustment method to further improve sparsity, allowing for a flexible trade-off between sparsity and accuracy based on task requirements. Furthermore, in speech recognition and reinforcement learning tasks, this invention maintains high performance while ensuring high sparsity, demonstrating its good generalization and robustness in multi-task scenarios, making it more suitable for deployment in resource-constrained neuromorphic hardware and edge computing devices.

Claims

1. A lightweight method for spiking neural networks based on nonlinear pruning, characterized in that, Includes the following steps: S1. Construct the NDI-LIF spiking neural network model; The NDI-LIF spiking neural network model consists of several NDI-LIF neurons; The NDI-LIF neuron, during the dynamic update of membrane potential, introduces a bilinear product term into the input current to simulate the nonlinear integration process of excitatory and inhibitory inputs in biological neurons, thereby enhancing the neuron's ability to represent complex spatiotemporal patterns. After introducing a bilinear product term into the input current I, the expression is as follows: I=Wx+b+(Wx)⊙(W n x) Where ⊙ represents the Hadamard product, x represents the input data, and W is the linear synaptic weight matrix. n Here, b is the non-linear integrated weight matrix, and b is the bias term; S2. Construct a nonlinear synaptic pruning mechanism; A nonlinear synaptic pruning mechanism was introduced to prune the connection weights in the constructed NDI-LIF spiking neural network model to obtain the NSPDI-SNN model. The pruning process of the nonlinear synaptic pruning mechanism includes: The linear synaptic weight matrix W and the nonlinear integrated weight matrix W n Each connection weight w is represented as a reparameterized function consisting of the reparameterized weight θ and the transformation gain coefficient a: w=sign(θ)·[a·(|θ|-d1)] + in,[] + This represents the ReLU function, where d1 is the preset state transition threshold; The connection weights w are determined and pruned based on the reparameterization function: when the θ value is lower than the preset state transition threshold d1, it indicates that the connection is inactive, and the corresponding connection weights w are pruned; when the θ value is higher than the preset state transition threshold d1, it indicates that the connection is active, and the corresponding connection weights w are retained. S3. Train the NSPDI-SNN model; The samples in the training set are converted into pulse sequences, and the constructed NSPDI-SNN model is trained to obtain a lightweight spiking neural network.

2. The lightweight spiking neural network method based on nonlinear pruning as described in claim 1, characterized in that, In step A3, the training process of the NSPDI-SNN model includes: Set the loss function to calculate the loss value; The gradient information of the NSPDI-SNN model network parameters is calculated using the loss value, and then the gradient information of the reparameterized weights θ and the transformation gain coefficient a is calculated and simultaneously optimized and updated.

3. The lightweight spiking neural network method based on nonlinear pruning as described in claim 2, characterized in that, Using the mean squared error as the loss function, the loss value L is calculated: Among them, y n Let s be the true label of the nth sample. t,n Let n be the output pulse of the nth sample at time t, where n = 1, 2, ..., N, N is the sample batch size, and t = 1, 2, ..., T, T is the time window size. The non-differentiable step function g(u) during the dynamic update of the membrane potential t It is approximated by a differentiable substitution function: The gradient information of the NSPDI-SNN model network parameters is calculated based on the differentiable substitution function; then, the gradient information of the reparameterized weights θ and the transformation gain coefficient a is calculated based on the gradient information of the connection weights contained in the gradient information of the network parameters. in, Let a, θ, and w represent the magnitudes of the gradients of a, θ, and w with respect to the loss value L, respectively.

4. The lightweight spiking neural network method based on nonlinear pruning as described in claim 3, characterized in that, Step S2 further includes threshold pruning; the threshold pruning is: when the absolute value of the connection weight w is less than a preset weight threshold d2, the corresponding connection weight w is pruned.

5. A lightweight spiking neural network method based on nonlinear pruning as described in claim 4, characterized in that, In step S1, the bilinear product terms introduced by the NDI-LIF neurons support deployment at the synaptic, channel, or hierarchical level.

6. The lightweight spiking neural network method based on nonlinear pruning as described in claim 4, characterized in that, In step S2, the nonlinear synaptic pruning mechanism supports deployment at the synaptic level, channel level, or hierarchical structure.

7. A lightweight spiking neural network method based on nonlinear pruning as described in any one of claims 3-6, characterized in that, In step S2, the state transition threshold d1 and weight threshold d2 are both adjusted through a progressive threshold scheduling strategy to achieve a smooth transition from a dense structure to a sparse structure, thereby improving the stability and performance retention of the final model.

Images