Pulse neural network pruning method based on conversion error
By modeling the spiking neural network pruning problem as a two-objective optimization problem of conversion error and accuracy, and using evolutionary algorithms to decompose and splice, the problems of insufficient accuracy and gradient instability in the existing spiking neural network pruning methods are solved, and a high-accuracy, low-power spiking neural network is achieved.
Patent Information
- Application Number
- CN202410323203.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-20
- Publication Date
- 2025-09-23
AI Technical Summary
Existing spiking neural network pruning methods cannot effectively combine accuracy optimization, and gradient-based methods are susceptible to gradient vanishing or gradient exploding, resulting in unsatisfactory pruning effects.
The spiking neural network pruning problem is modeled as a two-objective optimization problem of conversion error and accuracy, which is decomposed into a single-layer optimization problem using an evolutionary algorithm and automatically solved by the two-objective evolutionary algorithm. Finally, a high-performance spiking neural network is obtained by splicing.
It achieves a stable and highly accurate pulse neural network in the edge deployment scenario, avoids the influence of gradient instability, and has the advantages of low power consumption and high performance.
Smart Images
Figure CN120688564A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to pruning techniques for spiking neural networks, and in particular to a structured pruning method for convolutional spiking neural networks based on conversion errors. Background Art
[0002] Spiking neural networks, known as the third generation of artificial neural networks due to their biomimetic nature and low computational power consumption, are a hot research topic. Pruning spiking neural networks is crucial for reducing power consumption and has been widely studied both domestically and internationally.
[0003] Currently, many spiking neural network pruning methods use rule-inspired pruning methods and gradient-based methods. For rule-inspired methods, network parameters are pruned by manually designing pruning rules. For example, Shen et al. proposed the ESL-SNNs pruning method, which uses biologically inspired rules to select important weights and remove redundant weights. Chowdhury et al. proposed using principal component analysis to analyze the importance of different channels and prune unimportant weights. Within each layer of a trained convolutional spiking neural network, the output of each channel during forward inference is used as a one-dimensional feature. The channels at the back of the principal component analysis are pruned using principal component analysis combined with a set pruning rate. However, because this method does not explicitly combine the accuracy of the spiking neural network with pruning, the accuracy of the pruned spiking neural network is generally poor. Literature: Sayeed Shafayet Chowdhury, IshaGarg, and Kaushik Roy. Spatio-temporal pruning and quantization for low-latency spiking neural networks. Proceedings of the 2021 International Joint Conference on NeuralNetworks, 2021:1-9.
[0004] Regarding gradient-based methods, the Grad R method proposed by Chen et al. adds new weight coefficients to existing weights in the network, then automatically obtains the importance of the weights through training, and directly discards weights with low weight coefficients to obtain a pruned spiking neural network. However, due to the non-differentiable nature of spiking neural networks, the process of obtaining weights through training is extremely susceptible to gradient vanishing or gradient exploding, so this method has not yet achieved ideal results on deeper network structures. Reference: Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spiking neural networks through gradient rewiring. Proceedings of the 30th International Joint Conference on Artificial Intelligence, 2021: 1713-1721. Summary of the Invention
[0005] Purpose of the invention: To overcome the weaknesses of current rule-based spiking neural network pruning algorithms that cannot directly optimize performance-related objectives and gradient-based spiking neural network pruning methods that are susceptible to gradient vanishing or gradient exploding, based on the conversion error theory, the spiking neural network pruning problem is modeled as a two-objective optimization problem of conversion error and base network accuracy, and the optimization problem of the overall network is divided into optimization problems within a single layer, so that an evolutionary algorithm can be used to efficiently and automatically solve it.
[0006] To address the above technical issues, the present invention discloses a spiking neural network pruning method based on conversion error. This method can be used in on-device deployment scenarios such as target detection and speech recognition, and includes the following steps:
[0007] Step 1: Model the spiking neural network pruning as a subset selection optimization problem related to the transformation error and the performance of the artificial neural network;
[0008] Step 2: Decompose the conversion error calculation of the entire network into the output error calculation between the corresponding layers, and then decompose the optimization problem of the entire network into the optimization problem within each layer;
[0009] Step 3, calculating the conversion error between the artificial neural network and the spiking neural network layers of different network substructures and the accuracy of the artificial neural network on the training set;
[0010] Step 4: Use the two-objective evolutionary algorithm to automatically solve the optimal substructure of each layer, and splice the substructures of each layer to obtain a neat network substructure;
[0011] Step 5, fine-tune the substructure using the training set, and then transform the artificial neural network to obtain a pruned spiking neural network;
[0012] Step 6: Iterate steps 1 to 5 until the obtained spiking neural network pruning rate meets the set threshold.
[0013] Among them, the evolutionary algorithm is a gradient-free stochastic optimization algorithm.
[0014] In step 1, the pruning rate of the network is set to p. For the artificial neural network A trained on the training set S, L_ij is used to represent the jth convolution kernel of the i-th layer, m = {m ij |m ij ∈{0, 1}, i∈{1, 2, ..., n}, j∈{1, 2, ..., li}} represents the mask vector of the convolution kernel. For any of its substructures Modeled as an optimization problem between the conversion error and the accuracy of the artificial neural network:
[0015]
[0016] In step 2, based on the fact that the conversion error is limited by the output error of each layer, the output error of the corresponding layers of the artificial neural network and the pulse neural network is used to measure the conversion error between layers, thereby decomposing the large-scale optimization problem modeled on the entire network in step 1 into smaller-scale optimization problems that are easier to solve in each layer, and using m i ={m j |m j ∈{0,1},j∈{1,2,...,l i}} represents the mask vector of the i-th layer of the network, x i represents the output of the artificial neural network at layer i, Represents the output of the spiking neural network at layer i. The error LayerGap between the corresponding layers of the artificial neural network and the spiking neural network is calculated as follows:
[0017]
[0018] LayerGap measures the conversion error of the substructure of this layer, so the optimization problem of the overall network structure in step 1 is decomposed into the optimization problem within each layer:
[0019]
[0020] in, Indicates the accuracy of the overall network structure corresponding to this layer of substructure on the training set.
[0021] In step 3, the conversion error between the artificial neural network and the spiking neural network layers of different network substructures is calculated by performing forward inference on the training set, and the same data is forward inferred on the overall artificial neural network and spiking neural network corresponding to the substructure to calculate the accuracy and conversion error of the substructure.
[0022] In step 4, the two-objective evolutionary algorithm is used to solve the single-layer optimization problem modeled in step 3. Five substructures of the layer are randomly initialized as the evolution population. Five individuals are randomly selected from the population for random mutation to produce five offspring. The optimal five individuals are selected from the population and offspring through a comprehensive ranking of transformation error and accuracy to form the next evolution population. After 20 rounds of evolution, the optimal individual is selected from the final evolution population as the optimal substructure of the layer. For the comprehensive ranking of the two objectives, the evolutionary algorithm uses the average_rank ranking algorithm to sort the different substructures. The sorting method is:
[0023]
[0024] Among them, index1 is the ranking of the substructure on the optimization target of conversion error, and index2 is the ranking of the substructure on the optimization target of accuracy. All substructures are sorted, and the optimal substructure is selected as the solution for this layer. Then the solutions obtained from different layers are concatenated into the overall pruned network.
[0025] In step 5, the pruned network is fine-tuned using the warmup strategy, first with a linearly increasing learning rate for 50 epochs, and then with a cosine decay learning rate for 200 epochs. The trained artificial neural network is converted to a pruned spiking neural network through the conversion algorithm.
[0026] In step 6, for the pruned spiking neural network obtained in this round, if its pruning rate does not reach the set pruning rate, then the pruned artificial neural network in this round is used as the new base network, and steps 1 to 6 are repeated to prune it; if the pruned spiking neural network obtained in this round reaches the set pruning rate, the algorithm outputs spiking neural networks with different pruning rates and accuracy rates.
[0027] Beneficial effects: The significant advantage of the present invention is that the network performance is explicitly added to the pruning process for optimization, and by utilizing the converted pulse neural network training method, the influence of unstable gradients when directly training the pulse neural network is avoided. By minimizing the conversion error and maximizing the accuracy, a pulse neural network with high accuracy after pruning can be stably obtained. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.
[0029] Figure 1 The flowchart of obtaining the pruned pulse neural network of the present invention.
[0030] Figure 2 This is the classification accuracy of the present invention when performing instance verification on the CIFAR-10 dataset. DETAILED DESCRIPTION
[0031] In order to make the objectives, technical solutions and advantages of the present invention clearer, this chapter provides a further detailed description of the invention in conjunction with the accompanying drawings.
[0032] Figure 1 The flowchart of the present invention for obtaining a pruned spiking neural network includes 6 steps.
[0033] In the first step, the pruning rate of the network is set to p, and the pruning problem of the spiking neural network is modeled as a subset selection optimization problem related to the conversion error and the accuracy of the artificial neural network. For the artificial neural network A trained on the training set S, use L ij Represents the jth convolution kernel of the i-th layer, m={mi j |mi j ∈{0, 1}, i∈{1, 2, ..., n}, j∈{1, 2, ..., l i}} represents the mask vector of the convolution kernel, for any of its substructures The optimization problem of conversion error and artificial neural network accuracy is expressed as:
[0034]
[0035] In the second step, the large-scale optimization problem on the entire network is decomposed into smaller-scale optimization problems within each layer. Based on the fact that the conversion error is limited by the output error of each layer, the output error of the corresponding layers of the artificial neural network and the pulse neural network is used to measure the conversion error between layers, thereby decomposing the large-scale optimization problem modeled on the entire network in the first step into smaller-scale optimization problems that are easier to solve within each layer, and using m i ={m j |m j ∈{0,1},j∈{1,2,...,l i}} represents the mask vector of the i-th layer of the network, x i represents the output of the artificial neural network at layer i, Represents the output of the spiking neural network at layer i. The error LayerGap between the corresponding layers of the artificial neural network and the spiking neural network is calculated as follows:
[0036]
[0037] LayerGap measures the conversion error of the substructure of this layer, so the optimization problem of the overall network structure in the first step is decomposed into the optimization problem within each layer:
[0038]
[0039] in, Indicates the accuracy of the overall network structure corresponding to this layer of substructure on the training set.
[0040] In the third step, forward inference was performed on the training set to calculate the conversion error between the artificial neural network and spiking neural network layers of different network substructures. The same data was then forward inferred on the entire artificial neural network and spiking neural network corresponding to each substructure to calculate the accuracy and conversion error of each substructure. Forward inference was performed using the entire training set to obtain the accuracy, and forward inference was performed using five data sets to obtain the conversion error.
[0041] In the fourth step, a two-objective evolutionary algorithm is used to solve the single-layer optimization problem modeled in the third step. Five substructures of the layer are randomly initialized as the evolution population. Five individuals are randomly selected from the population and subjected to random mutation to produce five offspring. The next evolution population is constructed by selecting the best five individuals from the population and offspring based on a comprehensive ranking of conversion error and accuracy. After 20 rounds of evolution, the best individual is selected from the final evolution population as the optimal substructure of the layer. In the evolutionary algorithm, a 0-1 mask vector is used to represent the individuals corresponding to different substructures. The initial population is obtained by randomly flipping all-one vectors with probability p1. Mutation is also performed bit by bit with probability p1. Probability p2 is set as the pruning limit. When the pruning rate of the individual obtained after random mutation is lower than p2, the 1s are flipped to 0s bit by bit until the pruning rate limit is met.
[0042] For the comprehensive ranking of the two objectives, the evolutionary algorithm uses the average_rank sorting algorithm to sort different substructures. The sorting method is:
[0043]
[0044] Among them, index1 is the ranking of the substructure on the optimization target of conversion error, and index2 is the ranking of the substructure on the optimization target of accuracy. All substructures are sorted and the optimal substructure is selected as the solution for that layer. The solutions obtained at different layers are then concatenated into the overall pruned network. The substructure obtained by simultaneously optimizing the conversion error and the accuracy of the artificial neural network is converted into a high-performance spiking neural network.
[0045] In the fifth step, the pruned network is fine-tuned using the warmup strategy, first with a linearly increasing learning rate for 50 epochs, then with a cosine decay learning rate for 200 epochs. The trained artificial neural network is converted to a pruned spiking neural network through a conversion algorithm.
[0046] In the sixth step, for the pruned spiking neural network obtained in this round, if its pruning rate does not reach the set pruning rate, then the pruned artificial neural network in this round is used as the new base network, and steps 1 to 6 are repeated to prune it; if the pruned spiking neural network obtained in this round reaches the set pruning rate, the algorithm outputs spiking neural networks with different pruning rates and accuracy rates.
[0047] After the algorithm is completed, the spiking neural network with the corresponding pruning rate is selected for deployment according to the actual deployment requirements.
[0048] Example
[0049] In order to verify the effectiveness of the pruning algorithm, an example verification is conducted on the commonly used image classification dataset CIFAR-10. Each dataset contains a training set and a test set. For each dataset, the commonly used network structure VGG-16 is used. Figure 1 The algorithm flow shown in the figure is used for pruning, and the classification accuracy of the final pruned spiking neural network is calculated.
[0050] Take the CIFAR-10 dataset as an example. This dataset is a commonly used computer vision dataset for image classification tasks. It contains 10 different categories of image data, corresponding to airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Each category contains 6,000 images, for a total of 60,000 images. The size of each image is 32×32 pixels. 50,000 images were randomly divided as training data, and the remaining 10,000 images were used as test data. The 50,000 images in the training set were used for the spiking neural network pruning algorithm. The following steps were followed to prune the VGG-16:
[0051] 1. Define the structured pruning problem of spiking neural networks at the channel level and model it as an optimization problem about the conversion error and the accuracy of the base artificial neural network.
[0052] 2. The optimization problem of pruning the entire network is decomposed into the optimization problem within each layer. For VGG-16, its 1-13 layers are convolutional layers. The optimization problem is independently modeled in each layer, and the substructure within the layer with low conversion error and high performance potential is searched.
[0053] 3. Calculate the conversion error and accuracy on real data by performing forward inference on the CIFAR-10 training set;
[0054] 4. Use the binary evolution algorithm to solve the optimal substructures of layers 1-13 respectively, and then splice the substructures of each layer to obtain the substructure of the overall network;
[0055] 5. Fine-tune the substructure for 250 rounds using the training set, and then convert the fine-tuned artificial neural network to obtain the pruned spiking neural network;
[0056] 6. Iterate steps 1 to 5 until the obtained spiking neural network pruning rate meets the set threshold.
[0057] Finally, a spiking neural network that meets the pruning rate is obtained. For this spiking neural network, the CIFAR-10 test set is used for image classification. Under different spiking neural network hyperparameter simulation time steps, when the simulation time step is 8, the accuracy rate is achieved at a connection retention rate of 46.79%, which is 94.07%. Compared with the existing spiking neural network pruning method, the pruning rate is lower and the performance is applicable to general deep neural networks. Therefore, when used for the terminal side deployment of spiking neural networks, the present invention has the advantages of low power consumption and high performance.
[0058] like Figure 2The table shows the test results of the proposed spiking neural network pruning method on the CIFAR-10 dataset using the general convolutional spiking neural network structure VGG-16 as the basis, and compares the accuracy of the proposed method with that of existing methods on the test set. The first column of the table lists the existing methods for comparison, the second column lists the network structure used by the proposed method, and the last row shows the proposed method. ADMM-based method (reference: Lei Deng, Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufei Ding, Peng Li, and Yuan Xie. Comprehensive SNN compression using ADMM optimization and activity regularization. IEEE Transactions on Neural Networks and Learning Systems, 2023: 2791-2805.) and ESL-SNNs method (reference: Jiangrong Shen, Qi Xu, Jian K Liu, Yueming Wang, Gang Pan, and Huajin Tang. Esl-snns: Anevolutionary structure learning strategy for spiking neural networks. arXiv: 2306.03693, 2023.) Test results show that the present invention achieves both higher pruning rate and higher accuracy on a general convolutional spiking neural network structure, demonstrating the feasibility and effectiveness of this spiking neural network pruning method.
[0059] The present invention provides a method for pruning a spiking neural network based on conversion errors. There are many methods and approaches for implementing this technical solution. The above is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.
Claims
1. A spiking neural network pruning method based on conversion error, characterized in that: The steps include: Step 1: Model the spiking neural network pruning as a subset selection optimization problem related to the transformation error and the performance of the artificial neural network; Step 2: Decompose the conversion error calculation of the entire network into the output error calculation between the corresponding layers, and then decompose the optimization problem of the entire network into the optimization problem within each layer; Step 3, calculating the conversion error between the artificial neural network and the spiking neural network layers of different network substructures and the accuracy of the artificial neural network on the training set; Step 4: Use the two-objective evolutionary algorithm to automatically solve the optimal substructure of each layer, and splice the substructures of each layer to obtain a neat network substructure; Step 5, fine-tune the substructure using the training set, and then transform the artificial neural network to obtain a pruned spiking neural network; Step 6: Iterate steps 1 to 5 until the obtained spiking neural network pruning rate meets the set threshold.
2. The method according to claim 1, characterized in that Step 1 includes the following steps: Step 1-1, for the convolutional artificial neural network A trained on the training set S, L ij Represents the jth convolution kernel of the i-th layer, m={m ij |m ij ∈{0, 1}, i∈{1, 2,..., n}, j∈{1, 2,..., l i } represents the mask vector of the convolution kernel, for any of its substructures Modeled as an optimization problem between the conversion error and the accuracy of the artificial neural network: Step 1-2, manually set parameters: pruning rate p.
3. The method according to claim 2, characterized in that In step 2, the output error of the corresponding layers of the artificial neural network and the spiking neural network is used to measure the conversion error between layers, and m is used i ={m j |m j ∈{0,1},j∈{1,2,...,l i }} represents the mask vector of the i-th layer of the network, x i represents the output of the artificial neural network at layer i, represents the output of the spiking neural network at layer i. The conversion error between the corresponding layers of the artificial neural network and the spiking neural network is calculated as follows: The optimization problem of the overall network structure is decomposed into the optimization problem within each layer: in, Indicates the accuracy of the overall network structure corresponding to this layer of substructure on the training set.
4. The method according to claim 3, characterized in that In step 3, the conversion error between the artificial neural network and the spiking neural network layers of different network substructures is calculated on the training set, and the accuracy and conversion error of the substructure are calculated by performing forward inference on the overall artificial neural network and spiking neural network corresponding to the substructure with the same data.
5. The method according to claim 4, characterized in that In step 4, the evolutionary algorithm is used to automatically solve the two-objective optimization problem within a single layer. The average_rank sorting algorithm is used to sort different substructures within the evolutionary algorithm. The sorting method is: Among them, index1 is the ranking of the substructure on the optimization target of conversion error, and index2 is the ranking of the substructure on the optimization target of accuracy. All substructures are sorted, and the optimal substructure is selected as the solution for this layer. Then the solutions obtained from different layers are concatenated into the overall pruned network.
6. The method according to claim 5, characterized in that In step 5, the pruned network is fine-tuned using the warmup strategy, first with a linearly increasing learning rate for 50 epochs, and then with a cosine decay learning rate for 200 epochs. The trained artificial neural network is converted to a pruned spiking neural network through the conversion algorithm.
7. The method according to claim 6, characterized in that In step 6, for the pruned spiking neural network obtained in this round, if its pruning rate does not reach the set pruning rate, then the pruned artificial neural network in this round is used as the new base network, and steps 1 to 6 are repeated to prune it; if the pruned spiking neural network obtained in this round reaches the set pruning rate, the algorithm outputs spiking neural networks with different pruning rates and accuracy rates.