A spiking neural network compression method suitable for neuromorphic hardware
By constructing and optimizing spiking convolutional neural networks, and combining pre-training and pruning techniques, the problem of deploying spiking neural networks on embedded devices was solved, achieving efficient model compression and performance improvement.
Patent Information
- Application Number
- CN202310244958.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-14
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2043-03-14
AI Technical Summary
Deploying large-scale, high-efficiency spiking neural networks on embedded devices or neuromorphic chips faces challenges such as excessively large model size and computational load, making deployment difficult or even impossible.
By constructing a spiking convolutional neural network, and combining data preprocessing, pretraining, pre-pruning, and model fine-tuning, unnecessary convolutional channels are pruned by utilizing the statistical properties of spiking neurons and the statistical data of batch normalization layers. Sparsity is introduced during the model fine-tuning stage to optimize the model structure.
It achieves efficient model compression without adding redundant structures, reducing model size and computational cost, while improving the performance of the pruned model, enabling its deployment on resource-constrained neuromorphic hardware.
Smart Images
Figure CN116187418B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of computer vision and machine learning, and particularly relates to a spiking neural network compression method suitable for neuromorphic hardware. BACKGROUND
[0002] Spiking Neural Network (SNN) is a bio-inspired algorithm whose structure and function are inspired by biological neurons. Compared with artificial neural networks, SNN better simulates the operation mode of the biological nervous system by changing the information transmission mode of neurons from real numbers to pulse form. In SNN, information is transmitted between neurons through pulse signals. Whenever the cumulative input of a neuron exceeds a certain threshold, it will send out a pulse signal, which will be passed to the next neuron and affect its state. The input of SNN is usually analog continuous-time signals, which are converted into pulse signals and transmitted in the network, and the final output is also a pulse signal. The advantages of SNN include time-sensitive processing capability, fast processing speed, low power consumption, etc., which make it perform well in some specific tasks such as pattern recognition, control, etc. However, compared with traditional neural networks, there are still many challenges in the training method and model structure of SNN, which is one of the current research hotspots.
[0003] Due to its excellent spatio-temporal information expression ability and extremely low power consumption, SNN is highly expected, but due to the limitations of the learning algorithm of SNN and the number of neurons in neuromorphic chips, deploying large-scale and efficient SNN on embedded devices still faces major challenges.
[0004] In order to process complex image data, it is often necessary to design a multi-layer SNN, but multi-layer SNN often has extremely large model size and more computational load. The existence of these problems makes it very difficult or even impossible to deploy SNN on embedded devices or neuromorphic chips. SUMMARY
[0005] In order to solve the problems of the prior art, the present application provides a spiking neural network compression method suitable for neuromorphic hardware.
[0006] The technical solution of the present application is:
[0007] A spiking neural network compression method suitable for neuromorphic hardware, comprising the following steps:
[0008] Step 1, data preprocessing, using flip, rotation, random cropping and data standardization method to standardize and data enhance the data in the data set;
[0009] Step 2, constructing a pulse convolutional neural network, defining the network from input to output direction as spatial direction, and defining the network along different time step direction as time direction, then:
[0010] Along the spatial direction, the constructed pulse convolutional neural network sequentially includes an input encoding module, a residual module and a pulse output module; wherein,
[0011] The pulse encoding module comprises a first convolutional layer, a first batch normalization layer and a first pulse neuron layer along the spatial direction, specifically:
[0012] ①The input of the pulse encoding module is the image to be identified, which can be divided into RGB static data and pulse data according to the different types of image data. After the pulse encoding module processes the image to be identified, the output is pulse data.
[0013] ②The size of the convolution kernel of the first convolutional layer of the pulse encoding module is 3*3, and the input channel is consistent with the channel of the image to be identified. For the case of inputting RGB static data, the input channel is 3; for the case of inputting pulse data, the input channel is 2; the output channel of the first convolutional layer is 64.
[0014] ③The input of the first batch normalization layer of the pulse encoding module is the output of the first convolutional layer of the pulse encoding module, and the output is the batch normalized data of 64 channels.
[0015] ④The input of the first pulse neuron layer of the pulse encoding module is the output of the first batch normalization layer of the pulse encoding module, and the output is the pulse form output after the IF(Integrate and Fire) neuron operation.
[0016] The residual module comprises a first pulse residual block to a ninth pulse residual block along the spatial direction; specifically:
[0017] ①The pulse residual block contains two variable parameters of input channel number and output channel number, wherein the pulse residual block along the spatial direction comprises, in sequence: a first pulse convolutional layer in the block, a first batch normalization layer in the block, a first pulse neuron layer in the block, a first pulse discrimination layer in the block, a second pulse convolutional layer in the block, a second batch normalization layer in the block, a second pulse neuron layer in the block, a second pulse discrimination layer in the block and a residual accumulation layer.
[0018] ②The pulse residual block, the data input into the pulse residual block has two flow directions, the first data flow direction is that the input data sequentially passes through the first pulse convolution layer in the block, the first batch normalization layer in the block, the first pulse neuron layer in the block, the first pulse discriminator layer in the block, the second pulse convolution layer in the block, the second batch normalization layer in the block, the second pulse neuron layer in the block, the second pulse discriminator layer in the block, and reaches the residual accumulation layer. The second data flow direction is that if the input channel number of the pulse residual block is inconsistent with the output channel number, the input data needs to pass through the down-sampling pulse convolution layer and the down-sampling pulse neuron layer and then be connected to the residual accumulation layer; if the input channel number of the pulse residual block is consistent with the output channel number, the input data is directly connected to the residual accumulation layer. The output of the pulse residual block is the accumulation result of the output of the second pulse discriminator layer and the data output by the second data to the residual accumulation layer.
[0019] ③The input channel number of the first pulse convolution layer in the pulse residual block is the input channel number of the pulse residual block, the output channel number is the input channel number of the pulse residual block, and the convolution kernel size is 3*3. The input channel number and the output channel number of the second pulse convolution layer in the pulse residual block are equal to the output channel number of the pulse residual block, and the convolution kernel size is 3*3. The input channel number of the down-sampling pulse convolution layer of the pulse residual block is equal to the input channel number of the pulse residual block, the output channel number is equal to the output channel number of the pulse residual block, and the convolution kernel size is 1*1.
[0020] ④The input channel number and the output channel number of the residual module from the first pulse residual block to the ninth pulse residual block are (64, 64), (64, 64), (64, 64), (64, 128), (128, 128), (128, 128), (128, 256), (256, 256), and (256, 256) in sequence.
[0021] ⑤The input of the residual module is the pulse output of the pulse coding module, and the output is the pulse output of the ninth pulse residual block after the operation of the residual module.
[0022] The pulse output module sequentially comprises an output pooling layer, an output pulse neuron layer and an output linear layer along the spatial direction, and specifically comprises:
[0023] ①The output pooling layer is a two-dimensional average pooling layer, and the output size is (1, 1).
[0024] ②The output linear layer is a full connection layer, the input is 256, and the output is 10.
[0025] Step 3, pre-training the pulse neural network constructed in step 2, training a pulse neural network model with good performance using a back propagation algorithm based on gradient replacement, finally obtaining a pre-trained model that can achieve good recognition effect on a specific data set, and storing the pre-trained model. In this process, the pulse discrimination layer will record the pulse firing of each pulse neuron in the pulse discrimination layer before each iteration.
[0026] Step 4, pre-pruning stage. According to the set pruning threshold, calculate the number of pulse convolution channels of the convolution layer to be removed, and calculate the weight factor of each pulse convolution channel in each convolution layer according to the statistical data of the pulse discrimination layer and the pulse convolution channel scaling factor. According to the size of the channel weight factor value, mark which pulse convolution channel can be removed, and store the pre-pruned model.
[0027] Step 5, verify the pre-pruned pulse neural network model, record the impact of the removed channels in the pre-pruning on the performance of the pulse neural network model.
[0028] Step 6, pruning stage. Load the pre-trained model and channel mask matrix, for the pulse convolution channels marked as removable in the channel mask matrix, weight information and the connection of these pulse convolution channels with the remaining structure in the model, all of which will be removed from the pre-trained model and obtain the pruning model.
[0029] Step 7, adjust the loss function of the pulse neural network so that the loss calculation of the pulse neural network can introduce a certain sparsity.
[0030] Step 8, model fine-tuning stage. Fine-tune the pruning model to restore the accuracy of the pruning model and obtain a fine-tuned model.
[0031] Step 9, store the fine-tuned model, calculate its model performance, computational complexity, model parameters, and model volume relative to the pre-trained model.
[0032] Step 10, model deployment stage. Deploy the network structure and network parameters of the fine-tuned model to the neuromorphic hardware for inference.
[0033] Further, the pulse neural network construction stage of step 2 is characterized in that a pulse discrimination layer is inserted after each pulse neuron layer in the pulse neural network to be compressed.
[0034] The pulse neural network is a pulse neural network constructed using the ResNet structure, which includes 21 convolution layers, 21 pulse neuron layers and 19 batch normalization layers.
[0035] The input of each pulse neuron in the pulse neuron layer is the normalized pulse convolution channel output of the previous batch of normalization layers, and the output of each pulse neuron is the pulse output (specifically represented as pulse output and no pulse output) after IF neuron operation.
[0036] The pulse discrimination layer is mainly responsible for recording the pulse firing of the pulse neurons of the previous pulse discrimination layer and the channel scaling factor of the previous batch of normalization layers in the pre-training stage. In the pre-pruning stage, the pulse discrimination layer is mainly responsible for calculating the channel weight factor.
[0037] Further, the pulse neural network pre-training stage of step 3 is characterized in that the back propagation algorithm based on gradient substitution is used to train the pulse neural network and record the pulse firing of the pulse neurons during the training process.
[0038] The pulse neural network pre-training uses the training algorithm based on the back propagation algorithm based on gradient substitution and the SGD optimization algorithm for optimization.
[0039] The back propagation algorithm based on gradient substitution is different from the back propagation algorithm in artificial neural networks, which uses the gradient of the substitution function to replace the gradient of the pulse neuron, thereby realizing the continuity of the gradient and avoiding the gradient disappearance and gradient explosion caused by the discontinuity of the pulse.
[0040] The pulse firing of the pulse neuron is recorded, and at each pulse time step, the pulse neuron has two states: firing pulse and not firing pulse. The neuron that fires the pulse is recorded as 1, and the neuron that does not fire the pulse is recorded as 0.
[0041] The main purpose of the pulse neural network pre-training is to make the constructed pulse neural network have good classification performance for a specific data set through pre-training, which may need to be iterated multiple times (the specific number of iterations is related to the number of layers of the pre-training model and the complexity of the used data set).
[0042] Further, the pre-pruning stage of step 4 is characterized in that the pulse discrimination layer calculates the channel weight factor according to the pruning threshold and marks the pulse convolution channels to be removed.
[0043] The channel weight factor is a combination of the channel scaling factor of the previous batch of normalization layers and the neuron activity of the previous pulse neuron layer recorded in the pulse discrimination layer.
[0044] The meaning of the neuron activity of the pulse neuron layer is specifically: the ratio of the jth pulse neuron in the ith pulse neuron layer to the average pulse firing rate of the entire pulse neural network, and the mathematical expression is:
[0045]
[0046] wherein, is the neuron activity of the jth pulse neuron in the ith pulse neuron layer, Fr ij is the pulse firing rate statistical value of the jth pulse neuron in the ith pulse neuron layer, is the average pulse firing rate of the optimal pulse neural network model obtained after pre-training.
[0047] In the channel weight factor calculation process, the channel scaling factor and the neuron activity of the pulse neuron layer have been recorded and calculated in the pre-trained model. In the pre-pruning stage, the channel weight factor of each pulse convolution channel needs to be calculated using the channel scaling factor and the neuron activity of the pulse neuron layer, and the channel weight factor calculation method is:
[0048]
[0049] wherein, is the channel weight factor of the jth pulse neuron in the ith pulse neuron layer, pow is an exponential factor for adjusting the pruning sensitivity, and γ is the channel scaling factor.
[0050] After the channel weight factor is calculated, the channel weight factor needs to be sorted, and the channels to be retained are marked according to the set model compression rate to generate a channel mask matrix. For example, if the model compression rate is 0.3, 30% of the channels in the pre-trained model need to be removed, and at this time, the top 70% of the channels sorted from large to small in the channel weight factor will be marked as retained.
[0051] The channel mask matrix is a binary matrix of the same type as the pulse neuron layer, and its values are only 0 and 1. For the pulse convolution channels that need to be removed in the pruning stage, the corresponding positions in the channel mask matrix are marked as 0; for the pulse convolution channels that need to be retained in the pruning stage, the corresponding positions in the channel mask matrix are marked as 1.
[0052] Further, the pruning stage of step 6 is characterized in that the pre-trained model will remove the corresponding pulse convolution channels according to the channel mask matrix.
[0053] In the pruning stage, the pulse convolution channels marked as needing to be removed in the pre-trained model will be formally removed from the model in this stage. Accordingly, the inputs and outputs related to the removed pulse convolution channels will also be removed from the pre-trained model.
[0054] After the pruning stage, the model parameters, model structure and model operation of the pre-trained model will be greatly reduced, and a pruned model is obtained.
[0055] Further, the model fine-tuning stage of step 8 is characterized in that the loss function of model training is sparsified and trained to restore the accuracy of the pruned model.
[0056] The model fine-tuning stage adjusts the parameters obtained after the pre-training of the spiking neural network model, and the adjusted model is the pruned model. The adjustment target is to restore the model performance lost due to pruning through fine-tuning of the model parameters, and even exceed the performance of the pre-trained model.
[0057] The loss function used in the model fine-tuning stage is a sparsification loss function, which introduces sparsification to the batch normalization layer in the spiking neural network to improve the performance of the model, and its expression is:
[0058]
[0059] Where L is the loss function of the fine-tuned model, s is the sparsification factor introduced for the fine-tuning stage, L o is the loss function of the pre-trained model. g(γ) is the regularization function of the batch normalization layer, which can be L1 regularization or L2 regularization here.
[0060] Further, the model deployment stage of step 10 is characterized in that the volume and computational complexity of the compressed spiking neural network pre-trained model have been greatly reduced, but the performance has been slightly reduced or even improved, meeting the conditions for direct deployment to neuromorphic hardware.
[0061] The neuromorphic hardware is generally a special chip and its peripheral circuit designed specifically for running the spiking neural network. Limited by the operation mechanism of neurons, this type of hardware generally has a limited number of spiking neurons. The pre-trained model has too much model volume and neuron quantity for this type of hardware, while the fine-tuned model can be easily deployed to this type of hardware.
[0062] The beneficial effects of the present application: the pulse neural network compression method suitable for neuromorphic hardware of the present application realizes efficient model compression effect on the basis of not increasing too much redundant structure, and improves the performance of the pruned model in the model fine-tuning stage by introducing sparsity, so as to realize the deployment of deep pulse neural network on resource-limited neuromorphic hardware. Compared with the traditional neural network pruning technology, the pulse neural network compression method suitable for neuromorphic hardware can effectively reduce the model size of the pulse neural network, reduce the model operation amount, and ensure that the performance loss of the pulse neural network after pruning is within an acceptable range, and even at a certain compression rate, the model performance is improved. BRIEF DESCRIPTION OF DRAWINGS
[0063] Figure 1 is a ResNet structure pulse neural network structure diagram constructed by using the pulse neural network compression method suitable for neuromorphic hardware;
[0064] Figure 2 is a structure diagram of the pulse coding module of the pulse neural network suitable for neuromorphic hardware;
[0065] Figure 3 is a structure diagram of the residual module of the pulse neural network suitable for neuromorphic hardware;
[0066] Figure 4 is a structure diagram of the pulse residual block in the residual module of the pulse neural network suitable for neuromorphic hardware;
[0067] Figure 5 is a structure diagram of the pulse output module of the pulse neural network suitable for neuromorphic hardware;
[0068] Figure 6 is a network compression operation principle diagram of the pulse neural network compression method suitable for neuromorphic hardware;
[0069] Figure 7 is a work flow diagram of the pulse neural network compression method suitable for neuromorphic hardware;
[0070] Figure 8 is an effect comparison diagram of the pulse neural network compression method suitable for neuromorphic hardware in the model compression of the CIFAR-10, DVS-Gesture, and DVS-CIFAR10 data sets. DETAILED DESCRIPTION
[0071] The technical solutions of the present application will be described in detail below in combination with the drawings:
[0072] Pulse neural network is expected to be widely used due to its excellent spatio-temporal information expression ability and extremely low power consumption. However, due to the learning algorithm of the pulse neural network and the number of neurons in the neuromorphic chip, it still faces great challenges to deploy a large-scale and efficient pulse neural network on an embedded device. In order to process complex image data, a multi-layer pulse neural network is often designed, but the multi-layer pulse neural network often has an extremely large model size and a large amount of calculation. The existence of these problems makes it very difficult or even impossible to deploy a pulse neural network on an embedded device or a neuromorphic chip. In order to solve the problems of too large model size and too much calculation in the pulse neural network, the statistical characteristics of the pulse neurons in the pulse neural network and the statistical data in the batch normalization layer of the pulse neural network are comprehensively utilized, without adding too much redundant structure, efficient model compression effect is realized, and the performance of the pruned model is improved by introducing sparsity in the model fine-tuning stage.
[0073] To achieve the above purpose, the present application provides a pulse neural network compression method suitable for neuromorphic hardware, which includes the following implementation stages:
[0074] In the pulse neural network construction stage, IF neurons are used to construct pulse neural networks suitable for different data sets according to the type of input image. In this process, each IF pulse neuron layer is directly connected to a pulse discriminator layer that records the pulse activity statistics of the pulse neuron layer.
[0075] In the pulse neural network pre-training stage, a pulse neural network model with good performance is trained using a gradient replacement-based backpropagation algorithm. This model is called a pre-trained model. In this process, the pulse discriminator layer records the pulse firing frequency of each pulse neuron in the pulse neuron layer before the pulse discriminator layer in each iteration.
[0076] In the pre-pruning stage, according to the set pruning threshold, the number of pulse convolution channels of the convolution layer to be removed in the pre-trained model is calculated, and the weight factor of each pulse convolution channel in each convolution layer is calculated according to the statistical data of the pulse discriminator layer and the channel scaling factor of each batch normalization layer. According to the weight factor value, mark which pulse convolution channels can be removed.
[0077] In the pruning stage, the pulse convolution channels marked as removable will be formally removed from the pre-trained model, and the weights and connections related to the removed pulse convolution channels will also be removed from the pre-trained model, obtaining a pruned model.
[0078] In the model fine-tuning stage, the pruned model obtained after pruning will be properly fine-tuned to restore the accuracy of the pruned model. In this stage, the loss calculation of the model will be properly adjusted to effectively restore the impact of the pruning operation on the performance of the model.
[0079] In the model deployment stage, the fine-tuned model can be directly deployed on the neuromorphic hardware for inference.
[0080] For the purposes of the present invention, the technical solutions and advantages will be more clearly understood, and the following will be further described in detail by specific embodiments in conjunction with the accompanying drawings, it should be understood that the specific embodiments described herein are only used to explain the present invention, and not to limit the present invention.
[0081] As shown in Figure 7 , the following implementation steps are included:
[0082] Step 1, data preprocessing, using flip, rotation, random cropping and data standardization methods to standardize and data enhance the data in the data set;
[0083] Step 2, construct a pulse convolutional neural network, define the network from input to output direction as spatial direction, and define the network along different time step direction as time direction, then:
[0084] As shown in Figure 1 , along the spatial direction, the constructed pulse convolutional neural network sequentially includes an input encoding module, a residual module and a pulse output module;
[0085] As shown in Figure 2 , the pulse encoding module includes a first convolutional layer, a first batch normalization layer and a first pulse neuron layer along the spatial direction, specifically:
[0086] ①The input of the pulse encoding module is the image to be identified, which can be divided into RGB static data and pulse data according to the different types of image data. After the pulse encoding module processes the image to be identified, the output is pulse data.
[0087] ②The size of the convolution kernel of the first convolutional layer of the pulse encoding module is 3*3, and the input channel is consistent with the channel of the image to be identified. For the case of inputting RGB static data, the input channel is 3; for the case of inputting pulse data, the input channel is 2; the output channel of the first convolutional layer is 64.
[0088] ③The input of the first batch normalization layer of the pulse encoding module is the output of the first convolutional layer of the pulse encoding module, and the output is 64-channel batch normalized data.
[0089] ④ The input of the first pulse neuron layer of the pulse coding module is the output of the first batch of normalized layers of the pulse coding module, and the output is the pulse form output after the IF (Integrate and Fire) neuron operation.
[0090] like Figure 3 As shown, the residual module, along the spatial direction, consists of the first pulse residual block to the ninth pulse residual block; specifically:
[0091] ①The pulse residual block includes two variable parameters: the number of input channels and the number of output channels. The pulse residual block consists of the following layers in the spatial direction: the first pulse convolutional layer in the block, the first batch of normalization layers in the block, the first pulse neuron layer in the block, the first pulse discrimination layer in the block, the second pulse convolutional layer in the block, the second batch of normalization layers in the block, the second pulse neuron layer in the block, the second pulse discrimination layer in the block, and the residual accumulation layer.
[0092] ②For example Figure 4 As shown, the pulse residual block has two data flow directions. The first data flow direction involves the input data sequentially passing through the first pulse convolutional layer, the first batch of normalization layers, the first pulse neuron layer, the first pulse discrimination layer, the second pulse convolutional layer, the second batch of normalization layers, the second pulse neuron layer, and the second pulse discrimination layer before reaching the residual accumulation layer. The second data flow direction is as follows: if the number of input channels and the number of output channels in the pulse residual block are inconsistent, the input data needs to pass through a downsampled pulse convolutional layer and a downsampled pulse neuron layer before connecting to the residual accumulation layer; if the number of input channels and the number of output channels in the pulse residual block are consistent, the input data is directly connected to the residual accumulation layer. The output of the pulse residual block is the sum of the output of the second pulse discrimination layer and the data output from the second data path to the residual accumulation layer.
[0093] ③ The first pulse convolutional layer within the pulse residual block has the same number of input channels and output channels as the pulse residual block, with a kernel size of 3*3. The second pulse convolutional layer within the pulse residual block has the same number of input and output channels as the pulse residual block, with a kernel size of 3*3. The downsampling pulse convolutional layer of the pulse residual block has the same number of input and output channels as the pulse residual block, with a kernel size of 1*1.
[0094] ④ The number of input channels and output channels of the residual module from the first pulse residual block to the ninth pulse residual block are as follows: (64, 64), (64, 64), (64, 64), (64, 128), (128, 128), (128, 128), (128, 128), (128, 256), (256, 256), (256, 256).
[0095] V. the residual module inputs the pulse output of the pulse coding module and outputs the ninth pulse residual block pulse output after operation of the residual module.
[0096] As shown in Figure 5 , the pulse output module sequentially comprises an output pooling layer, an output pulse neuron layer and an output linear layer along the spatial direction, and specifically comprises:
[0097] I. the output pooling layer is a two-dimensional average pooling layer, and the output size is (1, 1).
[0098] II. the output linear layer is a fully connected layer, and the input is 256 and the output is 10.
[0099] Step 3: pre-training the pulse neural network constructed in step 2, training a pulse neural network model with good performance based on the back propagation algorithm based on gradient replacement, finally obtaining a pre-training model which can achieve good recognition effect on specific data set, and storing the pre-training model. In this process, the pulse discrimination layer will record the pulse firing of each pulse neuron in the pulse neuron layer before the pulse discrimination layer in each iteration.
[0100] Step 4: pre-pruning stage. According to the set pruning threshold, calculate the number of pulse convolution channels of the convolution layer to be removed, and calculate the weight factor of each pulse convolution channel in each convolution layer according to the statistical data of the pulse discrimination layer and the scaling factor of each pulse convolution channel. According to the size of the channel weight factor value, mark which pulse convolution channels can be removed, and store the pre-pruned model.
[0101] As shown in Figure 6 , the running principle of the pulse discrimination layer in the pre-training stage, the pre-pruning stage and the pruning stage is shown. In the figure, the kth convolution layer is taken as an example, wherein the kth convolution layer has n pulse convolution channels (C k1 to C kn ), corresponding to n channel scaling factors, and the k+1th convolution layer has 2 pulse convolution channels (C m1 to C m2). In the pre-training stage, the kth batch normalization layer after the kth convolutional layer calculates the channel scaling factor of the kth convolutional layer, the kth spiking neuron layer after the kth batch normalization layer receives the input of the kth batch normalization layer and generates spikes according to the IF neuron model, and at the same time, the spike discrimination layer responsible for monitoring the kth spiking neuron layer records the spike firing data of the kth spiking neuron layer and the channel scaling factor of the kth batch normalization layer. In the pre-pruning stage, the spike discrimination layer calculates the neuron activity of each pulse convolution channel according to the spike statistics and channel scaling factor data recorded in the pre-training stage. According to the set pruning threshold, the spike discrimination layer sorts the neuron activity of all pulse convolution channels, and marks the pulse convolution channels below the threshold as removable channels. In the pruning stage, the neuron connections and weights of the pulse convolution channels marked as removable are removed from the pre-trained model to obtain a pruned model. As shown in Figure 6 After pruning, the kth convolutional layer has only 2 pulse convolution channels, and the connections and neuron weights of the related neurons are removed from the pre-trained model.
[0102] Step 5, verify the pulse neural network model after pre-pruning, and record the impact of the removed channels in pre-pruning on the performance of the pulse neural network model.
[0103] Step 6, pruning stage. Load the pre-trained model and the channel mask matrix. For the pulse convolution channels marked as removable in the channel mask matrix, weight information and the connections of these pulse convolution channels with the remaining structures in the model will be removed from the pre-trained model to obtain a pruned model.
[0104] Step 7, adjust the loss function of the pulse neural network so that the loss calculation of the pulse neural network can introduce a certain sparsity.
[0105] Step 8, model fine-tuning stage. Fine-tune the pruned model to restore the accuracy of the pruned model and obtain a fine-tuned model.
[0106] Step 9, store the fine-tuned model and calculate the differences in model performance, computational complexity, model parameters, and model volume relative to the pre-trained model.
[0107] Step 10, model deployment stage. Deploy the network structure and network parameters of the fine-tuned model to neuromorphic hardware for inference.
[0108] As shown in Figure 8As shown, the best parameter compression ratio and calculation amount compression ratio achieved on different data sets in the experiment of verifying the spiking neural network compression method suitable for neuromorphic hardware proposed in the application on the CIFAR-10, DVS-Gesture and DVS-CIFAR10 data sets are shown.The experiment proves that the spiking neural network compression method suitable for neuromorphic hardware proposed in the application can achieve good model compression effect on dynamic data sets and static data sets, and on the premise of greatly reducing the model size and model operation amount, the performance of the model is also improved.
Claims
1. A method for spiking neural network compression suitable for neuromorphic hardware, the method comprising: The method comprises the following steps: Step 1, data preprocessing, using flip, rotation, random cropping and data standardization method to standardize and data enhance the data in the data set; Step 2, constructing a pulse convolutional neural network, defining the network from input to output direction as spatial direction, and defining the network along different time step direction as time direction, then: Along the spatial direction, the constructed pulse convolutional neural network comprises a pulse coding module, a residual module and a pulse output module in sequence; wherein, The pulse coding module comprises a first convolutional layer, a first batch normalization layer and a first pulse neuron layer in sequence along the spatial direction; The input of the pulse coding module is an image to be recognized, and the image to be recognized is divided into RGB static data and pulse data according to the different types of image data; the image to be recognized is output as pulse data after being processed by the pulse coding module; The convolution kernel size of the first convolutional layer of the pulse coding module is 3*3, the input channel is consistent with the channel of the image to be recognized; for the case of inputting RGB static data, the input channel is 3; for the case of inputting pulse data, the input channel is 2; the output channel of the first convolutional layer is 64; The input of the first batch normalization layer of the pulse coding module is the output of the first convolutional layer of the pulse coding module, and the output is 64-channel batch normalized data; The input of the first pulse neuron layer of the pulse coding module is the output of the first batch normalization layer of the pulse coding module, and the output is the pulse form output after IF neuron operation; The residual module comprises a first pulse residual block to a ninth pulse residual block in sequence along the spatial direction; The pulse residual block comprises two variable parameters of input channel number and output channel number, wherein the pulse residual block comprises, in sequence along the spatial direction, a first pulse convolutional layer in the block, a first batch normalization layer in the block, a first pulse neuron layer in the block, a first pulse discrimination layer in the block, a second pulse convolutional layer in the block, a second batch normalization layer in the block, a second pulse neuron layer in the block, a second pulse discrimination layer in the block and a residual accumulation layer; The input data of the pulse residual block has two flow directions, the first data flow direction is that the input data sequentially passes through the first pulse convolutional layer in the block, the first batch normalization layer in the block, the first pulse neuron layer in the block, the first pulse discrimination layer in the block, the second pulse convolutional layer in the block, the second batch normalization layer in the block, the second pulse neuron layer in the block, the second pulse discrimination layer in the block and reaches the residual accumulation layer; the second data flow direction is that if the input channel number and the output channel number of the pulse residual block are inconsistent, the input data needs to pass through a down-sampling pulse convolutional layer and a down-sampling pulse neuron layer and then be connected to the residual accumulation layer; if the input channel number and the output channel number of the pulse residual block are consistent, the input data is directly connected to the residual accumulation layer; the output of the pulse residual block is the accumulation result of the output of the second pulse discrimination layer and the data output from the second data to the residual accumulation layer. The input channel of the first pulse convolution layer in the block is the input channel number of the pulse residual block, the output channel number is the input channel number of the pulse residual block, and the convolution kernel size is 3*3; the input channel and the output channel of the second pulse convolution layer in the block are equal to the output channel number of the pulse residual block, and the convolution kernel size is 3*3; the input channel of the down-sampling pulse convolution layer of the pulse residual block is equal to the input channel number of the pulse residual block, the output channel is equal to the output channel number of the pulse residual block, and the convolution kernel size is 1*1; The input channel and the output channel of the residual module from the first pulse residual block to the ninth pulse residual block are (64, 64), (64, 64), (64, 64), (64, 128), (128, 128), (128, 128), (128, 256), (256, 256), and (256, 256) in sequence; The input of the residual module is the pulse output of the pulse coding module, and the output is the ninth pulse residual block pulse output after the operation of the residual module; The pulse output module along the spatial direction is sequentially an output pooling layer, an output pulse neuron layer, and an output linear layer; The output pooling layer is a two-dimensional average pooling layer, and the output size is (1, 1); The output linear layer is a fully connected layer, the input is 256, and the output is 10; Step 3, pre-training the pulse neural network constructed in step 2, specifically, using a back propagation algorithm based on gradient replacement for training to obtain a pre-training model, and storing the pre-training model; during the pre-training process, the pulse discrimination layer records the pulse firing of each pulse neuron in the pulse neuron layer before the pulse discrimination layer in each iteration; Step 4, pre-pruning stage, according to the set pruning threshold, calculate the number of pulse convolution channels of the convolution layer to be removed, and calculate the weight factor of each pulse convolution channel in each convolution layer according to the statistical data of the pulse discrimination layer and the scaling factor of each pulse convolution channel, mark which pulse convolution channels can be removed according to the channel weight factor value, and store the pre-pruned model; Step 5, verify the pre-pruned pulse neural network model, and record the influence of the removed channels in the pre-pruning on the performance of the pulse neural network model; Step 6, pruning stage, load the pre-training model and the channel mask matrix, remove all the pulse convolution channels, weight information and the connection of these pulse convolution channels with the remaining structures in the model marked as removable in the channel mask matrix from the pre-training model to obtain a pruning model; Step 7, adjust the loss function of the pulse neural network so that the loss calculation of the pulse neural network introduces sparsity; Step 8, model fine-tuning stage, properly fine-tune the pruning model to restore the accuracy of the pruning model to obtain a fine-tuned model; Step 9, store the fine-tuned model, calculate the differences in model performance, computational complexity, model parameters, and model volume relative to the pre-training model; Step 10, model deployment stage, deploy the network structure and network parameters of the fine-tuned model to neuromorphic hardware for inference.
2. The method of claim 1, wherein, In step 2, the pulse neural network is constructed using a ResNet structure, which includes a total of 21 convolutional layers, 21 pulse neuron layers, and 19 batch normalization layers; the input of each pulse neuron is the normalized pulse convolution channel output from the previous batch normalization layer, and the output of each pulse neuron is the pulse output after IF neuron operation; A pulse discrimination layer is inserted after each pulse neuron layer in the pulse neural network to be compressed. The pulse discrimination layer is responsible for recording the pulse firing of the pulse neuron in the previous pulse discrimination layer and the channel scaling factor of the previous batch normalization layer during the pre-training stage. During the pre-pruning stage, the pulse discrimination layer is mainly responsible for calculating the channel weight factor.
3. The method of claim 1, wherein, In step 4, the pulse discrimination layer calculates the channel weight factor according to the pruning threshold and marks the pulse convolution channels to be removed. Specifically: The channel weight factor is a combination of the channel scaling factor of the previous batch normalization layer and the neuron activity of the previous pulse neuron layer recorded in the pulse discrimination layer; The meaning of neuronal activity in the spiking neuron layer is specifically as follows: In the th spiking neuron layer The ratio of the firing rate of an individual spiking neuron to the average firing rate of the entire spiking neural network is expressed mathematically as follows: ， wherein, is the neuron activity of the i-th spiking neuron in the j-th spiking neuron layer, is the spike rate statistic of the i-th spiking neuron in the j-th spiking neuron layer, is the average spike rate of the optimal spiking neural network model obtained after pre-training. During the channel weight factor calculation process, the channel scaling factor and the neuron activity of the pulse neuron layer have been recorded and calculated in the pre-trained model. During the pre-pruning stage, the channel weight factor of each pulse convolution channel needs to be calculated using the channel scaling factor and the neuron activity of the pulse neuron layer. The channel weight factor calculation method is: ， wherein, is the channel weight factor for the i-th pulse neuron in the j-th pulse neuron layer, is the channel weight factor for the i-th pulse neuron in the j-th pulse neuron layer, is the channel weight factor for the i-th pulse neuron in the j-th pulse neuron layer, is an exponential factor for adjusting pruning sensitivity, is a channel scaling factor; After the channel weight factor is calculated, the channel weight factor needs to be sorted, and according to the set model compression rate, the channels to be retained are marked to generate a channel mask matrix. The channel mask matrix is a binary matrix of the same type as the pulse neuron layer, with only 0 and 1 values. For pulse convolution channels that need to be removed during the pruning stage, the corresponding position in the channel mask matrix is marked as 0. For pulse convolution channels that need to be retained during the pruning stage, the corresponding position in the channel mask matrix is marked as 1.
4. The method of claim 1, wherein, In step 8, the model fine-tuning stage, the adjusted model parameters are the parameters obtained after pre-training of the pulse neural network model, the adjusted model is the pruned model, and the adjustment target is to recover the model performance lost due to pruning and even exceed the performance of the pre-trained model by fine-tuning the model parameters; The loss function used in the model fine-tuning stage is a sparsification loss function, which introduces sparsification to the batch normalization layer in the pulse neural network to improve the performance of the model, and its expression is: ， wherein, is the loss function of the fine-tuned model, is a sparsification factor introduced for the fine-tuning phase, is the loss function of the pre-trained model, is a regularization function of the batch normalization layer, which here can be regularization or regularization.