An image classification method based on spiking residual neural network with gate mechanism
By introducing a pulse gate residual module with a binary selection gate mechanism into the pulse neural network, the gradient vanishing problem is solved and high-precision image classification is achieved. It is suitable for scenarios such as resource-constrained drones and unmanned vehicles.
Patent Information
- Application Number
- CN202310245207.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-15
- Publication Date
- 2025-09-26
- Estimated Expiration
- 2043-03-15
AI Technical Summary
Existing spiking neural network models suffer from the vanishing gradient problem during training, resulting in a small number of model layers, low capacity, and inability to achieve high-precision image processing.
A pulse gate residual module with a binary selection gate mechanism is introduced to ensure the normal back propagation of the gradient under the limitation of the pulse signal, thereby solving the gradient vanishing problem. Two types of pulse gate residual modules, basic and bottleneck, are adopted, and different module types are selected according to the network depth.
It significantly reduces energy consumption, improves image classification accuracy and computing speed, and is suitable for resource-constrained scenarios such as drones and unmanned vehicles.
Smart Images
Figure CN116258907B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of neural networks and image classification, and in particular relates to an image classification method based on a pulse residual neural network of a gate mechanism. Background Art
[0002] In recent years, spiking neural networks (SNs), a biomimetic neural network model inspired by biological neurons, have gained popularity. They use pulse signals and can run on brain-inspired computing chips with extremely low power consumption. Their high computational efficiency and low power consumption make SNs ideal for computing-constrained scenarios, such as drones, self-driving cars, and mobile robots.
[0003] The training method for spiking neural networks is similar to that of artificial neural networks, but during training, a surrogate gradient function is used to approximate the gradient of spiking neurons. Existing spiking neural network model structures used for image classification suffer from the vanishing gradient problem during training. The model has few layers and low capacity, making it impossible to achieve high-precision image processing. On the one hand, the mathematical model of spiking neurons is relatively complex, differing from the response pattern of artificial neurons. On the other hand, the spike signal is non-differentiable, and the surrogate gradient contains errors. Therefore, designing a deep spiking neural network model that can overcome the vanishing gradient problem is valuable and necessary. Previous work has explored the vanishing gradient problem from the perspective of residual learning. For example, Spiking ResNet directly replaces the artificial neurons in the ResNet structure with spiking neurons, and SEW ResNet designs an element-wise function to pulse the residual module. However, these works have not successfully designed a spiking neural network model that overcomes the vanishing gradient problem. Summary of the Invention
[0004] In response to the shortcomings of the prior art, the present invention provides an image classification method based on a pulse residual neural network with a gate mechanism. The main computing unit of the pulse residual neural network proposed in the present invention is a pulse gate residual module. The pulse gate residual module introduces a binary selection gate to select the residual signal, ensuring the normal back propagation of the gradient under the limitation of the pulse signal, thereby solving the gradient vanishing problem. Compared with artificial neural networks of the same depth, the present invention significantly reduces energy consumption, and compared with other directly trained pulse neural networks, the accuracy in image classification tasks is significantly improved.
[0005] The present invention is achieved through the following technical solutions: an image classification method based on a pulse residual neural network with a gate mechanism, wherein the method inputs an RGB image into the pulse residual neural network to obtain an image category;
[0006] The pulse residual neural network includes an image encoder, a pulse main network and a feature decoder;
[0007] The image encoder is used to convert the RGB image into a spike image, including a convolutional neural network layer, a batch normalization layer, a spike neuron layer and a maximum pooling layer;
[0008] The pulse main network is a main calculation unit for extracting high-dimensional features of pulse images. It is composed of four calculation stages connected in series, and each calculation stage is composed of a number of pulse gate residual modules stacked together. The pulse gate residual module includes two types: a basic pulse gate residual module Spiking Gate Basic Block and a bottleneck pulse gate residual module Spiking Gate Bottleneck Block. Both the basic pulse gate residual module and the bottleneck pulse gate residual module include a convolutional neural network layer, a batch normalization layer, a pulse neuron layer and a binary selection gate. The type of pulse gate residual module used is determined according to the network depth of the pulse main network. The first pulse gate residual module of the second, third and fourth calculation stages is provided with a downsampling layer to downsample the image features by two times.
[0009] The feature decoder is used to decode the high-dimensional features of the pulse image into the corresponding category, and includes an average pooling layer and a fully connected neural network layer.
[0010] Furthermore, the image encoder encodes the input RGB image to obtain a pulse image, specifically:
[0011] (1) The convolutional neural network layer and batch normalization layer perform preliminary transformation and normalization on the input image, and the output analog value image data dimension is: c×h×w, where c is the number of channels, h and w are the height and width of the image respectively;
[0012] (2) First, the analog value image data is copied along the time dimension, and then the analog value image is encoded into a pulse image using a pulse neuron layer. The pulse image dimension is: T×c×h×w, where T is the time step of the pulse residual neural network and the elements of the pulse image are binary pulse values;
[0013] (3) Use the maximum pooling layer to downsample the pulse image by two times, and the output pulse image dimension is:
[0014] Furthermore, the type of pulse gate residual module used is determined according to the network depth of the pulse main body network, specifically: a network depth threshold is set, and the network below the threshold adopts a basic pulse gate residual module, otherwise the bottleneck pulse gate residual module is adopted, thereby improving the image classification accuracy at each depth level.
[0015] Furthermore, the pulse main network is divided into five configurations according to the network depth: 18, 34, 50, 101, and 152. The number of module stacking in these five configurations is the same as that of ResNet; the 18 and 34 configurations use the basic pulse gate residual module; the 50, 101, and 152 configurations use the bottleneck pulse gate residual module.
[0016] Furthermore, the pulse gate residual module includes two branches, left and right, which respectively perform operations on the input X and then input the binary selection gate BSG. The two types of pulse gate residual modules only differ in the left branch, specifically:
[0017] (1) The left branch performs the transformation operation of the image features and outputs the transformed features X left The left branch contains almost all the parameters of the module. The convolutional neural network layer Conv, the batch normalization layer BN, and the pulse neuron layer SN are connected in series as submodules. The basic pulse gate residual module contains two series submodules, and the bottleneck pulse gate residual module contains three series submodules. The step size of the convolutional neural network layer Conv is 1.
[0018] (2) The right branch is the residual branch of the module, which outputs the identity mapping of X when there is no downsampling layer, that is, X right =X, when containing a downsampling layer, it is composed of a convolutional neural network layer Conv, a batch normalization layer BN, and a spike neuron layer SN in series, and the step size of the convolutional neural network layer Conv is 2;
[0019] (3) The binary select gate BSG selects the output Y from the inputs of the left and right branches according to the gate signal GATE, as follows:
[0020] Y=GATE·X left +(1-GATE)·X right
[0021] The gate signal GATE has two forms of expression, which can be flexibly selected according to the limitation of the computational load, as follows:
[0022] Form 1: GATE = Θ(W·X left +B-thr)
[0023] Form 2: GATE = X left
[0024] Where W and B are the weight and bias of the linear transformation; thr is the cutoff weight, which is set to a small value greater than 0; Θ is the Heaviside step function, which is used to binarize the gate signal;
[0025] Form 1 includes linear transformation and is usually used in situations where the amount of calculation is large, and can improve the accuracy of image classification. Form 2 does not include linear transformation and is usually used in situations where the amount of calculation is small, and can improve the calculation speed.
[0026] Furthermore, the feature decoder decodes the high-dimensional features of the image into categories and corresponding probabilities, specifically:
[0027] (1) Assume that the input high-dimensional feature dimension is: T×c f ×h f ×w f , where c f 、h f 、w f are the number of channels, height, and width of the feature image, respectively. The average pooling layer is used to compress the image height and width dimensions to obtain T×c f ;
[0028] (2) Use the fully connected neural network layer to perform linear transformation on the high-dimensional features so that they correspond to the image category dimension. The output dimension is: T×n class , where n class is the total number of categories;
[0029] (3) Average the time dimension and then use the softmax function to convert the output vector into the probability corresponding to all categories.
[0030] Furthermore, the image encoder and pulse main network structure in the pulse residual neural network can be used as a general feature extractor for feature extraction of other computer vision tasks.
[0031] Compared with the existing technology, the present invention has the following advantages: it is a directly trained pulse residual neural network with very low floating-point operations and energy consumption, and can run on brain-like computing chips; it proposes a binary selection gate mechanism, which is introduced into the pulse gate residual module to ensure the normal back propagation of the gradient under the limitation of the pulse signal, thereby solving the gradient vanishing problem and ensuring the normal training of the deep pulse residual neural network; it proposes two gate signal expression forms and multiple pulse residual neural network configurations, so that different application environments can flexibly choose different network configurations. The present invention can significantly improve the accuracy of image classification and increase the calculation speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A schematic diagram of the pulse residual neural network structure provided by an embodiment of the present invention;
[0033] Figure 2 A schematic diagram of the Spiking Gate Basic Block network structure provided in an embodiment of the present invention;
[0034] Figure 3 Schematic diagram of the Spiking Gate Basic Block network structure with a downsampling layer provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0037] This paper proposes an image classification method based on a pulse residual neural network with a gate mechanism. Figure 1 This is a schematic diagram of the structure of the pulse residual neural network, which is suitable for image classification tasks, especially in resource-constrained application scenarios such as drones, unmanned vehicles, and mobile robots that require image recognition and classification.
[0038] The pulse residual neural network includes an image encoder, a pulse main network and a feature decoder;
[0039] The image encoder is used to convert the RGB image into a spike image, including a convolutional neural network layer, a batch normalization layer, a spike neuron layer and a maximum pooling layer;
[0040] The pulse main network is a main calculation unit for extracting high-dimensional features of pulse images. It is composed of four calculation stages connected in series, and each calculation stage is composed of a number of pulse gate residual modules stacked together. The pulse gate residual module includes two types: a basic pulse gate residual module Spiking Gate Basic Block and a bottleneck pulse gate residual module Spiking Gate Bottleneck Block. Both the basic pulse gate residual module and the bottleneck pulse gate residual module include a convolutional neural network layer, a batch normalization layer, a pulse neuron layer and a binary selection gate. The type of pulse gate residual module used is determined according to the network depth of the pulse main network. The first pulse gate residual module of the second, third and fourth calculation stages is provided with a downsampling layer to downsample the image features by two times.
[0041] The feature decoder is used to decode the high-dimensional features of the pulse image into the corresponding category, and includes an average pooling layer and a fully connected neural network layer.
[0042] like Figure 1As shown in the figure, the RGB image is input into the image encoder to obtain the corresponding pulse image, which is further input into the pulse main network to obtain the high-dimensional features of the pulse image, and finally input into the feature decoder to obtain the object category of the image.
[0043] Furthermore, the encoding process of the image encoder for the RGB image is specifically as follows:
[0044] (1) The convolutional neural network layer and batch normalization layer perform preliminary transformation and normalization on the input image, and the output analog value image data dimension is: c×h×w, where c is the number of channels, h and w are the height and width of the image respectively;
[0045] (2) First, the analog value image data is copied along the time dimension, and then the analog value image is encoded into a pulse image using a pulse neuron layer. The pulse image dimension is: T×c×h×w, where T is the time step of the pulse residual neural network and the elements of the pulse image are binary pulse values;
[0046] (3) Use the maximum pooling layer to downsample the pulse image by two times, and the output pulse image dimension is:
[0047] Furthermore, the impulse main body network is the main computing unit, which is implemented as follows:
[0048] (1) The main pulse network is composed of four calculation stages in series, and each calculation stage is composed of a number of pulse gate residual modules stacked together. The pulse gate residual modules include two types: the basic pulse gate residual module Spiking Gate BasicBlock and the bottleneck pulse gate residual module Spiking Gate Bottleneck Block. In this embodiment, the SpikingGate Basic Block includes two convolutional neural network layers, two batch normalization layers, two pulse neuron layers and a binary selection gate. The specific network structure is as follows: Figure 2 As shown in Figure 2, the Spiking Gate Bottleneck Block contains three convolutional neural network layers, three batch normalization layers, three spiking neuron layers, and a binary selection gate.
[0049] (2) The pulse main network contains multiple configurations. In this embodiment, it is divided into five configurations according to the network depth: 18, 34, 50, 101, and 152. The number of module stacks in these five configurations is the same as that of ResNet, as shown in Table 1.
[0050] Table 1
[0051] Network Depth 18 34 50 101 152 Computation phase 1 2 3 3 3 3 Computation Phase 2 2 4 4 4 8 Computation phase 3 2 6 6 23 36 Calculation phase 4 2 3 2 3 3
[0052] (3) The pulse gate residual module type used in the 18 and 34 configurations is the Spiking Gate Basic Block.
[0053] (4) The pulse gate residual module type used in the three configurations of 50, 101, and 152 is the Spiking Gate Bottleneck Block.
[0054] (5) In all configurations, the first pulse gate residual module in the 2nd, 3rd, and 4th computation stages contains a downsampling layer that downsamples the image features by a factor of two.
[0055] Furthermore, the pulse gate residual module consists of two branches, left and right, which respectively perform operations on the input X and then input the binary selection gate BSG. The two types of pulse gate residual modules only differ in the left branch, specifically:
[0056] (1) The left branch performs the transformation operation of the image features and outputs the transformed features X left The left branch contains almost all the parameters of the module. The convolutional neural network layer Conv, the batch normalization layer BN, and the spiking neuron layer SN are connected in series as submodules. The Spiking Gate Basic Block contains two connected submodules, and the Spiking Gate Bottleneck Block contains three connected submodules. The step size of the convolutional neural network layer Conv is 1.
[0057] (2) The right branch is the residual branch of the module, which outputs the identity mapping of X when there is no downsampling layer, that is, X right =X, when containing a downsampling layer, it is composed of a convolutional neural network layer Conv, a batch normalization layer BN, and a spike neuron layer SN in series. The step size of the convolutional neural network layer Conv is 2. The specific network structure of the Spiking Gate Basic Block containing a downsampling layer is as follows: Figure 3 shown.
[0058] (3) The binary select gate BSG selects the output Y from the inputs of the left and right branches according to the gate signal GATE, as follows:
[0059] Y=GATE·X left +(1-GATE)·X right
[0060] The gate signal GATE has two forms of expression, which can be flexibly selected according to the limitation of the calculation amount, specifically:
[0061] Form 1: GATE = Θ(W·X left +B-thr)
[0062] Form 2: GATE = X left
[0063] Where W and B are the weight and bias of the linear transformation; thr is the cutoff weight, which is set to a small value greater than 0; Θ is the Heaviside step function, which is used to binarize the gate signal;
[0064] Form 1 includes linear transformation and is usually used in situations where the amount of calculation is large, and can improve the accuracy of image classification. Form 2 does not include linear transformation and is usually used in situations where the amount of calculation is small, and can improve the calculation speed.
[0065] Furthermore, the feature decoder decodes the high-dimensional features of the image into categories and corresponding probabilities, specifically:
[0066] (1) Assume that the input high-dimensional feature dimension is: T×c f ×h f ×w f , where c f 、h f 、w f are the channel, height, and width of the feature image, respectively. The average pooling layer is used to compress the image height and width dimensions to obtain T×c f .
[0067] (2) Use the fully connected layer to perform linear transformation on the high-dimensional features so that they correspond to the image category dimension. The output dimension is: T×n class , where n class is the total number of categories.
[0068] (3) Average the time dimension and then use the softmax function to convert the output vector into the probability corresponding to all categories.
[0069] The image encoder and spike body network structure in the spike residual neural network can be used as a general feature extractor for feature extraction in other computer vision tasks.
[0070] CIFAR100 is a commonly used dataset for image classification tasks, containing 60,000 32×32 RGB images. These images are diverse, including animals, flowers, fruits, vegetables, people, and vehicles. Each category is further divided into specific categories, such as roses and sunflowers. The CIFAR100 dataset contains 100 categories and can be used to verify the image classification capabilities of algorithms.
[0071] In order to demonstrate the progressiveness of the method proposed in the present invention, a comparative experiment was first conducted on the CIFAR100 dataset to compare the number of floating-point operations and energy consumption of the pulse residual neural network proposed in the present invention with those of an artificial neural network with the same structure and parameter amount, using the ResNet18 configuration. Then, the present invention was compared with the same type of pulse neural networks SpikingResNet and SEW ResNet, using the image classification accuracy as the indicator.
[0072] Table 2
[0073] Network Parameters MACs ACs Energy ANN ResNet18 11.22M 549.2M 0M <![CDATA[1.7×10 9 pJ]]> SG ResNet18(ours) 11.22M 1.8M 348.9M <![CDATA[4.1×10 7 pJ]]>
[0074] Table 2 compares the number of floating-point operations and energy consumption of the proposed Spike Residual Neural Network (SG ResNet) and the Artificial Neural Network (ANN ResNet). It can be seen that compared to the ANN, the proposed method significantly reduces the number of multiplication operations (MACs) with the same number of parameters, mostly converting them into energy-efficient addition operations (ACs). The ANN's energy consumption is 42 times that of the proposed network.
[0075] Table 3
[0076] Network Spiking ResNet SEW ResNet(IAND) SG ResNet(ours) ResNet10 73.00％ 71.96％ 72.68％ ResNet18 74.36％ 73.89％ 74.62％ ResNet34 32.05％ 73.8％ 75.01％
[0077] Table 3 compares the accuracy of the proposed spiking residual neural network (SG ResNet) on the CIFAR100 dataset with similar methods, Spiking ResNet and SEW ResNet (IAND). As can be seen, the accuracy of the proposed method gradually increases with the increase in network size, from 10 to 34 layers, overcoming the vanishing gradient problem. The ResNet34 network configuration achieves an accuracy of 75.01%, the highest among its peers.
[0078] The above description is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present invention without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment with equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.
Claims
1. An image classification method based on a pulse residual neural network with a gate mechanism, characterized in that: This method inputs RGB images into the pulse residual neural network to obtain image categories; The pulse residual neural network includes an image encoder, a pulse main network and a feature decoder; The image encoder is used to convert the RGB image into a spike image, including a convolutional neural network layer, a batch normalization layer, a spike neuron layer and a maximum pooling layer; The pulse main network is used to extract high-dimensional features of pulse images and is composed of four calculation stages connected in series, each of which is composed of a number of pulse gate residual modules stacked together; the pulse gate residual modules include two types: basic pulse gate residual modules and bottleneck pulse gate residual modules, and both types of pulse gate residual modules include a convolutional neural network layer, a batch normalization layer, a pulse neuron layer and a binary selection gate; the type of pulse gate residual module used is determined according to the network depth of the pulse main network, specifically: a network depth threshold is set, and the network below the threshold adopts the basic pulse gate residual module, otherwise the bottleneck pulse gate residual module is adopted, thereby improving the image classification accuracy at each depth level; the first pulse gate residual module in the second, third and fourth calculation stages is set with a downsampling layer to downsample the image features by two times; The pulse gate residual module consists of two branches, left and right, which respectively perform operations on the input X and then input the binary selection gate, specifically: (1) The left branch performs the transformation operation of the image features and outputs the transformed features X left The left branch consists of a convolutional neural network layer, a batch normalization layer, and a pulse neuron layer connected in series as submodules. The basic pulse gate residual module contains two connected submodules, and the bottleneck pulse gate residual module contains three connected submodules. (2) The right branch is the residual branch of the module, which outputs the identity mapping of X when there is no downsampling layer, that is, X right =X, which is composed of a convolutional neural network layer, a batch normalization layer, and a spiking neuron layer in series when it contains a downsampling layer; (3) The binary selection gate selects the output Y from the inputs of the left and right branches according to the gate signal GATE, as follows: Y=GATE·X left +(1-GATE)·X right The gate signal GATE has two forms of expression, which can be flexibly selected according to the limitation of the computational load, as follows: Form 1: GATE = Θ(W·X left +B-thr) Form 2: GATE = X left Where W and B are the weight and bias of the linear transformation; thr is the cutoff weight, which is set to a small value greater than 0; Θ is the Heaviside step function, which is used to binarize the gate signal; The feature decoder is used to decode the high-dimensional features of the pulse image into the corresponding categories, including an average pooling layer and a fully connected neural network layer.
2. The image classification method based on the pulse residual neural network with a gate mechanism according to claim 1, characterized in that: The image encoding process of the image encoder is specifically as follows: (1) The input image is initially transformed and normalized through the convolutional neural network layer and the batch normalization layer, and the output analog value image data dimension is: c×h×w, where c is the number of channels, h and w are the height and width of the image respectively; (2) The analog value image data is copied along the time dimension, and the analog value image is encoded into a pulse image using a pulse neuron layer. The pulse image dimension is: T×c×h×w, where T is the time step of the pulse residual neural network and the elements of the pulse image are binary pulse values; (3) Use the maximum pooling layer to downsample the pulse image by two times, and the output pulse image dimension is:
3. The image classification method based on the gate mechanism pulse residual neural network according to claim 1, characterized in that: The pulse main network is divided into five configurations according to the network depth: 18, 34, 50, 101, and 152. The number of module stacking in these five configurations is the same as that of ResNet; the 18 and 34 configurations use the basic pulse gate residual module; the 50, 101, and 152 configurations use the bottleneck pulse gate residual module.
4. The image classification method based on the pulse residual neural network with a gate mechanism according to claim 1, characterized in that: The step size of the convolutional neural network layer in the left branch is 1, and the step size of the convolutional neural network layer in the right branch is 2.
5. The image classification method based on the pulse residual neural network with a gate mechanism according to claim 1, characterized in that: The feature decoder decodes the high-dimensional features of the image into categories and corresponding probabilities, specifically: (1) Assume that the input high-dimensional feature dimension is: T×c f ×h f ×w f , where c f 、h f 、w f are the number of channels, height, and width of the feature image, respectively. The average pooling layer is used to compress the image height and width dimensions to obtain T×c f ; (2) Use the fully connected neural network layer to perform linear transformation on the high-dimensional features so that they correspond to the image category dimension. The output dimension is: T×n class , where n class is the total number of categories; (3) Average the time dimension and then use the softmax function to convert the output vector into the probability corresponding to all categories.
6. The image classification method based on a pulse residual neural network with a gate mechanism according to claim 1, characterized in that: The image encoder and pulse main network structure in the pulse residual neural network can be used as a general feature extractor for feature extraction of other computer vision tasks.