Image classification method based on S_ReLU spiking neural network

Abstract

The present invention discloses an image classification method based on an S_ReLU spiking neural network, primarily addressing the problems of the long convergence process of the loss function and the low firing rate of spiking neurons in existing models. The present invention constructs an S_ReLU spiking neural network and an S_ReLU artificial neural network, respectively. The S_ReLU artificial neural network is trained using a training set to obtain a transferable weight file, which is then loaded into the S_ReLU spiking neural network. The S_ReLU spiking neural network, loaded with the transferable weight file, is then used to classify images. The present invention reduces the convergence time of the training model's loss function, improves the matching degree between the activation rate of neurons in the artificial neural network and the pulse firing rate of spiking neurons in the spiking neural network, and thereby improves the accuracy of image classification using the spiking neural network.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and further relates to a spiking neural network image classification method based on the spiking rectified linear unit (S_ReLU) function in the field of image classification technology. The present invention can classify target objects in optical images. Background Art

[0002] Image classification is a common task in machine vision. A trained convolutional neural network (CNN) preprocesses images, extracts features, and uses a classifier to classify them, ultimately outputting a predicted class label. Currently, image classification methods based on artificial neural networks (ANNs) have been widely used for optical image classification. These methods eliminate the need for manual feature extraction and extensive expertise. Instead, they automatically and robustly extract image features through training the ANN model, achieving excellent classification results. However, ANNs are complex, require high-performance hardware (CPUs and GPUs), and consume significant energy, making them difficult to deploy on edge devices such as cameras and sensors for image classification. Spiking neural networks (SNs), a next-generation neural network model inspired by biological processes, feature event-driven features that are more compatible with hardware and more easily mimic the low-power, high-performance processing of the human brain. These SNs are more suitable for deployment on end devices for optical image classification than ANNs. However, there are currently many problems in converting artificial neural network models into spiking neural network models for training: on the one hand, the activation value of the artificial neuron and the pulse firing rate of the spiking neuron cannot be well fitted, resulting in a serious degradation in the performance of the converted spiking neural network and a reduction in the image classification accuracy; on the other hand, the spiking neural network has many parameters, and the training and inference speed of the model is slow, making it impossible to classify and detect images in real time.

[0003] The Institute of Computing Technology of the Chinese Academy of Sciences proposed a method for applying a converted spiking neural network to image classification in its patent application, "Spiking Neural Network Conversion Method and Related Conversion Chip" (Patent Application No.: 201910079637.7, Authorization Announcement No.: CN 110059800 B). This method first trains the artificial neural network to be converted to obtain the network's weight parameters. It then determines the layers and activation functions to be converted within the artificial neural network. Finally, the layers and activation functions of the artificial neurons to be converted are directly replaced with spiking neurons to obtain a spiking neural network. When applying the converted spiking neural network to image classification, the image dataset is first pulse-encoded to obtain pulse sequence data, which can then be used to classify the image. A drawback of this image classification method is that, because the spiking neural network conversion directly replaces the activation function with spiking neurons, the encoded image pulse sequence does not adequately stimulate the spiking neurons, resulting in a low pulse firing rate for the spiking neurons in the converted spiking neural network, which does not meet the activation rate of the artificial neurons required by the converted activation function. This ultimately results in very low image classification accuracy for the converted spiking neural network.

[0004] Yangfan Hu, Huajin Tang, and others proposed an image classification method based on a spiking deep residual network in their paper "Spiking Deep Residual Networks" (IEEE Transactions on Neural Networks and Learning Systems, 2021). This method introduces additional learnable parameters to scale weights and biases, designs a shortcut conversion model and compensation mechanism, and optimizes these parameters to scale the continuous values ​​of artificial neurons to match the pulse firing rate of spiking neurons, thereby reducing the error caused by the discretization of data by the converted spiking neural network. Although the spiking neural network obtained by this method has achieved an accuracy close to that of the original artificial neural network on multiple image classification datasets, the method still has the disadvantage that the shortcut conversion model and compensation mechanism proposed by it introduce a large number of learnable parameters. When iterating the parameters during model training, the convergence process of the model's loss function is prolonged, slow, and inefficient, which prolongs the time for model training and inference. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose an image classification method based on S_ReLU pulse neural network to solve the problems of low image classification accuracy and long convergence process of the model's loss function caused by the low pulse firing rate of pulse neurons in the pulse neural network.

[0006] The idea of ​​achieving the purpose of the present invention is that the present invention constructs an S_ReLU spiking neural network and an S_ReLU artificial neural network respectively, uses a training set to train the S_ReLU artificial neural network to obtain a transferable weight file, and loads the weight file into the S_ReLU spiking neural network to classify the image to be classified. The proposed S_ReLU spiking neural network uses multiple small-sized 3*3 convolution kernels to accept input, which reduces the complexity of the spiking neural network, reduces the amount of calculation of the model, and solves the problem of the long convergence process of the model's loss function. The proposed S_ReLU artificial neural network uses an S_ReLU layer, and the S_ReLU activation function of the S_ReLU layer can better activate the neurons of the convolution layer in the network, so that the transferable weights obtained by training the S_ReLU artificial neural network are more suitable for the S_ReLU spiking neural network, thereby improving the matching degree between the activation rate of neurons in the artificial neural network and the pulse firing rate of the spiking neurons in the spiking neural network, overcoming the problem of low image classification accuracy caused by insufficient pulse firing rate of spiking neurons in the prior art.

[0007] The specific steps of the present invention are as follows:

[0008] Step 1: Generate training set:

[0009] Step 1.1: Generate K optical images containing T target categories. Each target category corresponds to a sample set of I optical images, where T ≥ 2, K ≥ 100, and I ≥ 50.

[0010] Step 1.2: One-hot encode the true category value of each sample in the sample set to obtain the encoded label value of each sample;

[0011] Step 1.3, perform Poisson coding on the optical images in the sample set to form a training set;

[0012] Step 2: Construct the S_ReLU artificial neural network:

[0013] Step 2.1: Build an S_ReLU artificial neural network with the following structure: input layer, first residual block, second residual block, third residual block, fourth residual block, and output layer.

[0014] The structure of the input layer is: the first convolution layer, the second convolution layer, the third convolution layer, and the average pooling layer;

[0015] The structure of the first residual block is the first convolution layer, the first S_ReLU layer, the second convolution layer, the second S_ReLU layer, the third convolution layer, the third S_ReLU layer, the fourth convolution layer, and the fourth S_ReLU layer in sequence; the fifth convolution layer in the first residual block is connected between the average pooling layer of the input layer and the second S_ReLU layer of the first residual block, and the sixth convolution layer is connected between the second S_ReLU layer of the first residual block and the fourth S_ReLU layer of the first residual block;

[0016] The structure of the second residual block is the first convolution layer, the first S_ReLU layer, the second convolution layer, the second S_ReLU layer, the third convolution layer, the third S_ReLU layer, the fourth convolution layer, and the fourth S_ReLU layer in sequence; the fifth convolution layer in the second residual block is connected between the fourth S_ReLU layer of the first residual block and the second S_ReLU layer of the second residual block, and the sixth convolution layer is connected between the second S_ReLU layer of the second residual block and the fourth S_ReLU layer of the second residual block;

[0017] The structure of the third residual block is sequentially the first convolution layer, the first S_ReLU layer, the second convolution layer, the second S_ReLU layer, the third convolution layer, the third S_ReLU layer, the fourth convolution layer, and the fourth S_ReLU layer; the fifth convolution layer in the third residual block is connected between the fourth S_ReLU layer of the second residual block and the second S_ReLU layer of the third residual block, and the sixth convolution layer is connected between the second S_ReLU layer of the third residual block and the fourth S_ReLU layer of the third residual block;

[0018] The structure of the fourth residual block is sequentially the first convolution layer, the first S_ReLU layer, the second convolution layer, the second S_ReLU layer, the third convolution layer, the third S_ReLU layer, the fourth convolution layer, and the fourth S_ReLU layer; the fifth convolution layer in the fourth residual block is connected between the fourth S_ReLU layer of the third residual block and the second S_ReLU layer of the fourth residual block, and the sixth convolution layer is connected between the second S_ReLU layer of the fourth residual block and the fourth S_ReLU layer of the fourth residual block;

[0019] The output layer is composed of an average pooling layer and a fully connected layer in series;

[0020] Step 2.2, set the parameters of each layer of the S_ReLU artificial neural network as follows:

[0021] The convolution kernel size of the first to third convolution layers in the input layer is set to 3*3, the stride size is set to 2, and the output channels are set to 32, 32, and 64 respectively; the convolution kernel size of the average pooling layer is 3*3, the stride is 2, and the padding is 1;

[0022] The convolution kernel size of the first to fourth convolution layers in the first residual block is set to 3*3, the stride size is set to 1, the padding is set to 1, and the output channels are set to 64; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 64;

[0023] The convolution kernel size of the first to fourth convolution layers in the second residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 128; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 128;

[0024] The convolution kernel size of the first to fourth convolution layers in the third residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 256; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 256;

[0025] The convolution kernel size of the first to fourth convolution layers in the second residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 512; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 512;

[0026] All S_ReLU layers in the first to fourth residual blocks are implemented using the S_ReLU activation function;

[0027] The convolution kernel size of the average pooling layer in the output layer is set to 1*1, and the stride size is set to 1; the output of the fully connected layer is set to 10;

[0028] Step 3: Train the S_ReLU artificial neural network to obtain the transferable weight file:

[0029] Take batchsize samples from the training set each time and input them into the S_ReLU artificial neural network, where batchsize>10; output the predicted value of each sample category, input the predicted value of each sample category and its encoded label value into the S_ReLU artificial neural network loss function, and perform backpropagation of the loss value; iteratively update the network parameters through the mini-batch gradient descent optimizer until the loss function converges, and obtain the trained S_ReLU artificial neural network. Save the weight parameters of the trained S_ReLU artificial neural network into a transferable weight file;

[0030] Step 4: Build the S_ReLU spiking neural network and load the transferable weight file:

[0031] Step 4.1: Build an S_ReLU pulse neural network with the following structure: pulse input layer, first pulse residual block, second pulse residual block, third pulse residual block, fourth pulse residual block, and pulse output layer.

[0032] The structure of the pulse input layer is as follows: the first convolution layer, the first IF layer, the second convolution layer, the second IF layer, the third convolution layer, the third IF layer, the average pooling layer, and the fourth IF layer;

[0033] The structure of the first pulse residual block is sequentially a first convolution layer, a first IF layer, a second convolution layer, a second IF layer, a third convolution layer, a third IF layer, a fourth convolution layer, and a fourth IF layer; the fifth convolution layer in the first pulse residual block is connected between the fourth IF layer of the pulse input layer and the second IF layer of the first pulse residual block, and the sixth convolution layer is connected between the second IF layer of the first pulse residual block and the fourth IF layer of the first pulse residual block;

[0034] The structure of the second pulse residual block is sequentially a first convolution layer, a first IF layer, a second convolution layer, a second IF layer, a third convolution layer, a third IF layer, a fourth convolution layer, and a fourth IF layer; the fifth convolution layer in the second pulse residual block is connected between the fourth IF layer of the first pulse residual block and the second IF layer of the second pulse residual block, and the sixth convolution layer is connected between the second IF layer of the second pulse residual block and the fourth IF layer of the second pulse residual block;

[0035] The structure of the third pulse residual block is sequentially a first convolution layer, a first IF layer, a second convolution layer, a second IF layer, a third convolution layer, a third IF layer, a fourth convolution layer, and a fourth IF layer; the fifth convolution layer in the third pulse residual block is connected between the fourth IF layer of the second pulse residual block and the second IF layer of the third pulse residual block, and the sixth convolution layer is connected between the second IF layer of the third pulse residual block and the fourth IF layer of the third pulse residual block;

[0036] The structure of the fourth pulse residual block is sequentially a first convolution layer, a first IF layer, a second convolution layer, a second IF layer, a third convolution layer, a third IF layer, a fourth convolution layer, and a fourth IF layer; the fifth convolution layer in the fourth pulse residual block is connected between the fourth IF layer of the third pulse residual block and the second IF layer of the fourth pulse residual block, and the sixth convolution layer is connected between the second IF layer of the fourth pulse residual block and the fourth IF layer of the fourth pulse residual block;

[0037] The structure of the output layer is: average pooling layer, first IF layer, fully connected layer, second IF layer;

[0038] Step 4.2, set the parameters of each layer of the S_ReLU spiking neural network as follows:

[0039] The convolution kernel size of the first to third convolution layers in the pulse input layer is set to 3*3, the stride size is set to 2, and the output channels are set to 32, 32, and 64 respectively; the convolution kernel size of the average pooling layer is 3*3, the stride is 2, and the padding is 1;

[0040] The convolution kernel size of the first to fourth convolution layers in the first impulse residual block is set to 3*3, the stride size is set to 1, the padding is set to 1, and the output channels are set to 64; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 64;

[0041] The convolution kernel size of the first to fourth convolution layers in the second impulse residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 128; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 128;

[0042] The convolution kernel size of the first to fourth convolution layers in the third impulse residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 256; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 256;

[0043] The convolution kernel size of the first to fourth convolution layers in the second impulse residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 512; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 512;

[0044] The convolution kernel size of the average pooling layer in the pulse output layer is set to 1*1, and the stride size is set to 1; the output of the fully connected layer is set to 10;

[0045] Set the v_reset values ​​of all IF layers in the pulse input layer, the first to fourth pulse residual blocks, and the pulse output layer to 0, and the v_threshold values ​​to V in the S_ReLU layer. i value;

[0046] Step 4.3, parse the weight file saved in step 3.4 through the torch.load function, and use the net.load_state_dict function to load the parsed convolution kernel weights and fully connected layer weights into the S_ReLU spiking neural network;

[0047] Step 5: Classify the image using the S_ReLU spiking neural network after loading the transferable weight file:

[0048] Step 5.1, perform Poisson encoding on the optical image to be classified;

[0049] In step 5.2, the encoded image is input into the S_ReLU spiking neural network after the transferable weight file is loaded, and the classification result of the optical image is output.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] First, the S_ReLU artificial neural network constructed by the present invention uses an S_ReLU layer, and the S_ReLU activation function of the S_ReLU layer can better activate the neurons of the convolutional layer in the network, so that the transferable weights obtained by training the S_ReLU artificial neural network are more suitable for the S_ReLU pulse neural network, effectively improving the pulse firing rate of the pulse neurons in the pulse neural network, thereby improving the matching degree between the activation rate of the neurons in the artificial neural network and the pulse firing rate of the pulse neurons in the pulse neural network, solving the problem of mismatch between the pulse firing rate of the pulse neurons and the activation rate of the artificial neurons, and improving the accuracy of the pulse neural network in image classification.

[0052] Second, the S_ReLU pulse neural network constructed by the present invention uses multiple small-size 3*3 convolution kernels in the input layer, which reduces the computational cost of the network while ensuring the consistency of the output information, reduces the computational amount of the pulse neural network, and solves the problem of the long convergence process of the model's loss function, thereby accelerating the training and reasoning speed of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of the present invention;

[0054] Figure 2 : is a schematic diagram of the model structure of the S_ReLU artificial neural network of the present invention; wherein, Figure 2 (a) is a schematic diagram of the overall structure of the S_ReLU artificial neural network model; Figure 2 (b) is a schematic diagram of the residual block structure in the S_ReLU artificial neural network model;

[0055] Figure 3: is a schematic diagram of the model structure of the S_ReLU pulse neural network of the present invention; wherein, Figure 3 (a) is a schematic diagram of the overall structure of the S_ReLU pulse neural network model; Figure 3 (b) is a schematic diagram of the pulse residual block structure in the S_ReLU artificial neural network model;

[0056] Figure 4 This is a schematic diagram of the output results of the S_ReLU pulse neural network and the non-S_ReLU pulse neural network on the test set. DETAILED DESCRIPTION

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] Reference Figure 1 , further describing the implementation steps of the embodiment of the present invention.

[0059] Step 1: Generate training and test sets.

[0060] In step 1.1, this embodiment of the invention uses the existing MNIST dataset (Mixed National Institute of Standards and Technology database), a large database of handwritten digits compiled by the National Institute of Standards and Technology in the United States. It contains 6,000 images from ten categories, each of which is a 28x28 optical grayscale image. 50,000 images from each category are selected to form the training set, and the remaining 10,000 images form the test set.

[0061] In step 1.2, each image in the training and test sets is horizontally flipped once to obtain a new batch of images. Another batch of images is obtained by varying the brightness and contrast of the images in the training and test sets within the range [0, 100]. These new images are added to the training and test sets, tripling the number of samples. This results in an expanded training set of 150,000 images in ten categories (numbers 0 to 9) and an expanded test set of 30,000 images in ten categories (numbers 0 to 9).

[0062] In step 1.3, Poisson coding is used to encode the 28*28 input images in the expanded training sample set and the expanded test sample set into 0, 1 pulse sequences of length 28*28, forming the training set and test set of this embodiment.

[0063] Step 2: Construct the S_ReLU artificial neural network structure.

[0064] The weight file of the S_ReLU artificial neural network can be directly used on the S_ReLU spiking neural network, avoiding the problem of difficult training of spiking neural networks. Therefore, the weights for the S_ReLU spiking neural network can be obtained by training the S_ReLU artificial neural network.

[0065] Step 2.1, build an S_ReLU artificial neural network, the specific structure is as follows Figure 2 shown.

[0066] The structure of the S_ReLU artificial neural network is input layer, first residual block, second residual block, third residual block, fourth residual block, and output layer.

[0067] The following combination Figure 2 (a) and Figure 2 (b) Further describe the structure of the constructed S_ReLU artificial neural network.

[0068] The structure of the input layer is: the first convolution layer, the second convolution layer, the third convolution layer, and the average pooling layer.

[0069] The structure of the first residual block is the first convolutional layer, the first S_ReLU layer, the second convolutional layer, the second S_ReLU layer, the third convolutional layer, the third S_ReLU layer, the fourth convolutional layer, and the fourth S_ReLU layer. The fifth convolutional layer in the first residual block is connected between the average pooling layer of the input layer and the second S_ReLU layer of the first residual block, and the sixth convolutional layer is connected between the second S_ReLU layer of the first residual block and the fourth S_ReLU layer of the first residual block.

[0070] The structure of the second residual block is, in order, the first convolutional layer, the first S_ReLU layer, the second convolutional layer, the second S_ReLU layer, the third convolutional layer, the third S_ReLU layer, the fourth convolutional layer, and the fourth S_ReLU layer. The fifth convolutional layer in the second residual block is connected between the fourth S_ReLU layer of the first residual block and the second S_ReLU layer of the second residual block, and the sixth convolutional layer is connected between the second S_ReLU layer of the second residual block and the fourth S_ReLU layer of the second residual block.

[0071] The structure of the third residual block is, in order, the first convolutional layer, the first S_ReLU layer, the second convolutional layer, the second S_ReLU layer, the third convolutional layer, the third S_ReLU layer, the fourth convolutional layer, and the fourth S_ReLU layer. The fifth convolutional layer in the third residual block is connected between the fourth S_ReLU layer of the second residual block and the second S_ReLU layer of the third residual block, and the sixth convolutional layer is connected between the second S_ReLU layer of the third residual block and the fourth S_ReLU layer of the third residual block.

[0072] The structure of the fourth residual block is, in order, the first convolutional layer, the first S_ReLU layer, the second convolutional layer, the second S_ReLU layer, the third convolutional layer, the third S_ReLU layer, the fourth convolutional layer, and the fourth S_ReLU layer. The fifth convolutional layer in the fourth residual block is connected between the fourth S_ReLU layer of the third residual block and the second S_ReLU layer of the fourth residual block, and the sixth convolutional layer is connected between the second S_ReLU layer of the fourth residual block and the fourth S_ReLU layer of the fourth residual block.

[0073] The output layer is composed of an average pooling layer and a fully connected layer connected in series.

[0074] The parameters of each layer of the S_ReLU artificial neural network are set as follows: The convolution kernel size of the first to third convolution layers in the input layer is set to 3*3, the stride size is set to 2, and the output channels are set to 32, 32, and 64, respectively. The convolution kernel size of the average pooling layer is set to 3*3, the stride size is set to 2, and the padding is set to 1. The convolution kernel size of the first to fourth convolution layers in the first residual block is set to 3*3, the stride size is set to 1, the padding is set to 1, and the output channels are set to 64. The convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 64. The convolution kernel size of the first to fourth convolution layers in the second residual block is set to 3*3, the stride size is set to 2, 1, 1, 1, respectively, the padding is set to 1, and the output channels are set to 128. The convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 128. The convolution kernel size of the first to fourth convolutional layers in the third residual block is set to 3*3, with strides of 2, 1, 1, and 1, respectively, and padding of 1. The output channels are all set to 256. The convolution kernel size of the fifth and sixth convolutional layers is set to 1*1, with stride of 1, and output channels are all set to 256. The convolution kernel size of the first to fourth convolutional layers in the second residual block is set to 3*3, with strides of 2, 1, 1, and 1, respectively, and padding of 1. The output channels are all set to 512. The convolution kernel size of the fifth and sixth convolutional layers is set to 1*1, with stride of 1, and output channels are all set to 512. All S_ReLU layers in the first to fourth residual blocks are implemented using the S_ReLU activation function. The convolution kernel size of the average pooling layer in the output layer is set to 1*1, with stride of 1. The output of the fully connected layer is set to 10.

[0075] Step 2.2, construct the S_ReLU activation function as follows:

[0076]

[0077]

[0078] V i =c i ·max(W i ·F i-1 +b i-1 )

[0079] Among them, F i Represents the output of the S_ReLU layer in the S_ReLU artificial neural network, Y i V represents the actual output value of the neurons in the convolutional layer before the S_ReLU layer in the S_ReLU artificial neural network after the constraints are met. i Represents the maximum output value scaling c of the neurons in the convolutional layer before the S_ReLU layer in the S_ReLU artificial neural network i times the value, W i-1 represents the weight matrix from the neurons in layer i-1 to layer i, b i-1 Indicates the bias value from the neurons in layer i-1 to layer i. i It is a trainable parameter that represents the scaling factor for scaling the output of the convolutional layer neurons in the S_ReLU artificial neural network. The initial value is set to 1 and the value range is [0,1]. It needs to be converted into a trainable parameter type using the Parameter() function. When the parameters of the S_ReLU artificial neural network are updated, the optimizer can automatically update it.

[0080] Step 3: Train the S_ReLU artificial neural network to obtain a transferable weight file.

[0081] Step 3.1, construct the loss function for training the S_ReLU artificial neural network.

[0082] The loss function used in this invention is the cross entropy loss function CrossEntropyLoss(), as shown below:

[0083] loss(y i ,class)=-∑class·log(y i )

[0084] Among them, y i Represents the predicted value of the i-th sample output by the S_ReLU artificial neural network, class represents the label value after encoding each sample, loss(y i ,class) represents the loss value between the label and the predicted value, and log represents the logarithm with base 10.

[0085] Step 3.2, build an optimizer for training the S_ReLU artificial neural network.

[0086] The optimizer used in this paper is the Mini-Batch Gradient Descent (MBGD) optimizer. The parameter update principle of the MBGD optimizer is to use only a portion of the total data for each parameter update. Essentially, it uses a batch gradient descent strategy within each batch of data and a stochastic gradient descent strategy outside each batch of data. The parameters of the MBGD optimizer are set as follows: the momentum factor Momentum coefficient is set to 0.9, the weight decay coefficient weight_decay is set to 0.0001, the learning rate lr is set to 0.01, and other parameters use the default settings.

[0087] Step 3.3, specify the hyperparameters of the training network.

[0088] The basic hyperparameters include epoch (the number of times the training set data completes a full training of the network model) set to 70, batch_size (the number of training set data samples required for a backpropagation parameter update of the model weights) set to 50, and dropout (the random deactivation value of neurons in the network) set to 0.01.

[0089] Step 3.4: Train the S_ReLU artificial neural network to obtain a transferable weight file.

[0090] In each iteration, batchsize samples are taken from the training set and input into the S_ReLU artificial neural network, outputting the predicted value for each sample category until all data is taken from the training set. The predicted value for each sample category and its encoded label value are input into the S_ReLU artificial neural network loss function, and the loss value is backpropagated. The network parameters are iteratively updated using a mini-batch gradient descent optimizer until the loss function converges, resulting in a trained S_ReLU artificial neural network. The weight parameters of the trained S_ReLU artificial neural network are saved in a transferable weight file.

[0091] Step 4: Build the S_ReLU spiking neural network and load the transferable weight file.

[0092] Step 4.1, build an S_ReLU pulse neural network, the specific structure is as follows Figure 3 shown.

[0093] The S_ReLU pulse neural network structure is as follows: pulse input layer, first pulse residual block, second pulse residual block, third pulse residual block, fourth pulse residual block, pulse output layer.

[0094] The following combination Figure 3 (a) Figure 3(b) Further describe the structure of the constructed S_ReLU pulse neural network.

[0095] The structure of the pulse input layer is as follows: the first convolution layer, the first IF layer, the second convolution layer, the second IF layer, the third convolution layer, the third IF layer, the average pooling layer, and the fourth IF layer.

[0096] The structure of the first pulse residual block is, in order, a first convolutional layer, a first IF layer, a second convolutional layer, a second IF layer, a third convolutional layer, a third IF layer, a fourth convolutional layer, and a fourth IF layer. The fifth convolutional layer in the first pulse residual block is connected between the fourth IF layer of the pulse input layer and the second IF layer of the first pulse residual block, and the sixth convolutional layer is connected between the second IF layer of the first pulse residual block and the fourth IF layer of the first pulse residual block.

[0097] The structure of the second pulse residual block is, in order, a first convolutional layer, a first IF layer, a second convolutional layer, a second IF layer, a third convolutional layer, a third IF layer, a fourth convolutional layer, and a fourth IF layer. The fifth convolutional layer in the second pulse residual block is connected between the fourth IF layer of the first pulse residual block and the second IF layer of the second pulse residual block, and the sixth convolutional layer is connected between the second IF layer of the second pulse residual block and the fourth IF layer of the second pulse residual block.

[0098] The structure of the third pulse residual block is, in order, a first convolutional layer, a first IF layer, a second convolutional layer, a second IF layer, a third convolutional layer, a third IF layer, a fourth convolutional layer, and a fourth IF layer. The fifth convolutional layer in the third pulse residual block is connected between the fourth IF layer of the second pulse residual block and the second IF layer of the third pulse residual block, and the sixth convolutional layer is connected between the second IF layer of the third pulse residual block and the fourth IF layer of the third pulse residual block.

[0099] The structure of the fourth pulse residual block is, in order, a first convolutional layer, a first IF layer, a second convolutional layer, a second IF layer, a third convolutional layer, a third IF layer, a fourth convolutional layer, and a fourth IF layer. The fifth convolutional layer in the fourth pulse residual block is connected between the fourth IF layer of the third pulse residual block and the second IF layer of the fourth pulse residual block, and the sixth convolutional layer is connected between the second IF layer of the fourth pulse residual block and the fourth IF layer of the fourth pulse residual block.

[0100] The structure of the output layer is: average pooling layer, first IF layer, fully connected layer, and second IF layer.

[0101] The parameters of each layer of the S_ReLU spiking neural network are set as follows: The convolution kernel size of the first to third convolution layers in the spiking input layer is set to 3*3, the stride size is set to 2, and the output channels are set to 32, 32, and 64, respectively. The convolution kernel size of the average pooling layer is set to 3*3, the stride size is set to 2, and the padding is set to 1. The convolution kernel size of the first to fourth convolution layers in the first spiking residual block is set to 3*3, the stride size is set to 1, the padding is set to 1, and the output channels are set to 64. The convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 64. The convolution kernel size of the first to fourth convolution layers in the second spiking residual block is set to 3*3, the stride size is set to 2, 1, 1, 1, respectively, the padding is set to 1, and the output channels are set to 128. The convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 128. The convolution kernel size of the first to fourth convolution layers in the third impulse residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channels are set to 256. The convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 256. The convolution kernel size of the first to fourth convolution layers in the second impulse residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channels are set to 512. The convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 512. The convolution kernel size of the average pooling layer in the impulse output layer is set to 1*1, and the stride size is set to 1. The output of the fully connected layer is set to 10. Set the v_reset values ​​of all IF layers in the pulse input layer, the first to fourth pulse residual blocks, and the pulse output layer to 0, and the v_threshold values ​​to V in the S_ReLU layer. i value.

[0102] Step 4.2, load the transferable weight file into the S_ReLU spiking neural network.

[0103] The saved transferable weight file is parsed by the torch.load() function, and the parsed convolution kernel weights and fully connected layer weights are loaded into the S_ReLU spiking neural network using the net.load_state_dict() function.

[0104] Step 5: Use the S_ReLU spiking neural network to classify the image.

[0105] The obtained test set is sent to the S_ReLU pulse neural network to perform image classification on the test set to obtain the category prediction value of each test set sample.

[0106] The effects of the present invention are further described below in conjunction with simulation experiments:

[0107] 1. Simulation experiment conditions:

[0108] The hardware platform of the simulation experiment of the present invention is: the processor is Intel (R) Xeon (R) Gold 5218 CPU, the main frequency is 2.30GHz, and the memory is 128GB.

[0109] The software platforms of the simulation experiment of the present invention are: Ubuntu operating system and spikingjelly0.0.0.0.12.

[0110] The input images used in the simulation experiments of the present invention are the MNIST dataset of handwritten digits. This dataset is a dataset initiated and compiled by the National Institute of Standards and Technology (NIST) of the United States. A total of 250 different people's handwritten digits are counted. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in the images. The images are of a fixed size (28x28 pixels) and their values ​​range from 0 to 9.

[0111] 2. Simulation content and result analysis:

[0112] The simulation experiment of the present invention adopts S_ReLU spiking neural network and non-S_ReLU spiking neural network to classify the input handwritten image data respectively.

[0113] An image is randomly selected from each category for intuitive display. Below the image are the classification prediction results of the two networks for the image. The prediction results of the two networks are as follows: Figure 4 As shown. Figure 4 It can be seen that the S_ReLU spiking neural network classified all the numbers correctly, while the non-S_ReLU spiking neural network classified the image of the number 2 as 3 and the image of the number 5 as 6.

[0114] The classification results were evaluated using two evaluation metrics (TOP-5 error rate and TOP-1 error rate). TOP-1 and TOP-5 error rates are commonly used metrics for evaluating network classification performance. The TOP-5 error rate compares the top five labels predicted by the network with the correct result. If one of the five is correct, the classifier's prediction is considered correct. The TOP-1 error rate compares the label predicted by the network with the correct result. If the predicted label is the same as the correct result, the classifier's prediction is considered correct. As shown in Table 1, the second column shows the TOP-1 accuracy of the three networks on the test set, and the third column shows the TOP-5 accuracy of the three networks on the test set.

[0115] Table 1 Network accuracy results on the test set

[0116] Algorithm Model TOP-1, % TOP-5, % S_ReLU Artificial Neural Network 98.12 99.79 Non-S_ReLU Spiking Neural Network 96.35 98.56 S_ReLU Spiking Neural Network 97.52 99.26

[0117] As shown in Table 1, the S_ReLU spiking neural network of the present invention achieves a Top-1 error rate of 96.35% and a Top-5 error rate of 99.26%. Both of these metrics outperform non-S_ReLU spiking neural network methods and are closer to the classification accuracy of S_ReLU artificial neural networks, demonstrating that the present invention can achieve higher image classification accuracy.

[0118] The above simulation experiments show that the S_ReLU spiking neural network constructed by the present invention effectively improves the pulse firing rate of spiking neurons in the spiking neural network, thereby improving the matching degree between the activation rate of neurons in the artificial neural network and the pulse firing rate of spiking neurons in the spiking neural network, effectively improving the accuracy of spiking neural network image classification, and is a very practical classification method.

Claims

1. An image classification method based on S_ReLU pulse neural network, characterized in that: An S_ReLU spiking neural network and an S_ReLU artificial neural network are constructed respectively, the S_ReLU artificial neural network is trained using a training set to obtain a transferable weight file, and the weight file is loaded into the S_ReLU spiking neural network. The steps of the image classification method include the following: Step 1: Generate training set: Step 1.1: Generate K optical images containing T target categories. Each target category corresponds to a sample set of I optical images, where T ≥ 2, K ≥ 100, and I ≥ 50. Step 1.2: One-hot encode the true category value of each sample in the sample set to obtain the encoded label value of each sample; Step 1.3, perform Poisson coding on the optical images in the sample set to form a training set; Step 2: Construct the S_ReLU artificial neural network: Step 2.1: Build an S_ReLU artificial neural network with the following structure: input layer, first residual block, second residual block, third residual block, fourth residual block, and output layer. The structure of the input layer is: the first convolution layer, the second convolution layer, the third convolution layer, and the average pooling layer; The structure of the first residual block is the first convolution layer, the first S_ReLU layer, the second convolution layer, the second S_ReLU layer, the third convolution layer, the third S_ReLU layer, the fourth convolution layer, and the fourth S_ReLU layer in sequence; the fifth convolution layer in the first residual block is connected between the average pooling layer of the input layer and the second S_ReLU layer of the first residual block, and the sixth convolution layer is connected between the second S_ReLU layer of the first residual block and the fourth S_ReLU layer of the first residual block; The structure of the second residual block is the first convolution layer, the first S_ReLU layer, the second convolution layer, the second S_ReLU layer, the third convolution layer, the third S_ReLU layer, the fourth convolution layer, and the fourth S_ReLU layer in sequence; the fifth convolution layer in the second residual block is connected between the fourth S_ReLU layer of the first residual block and the second S_ReLU layer of the second residual block, and the sixth convolution layer is connected between the second S_ReLU layer of the second residual block and the fourth S_ReLU layer of the second residual block; The structure of the third residual block is sequentially the first convolution layer, the first S_ReLU layer, the second convolution layer, the second S_ReLU layer, the third convolution layer, the third S_ReLU layer, the fourth convolution layer, and the fourth S_ReLU layer; the fifth convolution layer in the third residual block is connected between the fourth S_ReLU layer of the second residual block and the second S_ReLU layer of the third residual block, and the sixth convolution layer is connected between the second S_ReLU layer of the third residual block and the fourth S_ReLU layer of the third residual block; The structure of the fourth residual block is sequentially the first convolution layer, the first S_ReLU layer, the second convolution layer, the second S_ReLU layer, the third convolution layer, the third S_ReLU layer, the fourth convolution layer, and the fourth S_ReLU layer; the fifth convolution layer in the fourth residual block is connected between the fourth S_ReLU layer of the third residual block and the second S_ReLU layer of the fourth residual block, and the sixth convolution layer is connected between the second S_ReLU layer of the fourth residual block and the fourth S_ReLU layer of the fourth residual block; The output layer is composed of an average pooling layer and a fully connected layer in series; Step 2.2, set the parameters of each layer of the S_ReLU artificial neural network as follows: The convolution kernel size of the first to third convolution layers in the input layer is set to 3*3, the stride size is set to 2, and the output channels are set to 32, 32, and 64 respectively; the convolution kernel size of the average pooling layer is 3*3, the stride is 2, and the padding is 1; The convolution kernel size of the first to fourth convolution layers in the first residual block is set to 3*3, the stride size is set to 1, the padding is set to 1, and the output channels are set to 64; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 64; The convolution kernel size of the first to fourth convolution layers in the second residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 128; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 128; The convolution kernel size of the first to fourth convolution layers in the third residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 256; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 256; The convolution kernel size of the first to fourth convolution layers in the second residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 512; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 512; All S_ReLU layers in the first to fourth residual blocks are implemented using the S_ReLU activation function; The convolution kernel size of the average pooling layer in the output layer is set to 1*1, and the stride size is set to 1; the output of the fully connected layer is set to 10; Step 3: Train the S_ReLU artificial neural network to obtain the transferable weight file: Take batchsize samples from the training set each time and input them into the S_ReLU artificial neural network, where batchsize>10; output the predicted value of each sample category, input the predicted value of each sample category and its encoded label value into the S_ReLU artificial neural network loss function, and perform backpropagation of the loss value; iteratively update the network parameters through the mini-batch gradient descent optimizer until the loss function converges, and obtain the trained S_ReLU artificial neural network. Save the weight parameters of the trained S_ReLU artificial neural network into a transferable weight file; Step 4: Build the S_ReLU spiking neural network and load the transferable weight file: Step 4.1: Build an S_ReLU pulse neural network with the following structure: pulse input layer, first pulse residual block, second pulse residual block, third pulse residual block, fourth pulse residual block, and pulse output layer. The structure of the pulse input layer is as follows: the first convolution layer, the first IF layer, the second convolution layer, the second IF layer, the third convolution layer, the third IF layer, the average pooling layer, and the fourth IF layer; The structure of the first pulse residual block is sequentially a first convolution layer, a first IF layer, a second convolution layer, a second IF layer, a third convolution layer, a third IF layer, a fourth convolution layer, and a fourth IF layer; the fifth convolution layer in the first pulse residual block is connected between the fourth IF layer of the pulse input layer and the second IF layer of the first pulse residual block, and the sixth convolution layer is connected between the second IF layer of the first pulse residual block and the fourth IF layer of the first pulse residual block; The structure of the second pulse residual block is sequentially a first convolution layer, a first IF layer, a second convolution layer, a second IF layer, a third convolution layer, a third IF layer, a fourth convolution layer, and a fourth IF layer; the fifth convolution layer in the second pulse residual block is connected between the fourth IF layer of the first pulse residual block and the second IF layer of the second pulse residual block, and the sixth convolution layer is connected between the second IF layer of the second pulse residual block and the fourth IF layer of the second pulse residual block; The structure of the third pulse residual block is sequentially a first convolution layer, a first IF layer, a second convolution layer, a second IF layer, a third convolution layer, a third IF layer, a fourth convolution layer, and a fourth IF layer; the fifth convolution layer in the third pulse residual block is connected between the fourth IF layer of the second pulse residual block and the second IF layer of the third pulse residual block, and the sixth convolution layer is connected between the second IF layer of the third pulse residual block and the fourth IF layer of the third pulse residual block; The structure of the fourth pulse residual block is sequentially a first convolution layer, a first IF layer, a second convolution layer, a second IF layer, a third convolution layer, a third IF layer, a fourth convolution layer, and a fourth IF layer; the fifth convolution layer in the fourth pulse residual block is connected between the fourth IF layer of the third pulse residual block and the second IF layer of the fourth pulse residual block, and the sixth convolution layer is connected between the second IF layer of the fourth pulse residual block and the fourth IF layer of the fourth pulse residual block; The structure of the output layer is: average pooling layer, first IF layer, fully connected layer, second IF layer; Step 4.2, set the parameters of each layer of the S_ReLU spiking neural network as follows: The convolution kernel size of the first to third convolution layers in the pulse input layer is set to 3*3, the stride size is set to 2, and the output channels are set to 32, 32, and 64 respectively; the convolution kernel size of the average pooling layer is 3*3, the stride is 2, and the padding is 1; The convolution kernel size of the first to fourth convolution layers in the first impulse residual block is set to 3*3, the stride size is set to 1, the padding is set to 1, and the output channels are set to 64; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channels are set to 64; The convolution kernel size of the first to fourth convolution layers in the second impulse residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 128; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 128; The convolution kernel size of the first to fourth convolution layers in the third impulse residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 256; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 256; The convolution kernel size of the first to fourth convolution layers in the second impulse residual block is set to 3*3, the stride size is set to 2, 1, 1, 1 respectively, the padding is set to 1, and the output channel is set to 512; the convolution kernel size of the fifth and sixth convolution layers is set to 1*1, the stride size is set to 1, and the output channel is set to 512; The convolution kernel size of the average pooling layer in the pulse output layer is set to 1*1, and the stride size is set to 1; the output of the fully connected layer is set to 10; Set the v_reset values ​​of all IF layers in the pulse input layer, the first to fourth pulse residual blocks, and the pulse output layer to 0, and the v_threshold values ​​to V in the S_ReLU layer. i value; Step 4.3, parse the weight file saved in step 3.4 through the torch.load function, and use the net.load_state_dict function to load the parsed convolution kernel weights and fully connected layer weights into the S_ReLU spiking neural network; Step 5: Classify the image using the S_ReLU spiking neural network after loading the transferable weight file: Step 5.1, perform Poisson encoding on the optical image to be classified; In step 5.2, the encoded image is input into the S_ReLU spiking neural network after the transferable weight file is loaded, and the classification result of the optical image is output.

2. The image classification method based on the S_ReLU spiking neural network according to claim 1, characterized in that: The S_ReLU activation function described in step 2.2 is as follows: V i =c i ·max(W i ·F i-1 +b i-1 ) Among them, F i Represents the output of the S_ReLU layer in the S_ReLU artificial neural network, Y i Indicates that the neurons in the previous convolutional layer of the S_ReLU layer in the S_ReLU artificial neural network pass through Y i The actual output value after the limiting condition, V i Represents the maximum output value scaling c of the neurons in the convolutional layer before the S_ReLU layer in the S_ReLU artificial neural network i The value after multiplication, W i-1 represents the weight matrix from the neurons in the i-1th layer to the i-th layer, b i-1 Represents the deviation value from the i-1th layer neuron to the i-th layer; the c i Represents the scaling factor for scaling the output of the convolutional layer neurons in the S_ReLU artificial neural network. The initial value is set to 1 and the value range is [0,1]. Use the Parameter function to convert it into a trainable parameter type. When the parameters of the S_ReLU artificial neural network are updated, the optimizer can automatically update it.

3. The image classification method based on the S_ReLU pulse neural network according to claim 1, characterized in that The loss function of the S_ReLU artificial neural network described in step 3 is as follows: loss(y i ,class)=-∑class·log(y i ) Among them, y i Represents the predicted value of the i-th sample output by the S_ReLU artificial neural network, class represents the label value after encoding each sample, loss(y i ,class) represents the loss value between the label and the predicted value, and log represents the logarithm with base 10.

4. The image classification method based on the S_ReLU spiking neural network according to claim 1, characterized in that: The mini-batch gradient descent optimizer described in step 3 is the mini-batch gradient descent optimizer MBGD. The parameters of the MBGD optimizer are set as follows: the momentum factor Momentum coefficient is set to 0.9, the weight decay coefficient weight_decay is set to 0.0001, the learning rate lr is set to 0.01, and other parameters use the default settings.

Images