Image recognition method based on spiking neural network fusion of probability and attention mechanisms

Abstract

This invention discloses a spiking neural network image recognition method based on the fusion of probability and attention mechanisms, belonging to the fields of computer vision and neuromorphic computing. The method uses probabilistic information calculated based on membrane potential and discharge threshold to continuously represent non-differentiable pulse discharge activity, and combines probability with an adaptive discharge threshold to generate output information. Simultaneously, an attention mechanism is used to further refine the input information received by the spiking neurons and incorporate membrane potential. Spiking neurons are constructed using this method to form a deep spiking neural network, and the parameters in the spiking neural network are iteratively updated using a backpropagation algorithm over time. Compared to existing technologies, the method of this invention can accurately identify the category of static images and event data based on asynchronous pulses in scenarios with extremely low time delay.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and neuromorphic computing, and specifically relates to a spiking neural network image recognition method based on the fusion of probability and attention mechanisms. Background Technology

[0002] Artificial Neural Networks (ANNs) based on deep learning have benefited from the rapid development of data and hardware performance, and are widely used in fields such as image recognition, object detection, natural language processing, and autonomous driving. However, high-performance ANNs require extremely high computational resources, which greatly limits their practical deployment on resource-constrained devices.

[0003] Spiking Neural Networks (SNNs) transmit information through discrete 0-1 pulses. During computation, they can replace cumbersome multiply-accumulate operations with accumulation operations, significantly reducing computational resource overhead. Furthermore, SNNs maintain a certain membrane potential state that changes continuously over time, allowing them to capture the temporal characteristics of the data. Compared to ANNs, SNNs have attracted considerable attention due to their low power consumption and potential high-performance computing capabilities. However, the activity of spiking neurons continuously accumulating membrane potential and generating pulses when the membrane potential reaches a firing threshold does not satisfy the condition of continuous differentiability, making it impossible to directly optimize using the backpropagation algorithm and hindering the development of high-performance SNNs.

[0004] Currently, common methods for obtaining high-performance SNNs include ANN-to-SNN conversion and surrogate gradient-based training. The ANN-to-SNN method pre-trains an ANN and then replaces its internal activation function with spiking neurons, thus converting it into an SNN. While this method avoids non-differentiable spiking activity, the resulting SNN cannot extract latent temporal features from the data, limiting its applications. The surrogate gradient-based method uses a surrogate gradient to approximate the derivative of the spiking activity during backpropagation, allowing the backpropagation algorithm to optimize the network parameters. Although this method can directly use the backpropagation algorithm to optimize the network parameters, the difference between the surrogate gradient and the true gradient value means its performance cannot compare to that of an ANN. Therefore, a method that is both easy to train and capable of producing high-performance SNNs is needed. Summary of the Invention

[0005] To address the issues of errors between surrogate gradients and true gradients during SNN training, resulting in weak performance of the trained SNN, this invention provides a spiking neural network image recognition method based on a fusion of probability and attention mechanisms. By using continuous probability and adaptive firing thresholds, the non-differentiable spiking activity of spiking neurons is smoothed, allowing direct optimization of the network using a time-varying backpropagation algorithm. Furthermore, to enhance the feature extraction capability of spiking neurons from input information, this invention refines the input using an attention mechanism, further improving the overall network performance. The technical problem addressed by this invention is solved as follows:

[0006] The image recognition method based on the fusion of probability and attention mechanisms using spiking neural networks includes the following steps:

[0007] Step 1: The image to be recognized is preprocessed by random horizontal flipping, random cropping, and regularization, and then input into the spiking neural network.

[0008] Step 2: The convolutional and pooling layers in the spiking neural network extract shallow features I from the image;

[0009] Step 3: The feature I obtained in Step 2 is passed as input to the spiking neuron based on the fusion of probability and attention mechanism. The input is refined by combining the attention mechanism and fused into the membrane potential H of the spiking neuron.

[0010] Step 4: Calculate the probability P(s=1) that the spiking neuron will generate a pulse based on its membrane potential and firing threshold. Combine the probability with the firing threshold V. th Calculate its output and reset the membrane potential;

[0011] Step 5: Repeat steps 2 to 4 multiple times to extract high-dimensional features;

[0012] Step 6: The extracted high-dimensional features are passed as input into the fully connected layer to obtain the final classification result O;

[0013] Step 7: Calculate the loss function using the final classification result O and the true image label, and continuously optimize the parameters in the SNN using the backpropagation algorithm over time;

[0014] Step 8: Repeat steps 1 to 7 to continue training the SNN until its performance converges.

[0015] Step 9: Input the image to be recognized into the trained SNN to obtain the recognition result.

[0016] Step 3 of the present invention includes:

[0017] Step 3-1: Using an attention mechanism, evaluate the importance of the information contained in feature I extracted in step 2, and obtain the importance score d for feature I:

[0018]

[0019] Where conv(.) represents the convolution operation. Represents the sigmoid function;

[0020] Step 3-2: Perform a dot product operation between the d obtained in Step 3-1 and the initial input feature I to obtain the refined feature.

[0021]

[0022] Where ⊙ represents element-wise multiplication;

[0023] Step 3-3, Refined Features The membrane potential state H(t) of the neuron at time t is updated by accumulating the input into the spiking neuron's membrane potential H:

[0024]

[0025] Where τ represents the time constant of the spiking neuron's membrane potential, controlling the degree of change of the spiking neuron's membrane potential over time.

[0026] Step 4 of the present invention includes:

[0027] Step 4-1, based on the Boltzmann distribution, according to the neuron membrane potential H and the firing threshold V th Calculate the probability P(s=1) that it will discharge and generate a pulse:

[0028]

[0029] Where E(s) represents the probability that the neuron will be in state s. This represents the sigmoid function, where s=1 indicates that a pulse will be generated, and s=0 indicates that no pulse will be generated.

[0030] Step 4-2, combine the neuron firing probability P(s=1) and firing threshold V obtained in step 4-1. th Multiply by each other to obtain the output value S:

[0031] S = P(s = 1) · V th ,

[0032] Where · represents the product operation;

[0033] Step 4-3: Limit the output range of the neuron so that its output value S does not exceed its membrane potential H.

[0034] S = min(S, H),

[0035] Where min(S, H) means taking the minimum value between the output value S and the membrane potential H as the output;

[0036] Step 4-4: After the neuron generates an output, its membrane potential H is reset.

[0037] H = HS.

[0038] Step 7 of the present invention includes:

[0039] Step 7-1, the final classification result O = (o1, o2, ..., o m This contains the average output of the output layer neurons over multiple time steps, where m represents the number of classes present, and O is the sum of the true labels Y = (y1, y2, ..., y3). m The loss value L(O,Y) is calculated using the cross-entropy function:

[0040] Step 7-2: Based on the loss value calculated in Step 7-1, update the parameters in the network using the backpropagation algorithm performed over time.

[0041]

[0042] Among them, w n H represents the weights between the (n-1)th and nth layers. t，n and S t，n Let represent the membrane potential and output value of the neuron in the nth layer at time t, and T represent the total time step of the SNN; for a neuron in the nth layer, if the output value is not less than the membrane potential, the derivative of its output value with respect to the membrane potential at time t can be calculated as:

[0043]

[0044] in This represents the firing threshold of the spiking neuron in the nth layer; if the output value is less than the membrane potential, the derivative of its output value with respect to the membrane potential at time t can be calculated as:

[0045]

[0046] Based on the above derivative calculation formula, the optimization of the SNN intrinsic parameters can be achieved through an automatic differentiation framework;

[0047] Step 7-3, based on the pre-set upper limit of the discharge threshold. and the lower realm The firing threshold of each layer of spiking neurons is adjusted based on a backpropagation algorithm optimized over time.

[0048]

[0049] in Indicates when the discharge threshold Below At that time, Adjusted to When the discharge threshold Higher than At that time, Adjusted to

[0050] This invention proposes a spiking neural network image recognition method based on the fusion of probability and attention mechanisms, with the following advantages:

[0051] This invention generates output by combining the firing probability and firing threshold of spiking neurons, thus continuousizing the pulse generation process of spiking neurons. It can directly use the backpropagation algorithm over time to optimize the parameters in the network. This invention utilizes convolution operations and the sigmoid function to construct an attention mechanism, further refining the input received by spiking neurons, improving the ability of spiking neurons to recognize input information, and improving the overall performance of image recognition using SNN. Attached Figure Description

[0052] Figure 1 The accompanying drawings are the abstract of this invention and the overall flowchart of the method of this invention;

[0053] Figure 2 This is a flowchart illustrating the overall SNN training and testing process according to an embodiment of the present invention.

[0054] Figure 3 This is a flowchart illustrating the SNN image input reception and training process according to an embodiment of the present invention.

[0055] Figure 4 This is a detailed input diagram illustrating the attention mechanism in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram illustrating the combination of probability mechanism and threshold to generate output in an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram illustrating the change in accuracy during the training process according to an embodiment of the present invention; Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] The spiking neural network image recognition method based on the fusion of probability and attention mechanisms shown in this embodiment of the invention uses the ResNet20 network structure, combines the probability and attention mechanism fusion method with the traditional Leaky Integrate-and-Fire (LIF) spiking neuron model, sets the total time step T of the SNN to 4, and sets the initial firing threshold V of each layer of spiking neurons to... th Set to 1.0 and conduct recognition tests on the CIFAR10 dataset. The overall training and testing process of SNN is as follows: Figure 2 As shown, the network structure for inputting images into the SNN for recognition is as follows: Figure 3 As shown. The specific construction steps of image recognition in this embodiment of the invention are as follows:

[0060] Step 1: Perform data augmentation on the training set images in the CIFAR10 dataset using random horizontal flipping, random cropping, and regularization;

[0061] Step 2: Input the enhanced image obtained in Step 1 into the SNN with the ResNet20 structure. The convolutional layer and pooling layer in the SNN extract the shallow features I in the image.

[0062] Step 3: The feature I obtained in Step 2 is passed as input to the spiking neuron based on the fusion of probability and attention mechanisms. The input is refined by combining the attention mechanism and fused into the membrane potential H of the spiking neuron. The process is as follows: Figure 4 As shown;

[0063] Step 4: Calculate the probability P(s=1) that the spiking neuron will generate a pulse based on its membrane potential and firing threshold. Combine the probability with the firing threshold V. th The process of calculating its output and resetting the membrane potential is as follows: Figure 5 As shown;

[0064] Step 5: Repeat steps 2 to 4 multiple times to extract high-dimensional features;

[0065] Step 6: The extracted high-dimensional features are passed as input into the fully connected layer to obtain the final classification result O;

[0066] Step 7: Calculate the loss function using the final classification result O and the true image label, and continuously optimize the parameters in the SNN using the backpropagation algorithm over time;

[0067] Step 8: Repeat steps 1 to 7 to continue training the SNN until its performance converges.

[0068] Step 9: Input the image to be recognized into the trained SNN to obtain the recognition result; in this embodiment, the final recognition accuracy of 10,000 images in the CIFAR10 dataset is 94.88%. The change in recognition accuracy during the training process of the SNN is as follows: Figure 6 As shown.

[0069] Step 3 of the present invention includes:

[0070] Step 3-1: Using an attention mechanism, evaluate the importance of the information contained in feature I extracted in step 2, and obtain the importance score d for feature I:

[0071]

[0072] Where conv(.) represents the convolution operation. This represents the sigmoid function, which uses a convolution kernel of size 3 during implementation, sets the convolution padding size to 1, and sets the stride between each convolution to 1.

[0073] Step 3-2: Perform a dot product operation between the d obtained in Step 3-1 and the initial input feature I to obtain the refined feature.

[0074]

[0075] Where ⊙ represents element-wise multiplication;

[0076] Step 3-3, Refined Features The membrane potential state H(t) of the neuron at time t is updated by accumulating the input into the spiking neuron's membrane potential H:

[0077]

[0078] Where τ represents the time constant of the spiking neuron's membrane potential, controlling the degree of change of the spiking neuron's membrane potential over time. During implementation, the following settings are configured: It is 0.25.

[0079] Step 4 of the present invention includes:

[0080] Step 4-1, based on the Boltzmann distribution, according to the neuron membrane potential H and the firing threshold V th Calculate the probability that it will discharge and generate a pulse:

[0081] E(s)=-s(HV th ),

[0082]

[0083] P(s=1=1-σ(V)th -H),

[0084] Where E(s) represents the probability that the neuron will be in state s. This represents the sigmoid function, where s=1 indicates that a pulse will be generated, and s=0 indicates that no pulse will be generated.

[0085] Step 4-2, combine the neuron firing probability P(s=1) and firing threshold V obtained in step 4-1. th Multiply by each other to obtain the output value S:

[0086] S = P(s = 1) · V th ,

[0087] Where · represents the product operation;

[0088] Step 4-3: Limit the output range of the neuron so that its output value S does not exceed its membrane potential H.

[0089] S = min(S, H),

[0090] Where min(S, H) means taking the minimum value between the output value S and the membrane potential H as the output;

[0091] Step 4-4: After the neuron generates an output, its membrane potential H is reset.

[0092] H = HS.

[0093] Step 7 of the present invention includes:

[0094] Step 7-1, the final classification result O = (o1, o2, ..., o2) m This contains the average output of the output layer neurons over multiple time steps, where m represents the number of classes present. In this embodiment, m = 10, and 0 corresponds to the true label Y = (y1, y2, ..., y3). m The loss value L(O, Y) is calculated using the cross-entropy function:

[0095]

[0096] Step 7-2: Based on the loss value calculated in Step 7-1, update the parameters in the network using the backpropagation algorithm performed over time.

[0097]

[0098] Among them, w n H represents the weights between the (n-1)th and nth layers. t，n and S t，nLet represent the membrane potential and output value of the neuron in the nth layer at time t, and T represent the total time step of the SNN. For a neuron in the nth layer, when its output value at time t is not less than its membrane potential, the derivative of its output value at time t with respect to the membrane potential can be calculated as follows:

[0099]

[0100] in Let represent the firing threshold of the spiking neuron in the nth layer; when its output value at time t is less than the membrane potential, the derivative of its output value at time t with respect to the membrane potential can be calculated as:

[0101]

[0102] Based on the above derivative calculation formula, the SNN intrinsic parameters are optimized using the automatic differentiation framework PyTorch. In this embodiment, the parameters are optimized using the cross-entropy loss function and a stochastic gradient descent optimizer with a momentum of 0.9, combined with the PyTorch framework. The initial learning rate is set to 0.1, and the learning rate is reduced to 1 / 10 of the previous rate every forty iterations, for a total of 120 iterations.

[0103] Step 7-3, based on the pre-set upper limit of the discharge threshold. and the lower realm The firing threshold of each layer of spiking neurons is adjusted based on a backpropagation algorithm optimized over time.

[0104]

[0105] in Indicates when the discharge threshold Below At that time, Adjusted to When the discharge threshold Higher than At that time, Adjusted to In this embodiment, an upper bound for the threshold is set. Set the lower bound of the threshold to 10. It is 0.1.

[0106] This invention achieves image recognition by using a spiking neural network based on the fusion of probability and attention mechanisms. This not only improves the accuracy of image recognition using SNN, but also reduces the time step required for image recognition using SNN.

[0107] This invention provides a spiking neural network image recognition method based on the fusion of probability and attention mechanisms. The automatic differentiation framework, loss function, optimizer and specific parameter values ​​used in the above embodiments are specific implementations combined with the principles of this invention. The scope of protection of this invention is not limited to such specific implementations.

Claims

1. A spiking neural network image recognition method based on the fusion of probability and attention mechanisms, characterized in that, Includes the following steps: Step 1: The image to be recognized is preprocessed by random horizontal flipping, random cropping, and regularization, and then input into the spiking neural network. Step 2: The convolutional and pooling layers in the spiking neural network extract shallow features I from the image; Step 3: The feature I obtained in Step 2 is passed as input to the spiking neuron based on the fusion of probability and attention mechanisms. The input is refined by combining the attention mechanism and fused into the membrane potential H of the spiking neuron. The specific steps are steps 3-1 to 3-3: Step 3-1: Using an attention mechanism, evaluate the importance of the information contained in feature I extracted in step 2, and obtain the importance score d for feature I: Where conv(.) represents the convolution operation. Represents the sigmoid function; Step 3-2: Perform a dot product operation between the d obtained in Step 3-1 and the initial input feature I to obtain the refined feature. Where ⊙ represents element-wise multiplication; Step 3-3, Refined Features The membrane potential state H(t) of the neuron at time t is updated by accumulating the input into the spiking neuron's membrane potential H: Where τ represents the time constant of the membrane potential of the spiking neuron, which controls the degree of change of the membrane potential of the spiking neuron with time; Step 4: Calculate the probability P(s=1) that the spiking neuron will generate a pulse based on its membrane potential and firing threshold. Combine the probability with the firing threshold V. th Calculate its output and reset the membrane potential; Step 5: Repeat steps 2 to 4 multiple times to extract high-dimensional features; Step 6: The extracted high-dimensional features are passed as input into the fully connected layer to obtain the final classification result O; Step 7: Calculate the loss function using the final classification result O and the true image label, and continuously optimize the parameters in the SNN using the backpropagation algorithm over time; Step 8: Repeat steps 1 to 7 to continue training the SNN until its performance converges. Step 9: Input the image to be recognized into the trained SNN to obtain the recognition result.

2. The method according to claim 1, characterized in that... Step 4 includes: Step 4-1, based on the Boltzmann distribution, according to the neuron membrane potential H and the firing threshold V th Calculate the probability P(s=1) that it will discharge and generate a pulse: E(s)=-s(H-V th ), P(s=1)=1-σ(V th -H), Where E(s) represents the probability that the neuron will be in state s. This represents the sigmoid function, where s=1 indicates that a pulse will be generated, and s=0 indicates that no pulse will be generated. Step 4-2, combine the neuron firing probability P(s=1) and firing threshold V obtained in step 4-1. th Multiply by each other to obtain the output value S: S=P(s=1)·V th , Where · represents the product operation; Step 4-3: Limit the output range of the neuron so that its output value S does not exceed its membrane potential H. S = min(S, H), Where min(S, H) means taking the minimum value between the output value S and the membrane potential H as the output; Step 4-4: After the neuron generates an output, its membrane potential H is reset. H = HS.

3. The method according to claim 1, characterized in that... Step 7 includes: Step 7-1, the final classification result O = (o1, o2, ..., o m This contains the average output of the output layer neurons over multiple time steps, where m represents the number of classes present, and O is the sum of the true labels Y = (y1, y2, ..., y3). m The loss value L(O,Y) is calculated using the cross-entropy function: Step 7-2: Based on the loss value calculated in Step 7-1, update the parameters in the network using the backpropagation algorithm performed over time. Among them, w n H represents the weights between the (n-1)th and nth layers. t,n and S t,n Let represent the membrane potential and output value of the neuron in the nth layer at time t, and T represent the total time step of the SNN; for the neuron in the nth layer, if the output value is not less than the membrane potential value, then the derivative of its output value with respect to the membrane potential at time t is calculated as follows: in This represents the firing threshold of the spiking neuron in the nth layer; if the output value is less than the membrane potential, then the derivative of its output value with respect to the membrane potential at time t is calculated as follows: Based on the above derivative calculation formula, the optimization of the SNN intrinsic parameters can be achieved through an automatic differentiation framework; Step 7-3, based on the pre-set upper limit of the discharge threshold. and the lower realm The firing threshold of each layer of spiking neurons is adjusted based on a backpropagation algorithm optimized over time. in Indicates when the discharge threshold Below At that time, Adjusted to When the discharge threshold Higher than At that time, Adjusted to

Images