A dynamic image recognition method based on spatiotemporal fusion attention spiking neural network

Abstract

The present invention discloses a dynamic image recognition method based on a spatiotemporal fusion attention pulse neural network, which is applied to the field of dynamic image recognition. The prior art has no way to process the noise and interference with spatiotemporal characteristics transmitted in the time direction of dynamic data with time information. The present invention draws on the predictive feature remapping theory of biology to convert the extracted spatiotemporal features into a spatiotemporal attention tensor through an attention module, and adjusts the spatiotemporal feature map of the next moment in the time direction through the membrane potential inheritance process. The present invention can efficiently generate a spatiotemporal attention tensor under a network structure with a small number of parameters, thereby improving the difficulty of the pulse neural network in extracting spatiotemporal features in spatiotemporal dynamic image recognition, and while guiding feature extraction, reduces noise interference along the time direction, so that the pulse neural network can efficiently recognize spatiotemporal images.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a dynamic image recognition technology. Background Art

[0002] Spiking Neural Network (SNN) is a third-generation neural network inspired by more sophisticated biological neural networks. Compared with traditional neural networks, SNN is driven by pulse events, carries information in the spatiotemporal relationship of pulses, and has precise discharge time and discharge position. Therefore, SNN can reduce energy consumption while robustly and efficiently integrating spatiotemporal information. However, since the information transmission in the temporal direction of naive SNN only exists inside neurons and cannot interact between neurons, it is difficult to cope with the spatiotemporal information redundancy of environmental and system noise and high dynamic background in the process of dynamic data recognition with temporal information such as actions, gestures, and dynamic pictures.

[0003] Traditional neural networks often use recurrent network structures and attention mechanisms to solve the above problems. The recurrent network structure adds recurrent connections to the original classification network, thereby increasing the model's ability to extract temporal features. However, this model also introduces a large number of parameters and noise in the temporal direction, which is not conducive to the classification of spatiotemporal dynamic data. The attention mechanism can ignore irrelevant information and focus on key information, thereby weakening the impact of data noise on the recognition process. Inserting an attention module into the model automatically searches for key information in the features by assigning different weights to feature maps and feature vectors, ignoring noise interference. However, the existing attention mechanism models mainly focus on single-dimensional or simply superimposed composite-dimensional processes such as channels, space, branches, and time. When faced with dynamic data with temporal information, there is no way to deal with the noise and interference with spatiotemporal characteristics transmitted in the temporal direction of the dynamic data, and it cannot play a role in SNN. Therefore, it is necessary to further design an attention pulse neural network recognition method suitable for processing dynamic images. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a dynamic image recognition method based on a spatiotemporal fusion attention pulse neural network, which draws on the predictive feature remapping theory of biology and efficiently recognizes dynamic data by introducing a small amount of parameters.

[0005] The technical solution adopted by the present invention is: a dynamic image recognition method based on spatiotemporal fusion attention pulse neural network, comprising:

[0006] A1. Obtain the dynamic image to be processed;

[0007] A2. Construct a spiking neural network based on spatiotemporal fusion attention;

[0008] A3. Input the dynamic image to be processed in step A1 into the pulse neural network based on spatiotemporal fusion attention constructed in step A2 to obtain the recognition result.

[0009] Step A2 constructs a pulse neural network based on spatiotemporal fusion attention, including: an input module, a spatiotemporal feature extraction module, an attention module and an output module; the input module is used to divide the input dynamic image into frames and input it into the spatiotemporal feature extraction module in chronological order, and the spatiotemporal feature extraction module is used to extract spatiotemporal features from the input data; the attention module converts the spatiotemporal features into attention tensors; the spatiotemporal feature extraction module also includes a spatiotemporal feature adjustment unit, and the spatiotemporal feature adjustment unit multiplies the attention tensor with the spatiotemporal features to obtain adjusted spatiotemporal features; the output module outputs the recognition result according to the adjusted spatiotemporal features.

[0010] Beneficial effects of the present invention: The present invention converts the spatiotemporal features extracted by the spatiotemporal feature extraction module into a spatiotemporal attention tensor through an attention module, and adjusts the spatiotemporal feature map of the next moment in the time direction during the membrane potential inheritance process, thereby suppressing noise under the guidance of historical information and strengthening the attention level of key areas. The present invention introduces a small parameter structure of the attention module to efficiently generate a spatiotemporal attention tensor, improves the difficulty of the pulse neural network in extracting spatiotemporal features in spatiotemporal dynamic image recognition, and reduces noise interference along the time direction while guiding feature extraction, so that the pulse neural network can efficiently recognize spatiotemporal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a pulse neural network structure based on spatiotemporal fusion attention.

[0012] Figure 2 It is the structure of spatiotemporal fusion attention and the process of spatiotemporal feature extraction.

[0013] Figure 3 It is a flow chart of pulse neural network training based on spatiotemporal fusion attention.

[0014] Figure 4 This is the result display of spatiotemporal fusion attention pulse neural network. DETAILED DESCRIPTION

[0015] The present invention mainly adopts the simulation experiment method for verification, and all steps and conclusions are verified to be correct on Pytorch1.7. Figure 1-4 The content of the present invention is explained in detail.

[0016] like Figure 1As shown, the pulse neural network structure involved in the present invention is a convolutional pulse neural network structure, which is composed of an input module, a spatiotemporal feature extraction module, a classification module and an attention module, as follows:

[0017] 1. Input module: The data that can be processed by the present invention include video data, event stream data and other dynamic image data. In the embodiment of the present invention, action video data is used as an example to illustrate that the video data is filtered, mirrored, rotated and other operations are performed to input the video frames into the network in time sequence.

[0018] 2. Spatiotemporal feature extraction module: It mainly uses convolutional or fully connected structures to extract features, uses membrane potential decay and attention conversion to connect spatiotemporal information, and sets the number of convolutional layers and fully connected layers according to specific tasks and requirements, such as Figure 2 As shown in the figure, in the spatial direction, they are connected by convolution or fully connected structures, and in the temporal direction, they are connected by recurrent connections and attention transformations. The input of the spatiotemporal feature extraction module is the feature map of the previous layer or the input data adjusted by the spatiotemporal attention; the spatiotemporal features obtained after nonlinear transformation are output to the subsequent layer and the attention module.

[0019] 3. Output module: Use full convolution, recurrent connection or full connection structure to count the output at each moment and jointly determine the result. In the classification embodiment of the present invention, the last layer of neurons in the output module each corresponds to a certain type of data. Within the input time window, the average output value of the neurons is counted, and the largest neuron determines the output classification result.

[0020] 4. Attention module: Use attention modules such as channel attention, spatial attention, self-attention or channel-space attention to convert the pulse features generated by neurons into attention tensors. After the attention tensor acts on the features transmitted on the recurrent connection, it is put into the same layer neurons at the next moment as input. In the embodiment of the pulse neural network of the present invention, the generated attention tensor is multiplied by the neuron membrane potential to adjust the attenuation of the neuron membrane potential.

[0021] like Figure 3 As shown, in a specific embodiment of the present invention, the pulse neural network training process based on spatiotemporal fusion attention is as follows:

[0022] S1: Build a pulse neural network. Select the specific structure of the spatiotemporal feature extraction module and the output rules of the output module according to the specific task situation, and define the loss function. In one embodiment of the present invention, for the gesture recognition task of the DVS Gesture data set, the spatiotemporal feature extraction module includes 5 convolutional layers and 2 fully connected layers, and each convolutional layer corresponds to BatchNorm and an attention module. Among them, the input module compresses the DVS data into frames according to a fixed duration and inputs them into the network; the feature extraction module is the convolutional layer and the fully connected layer in the network, which extracts image features layer by layer. The output module consists of 11 pulse neurons, corresponding to 11 classification labels. Epoch is set to 200, batch size is 16, the learning optimizer is Adam, the learning rate is decayed with an exponential decay strategy with a decay factor of 0.98, and the learning rate is initialized to 0.001.

[0023] Of course, the pulse neural network in the present invention can also be other structures, such as adjusting the number of convolutional layers and fully connected layers in the feature extraction module; or adopting a general structure such as ResNet18, VGG16, etc.

[0024] S2: Data preprocessing: Data is input into the model in chronological order. In one embodiment of the present invention, for the gesture recognition task of the DVS Gesture dataset, the input module compresses the input event data into one frame every 200 milliseconds, for a total of 10 frames. The data is input into the model in chronological order.

[0025] S3: The attention tensor adjusts the input feature map. The attention tensor generated by the spatiotemporal attention module is multiplied by the spatiotemporal features transmitted by the recurrent connection to adjust the spatiotemporal features. In some embodiments of the present invention, for the convolutional layer of the spiking neural network, the attention tensor U generated by the l layer at time t is l,t , acting on the membrane potential inheritance process of the pulse neural network, the formula is as follows:

[0026]

[0027] in, is the Hadamard product, which means the corresponding elements of the matrix are multiplied, V l,t is the membrane potential tensor of all neurons in layer l at time t, S l,t is the activation state tensor of all neurons in layer l at time t, κ τ is the membrane potential decay coefficient, i.e., the recurrent connection weight in the spiking neural network, W j is the synaptic connection weight from neuron j in layer l-1 to layer l.

[0028] S4: The spatiotemporal feature extraction module extracts the spatiotemporal features, and the output module completes the task output. The data is sequentially recognized in the time window, and the recognition results obtained from all time windows are integrated. The gesture categories in this embodiment include: clapping, waving the left hand, waving the right hand, rotating the left hand clockwise, rotating the right hand clockwise, rotating the left hand counterclockwise, rotating the right hand counterclockwise, air guitar, air drumming, rotating the arm, and a total of eleven other categories. The recognition result is the specific gesture category among these eleven.

[0029] S5: The spatiotemporal features enter the attention module and are converted into an attention tensor. In this embodiment of the present invention, we use a channel-space attention module to generate a three-dimensional attention tensor in the channel and spatial dimensions of the spatiotemporal feature map. The formula is as follows:

[0030]

[0031] in is the channel attention, It is the spatial attention. Those skilled in the art should note that only the channel attention module or the spatial attention module can be used here, and the corresponding generated attention tensor is two-dimensional or one-dimensional.

[0032] Channel attention is generated by spatially compressed channel activations:

[0033]

[0034] Among them, H and W are the height and width of the feature map respectively. and is the linear function weight, is a set of real numbers, C is the number of channels, and r is the compression factor.

[0035] Spatial attention is generated by channel-compressed spatial activation:

[0036]

[0037] in, is the convolution weight and b is the convolution bias.

[0038] The attention tensor obtained by synthesizing channel attention and spatial attention is used to adjust the spatiotemporal features transmitted in the time direction, strengthen the attention of key areas, and weaken the influence of areas irrelevant to the classification results, thereby suppressing the noise in the time direction and extracting effective features.

[0039] S6: Calculate loss value and back propagate. In a specific embodiment of the present invention, for action recognition on the DVS Gesture dataset, the mean square error (MSE) is selected as the loss function, and the formula is as follows:

[0040]

[0041] We use the gradient descent method to update the model parameters. According to the chain rule, the partial derivative formula of the loss function with respect to neuronal pulse firing and neuronal membrane potential is as follows:

[0042]

[0043]

[0044] From this, the partial derivatives of the weights in the attention module can be calculated:

[0045]

[0046] Since spatiotemporal attention consists of channel attention and spatial attention, the partial derivatives of attention and spatial attention can be expressed as:

[0047]

[0048]

[0049] Therefore, we can calculate the gradient of the required parameters based on the above partial derivatives and then update the parameters.

[0050] S7: Test network accuracy: If the network accuracy is stable, stop network training. When the network accuracy is stable or the maximum number of training times is reached, stop training and save the model with the best accuracy in the test set. In this embodiment of the present invention, we save the model with the highest classification accuracy in the test set as the final DVS Gesture classifier required to recognize gesture actions.

[0051] The maximum number of training times is generally set to 200.

[0052] like Figure 4 As shown in the figure, in DVS Gesture and Cifar10-DVS, after adding attention, the network model achieved 5.03% and 1.31% improvement from the training curve without attention. For the two datasets, the number of parameters increased by 4.6% and 0.9% respectively after adding attention.

[0053] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A dynamic image recognition method based on spatiotemporal fusion attention pulse neural network, characterized in that: include: A1. Obtain the dynamic image to be processed; Step A2 constructs a pulse neural network based on spatiotemporal fusion attention, including: an input module, a spatiotemporal feature extraction module, an attention module and an output module; the input module is used to divide the input dynamic image into frames and input it into the spatiotemporal feature extraction module in time sequence, and the spatiotemporal feature extraction module is used to extract spatiotemporal features from the input data; the attention module converts the spatiotemporal features into attention tensors; the spatiotemporal feature extraction module also includes a spatiotemporal feature adjustment unit, and the spatiotemporal feature adjustment unit adjusts the spatiotemporal features according to the attention tensor; the output module outputs the recognition result according to the adjusted spatiotemporal features; The spatiotemporal feature adjustment unit multiplies the attention tensor by the spatiotemporal feature to obtain an adjusted spatiotemporal feature; A2. Construct a spiking neural network based on spatiotemporal fusion attention; A3. Input the dynamic image to be processed in step A1 into the pulse neural network based on spatiotemporal fusion attention constructed in step A2 to obtain the recognition result.

2. The dynamic image recognition method based on spatiotemporal fusion attention pulse neural network according to claim 1 is characterized in that: The spatiotemporal feature extraction module includes several convolutional layers and several fully connected layers, and the cross-image features are extracted layer by layer through the convolutional layers. Each convolutional layer corresponds to a BatchNorm layer and an attention module.

3. The dynamic image recognition method based on spatiotemporal fusion attention pulse neural network according to claim 2 is characterized in that: The spatiotemporal feature adjustment unit adjusts the spatiotemporal feature map of the next moment in the time direction through the membrane potential inheritance process; specifically: for the convolution layer of the spatiotemporal feature extraction module, the attention tensor U generated by the l layer at time t is l,t , acting on the membrane potential inheritance process of the pulse neural network, the formula is as follows: in, is the Hadamard product, which means the corresponding elements of the matrix are multiplied, V l,t is the membrane potential tensor of all neurons in layer l at time t, S l,t is the activation state tensor of all neurons in layer l at time t, κ τ is the membrane potential decay coefficient, i.e., the recurrent connection weight in the spiking neural network, W j is the synaptic connection weight from neuron j in layer l-1 to layer l.

4. The dynamic image recognition method based on spatiotemporal fusion attention pulse neural network according to claim 3 is characterized in that: The attention module converts the spatiotemporal features into attention tensors, which are specifically: attention tensors of channel and spatial dimensions. The expression of attention tensor is: in, is the channel attention, It is spatial attention.

5. The dynamic image recognition method based on spatiotemporal fusion attention pulse neural network according to claim 4 is characterized in that: Channel attention is generated by spatially compressed channel activations: Among them, H and W are the height and width of the feature map respectively. and is the linear function weight, is a set of real numbers, and r is the compression coefficient.

6. The method for dynamic image recognition based on spatiotemporal fusion attention pulse neural network according to claim 5, characterized in that: Spatial attention is generated by channel-compressed spatial activation: in, is the convolution weight and b is the convolution bias.

Images