A behavior recognition method based on SimAM attention mechanism

Abstract

The present invention discloses a behavior recognition method based on SimAM attention mechanism, which makes full use of the parameter-free characteristics of the parameter-free attention module SimAM and organically combines it with a dual-stream network model. It can reduce network parameters and training difficulty while improving recognition accuracy, and can adaptively adopt different fusion methods according to the different high- and low-layer feature contents (i.e., resolution) of the input video to improve the perception of details, thereby obtaining a high-performance behavior recognition algorithm, more effectively optimizing the dual-stream network model, and increasing the engineering and accuracy of the behavior recognition algorithm of this scheme.

Description

Technical Field

[0001] The patent of this invention belongs to computer vision technology in the field of artificial intelligence, and in particular, relates to a behavior recognition method based on the SimAM attention mechanism. Technical Background

[0002] In the era of Internet big data, more and more videos are shared. How to quickly extract information from massive video resources has extremely high research and application value. Human behavior recognition in videos has gradually become a major research hotspot in the field of computer vision, and has been widely used in public video surveillance, human-computer interaction, scientific cognition, medical rehabilitation and other fields. In recent years, with the continuous improvement of computer computing power, deep learning has been deeply developed, and behavior recognition algorithms based on deep learning have gradually emerged.

[0003] At present, behavior recognition algorithms based on deep learning are mainly divided into three categories:

[0004] (1) Single-stream network model method. The most widely used network model method is the three-dimensional convolutional neural network, or 3D CNN, proposed by Shuiwang Ji et al. Its convolutional network structure includes a hard connection layer, three three-dimensional convolutional layers, two pooling layers, and a fully connected layer.

[0005] (2) Two-stream network model method. Inspired by the two-stream hypothesis in neuroscience, Simonyan et al. first proposed the two-stream network model in 2014. The two-stream hypothesis assumes that the visual cortex has a ventral pathway and a dorsal pathway. The ventral pathway is sensitive to the shape and color of the target, and the dorsal pathway is sensitive to the spatial transformation caused by the movement of the target. The two-stream network architecture imitates the visual cortex to establish temporal information pathways and spatial information pathways, and uses independent parallel CNN networks to extract the temporal and spatial features of the video, and finally performs feature fusion.

[0006] (3) Multi-stream network model method. This network model method is an extension of the two-stream network model. Other CNN networks are added to the two-stream network model to extract different features.

[0007] Looking at the development of deep learning, major breakthroughs and discoveries are often proposed on the basis of certain neuroscience theories. In visual neuroscience, the neurons with the richest information are usually those that show unique firing patterns of surrounding neurons. In addition, active neurons may also inhibit the activity of surrounding neurons, a phenomenon known as spatial inhibition. In visual processing, neurons that show obvious spatial inhibition effects should be given priority. The simplest way to find these neurons is to measure the linear separability between a target neuron and other neurons. Recently, Linxiao Yang et al. proposed a parameter-free attention module (SimAM, A Simple, Parameter Free Attention Module, "SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks") for convolutional neural networks based on the spatial inhibition effect of such neurons. The energy function of each neuron is calculated to study the importance of each neuron and form the attention of the corresponding neuron.

[0008] The two-stream network model method mentioned above has good generalization and scalability. However, people have gradually discovered that the two-stream network model uses late fusion to fuse the recognition results of the time stream and the space stream, and has poor perception of details. Therefore, people have proposed many improved models based on this model, such as adding an attention mechanism module to the two-stream network model to form a "two-stream network model + attention mechanism" structure. The existing attention mechanisms include channel attention, spatial attention, etc. The advantage of combining the two-stream network model with these attention mechanisms is that it effectively improves the recognition accuracy. The disadvantage is that it greatly increases the complexity and training difficulty of this type of model, affecting its further application. Summary of the invention

[0009] The technical problem to be solved by the present invention is to address the defects of the existing behavior recognition algorithm involved in the technical background, and based on the existing artificial intelligence and machine vision technology, a behavior recognition method based on the SimAM attention mechanism is proposed.

[0010] The principle is:

[0011] The existing two-stream network model is used to extract video features, which are further enhanced by combining with the lightweight SimAM attention mechanism. Different feature fusion methods are used according to the different high-level and low-level feature contents in the video. This improves the recognition accuracy while reducing network parameters and training difficulty, and improves the perception of details.

[0012] A behavior recognition method based on SimAM attention mechanism, including:

[0013] Sampling the input video, randomly extracting a sample feature map, the sample feature map includes: a single-frame first video image and a plurality of consecutive frames of second video images, scaling and cropping the sample feature map;

[0014] Spatial stream feature extraction and temporal stream feature extraction: decompose the first video image into three primary color channels to obtain a three primary color channel image; calculate the stacked optical flow of the second video image; input a single frame of the three primary color channel image into the spatial stream CNN to extract the spatial stream feature map, and input the stacked optical flow into the temporal stream CNN to extract the temporal stream feature map;

[0015] Calculate the SimAM attention of the spatial stream CNN and the temporal stream CNN respectively, and fuse them with the corresponding spatial stream feature map and temporal stream feature map into the spatial stream attention fusion feature map and the temporal stream attention fusion feature map respectively;

[0016] Feature early fusion and output: For input videos with more high-level features, early fusion is used to fuse features. More high-level features are single-frame input videos with a resolution greater than 1080p. Finally, the fused features of the spatial stream attention fusion feature map and the temporal stream attention fusion feature map are input into the fully connected layer for classification and the results are output through the softmax function.

[0017] Results: Late fusion and output: For input videos with more low-level features, late fusion is used to fuse features. For more low-level features, the resolution of a single-frame input video is less than 1080p. In the spatial stream CNN, the spatial stream attention fusion feature map is input into the fully connected layer for classification. In the temporal stream CNN, the temporal stream attention fusion feature map is input into the fully connected layer for classification, and the mean fusion output value is used. Finally, the output value is input into the softmax function to output the result.

[0018] Low-level features refer to features such as borders, contours, positions, and sizes, while high-level features refer to features such as textures. Generally, the higher the resolution, the more high-level features it contains.

[0019] Furthermore, the specific steps of scaling and cropping the sample feature map are: scaling the sample feature map to a size of N*N, and then randomly cropping the sample feature map to a size of N′*N′, wherein: N represents the pixel size of the image, and N′ represents the pixel size of the image after random cropping.

[0020] Furthermore, the three-primary color channel image acquisition method specifically includes the following steps:

[0021] Decompose the scaled and cropped first video image into three channels: red, green, and blue, and obtain the image X under the three channels z (x,y),

[0022] Wherein, z represents the zth channel, z is an integer greater than or equal to 1 and less than or equal to 3, and the 1st, 2nd, and 3rd channels represent the red, green, and blue channels respectively;

[0023] x is the horizontal coordinate of the pixel in the image, and y is the vertical coordinate of the pixel in the image.

[0024] Furthermore, calculating the stacked optical flow specifically includes the following steps:

[0025] The optical flow is regarded as a set of displacement vector fields between consecutive frames t and t+1; the point (u, v) in the tth frame is recorded, and the optical flow of the tth frame is I τ , and its calculation formula is as follows:

[0026]

[0027]

[0028] Where: u=[1;w], v=[1:h], k=[1;L]; w is the width of the second video image, h is the height of the second video image, and L is the number of frames of the second video image.

[0029] Finally, we get a stacked optical flow of N′*N′*2L.

[0030] Further, extracting the spatial stream feature map or the temporal stream feature map specifically includes the following steps:

[0031] Step 1: Perform a filling operation on the input three-primary color channel image or the stacked optical flow, expanding from N'*N' to (N'+7)*(N'+7), and fill the expanded part with 0 value; then use the 7*7*96 convolution kernel with a step size of 2 to perform convolution on the three-primary color channel image or the stacked optical flow to generate a feature map;

[0032] Step 2: Use ReLU to perform linear rectification on the feature map generated in step 1, where the formula of the ReLU function is:

[0033] ReLU(m)=max(0,m)

[0034] Where m is the independent variable; max refers to the maximum value between 0 and m;

[0035] Step 3: Perform a maximum pooling operation on the linearly rectified feature map with a pooling size of 2*2;

[0036] Step 4: Convolve the feature map generated in step 3, set the convolution kernel size to 5*5*256, the step size to 2, and the pooling size to 2*2;

[0037] Step 5: Convolve the feature map generated in step 4, set the convolution kernel size to 3*3*512, and the step size to 1;

[0038] Step 6: Convolve the feature map generated in step 5, set the convolution kernel size to 3*3*512, and the step size to 1;

[0039] Step 7: Convolve the feature map generated in step 6, set the convolution kernel size to 3*3*512, the step size to 1, and the pooling size to 2*2;

[0040] Finally, 512 spatial stream feature maps S are obtained. p , regarded as 512 channels, where: p∈[1,512], each S p There are (N′ / 32)*(N′ / 32) neurons;

[0041] Time flow feature map S q , regarded as 512 channels, where: q∈[1,512], each S q There are (N′ / 32)*(N′ / 32) neurons.

[0042] Furthermore, the SimAM attention of the spatial stream CNN and the temporal stream CNN are calculated respectively, and compared with the corresponding spatial stream feature map S p And the time flow feature map S q They are fused into the spatial stream attention fusion feature map S′ p And the time stream attention fusion feature map S′ q The specific steps include:

[0043] Step 1: SimAM attention calculation of spatial stream CNN and temporal stream CNN:

[0044] Calculate the energy function e of each neuron r :

[0045]

[0046] Where r represents the target neuron in a single input channel; q i Represents other neurons in the input channel, i is the serial number; w r Linear transformation of weights; b r is the linear transformation of the bias; M = (N' / 32)*(N' / 32) is the number of other neurons on the channel; γ is a variable; λ is a coefficient;

[0047] Use the following formula to calculate w r :

[0048]

[0049] Use the following formula to calculate b r :

[0050]

[0051] μ r It represents the average value, and its calculation formula is:

[0052]

[0053] σ r 2 It represents the variance, and its calculation formula is:

[0054]

[0055] Calculate the minimum neuron energy

[0056]

[0057] The smaller the energy of each neuron, the more important the neuron is compared with other neurons in the same channel. The inverse of the minimum neuron energy is is the weight of the neuron;

[0058] One neuron corresponds to one neuron energy The energy of all neurons in a single channel constitutes the energy matrix E of the channel. The attention weight matrix E′ of the channel is obtained by taking the inverse of each element in the energy matrix E and normalizing it with the sigmoid function. Its calculation formula is:

[0059]

[0060] Step 2: The SimAM attention of the spatial stream CNN and the temporal stream CNN are fused with the spatial stream feature map and the temporal stream feature map respectively, including the following steps:

[0061] The calculation formula of the fused feature map S′ is:

[0062] S′=S·E′

[0063] Among them, S represents the spatial stream feature map S p Or the spatial stream feature map S q One of them, S′ represents the spatial stream attention fusion feature map S′ p Or the time stream attention fusion feature map S′ q One of them.

[0064] Furthermore, the dual-stream attention fusion feature map fused by the spatial stream attention fusion feature map and the temporal stream attention fusion feature map specifically includes:

[0065] Attention fusion feature map S′ for spatial stream p And the time stream attention fusion feature map S′ q Cascade fusion,

[0066]

[0067]

[0068] in: The feature map S′ is output by the spatial stream network p A certain position (i, j, d) on the surface, s represents space, The feature map S′ output by its corresponding temporal stream network q A certain position (i, j, d) on the graph, t represents the time, To obtain a certain position on the feature map for cascade fusion, To obtain a certain position (i, j, 2d-1) on the feature map by cascade fusion, d represents the dth feature map;

[0069] Get the cascade fusion feature map: the size is (N' / 32)*(N' / 32)*1024;

[0070] The above cascade fusion feature map is subjected to three layers of convolution successively, and the convolution kernel sizes are 3*3*512, 3*3*1024, and 1*1*512 respectively. The last layer of convolution plays a role in reducing the dimension, and finally the dual-stream attention fusion feature map S is obtained. d The size is (N′ / 32)*(N′ / 32)*512.

[0071] Furthermore,

[0072] Fully connected layers are classified into:

[0073] output=f(w T A+b)

[0074] Where w is the weight vector, T represents transpose, A is the input vector, b is the bias vector, and the output dimension is H*1;

[0075] The input vector A is the dual-stream attention fusion feature map S d , time stream attention fusion feature map S′ q Or the spatial stream attention fusion feature map S′ p One of;

[0076] The obtained output is one of the output s_output of the spatial stream network model, the output t_output of the temporal stream network model, or the output d_output of the dual stream network model;

[0077] Further;

[0078] Solving the final behavior recognition results specifically includes:

[0079] The softmax function expression is:

[0080]

[0081] Where: output i Represents the i-th element in output, output k represents the Kth element in output, P represents probability, exp() represents the exponential function with natural logarithm e as base; H is the number of elements;

[0082] The output fed into softmax is d_output, or the mean of s_output and t_output;

[0083] The behavior classification label corresponding to the element with the highest probability among the H elements is the final behavior recognition result.

[0084] Further: the spatial stream CNN or spatial stream CNN is trained using a stochastic gradient descent algorithm;

[0085] Calculate the gradient of the output layer, It represents the gradient of the Kth node in the output layer (Kth layer), and its calculation formula is as follows:

[0086]

[0087] Among them k represents the output of the kth node in the Kth layer, t k Represents the label of the k-th node in the K-th layer;

[0088] Calculate the gradient of the hidden layer, represents the gradient of the ith node in the ith hidden layer, and its calculation formula is as follows:

[0089]

[0090] Among them i represents the label value of the ith node in the ith hidden layer, represents the gradient of the i-th node in the previous layer (J-th layer) of the i-th hidden layer, w ab Represents the value of the ath row and bth column of the weight matrix; and updates the parameters based on the calculated gradient.

[0091] The main innovative points of the present invention are as follows:

[0092] (1) Combining the two-stream network model with the lightweight SimAM attention mechanism;

[0093] (2) Different feature fusion methods are used according to the different high-level and low-level feature contents of the video.

[0094] Compared with the prior art, the present invention has the following advantages: the behavior recognition method based on the SimAM attention mechanism proposed in the present invention uses the CNN-M-1024 network in the spatial stream and the temporal stream respectively, which can fully extract the spatial information and temporal information of the video, and integrate with the SimAM attention mechanism to improve the accuracy of video behavior recognition without increasing network parameters, and achieve a balance between model accuracy and training complexity. The present invention adaptively adopts different fusion methods according to the different high- and low-level feature contents of the input video, further improving the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 This is a flow chart of an embodiment of the present invention using an early fusion method;

[0096] Figure 2 This is a flow chart of a late fusion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0097] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0098] The present invention can be implemented in many different forms and should not be considered to be limited to the embodiments described herein. On the contrary, these embodiments are provided to make this disclosure thorough and complete, and will fully express the scope of the present invention to those skilled in the art. In the accompanying drawings, components are enlarged for clarity.

[0099] Provide the detailed steps of the present invention below:

[0100] The behavior recognition method based on SimAM attention mechanism includes the following steps:

[0101] Preprocessing part (step 1 to step 2):

[0102] Step 1) Use the public UCF-101 dataset to sample the input video. First, randomly extract a frame of image, denoted as img a , where a is the group symbol. Then randomly select ten consecutive frames of images, denoted as img bi , where i∈[1,10], b is the group symbol;

[0103] Step 2), the eleven frames of sampled images are scaled and cropped in the same manner; first, the images are scaled to a size of 256*256, and then randomly cropped to a size of 224*224;

[0104] Spatial stream feature extraction part (step 3 to step 10):

[0105] Step 3), input image img a Decompose into three channels: red, green, and blue, and obtain the image X under three channels z (x, y), where z represents the zth channel, z is an integer greater than or equal to 1 and less than or equal to 3, the first, second, and third channels represent the red, green, and blue channels respectively; x and y are the horizontal and vertical coordinates of the pixel in the image respectively;

[0106] Step 4), for X z (x, y) performs padding operation to expand the image from 224*224 to 231*231, and fill the expanded part with 0 value; use 7*7*96 convolution kernel with step size 2 to perform padding on X z Convolution, the convolution kernel is conv 1j (m,n), where j∈[1,96]; m,n∈[0,6], j represents the jth convolution kernel; m and n are the subscripts of the mth row and nth column elements in the convolution kernel respectively. The convolution process can be represented by the following steps:

[0107] Step 4.1), calculate the convolution of a single channel, the process can be expressed by the following formula:

[0108]

[0109] Where x∈[0,112],y∈[0,112], the symbol · represents the multiplication of corresponding elements of the matrix;

[0110] Step 4.2), calculate the sum of the convolution of the three channels, the process can be expressed by the following formula:

[0111]

[0112] Step 5), generate the feature map F j (x, y) is linearly rectified, and the process can be expressed by the following formula:

[0113] F j ′(x,y)=ReLU(F j ′(x,y))

[0114] The formula of ReLU is:

[0115] ReLU(x)=max(0,x)

[0116] Step 6), the linearly rectified feature map F j ′(x,y) performs the maximum pooling operation with a pooling size of 2*2, and finally generates a 56*56 feature map Fj ″(x,y), the process can be expressed by the following formula:

[0117] F j ″(x,y)=max(F j ′(2x,2y),F j ′(2x+1,2y),F j ′(2x,2y+1),F j ′(2x+1,2y+1))

[0118] Where x∈[0,56],y∈[0,56];

[0119] Step 7), convolve the feature map, set the convolution kernel size to 5*5*256, the step size to 2, the pooling size to 2*2, and obtain a 14*14*256 feature map. The convolution process is the same as steps 4 to 6;

[0120] Step 8), convolve the feature map, set the convolution kernel size to 3*3*512, the step size to 1, and obtain a 14*14*512 feature map. The convolution process is the same as step 4;

[0121] Step 9), convolve the feature map, set the convolution kernel size to 3*3*512, the step size to 1, and obtain a 14*14*512 feature map. The convolution process is the same as step 4;

[0122] Step 10), convolve the feature map, set the convolution kernel size to 3*3*512, the step size to 1, the pooling size to 2*2, and finally obtain 7*7*512 feature maps. The convolution process is the same as steps 4 to 6.

[0123] Spatial stream SimAM part (step 11 to step 13):

[0124] Step 11), the 512 spatial stream feature maps of the spatial stream network obtained in step 10 are regarded as 512 channels, denoted as S p , where p∈[1,512]; each S p There are 7*7 neurons, totaling 49 neurons; calculate the energy function e of each neuron r , and its calculation formula is as follows:

[0125]

[0126] Where r represents the target neuron in a single input channel; q i Represents other neurons in the input channel, i is the serial number; w r Linear transformation of weights; b r is the linear transformation of the bias; M = 49 is the number of other neurons on this channel; γ is a variable; λ is a coefficient.

[0127] Use the following formula to calculate w r :

[0128]

[0129] Use the following formula to calculate b r :

[0130]

[0131] μ r It represents the average value, and its calculation formula is:

[0132]

[0133] σ r 2 It represents the variance, and its calculation formula is:

[0134]

[0135] Calculate the minimum neuron energy

[0136]

[0137] The smaller the energy of each neuron, the more important it is compared with other neurons in the same channel. Therefore, the inverse of the minimum neuron energy is used. Represents the weight of the neuron.

[0138] Step 12) The energy of all neurons in a single channel constitutes the energy matrix E of the channel. The attention weight matrix E′ of the channel is obtained by taking the inverse of each element in the energy matrix E and normalizing it with the sigmoid function. The calculation formula is:

[0139]

[0140] Step 13), the feature map is fused with the channel attention weight, and the fused feature map S′ is calculated p , and its calculation formula is:

[0141] S′ p =S p E′,p∈[1,512]

[0142] Time stream feature extraction part (step 14 to step 15):

[0143] Step 14), calculate the stack of optical flows; dense optical flows can be regarded as a set of displacement vector fields between consecutive frames t and t+1; the point (u, v) in the tth frame, the optical flow of the tth frame is Iτ , and its calculation formula is as follows:

[0144]

[0145]

[0146] Wherein u=[1;w], v=[1:h], k=[1;L], w is the width of the second video image, h is the height of the second video image, and L is the number of frames of the second video image.

[0147] The frame number L is set to 10, and the final result is a stacked optical flow input of 224*224*20.

[0148] Step 15), perform 5 convolutions on the feature map, and the convolution steps are the same as steps 4 to 10;

[0149] Time flow SimAM part (step 16 to step 17):

[0150] Step 16), calculate the attention in the temporal stream network model, the same as steps 11 to 12;

[0151] Step 17), the feature map is fused with the channel attention weight, same as step 13;

[0152] Feature fusion and output part (step 18 to step 23):

[0153] If the resolution of one dimension of the input video is greater than 1080p, use steps 18 to 21;

[0154] Early feature fusion part (step 18 to step 19):

[0155] Step 18), record the value of a certain position (i, j, d) on the feature map output by the spatial stream network as s represents space, and the value of a certain position (i, j, d) on the feature map output by the corresponding time stream network is t represents time, and the position obtained by cascade fusion is Cat represents cascade, and its calculation formula is as follows:

[0156]

[0157]

[0158] The size of the feature map obtained by cascade fusion is 7*7*1024;

[0159] Step 19), perform three layers of convolution on the feature map obtained by cascade fusion, and the convolution kernel sizes are 3*3*512, 3*3*1024, and 1*1*512 respectively. The last layer of convolution plays a role in reducing the dimension, and the size of the output feature map is 7*7*512, which is consistent with the size of the original feature map; the convolution process is the same as step 4 to step 6;

[0160] Full connection (step 20):

[0161] Step 20), the feature map of the fusion SimAM attention mechanism with a size of 7*7*512 is sent to two fully connected layers containing 1024 and 101 neurons respectively, and the output d_output of the two-stream network model is obtained, where d represents the time and space dual streams, and its calculation formula is,

[0162] output=f(w T A+b)

[0163] Where w is the weight vector, T represents transpose, A is the input vector, b is the bias vector, and the output dimension is 101*1.

[0164] Send the output value d_output of the output to the softmax function to get the final classification result;

[0165] If the resolution of all dimensions of the input video is less than 1080p, use steps 22 to 23;

[0166] Result late fusion (step 21 to step 23):

[0167] Step 21), send the feature map of 7*7*512 fused with SimAM attention mechanism into two fully connected layers containing 1024 and 101 neurons respectively, and get the output s_output of the spatial stream network model, where s represents space, the same as step 20;

[0168] Step 22), send the feature map of 7*7*512 fused with SimAM attention mechanism into two fully connected layers containing 1024 and 101 neurons respectively, and obtain the output t_output of the time stream network model, where t represents time, the same as step 20;

[0169] Step 23), the outputs of the spatial stream network model and the temporal stream network model are averaged and fused to obtain the output value of the dual stream network model Send it to the softmax function to get the final behavior recognition result.

[0170] Output section (step 24)

[0171] Step 24), the softmax function expression is:

[0172]

[0173] Where: output i Represents the i-th element in output, output k represents the kth element in output, P represents probability, and exp() represents an exponential function with the natural logarithm e as the base; the behavior classification label corresponding to the element with the largest probability among the 101 elements is the final behavior recognition result.

[0174] Neural network training part:

[0175] Step 25), using stochastic gradient descent (SGD) algorithm to train the neural network;

[0176] Step 25.1), calculate the gradient of the output layer, It represents the gradient of the kth node in the output layer (Kth layer), and its calculation formula is as follows:

[0177]

[0178] Among them k represents the output of the kth node in the Kth layer, t k Represents the label of the k-th node in the K-th layer.

[0179] Step 25.2), calculate the gradient of the hidden layer, represents the gradient of the ith node in the ith hidden layer, and its calculation formula is as follows:

[0180]

[0181] Among them i represents the label value of the ith node in the ith hidden layer, represents the gradient of the i-th node in the previous layer (J-th layer) of the i-th hidden layer, w ab Represents the value in the ath row and bth column of the weight matrix.

[0182] Step 25.3), update the parameters according to the calculated gradient.

[0183] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0184] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A behavior recognition method based on SimAM attention mechanism, characterized in that: include: Sampling the input video, randomly extracting a sample feature map, the sample feature map includes: a single-frame first video image and a plurality of consecutive frames of second video images, scaling and cropping the sample feature map; Decomposing the first video image into three primary color channels to obtain a three primary color channel image; calculating the stacked optical flow of the second video image; Input a single frame of the three primary color channel image into the spatial stream CNN to extract the spatial stream feature map, and input the stacked optical flow into the temporal stream CNN to extract the temporal stream feature map; Calculate the SimAM attention of the spatial stream CNN and the temporal stream CNN, and fuse them with the corresponding spatial stream feature map and temporal stream feature map into the corresponding spatial stream attention fusion feature map and temporal stream attention fusion feature map respectively; For input videos with more high-level features, a dual-stream attention fusion feature map fused with a spatial stream attention fusion feature map and a temporal stream attention fusion feature map is input into a fully connected layer classification, and its output value is input into a softmax function to solve the final behavior recognition result; wherein, the presence of more high-level features is manifested as a single-frame input video resolution greater than 1080p; For input videos containing more low-level features, the spatial stream attention fusion feature map is input into the fully connected layer classification in the spatial stream CNN, and the temporal stream attention fusion feature map is input into the fully connected layer classification in the temporal stream CNN. The output values ​​of the two networks are fused by the mean and then input into the softmax function to solve the final behavior recognition result.

2. According to claim 1, a behavior recognition method based on SimAM attention mechanism is characterized in that: The scaling and cropping of the sample feature map includes: scaling the sample feature map to a size of N*N, and then randomly cropping the sample feature map to a size of N′*N′, wherein: N represents the pixel size of the image, and N′ represents the pixel size of the image after random cropping.

3. According to claim 2, a behavior recognition method based on SimAM attention mechanism is characterized in that: The three-primary color channel image acquisition method comprises: Decompose the scaled and cropped first video image into three channels: red, green, and blue, and obtain the image X under the three channels z (x,y), Wherein, z represents the zth channel, z is an integer greater than or equal to 1 and less than or equal to 3, and the 1st, 2nd, and 3rd channels represent the red, green, and blue channels respectively; x is the horizontal coordinate of the pixel in the image, and y is the vertical coordinate of the pixel in the image.

4. The behavior recognition method based on SimAM attention mechanism according to claim 2 is characterized in that: The optical flow of the computation stack includes: The optical flow is regarded as a set of displacement vector fields between consecutive frames t and t+1; the point (u, v) in the tth frame is recorded, and the optical flow of the tth frame is I τ , and its calculation formula is as follows: Where: h∈[1,w],v∈[1,h],k∈[1,L]; w is the width of the second video image, h is the height of the second video image, L is the number of frames of the second video image, and k is the offset sequence number of the frame, and finally a stacked optical flow of N′*N′*2L is obtained.

5. The behavior recognition method based on SimAM attention mechanism according to claim 2 is characterized in that: Extracting a spatial stream feature map or extracting a temporal stream feature map comprises the following steps: Step 1: Perform a filling operation on the input three-primary color channel image or the stacked optical flow, expanding from N'*N' to (N'+7)*(N'+7), and fill the expanded part with 0 value; then use the 7*7*96 convolution kernel with a step size of 2 to perform convolution on the three-primary color channel image or the stacked optical flow to generate a feature map; Step 2: Use ReLU to perform linear rectification on the feature map generated in step 1, where the formula of the ReLU function is: ReLU(m)=max(0,m) Where m is the independent variable; Step 3: Perform a maximum pooling operation on the linearly rectified feature map with a pooling size of 2*2; Step 4: Convolve the feature map generated in step 3, set the convolution kernel size to 5*5*256, the step size to 2, and the pooling size to 2*2; Step 5: Convolve the feature map generated in step 4, set the convolution kernel size to 3*3*512, and the step size to 1; Step 6: Convolve the feature map generated in step 5, set the convolution kernel size to 3*3*512, and the step size to 1; Step 7: Convolve the feature map generated in step 6, set the convolution kernel size to 3*3*512, the step size to 1, and the pooling size to 2*2; The obtained 512 spatial stream feature maps S p Considered as 512 channels, where: p∈[1,512], each S p There are (N′ / 32)*(N′ / 32) neurons; The 512 time stream feature graphs S q Considered as 512 channels, where: q∈[1,512], each S q There are (N′ / 32)*(N′ / 32) neurons.

6. The behavior recognition method based on SimAM attention mechanism according to claim 5, characterized in that: Calculate the SimAM attention of the spatial stream CNN and the temporal stream CNN, and compare them with the corresponding spatial stream feature map S p And the time flow feature map S q They are fused into the spatial stream attention fusion feature map S′ p And the time stream attention fusion feature map S′ q The specific steps include: SimAM attention calculation of spatial stream CNN and temporal stream CNN: Calculate the energy function e of each neuron r : Where r represents the target neuron in a single input channel; q i Represents other neurons in the input channel, i is the serial number; w r Linear transformation of weights; b r is the linear transformation of the bias; M = (N' / 32)*(N' / 32) is the number of other neurons on the channel; γ is a variable; λ is a coefficient; Calculate w r : Calculate b r : μ r represents the average value, and μ is calculated r : σ r 2 Represents the variance, and σ is calculated r 2 : Calculate the minimum neuron energy The inverse of the minimum neuron energy is the weight of the neuron; One neuron corresponds to one neuron energy The energy of all neurons in a single channel constitutes the energy matrix E of the channel. The attention weight matrix E′ of the channel is obtained by taking the inverse of each element in the energy matrix E and normalizing it with the sigmoid function. Its calculation formula is: The SimAM attention of spatial stream CNN and temporal stream CNN is fused with spatial stream feature map and temporal stream feature map respectively, including: The calculation formula of the fused feature map S′ is: S′=S·E′ Among them, S represents the spatial stream feature map S p Or the spatial stream feature map S q One of them, S′ represents the spatial stream attention fusion feature map S′ p Or the time stream attention fusion feature map S′ q One of them.

7. The behavior recognition method based on SimAM attention mechanism according to claim 6 is characterized in that: The dual-stream attention fusion feature map of the spatial stream attention fusion feature map and the temporal stream attention fusion feature map includes the following steps: Attention fusion feature map S′ for spatial stream p And the time stream attention fusion feature map S′ q To perform cascade fusion: in: The feature map S′ is output by the spatial stream network p The upper position is (i, j, d), s represents space, The feature map S′ output by its corresponding temporal stream network q The upper position is (i, j, d), t represents the time, To obtain the position (i, j, 2d) on the feature map for cascade fusion, The position (i, j, 2d-1) on the feature map is obtained by cascade fusion, where d represents the dth feature map; The cascade fusion feature map is obtained: the size is (N′ / 32)*(N′ / 32)*1024; the above cascade fusion feature map is successively convolved with three layers, and the convolution kernel sizes are 3*3*512, 3*3*1024, and 1*1*512 respectively, and the dual-stream attention fusion feature map S is obtained. d The size is (N′ / 32)*(N′ / 32)*512.

8. The behavior recognition method based on SimAM attention mechanism according to claim 7, characterized in that: Fully connected layers are classified into: output=f(w T A+b) Where w is the weight vector, T represents transposition, A is the input vector, b is the bias vector, and the output dimension is H*1; the input vector A is the dual-stream attention fusion feature map S d , time stream attention fusion feature map S′ q Or the spatial stream attention fusion feature map S′ p One of; output is one of the spatial stream network model output s_output, the temporal stream network model output t_output, or the dual stream network model output d_output.

9. A behavior recognition method based on SimAM attention mechanism according to claim 8, characterized in that , Solving the final behavior recognition results specifically includes: The softmax function expression is: Where: output i Represents the i-th element in output, output k represents the kth element in output, P represents probability, exp() represents the exponential function with natural logarithm e as base; H is the number of elements; The output fed into softmax is d_output, or the mean of s_output and t_output; The behavior classification label corresponding to the element with the highest probability among the H elements is the final behavior recognition result.

10. The behavior recognition method based on SimAM attention mechanism according to claim 1, characterized in that: The spatial stream CNN or spatial stream CNN is trained using a stochastic gradient descent algorithm; Calculate the gradient of the output layer, It represents the gradient of the kth node in the output layer (Kth layer), and its calculation formula is as follows: Among them k represents the output of the kth node in the Kth layer, t k Represents the label of the k-th node in the K-th layer; Calculate the gradient of the hidden layer, represents the gradient of the ith node in the ith hidden layer, and its calculation formula is as follows: Among them i represents the label value of the ith node in the ith hidden layer, represents the gradient of the i-th node in the previous layer (J-th layer) of the i-th hidden layer, w ab Represents the value of the ath row and bth column of the weight matrix; and updates the parameters based on the calculated gradient.

Images