Action recognition method based on decoupled 3D network of multi-scale self-attention mechanism

Abstract

The application discloses a kind of action recognition methods of decoupled 3D network based on multi-scale self-attention mechanism.The method comprises the following steps: constructing action training set after pre-processing action video with action category label;Decoupled 3D network of multi-scale self-attention mechanism is established, and action training set is inputted for training;Collect the action video to be detected and input, output the action category of the action video to be detected, complete action recognition.The method can pay attention to the global spatiotemporal features on the multi-scale feature map, can achieve better balance in space and time dimensions, can fully utilize feature information, more comprehensively model the spatiotemporal characteristics of video data, improve the expression ability of video data, improve the understanding and interpretation ability of video content, improve the performance of video action classification.

Description

Technical Field

[0001] This invention relates to an action recognition method, specifically to an action recognition method based on a decoupled 3D network using a multi-scale self-attention mechanism, within the field of deep learning action recognition technology. Background Technology

[0002] With the continuous development of deep learning technology and the increasing application scenarios of action recognition, deep learning-based action recognition technology is gradually becoming a key field in the field of action recognition.

[0003] Action recognition technology is an important research area in computer vision, with wide applications in human-computer interaction. In recent years, action recognition technology has gradually become a major research topic in the field of computer vision. Research methods for action recognition can be broadly divided into two types: methods based on manual feature extraction using traditional machine learning, and recognition methods based on feature learning using deep networks.

[0004] Traditional machine learning methods for manually extracting features mainly involve preprocessing videos using conventional machine learning algorithms, extracting video features, vectorizing these features, training the model, and predicting action classifications. However, video action recognition is affected by factors such as lighting, background changes, and relationships between video frames, making it difficult to fully extract gesture features and resulting in poor robustness.

[0005] For deep learning-based action recognition methods, compared to two-dimensional images, temporal information is added. Therefore, deep learning-based action recognition methods can learn not only spatial sequence information but also recognize temporal sequence features. For example, Simonyan first proposed the classic two-stream CNN, which uses spatial-stream and temporal-stream networks respectively to learn spatial and temporal features, and finally fuses the two. This method compensates for the loss of temporal feature information in traditional machine learning methods. Another example is the C3D model proposed by Tran et al. to extract spatiotemporal features from videos; however, this method suffers from excessive parameters and high computational cost. Furthermore, traditional convolutional networks have relatively limited ability to model long-term dependencies, making it difficult to capture semantic information over longer time spans in video sequences. Summary of the Invention

[0006] To address the problems existing in the background technology, this invention provides a decoupled 3D network using a multi-scale self-attention mechanism (MSA_(2+1)DNet). This method specifically addresses the difficulties in effectively extracting complex features, the large computational cost and parameters of traditional 3D convolutional neural networks, and the inability to simultaneously consider both long-term and short-term spatiotemporal information. While 3DCNN network structures can capture the spatiotemporal features of video information well, they suffer from a large number of network parameters. To address this problem, a (2+1)D network model is adopted. By splitting the 3D convolutional kernel into a two-dimensional convolution in the spatial dimension and a one-dimensional convolution in the temporal dimension, this method can effectively reduce the computational cost. Furthermore, an Attention-Mechnism model is designed. This model divides the action video into video blocks of equal size and introduces spatiotemporal encoding to preserve the positional and temporal information of the video blocks in the original video. Global spatiotemporal information of the action video is obtained through self-attention calculation between video blocks, and this attention mechanism is applicable to feature maps of multiple scales generated by convolution. Furthermore, the designed (2+1)D Residual Module not only integrates the global spatiotemporal features obtained by the attention mechanism, but also further extracts the local spatiotemporal features of video actions, enabling the network to effectively take into account both long-term and short-term spatiotemporal features.

[0007] The technical solution adopted in this invention is:

[0008] The action recognition method of the present invention based on a decoupled 3D network with a multi-scale self-attention mechanism includes:

[0009] Step 1) After preprocessing the action videos of several action categories with action category labels, an action training set is formed. In actual operation, it is divided into a training set and a test set according to a preset ratio.

[0010] Step 2) Build a decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism. Input the action training set into the decoupled 3D network MSA_(2+1)DNet and train it to obtain the trained decoupled 3D network MSA_(2+1)DNet.

[0011] Step 3) Acquire the action video to be detected and input it into the trained decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism. The decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism outputs the action category of the action video to be detected, thus completing action recognition.

[0012] In step 1), the action videos of several action categories with action category labels are preprocessed. For each action video with action category label, the video frames of the action video are first extracted using a sparse sampling method to extract a preset number of video frame sequences. Then, the size of each video frame in the extracted video frame sequence is normalized. Finally, the preprocessed action videos are used to form an action training set.

[0013] In step 2), the decoupled 3D network MSA_(2+1)D Net with multi-scale self-attention mechanism includes a (2+1)D stem guiding module, a (2+1)D attention module group, a global average pooling layer, a vector one-dimensionalization operation Flatten operation, a fully connected layer FC, and a Softmax classifier connected in sequence. The (2+1)D attention module group includes four attention mechanism modules and four (2+1)D residual modules. The four attention mechanism modules and the four (2+1)D residual modules are interleaved and connected in sequence to form the (2+1)D attention module group. The first module in the (2+1)D attention module group is the attention mechanism module.

[0014] The input to the (2+1)D stem module is a sequence of video frames. After the video frame sequence passes through the first convolutional layer, a feature map of low-level spatial information is extracted. Then, it is input into the second convolutional layer. The feature map of the second convolutional layer is subjected to a one-dimensional convolution operation in the time dimension to obtain a feature map of low-level spatiotemporal information.

[0015] The (2+1)D Residual module contains two residual modules: In the first residual module, the input feature map is processed through two 3D convolutional layers to extract local spatiotemporal features, and the obtained local spatiotemporal features are added to the features input to the first residual module to obtain the output of the first residual module. This fused feature is then downsampled to increase the number of channels and reduce the height and width of the feature map. In the second residual module, the output of the first residual module is used as the input, and the input feature map is processed through two 3D convolutional layers to further extract local spatiotemporal features. These features are then added to the features input to the second residual module to obtain the output of the second residual module.

[0016] The spatiotemporal features extracted by the (2+1)D Residual Module are further extracted into high-level global spatiotemporal features through a global average pooling layer. Then, the multidimensional features are reduced to one dimension through the Flatten operation. After passing through a fully connected layer and using the SoftMax classifier for action classification, the action classification results of the MSA_(2+1)DNet model are finally obtained.

[0017] The attention mechanism module includes patch_embedding operation, first vector addition operation, spatiotemporal encoding generation operation, neuron random deactivation Dropout operation, second linear fully connected layer MLP, third linear fully connected layer MLP, fourth linear fully connected layer MLP, first matrix multiplication Multiply operation, exponential function softmax normalization operation, vector division Divide operation, second matrix multiplication Multiply operation, upsampling operation, and second vector addition operation.

[0018] The input to the attention mechanism module first undergoes a patch embedding operation to obtain a feature map. This feature map is then generated through spatiotemporal encoding to obtain learnable parameters. These parameters and the feature map are then combined using a first vector addition operation followed by a dropout operation to randomly deactivate neurons. The resulting inputs are then fed into the second, third, and fourth linear fully connected layers (MLP) for processing. The outputs from the second and third MLPs are sequentially processed by a first matrix multiplication (Multiply), an exponential function softmax normalization operation, and a vector division (Divide) operation. These outputs are then combined with the output of the fourth MLP and subjected to a second matrix multiplication (Multiply) operation. The resulting output is then upsampled and combined with the input to the attention mechanism module using a second vector addition operation. The output of this second upsampling operation serves as the output of the attention mechanism module.

[0019] After the feature map undergoes a spatiotemporal encoding generation operation, a learnable parameter space_temporal_embedding containing spatiotemporal encoding information is generated based on the shape of the feature map F(patch_embedding). Then, an Add operation is performed on the space_temporal_embedding and the feature map F(patch_embedding) to obtain a feature map F′ containing spatiotemporal location encoding.

[0020] The patch_embedding operation includes a first 3D convolutional layer Conv3d, a feature map shape reshaping operation Rearrange, and a first linear fully connected layer MLP, connected sequentially.

[0021] The upsampling operation includes a fifth linear fully connected layer (MLP), a second feature map shape reshaping operation (Rearrange), and a second three-dimensional convolutional layer (Conv3d).

[0022] This invention first preprocesses each action category by reading the action video and obtaining the category label. Secondly, it preprocesses each action video by using a (2+1)D Stem module to extract and downsample the video frames to obtain the spatiotemporal features of the action video. The attention mechanism module designed in this invention can focus on global spatiotemporal features on multi-scale feature maps. The subsequently integrated (2+1)D Residual module not only further extracts local spatiotemporal features but also integrates global spatiotemporal feature information into the output features. This allows the network to fully consider multi-scale global and local spatiotemporal feature information at multiple stages. The introduced residual structure effectively solves the gradient vanishing problem caused by increasing the number of network layers.

[0023] The beneficial effects of this invention are:

[0024] 1) The Attention-Mechnism module designed in this invention can capture the global spatiotemporal features of the network on the multi-scale feature maps generated by multi-stage convolution, thereby enabling the network to make up for the global spatiotemporal features at each stage and improve the performance of model classification.

[0025] 2) Multi-scale self-attention mechanisms can help networks model context when processing videos. By focusing on features at different scales, networks can better understand the contextual information of objects or actions in videos, thereby improving their ability to understand and interpret video content.

[0026] 3) The (2+1)D Residual Module used not only further extracts local spatiotemporal features, but also integrates long-term and short-term spatiotemporal feature information in the final output, enabling the network to fully utilize feature information. The (2+1)D network combined with a multi-scale self-attention mechanism can achieve a better balance in spatial and temporal dimensions. 2D convolution operations capture the spatial information of the image, while 1D convolution operations capture the temporal relationship between frames. This balance enables the network to more comprehensively model the spatiotemporal characteristics of video data and improve its expressive power for video data. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the technical solution of the method of the present invention;

[0028] Figure 2 This is a structural diagram of the Attention-mechnism module of the present invention;

[0029] Figure 3This is a graph showing the accuracy of the validation set in this invention. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] The action recognition method of the present invention based on a decoupled 3D network with a multi-scale self-attention mechanism includes:

[0032] Step 1) After preprocessing the action videos of several action categories with action category labels, an action training set is formed. In actual operation, it is divided into a training set and a test set according to a preset ratio.

[0033] In step 1), the action videos of several action categories with action category labels are preprocessed. For each action video with action category label, the video frames of the action video are first extracted using a sparse sampling method to extract a video frame sequence of a preset number of frames. Then, the size of each video frame in the extracted video frame sequence is normalized. Finally, the preprocessed action videos are used to form an action training set.

[0034] Step 2) Build a decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism. Input the action training set into the decoupled 3D network MSA_(2+1)DNet and train it to obtain the trained decoupled 3D network MSA_(2+1)DNet.

[0035] In step 2), the decoupled 3D network MSA_(2+1)D Net with multi-scale self-attention mechanism includes a (2+1)D stem guiding module, a (2+1)D attention module group, a global average pooling layer, a vector one-dimensionalization operation Flatten operation, a fully connected layer FC, and a Softmax classifier connected in sequence. The (2+1)D attention module group includes four attention mechanism modules and four (2+1)D residual modules. The four attention mechanism modules and the four (2+1)D residual modules are interleaved and connected in sequence to form the (2+1)D attention module group. The first module in the (2+1)D attention module group is the attention mechanism module.

[0036] The input to the (2+1)D stem module is a sequence of video frames. After the video frame sequence passes through the first convolutional layer, a feature map of low-level spatial information is extracted. Then, it is input into the second convolutional layer. The feature map of the second convolutional layer is subjected to a one-dimensional convolution operation in the time dimension to obtain a feature map of low-level spatiotemporal information.

[0037] The (2+1)D Residual module contains two residual modules: In the first residual module, the input feature map is processed through two 3D convolutional layers to extract local spatiotemporal features, and the obtained local spatiotemporal features are added to the features input to the first residual module to obtain the output of the first residual module. This fused feature is then downsampled to increase the number of channels and reduce the height and width of the feature map. In the second residual module, the output of the first residual module is used as the input, and the input feature map is processed through two 3D convolutional layers to further extract local spatiotemporal features. These features are then added to the features input to the second residual module to obtain the output of the second residual module.

[0038] The spatiotemporal features extracted by the (2+1)D Residual Module are further extracted into high-level global spatiotemporal features through a global average pooling layer. Then, the multidimensional features are reduced to one dimension through the Flatten operation. After passing through a fully connected layer and using the SoftMax classifier for action classification, the action classification results of the MSA_(2+1)DNet model are finally obtained.

[0039] The attention mechanism module includes patch_embedding operation, first vector addition operation, spatiotemporal encoding generation operation, neuron random deactivation Dropout operation, second linear fully connected layer MLP, third linear fully connected layer MLP, fourth linear fully connected layer MLP, first matrix multiplication Multiply operation, exponential function softmax normalization operation, vector division Divide operation, second matrix multiplication Multiply operation, upsampling operation, and second vector addition operation.

[0040] The input to the attention mechanism module first undergoes a patch embedding operation to obtain a feature map. This feature map is then generated through spatiotemporal encoding to obtain learnable parameters. These parameters and the feature map are then combined using a first vector addition operation followed by a dropout operation to randomly deactivate neurons. The resulting inputs are then fed into the second, third, and fourth linear fully connected layers (MLP) for processing. The outputs from the second and third MLPs are sequentially processed by a first matrix multiplication (Multiply), an exponential function softmax normalization operation, and a vector division (Divide) operation. These outputs are then combined with the output of the fourth MLP and subjected to a second matrix multiplication (Multiply) operation. The resulting output is then upsampled and combined with the input to the attention mechanism module using a second vector addition operation. The output of this second upsampling operation serves as the output of the attention mechanism module.

[0041] After the feature map undergoes a spatiotemporal encoding generation operation, a learnable parameter space_temporal_embedding containing spatiotemporal encoding information is generated based on the shape of the feature map F(patch_embedding). Then, an Add operation is performed on the space_temporal_embedding and the feature map F(patch_embedding) to obtain a feature map F′ containing spatiotemporal location encoding.

[0042] The patch_embedding operation consists of a first 3D convolutional layer Conv3d, a feature map shape reshaping operation Rearrange, and a first linear fully connected layer MLP, which are connected in sequence.

[0043] The upsampling operations include the fifth linear fully connected layer MLP, the second feature map shape reshaping Rearrange operation, and the second three-dimensional convolutional layer Conv3d.

[0044] Step 3) Acquire the action video to be detected and input it into the trained decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism. The decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism outputs the action category of the action video to be detected, thus completing action recognition.

[0045] Specific embodiments of the present invention are as follows:

[0046] This invention primarily focuses on video action recognition. Since the recognition object is video action sequences, the UCF101 dataset, released by the University of Central Florida, was selected for recognition. This dataset contains 101 different categories of human movement behaviors, including videos related to human-object interaction, human movements, human-human interaction, musical instrument performance, and sports. Action categories include archery, cycling, diving, hairdressing, cello playing, rowing, typing, and writing.

[0047] like Figure 1As shown, firstly, class labels are obtained by reading videos of each category from the UCF101 dataset; then, preprocessing is performed on each video: sparse sampling is used to extract video frames with specified parameters, followed by size normalization; continuous video frame sequences are used as input to the (2+1)D stem for low-level spatiotemporal feature extraction and downsampling; an attention mechanism module, Attention-mechnism, is designed to focus on the global spatiotemporal information of feature maps of different scales generated during convolution; the designed (2+1)D Residual Module further extracts the local spatiotemporal features of video actions and fuses them with the attention mechanism to obtain spatiotemporal features that take into account both global and local information; after multiple downsampling and four spatiotemporal feature fusions of the attention mechanism and the (2+1)D Residual Module, the output features are processed by a global pooling layer to extract high-level spatiotemporal information, then connected to a fully connected layer (FC), and finally classified by a SoftMax classifier to obtain the classification result.

[0048] The specific steps of this invention are as follows:

[0049] 1) Read videos of each category, obtain category labels, and then perform preprocessing operations on each video to obtain the number of frames for each video action; then use sparse sampling to obtain the specified video frames; finally, normalize the size of the extracted video frames.

[0050] The input to the (2+1)D stem module is a sequence of video frames. After passing through the first convolutional layer, the video frame sequence yields a feature map with extracted spatial information. The first convolutional layer is a 3D convolution with a kernel size of 1×7×7 and a step rate of (1,2,2). It is followed by a 3D batch normalization (BatchNorm3d) function and a ReLU activation function. This feature map is then input into the second convolutional layer. The feature map after passing through the second convolutional layer is subjected to a one-dimensional convolution operation in the time dimension to obtain a feature map with spatiotemporal information. The second convolutional layer is a 3D convolution with a step rate of (1,1,1). It is followed by a 3D batch normalization (BatchNorm3d) function and a ReLU activation function.

[0051] 3) such as Figure 2As shown, the attention mechanism module takes the feature map F obtained from the previous module as input and performs a patch_embedding operation to obtain a feature map F(patch_embedding) containing spatiotemporal information in video blocks. The patch_embedding operation sequentially includes a 3D convolutional layer, a feature map shape reshaping Rearrange operation, and a linear layer. The number of output channels of the 3D convolutional layer is half the number of input channels, and the convolutional kernel is 1×1×1. The feature map shape reshaping Rearrange operation specifically uses the Python library function Rearrange to cut the feature map shape into small blocks in both the temporal and spatial dimensions. That is, the feature map is transformed from T×W×H into t×h×w blocks with a time length of pt, a height of ph, and a width of pw. T is the original length of the feature map in the temporal dimension, W is the original width of the feature map in the spatial dimension, H is the original height of the feature map in the spatial dimension, c is the number of channels, and pt is the length of the feature map in the temporal dimension. Let be the number of frames in the video patch, ph be the height of the video patch, pw be the width of the video patch, t be the number of video patches in the temporal dimension, and h and w be the number of video patches in the spatial dimension (height and width, respectively). Here, t is set to 4, h to 7, and w to 7. The input to the linear layer is pt×ph×pw×c, and the number of neurons output is set to 192. The shape of the feature map F(patch_embedding) obtained by the patch_embedding operation generates a learnable parameter space_temporal_embedding containing spatiotemporal coding information with the same shape. Specifically, it is generated using the PyTorch library function Parameter, where the number of patches num_patches is t×h×w, and the dimension dim is 192. Then, a vector addition operation is performed between this and the feature map F(patch_embedding) to obtain a feature map F′ containing spatiotemporal location coding. The feature map F′ is input into three linear layers to obtain the query matrix M. q Key matrix M k Sum matrix M v The three linear layers have 192 input neurons and 512 output neurons; matrix M... q Bond matrix M k The feature map obtained by matrix multiplication is then normalized by an exponential function and then by softmax normalization, and finally combined with the value matrix M. v Perform matrix multiplication to obtain a feature map containing global information, then combine this feature map with a factor d that scales the attention weights. kThe division operation yields the feature map F″. The feature map F″ is then upsampled to become a feature map F(attention) with the same shape as feature map F. This upsampling operation sequentially includes a linear layer, a feature map reshaping Rearrange operation, and a 3D convolutional layer. The linear layer has 192 input neurons and outputs pt×ph×pw×c. The Rearrange operation uses the Python library function Rearrange to reshape the feature map from its initial shape, transforming the t×h×w blocks (time length pt, height ph, width pw) into t×h×w feature maps with the same time length as the previous T×W×H. The 3D convolutional layer outputs twice the number of input channels, with a 1×1×1 kernel. Finally, the feature map F(attention) and feature map F are added together to obtain the output feature map F″′.

[0052] The attention mechanism module obtains the output features using the following formula:

[0053] F(patch_embedding)=MLP(Rerrange(Conv3D(F)))

[0054] F′=F(patch_embedding)+space_temporal_embedding

[0055] M q =MLP(F′)

[0056] M k =MLP(F′)

[0057] M v =MLP(F′)

[0058]

[0059] F(attention)=Conv3D(Rerrange(MLP(F)))

[0060] F″′=F(attention)+F

[0061] Where F represents the input features of the Attention-Mechnism module, F(patch_embedding) is the video feature map after segmentation, F′ is the feature map with spatiotemporal encoding, and M... q For querying the matrix, M k M is the key matrix. vF″ is the value matrix, F(attention) is the feature map after self-attention, F(attention) is the feature map after upsampling, F″′ is the fused feature map, Conv3D represents 3D convolution operation, Rerrange represents matrix cutting operation, MLP represents a linear layer, Softmax represents the softmax activation function, and × represents matrix multiplication.

[0062] 4) In the first residual module, the feature map obtained by the attention mechanism is used as input, and local spatiotemporal features are extracted through two layers of three-dimensional convolutional layers. The kernel of the first three-dimensional convolutional layer is 1×3×3, the step frequency is (1,2,2), followed by the ReLU activation function. The kernel of the second three-dimensional convolutional layer is 3×1×1, and the step frequency is (2,1,1). The local spatiotemporal features obtained by the above residual module are added to the features input by the first residual module to obtain the output of the first residual module. In the second residual module, the output of the first residual module is used as input. The input feature map is further extracted through two layers of three-dimensional convolutional layers. The local spatiotemporal features are added to the features input by the second residual module to obtain the output of the second residual module. The kernel of the first three-dimensional convolutional layer is 1×3×3, the step frequency is (1,1,1), followed by the ReLU activation function. The kernel of the second three-dimensional convolutional layer is 3×1×1, and the step frequency is (1,1,1).

[0063] 5) The global spatiotemporal features obtained from the attention mechanism and the spatiotemporal features extracted by the (2+1)D Residual Module are fused multiple times to obtain a feature map as input. A global pooling layer is used to further extract high-level spatiotemporal features, and then the multidimensional features are flattened to one dimension through a Flatten operation. After that, a fully connected layer is passed and a SoftMax classifier is used for action classification. Finally, the action classification result of the MSA_(2+1)DNet model is obtained, such as... Figure 3 As shown in Table 1.

[0064] Table 1 Comparison of results between the present invention and other models

[0065]

[0066] The Attention-Mechnism module designed in this invention can capture global spatiotemporal features on multi-scale feature maps generated by multi-stage convolutions, thereby enabling the network to compensate for global spatiotemporal features at each stage and improve the performance of model classification. The multi-scale self-attention mechanism can help the network model the context when processing videos. By focusing on features at different scales, the network can better understand the contextual information of objects or actions in the video, thereby improving the understanding and interpretation of video content. Finally, the integrated (2+1)D Residual Module can not only further extract local spatiotemporal features, but also integrate long-term and short-term spatiotemporal feature information in the final output features, thereby enabling the network to make full use of feature information. The decoupled 3D network combined with the multi-scale self-attention mechanism can achieve a better balance in the spatial and temporal dimensions. 2D convolution operations capture the spatial information of the image, while 1D convolution operations capture the temporal relationship between frames. This balance enables the network to more comprehensively model the spatiotemporal characteristics of video data and improve the expressive power of video data.

[0067] The above-described specific technical process is used to explain and illustrate the present invention, rather than to limit it. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. An action recognition method based on a decoupled 3D network using a multi-scale self-attention mechanism, characterized in that, include: Step 1) Preprocess the motion videos of several motion categories with motion category labels to form a motion training set; Step 2) Build a decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism. Input the action training set into the decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism for training to obtain the trained decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism. a) The MSA_(2+1)DNet includes a (2+1)D stem guiding module, a (2+1)D attention module group, a global average pooling layer, a vector one-dimensionalization operation Flatten operation, a fully connected layer FC, and a Softmax classifier connected in sequence. b) The (2+1)D attention module group includes four attention mechanism modules and four (2+1)D residual modules. The attention mechanism modules and residual modules are interleaved and connected in sequence, and the first module is the attention mechanism module. c) The attention mechanism module includes patch_embedding operation, first vector addition operation, spatiotemporal encoding generation operation, neuron random deactivation Dropout operation, second MLP, third MLP, fourth MLP, first matrix multiplication multiply operation, exponential function softmax normalization operation, vector division Divide operation, second matrix multiplication multiply operation, upsampling operation, and second vector addition operation; d) The input of the attention mechanism module is first processed by patch_embedding to obtain a feature map. The feature map is then processed by spatiotemporal encoding to obtain learnable parameters. The learnable parameters and the feature map are then processed by the first vector addition operation, followed by the neuron random deactivation dropout operation. The results are then fed into the second MLP, the third MLP, and the fourth MLP for processing. The outputs of the second and third MLPs are processed by the first matrix multiplication operation, the exponential function softmax normalization operation, and the vector division operation, respectively. The outputs are then processed by the fourth MLP and processed by the second matrix multiplication operation. The outputs are then upsampled and processed by the second vector addition operation, followed by the input of the attention mechanism module. The outputs are then used as the output of the attention mechanism module. Step 3) Acquire the action video to be detected and input it into the trained decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism. The decoupled 3D network MSA_(2+1)DNet with multi-scale self-attention mechanism outputs the action category of the action video to be detected, thus completing action recognition.

2. The action recognition method based on a decoupled 3D network using a multi-scale self-attention mechanism according to claim 1, characterized in that: In step 1), the action videos of several action categories with action category labels are preprocessed. For each action video with action category label, the video frames of the action video are first extracted using a sparse sampling method to extract a preset number of video frame sequences. Then, the size of each video frame in the extracted video frame sequence is normalized. Finally, the preprocessed action videos are used to form an action training set.

3. The action recognition method based on a decoupled 3D network using a multi-scale self-attention mechanism according to claim 1, characterized in that: The patch_embedding operation includes a first 3D convolutional layer Conv3d, a feature map shape reshaping operation Rearrange, and a first MLP connected in sequence.

4. The action recognition method based on a decoupled 3D network using a multi-scale self-attention mechanism according to claim 1, characterized in that: The upsampling operation includes a fifth MLP, a second feature map shape reshaping Rearrange operation, and a second three-dimensional convolutional layer Conv3d.

Images