Video action recognition method based on local and global attention mechanisms

Abstract

This invention discloses a video action recognition method based on local and global attention mechanisms. The method includes the following steps: designing a local modeling module that computes self-attention locally within a non-overlapping window, which divides a label mapping and extracts features from the local spatiotemporal context of a small 3D domain; embedding a convolutional feedforward neural network in the local modeling module to capture local continuity and positional information; designing a global modeling module that applies an attention mechanism to pixel-level labels, where each transposed label abstracts global information; embedding a skip-connected convolutional layer in the global modeling module to form a network LGATNet based on convolutional neural networks and Transformers; and learning model parameters for LGATNet on a publicly available action recognition dataset without pre-training, and outputting corresponding test results on a test set. This invention effectively utilizes multi-scale feature information, significantly improving the accuracy and real-time performance of action recognition.

Description

Technical Field

[0001] This invention relates to the field of video action recognition technology, specifically a video action recognition method based on local and global attention mechanisms. Background Technology

[0002] With the development of convolutional neural networks (CNNs), significant breakthroughs have been achieved in the field of visual recognition. The introduction of residual networks has made large-scale deep networks easier to optimize. Three-dimensional (3D) CNNs are a natural extension of 2D CNNs when processing video data. Due to the release of large-scale video datasets, 3D CNNs based on massive parameters have become popular. However, due to the limitation of parameter size, 3D CNNs face the problem of consuming large amounts of computational resources, which limits their application in mobile scenarios. In recent years, researchers have proposed decomposing 3D convolution into 2D spatial convolution and 1D temporal convolution to reduce computational resource consumption while obtaining lightweight models.

[0003] However, due to the limitation of the receptive field, convolutional neural networks process a local neighborhood, which hinders their dependency modeling in video processing. Inspired by the successful application of Transformers in natural language processing, researchers proposed the Visual Transformer to compute global self-attention among all tokens, thereby establishing long-term dependencies from distant regions or frames. This pioneering work led subsequent research, enabling advancements in both image and video processing. However, the Transformer architecture still requires significant computation, and when trained on insufficient data, the accuracy of Transformer models is lower than that of residual networks with similar parameter sizes. It seems that the Transformer lacks some of the inductive bias inherent in convolutional neural networks. Therefore, the Transformer can only be trained on large-scale datasets to overcome the inductive bias and then transferred to smaller datasets. Furthermore, the Visual Transformer is sensitive to hyperparameter settings during training, such as learning rate, number of training epochs, optimizer, and data augmentation.

[0004] To address the issues of Transformer requiring massive amounts of data for model parameter learning and the inability of 3D convolutional neural networks to establish long dependencies in video processing, this invention proposes an integrated approach that integrates local and global features with minimal learning cost and parameter optimization, thereby achieving accurate video action recognition. Specifically, in the shallow layers, 3D deep convolutions are used to reduce computational burden and provide local connectivity and translation variance. To further compute local feature correlations, a self-attention mechanism is applied within a small spatial window around each pixel. In the deep layers, global self-attention is used to construct long-term labeled dependencies from distant frames in the video. Specifically, spatial and channel attention are used to capture global interactions. Finally, a general and robust video action recognition backbone network is built by progressively integrating blocks in a hierarchical manner. Summary of the Invention

[0005] This invention proposes a video action recognition method based on local and global attention mechanisms. This method combines convolutional neural networks and Transformers. It extracts local features by utilizing local attention in shallow layers to reduce parameters and eliminate redundancy, and embeds spatial and channel self-attention mechanisms in deep layers to achieve global dependence of high-level features. The technical solution employed in this method includes the following steps:

[0006] Step 1: Design a local modeling module that computes self-attention locally within a non-overlapping window. This window is divided into a label map that extracts features from the local spatiotemporal context of a small 3D domain.

[0007] Step 2: Embed a convolutional feedforward neural network in the local modeling module to capture local continuity and location information;

[0008] Step 3: Design a global modeling module that applies an attention mechanism to pixel-level markers, where each transposed marker abstracts global information;

[0009] Step 4: Embed convolutional layers with skip connections in the global modeling module to form LGATNet, a network based on convolutional neural networks and Transformers; Step 5: Without pre-training, LGATNet learns model parameters on a public dataset for action recognition and outputs corresponding test results on the test set;

[0010] The video action recognition method based on local and global attention mechanisms proposed in this invention has the following advantages over traditional video action recognition methods:

[0011] (1) This invention proposes a video action recognition method that combines convolutional neural networks and Transformer. By using local attention to extract local features in shallow structures and embedding spatial and channel self-attention mechanisms in deep structures, the method achieves global dependence of high-level features, thereby improving the accuracy and real-time performance of video action recognition.

[0012] (2) The local modeling module and global modeling module proposed in this invention can fully explore the local and global features of video information while reducing the computational resources required by the model, thereby improving the applicability of the model. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of video action recognition based on local and global attention mechanisms proposed in this invention.

[0014] Figure 2 This is a schematic diagram of a local modeling module.

[0015] Figure 3 A schematic diagram of the global modeling module. Detailed Implementation

[0016] The method described in this invention will be explained in detail with reference to the accompanying drawings and embodiments.

[0017] A diagram illustrating video action recognition based on local and global attention mechanisms is shown below. Figure 1 As shown. The general process of this method is as follows: First, multiple frames of images are sampled from the input video and then input into LGATNet. The local modeling module calculates self-attention locally within a non-overlapping window and extracts features from the local spatiotemporal context of a small 3D domain. Then, the global modeling module applies an attention mechanism to pixel-level labels and abstracts global information through transposed labels. Finally, these feature information completes action recognition through mean pooling layers and fully connected layers.

[0018] Step 1: Design a local modeling module that computes self-attention locally within a non-overlapping window. This window is divided into a label map that extracts features from the local spatiotemporal context of a small 3D domain.

[0019] Given a feature map consisting of labels of size T×H×W and a 3D window of size P×M×M, where the window uniformly divides the input in a non-overlapping manner, i.e., the input labels are divided into... Multiple non-overlapping 3D windows are used, and multi-head attention is deployed in each window. Given an R... L×D The visual features are defined in dimensionality, where L is the total number of slices and D is the total number of channels. Assume there are N... W Several different windows, each containing L W There are N cut blocks, where L = NW ×L W First, the attention level for each head is defined as:

[0020]

[0021] Where Q = XW Q K = XW K V = XW V It is a linear projection of the input X. W i (i∈Q,K,V) represents the projected weights of Q,K,V. Since L is large in shallow layers, leading to high computational cost, the window attention is designed as follows:

[0022]

[0023] in Is input X W Linear projection of.

[0024] Step 2: Embed a convolutional feedforward neural network in the local modeling module to capture local continuity and location information;

[0025] Before the first fully connected layer and the GELU activation function, a convolutional layer with a kernel size of 3×3×3, a padding size of 1, and skip connections is added to capture the local continuity and location information of the features.

[0026] Step 3: Design a global modeling module that applies an attention mechanism to pixel-level markers, where each transposed marker abstracts global information;

[0027] Spatial attention is defined as:

[0028]

[0029] Q s =XW s Q K s =XW s K V s =XW s V It is a linear projection of the input X. Channel attention is defined as:

[0030]

[0031] Q c =XW c Q K c =XW c K V c =XWc V It is a linear projection of the input X. Next, spatial attention and channel attention are fused as follows:

[0032] Attention(Q,K,V)=Attention s (Q,K,V)+Attention c (Q,K,V) (5)

[0033] Step 4: Embed convolutional layers with skip connections in the global modeling module to form LGATNet, a network based on convolutional neural networks and Transformers;

[0034] Before the first fully connected layer, a convolutional layer with a kernel size of 3×3×3, a padding size of 1, and skip connections is added to form LGATNet, a network based on convolutional neural networks and Transformers.

[0035] Step 5: Without pre-training, LGATNet learns model parameters on a public dataset for action recognition and outputs the corresponding test results on the test set;

[0036] LGATNet is not pre-trained on the large-scale dataset ImageNet, but only trained on the dataset Kinetics400, and then transferred to the dataset Something-Something V1 & V2 for fine-tuning and testing to verify its action recognition performance.

Claims

1. A video action recognition method based on local and global attention mechanisms, characterized in that, In shallow layers, 3D deep convolutions are used to reduce computational burden, and convolutions provide local connectivity and translation variance. Self-attention is applied within a small spatial window around each pixel. In deep layers, global self-attention is used to construct the network, and spatial and channel attention is used to capture global interactions. By progressively integrating blocks in a hierarchical manner, a video action recognition backbone network is built, specifically including the following steps: 1) Design a local modeling module that computes self-attention locally within a non-overlapping window. This window is divided into a label map that extracts features from the local spatiotemporal context of a small 3D domain. The local modeling module includes a first deep 3D convolutional layer, a first dimension reset layer, a first linear block, multiple first linear layers, a second linear layer, a second dimension reset layer, a first batch of normalization layers, a second deep 3D convolutional layer, a third deep 3D convolutional layer, a first activation function, and a fourth deep 3D convolutional layer. 2) Embed a convolutional feedforward neural network in the local modeling module to capture local continuity and location information; 3) Design a global modeling module that applies an attention mechanism to pixel-level labels. Each transposed label abstracts global information. The global modeling module includes a fifth deep 3D convolutional layer, a third-dimensional reset layer, a second linear block, multiple third linear layers, a fourth linear layer, a fourth-dimensional reset layer, a fifth linear layer, a second batch normalization layer, a fifth-dimensional reset layer, a sixth deep 3D convolutional layer, a sixth-dimensional reset layer, a sixth linear layer, a second activation function, and a seventh linear layer. 4) Embed convolutional layers with skip connections in the global modeling module to form LGATNet, a network based on convolutional neural networks and Transformers. LGATNet includes a first embedding layer, the local modeling module, a second embedding layer, the local modeling module, a third embedding layer, the global modeling module, a fourth embedding layer, the global modeling module, and a classification layer. 5) Without pre-training, LGATNet learns model parameters on public datasets for action recognition and outputs corresponding test results on the test set.

2. The video action recognition method based on local and global attention mechanisms as described in claim 1, characterized in that, Given a by Size markings and The feature map consists of three-dimensional windows of varying sizes, where the windows uniformly divide the input in a non-overlapping manner, i.e., the input labels are divided into... Given a non-overlapping 3D window, and deploying multi-head attention in each window, given a... Dimensional visual features, among which That is the total number of pieces. This is the total number of channels, assuming there are... Several different windows, each containing One piece, of which First, the attention of each head is defined as: ， in , , It is input linear projection, express The projection weights, due to the shallow layers The large size leads to high computational cost, therefore the window attention is designed to be... ， in , , It is input Linear projection of.

3. The video action recognition method using local and global attention mechanisms as described in claim 1, characterized in that, Add a kernel of size [size missing] before the first fully connected layer and the GELU activation function. A convolutional layer with padding size of 1 and containing skip connections is used to capture the local continuity and location information of features.

4. The video action recognition method based on local and global attention mechanisms as described in claim 1, characterized in that, Spatial attention is defined as: ， in , , It is input The linear projection of the channel attention is defined as: ， in , , It is input The linear projection, then the fusion of spatial attention and channel attention, is as follows: 。 5. The video action recognition method based on local and global attention mechanisms as described in claim 1, characterized in that, Add a kernel of size [size missing] before the first fully connected layer. LGATNet, a network based on convolutional neural networks and Transformers, consists of convolutional layers with a padding size of 1 and containing skip connections.

6. The video action recognition method based on local and global attention mechanisms as described in claim 1, characterized in that, LGATNet is not pre-trained on the large-scale dataset ImageNet, but only trained on the dataset Kinetics400, and then transferred to the dataset Something-Something V1 & V2 for fine-tuning and testing to verify its action recognition performance.

Images