Action recognition system and method based on temporal aggregation and gated Transformer

Abstract

The invention discloses an action recognition system and method based on time series aggregation and gated Transformer. The system comprises: a feature extraction module, a first 3D-ResNet network, a first downsampling module, a second 3D-ResNet network, a second downsampling module, a first Transformer network, a third downsampling module, a second Transformer network, a global average pooling module and a fully connected layer connected in sequence; the method comprises: feature extraction; local feature enhancement; global feature enhancement; action recognition; the invention introduces a depth-separable convolution module into a shallow network, designs a convolution operator, and introduces a gated unit into a deep network, so as to effectively integrate the advantages of a convolutional neural network and a Transformer network, establish an action recognition system that takes into account both local and global spatiotemporal relationships, realize effective extraction of appearance features and motion features of video actions, and has the characteristics of reducing background information interference, reducing model calculation amount, and improving action recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of video understanding and computer vision technology, and in particular to an action recognition system and method based on temporal aggregation and gated Transformer. Background Art

[0002] According to the different methods of obtaining video features by network models, action recognition methods can be divided into two categories: traditional machine learning methods and deep learning-based methods. In recent years, with the rapid development of professional computing chips, computing costs have dropped rapidly, making it possible to use deep learning algorithms on a large scale, thereby significantly improving the accuracy of action recognition without increasing costs. Therefore, action recognition methods based on deep learning algorithms have received widespread attention from scholars.

[0003] In action recognition methods based on deep learning algorithms, most methods only consider spatial features and ignore the temporal relationship between video frames, while the actions in the video have a temporal relationship. Spatial features have rich action details, while temporal features have a temporal relationship. Therefore, on the basis of spatial modeling, it is necessary to perform temporal modeling on video features to fuse the features of each frame of the video. Existing methods directly learn spatial features and temporal features between adjacent frames from video clips, and then use the learned high-level features for classification. However, since video data has similarities in local spatiotemporal neighborhoods, this local property easily introduces a large number of inefficient calculations.

[0004] Li K et al. proposed a unified framework combining 3D convolution and spatiotemporal attention in the paper "UniFormer: Unified Transformer for Efficient Spatial-Temporal Representation Learning" (International Conference on LearningRepresentations). When extracting spatiotemporal features of videos, the existing technology lacks the ability to capture global dependencies due to the limited receptive field of convolution and pooling operations. The attention network can model long time series and model the spatiotemporal features of videos by combining convolution and attention operations. The specific steps are as follows: ① The original image is subjected to a three-dimensional deep convolution operation to extract the spatiotemporal features of the video; ② The traditional spatial downsampling operation is replaced by spatiotemporal downsampling to generate a four-dimensional feature map with time information; ③ In the shallow layer of the backbone network, three-dimensional deep convolution is used to extract local neighborhood information and learn spatiotemporal feature representation; ④ In the deep layer of the backbone network, the spatiotemporal Transformer network is used to obtain the features of the global video frame and establish long-term temporal dependencies; ⑤ In the PatchEmbedding stage, a convolution-style dynamic position encoding is used to unify the encoded position information; ⑥ Through the average pooling layer and the linear classification layer, the final prediction result is output.

[0005] TianY et al. proposed an event adaptive network that can be plugged and played in convolutional networks in "EAN: Event Adaptive Network for Enhanced Action Recognition" (International Journal of Computer Vision, 2022, 130(10): 2453-2471). The content is: ① The Event Adaptive Block (EAB) is proposed to enhance the scale adaptive modeling capability of the convolution operator. This module can input the scale information of key events in the video and adaptively and dynamically select the fusion of fixed-scale spatiotemporal convolution kernels; ② The Sparse Object Interaction Transformer (SOI-Tr) is proposed to adaptively select the most important objects in the action to construct a sparse interaction graph. That is, for a given deep video feature, the sparse object localization network first outputs saliency maps of several objects, and then uses Transformer to model the long-range interactions between these objects, averages the original video features and interaction features, and generates a global video representation; ③ A latent motion coding (LMC) module is proposed to capture short-term motion information in local video clips, which promotes the ability to distinguish actions with only subtle differences.

[0006] Hao Y et al. proposed an attention in attention (AIA) method in the paper "Attention in attention: Modeling context correlation for efficient video classification" (IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(10): 7120-7132), which inserts channel context into the spatiotemporal attention module and its reverse variant. The specific contents are as follows: ① The context instance information of video features is aggregated along a specific dimension using global average pooling and maximum pooling operations. The first attention block of the AIA module uses context information to guide the calculation of the gating weight of another attention; ② Design a module that inserts channel attention into spatiotemporal attention to enhance the feature extraction capability of each dimension; ③ Design a reverse variant of the above module to insert spatiotemporal attention into channel attention; ④ The AIA module is used in convolutional neural networks and Transformer networks, which can effectively improve the accuracy of the original model.

[0007] In the paper "Stand-alone inter-frame attention in video models" (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2022:3192-3201), Long F et al. designed a stand-alone inter-frame attention (SIFA) module that can be inserted into common convolutional neural networks and Transformer networks. The specific content of this method is: ① Each spatial position of the current frame is used as a query (Query, Q), and the local neighborhood in the next frame is regarded as keys / values ​​(K / V); ② The similarity measure between Q and V is learned as an independent attention, and all neighborhoods in the local area of ​​adjacent frames are aggregated to weighted average time aggregation features; ③ A two-dimensional convolution operator is used to predict the offset of each spatial position in the neighborhood centered on the query position to increase the spatiotemporal features of the video frame; ④ The SIFA module is used in convolutional neural networks and Transformer networks, which can effectively improve the accuracy of the original model.

[0008] In the paper "Global Temporal Difference Network for Action Recognition" (IEEE Transactions on Multimedia, 2022), Xie Z et al. proposed a global temporal difference (GTD) module for action recognition, which consists of two submodules, namely the global aggregation module (GAM) and the global difference module (GTM). The content is: ① In the global aggregation module, the global prior knowledge is learned by aggregating the visual feature sequence of the video into a global vector, and different time dependency strategies are used to learn more global features; ② In the global difference module, the difference vector sequence of the video is obtained by subtracting each local vector from the global vector, and the temporal dependency of the features is learned using the channel self-attention operation; ③ The global temporal difference module is plug-and-play, which is embedded in the ResNet-50 network, replacing the original residual block, and using the fully connected layer as the classification layer to provide predictions for the action recognition results.

[0009] However, the prior art has the following problems:

[0010] (1) When extracting video features through convolutional neural networks, the existing technology does not fully consider the temporal dependency in the feature graphs obtained by operations such as convolution and downsampling, making it difficult to effectively identify video actions;

[0011] (2) Although 3D convolution has the advantage of being able to learn video temporal features compared to 2D convolution, 3D convolution can only learn local temporal information in video clips, and has problems such as slow convergence speed and high computational cost;

[0012] (3) When using the Transformer network to capture long-distance temporal dependencies, since there is a lot of similarity information in the video and the influence of background information is not taken into account, a large amount of inefficient calculations will be introduced, resulting in a large number of model parameters and slow inference speed. Summary of the invention

[0013] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an action recognition system and method based on temporal aggregation and gated Transformer. By introducing a depth-separable convolution module in a shallow network and designing a convolution operator, and introducing a gated unit in a deep network, the advantages of the convolutional neural network and the Transformer network are effectively integrated to establish an action recognition system that takes into account both local and global spatiotemporal relationships, thereby achieving effective extraction of video action appearance features and motion features, reducing background information interference, reducing model calculations, and improving action recognition accuracy.

[0014] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0015] An action recognition system based on temporal aggregation and gated Transformer, comprising a feature extraction module, a first 3D-ResNet network, a first downsampling module, a second 3D-ResNet network, a second downsampling module, a first Transformer network, a third downsampling module, a second Transformer network, a global average pooling module and a fully connected layer connected in sequence;

[0016] Feature extraction module: extracts the spatiotemporal features of the input video, compresses the temporal and spatial dimensions of the input data, performs layer normalization, and inputs the extracted spatiotemporal features into the first 3D-ResNet network;

[0017] The first 3D-ResNet network: enhances the local spatiotemporal feature information of the video, and inputs the enhanced local spatiotemporal features into the first downsampling module;

[0018] The first downsampling module: extracts spatiotemporal features, compresses the spatial dimension of the input data, and performs layer normalization operations to reduce the computational complexity of the second 3D-ResNet network and ensure accuracy;

[0019] The second 3D-ResNet network: enhances the local spatiotemporal feature information of the output features of the first downsampling module, and inputs the enhanced local spatiotemporal features into the second downsampling module;

[0020] The second downsampling module: extracts spatiotemporal features, compresses the spatial dimension of the input data, and performs layer normalization operations to reduce the amount of calculation of the first Transformer network and ensure accuracy;

[0021] The first Transformer network: enhances the global spatiotemporal feature information of the output features of the second downsampling module, and inputs the enhanced global spatiotemporal features into the third downsampling module;

[0022] The third downsampling module: extracts spatiotemporal features, compresses the spatial dimension of the input data, and performs layer normalization operations to reduce the amount of calculation of the second Transformer network and ensure accuracy;

[0023] Second Transformer network: enhances the global spatiotemporal feature information of the output features of the third downsampling module, and inputs the enhanced global spatiotemporal features into the global average pooling module and the fully connected layer;

[0024] Global average pooling module and fully connected layer: classify the spatiotemporal features output by the second Transformer network and output action recognition results.

[0025] The first 3D-ResNet network includes three local feature aggregation modules connected in sequence, specifically:

[0026] The first local feature aggregation module: enhances the local spatiotemporal feature information of the spatiotemporal features extracted by the feature extraction module, and inputs the enhanced local spatiotemporal features into the second local feature aggregation module;

[0027] The second local feature aggregation module performs local spatiotemporal feature information enhancement on the local spatiotemporal features output by the first local feature aggregation module, and inputs the enhanced local spatiotemporal features into the third local feature aggregation module;

[0028] The third local feature aggregation module enhances the local spatiotemporal feature information of the local spatiotemporal features output by the second local feature aggregation module, and inputs the enhanced local spatiotemporal features into the first downsampling module.

[0029] The second 3D-ResNet network includes four local feature aggregation modules connected in sequence, specifically:

[0030] The fourth local feature aggregation module: enhances the local spatiotemporal feature information of the spatiotemporal features output by the first downsampling module, and inputs the enhanced local spatiotemporal features into the fifth local feature aggregation module;

[0031] The fifth local feature aggregation module performs local spatiotemporal feature information enhancement on the local spatiotemporal features output by the fourth local feature aggregation module, and inputs the enhanced local spatiotemporal features into the sixth local feature aggregation module;

[0032] The sixth local feature aggregation module: performs local spatiotemporal feature information enhancement on the local spatiotemporal features output by the fifth local feature aggregation module, and inputs the enhanced local spatiotemporal features into the seventh local feature aggregation module;

[0033] The seventh local feature aggregation module enhances the local spatiotemporal feature information of the local spatiotemporal features output by the sixth local feature aggregation module, and inputs the enhanced local spatiotemporal features into the second down-sampling module.

[0034] The first Transformer network includes a first gating unit, a first attention module, a second attention module, a third attention module, a second gating unit, a fourth attention module, a fifth attention module and a sixth attention module connected in sequence;

[0035] The first gating unit: enhances the motion feature information of the spatiotemporal features output by the second downsampling module, and inputs the obtained motion features into the first attention module;

[0036] The first attention module: enhances the global spatiotemporal feature information of the motion features output by the first gating unit, and inputs the obtained global spatiotemporal features into the second attention module;

[0037] The second attention module: enhances the global spatiotemporal feature information of the global spatiotemporal feature output by the first attention module, and inputs the obtained global spatiotemporal feature into the third attention module;

[0038] The third attention module: enhances the global spatiotemporal feature information of the spatiotemporal features output by the second attention module, and inputs the obtained spatiotemporal features into the second gating unit;

[0039] The second gating unit: enhances the motion feature information of the global spatiotemporal features output by the third attention module, and inputs the obtained motion features into the fourth attention module;

[0040] The fourth attention module: enhances the global spatiotemporal feature information of the motion features output by the second gating unit, and inputs the obtained global spatiotemporal features into the fifth attention module;

[0041] The fifth attention module: enhances the global spatiotemporal feature information of the global spatiotemporal feature output by the fourth attention module, and inputs the obtained global spatiotemporal feature into the sixth attention module;

[0042] The sixth attention module: enhances the global spatiotemporal feature information of the global spatiotemporal features output by the fifth attention module, and inputs the obtained global spatiotemporal features into the third downsampling module.

[0043] The second Transformer network includes a third gating unit, a seventh attention module and an eighth attention module connected in sequence;

[0044] The third gating unit: enhances the motion feature information of the spatiotemporal features output by the third downsampling module, and inputs the obtained motion features into the seventh attention module;

[0045] The seventh attention module: enhances the global spatiotemporal feature information of the motion features output by the third gating unit, and inputs the obtained global spatiotemporal features into the eighth attention module;

[0046] The eighth attention module: enhances the global spatiotemporal feature information of the global spatiotemporal features output by the seventh attention module, and inputs the obtained global spatiotemporal features into the global average pooling module and the fully connected layer.

[0047] The local feature aggregation module specifically includes:

[0048] The first depth-wise separable convolution module: uses a convolution operator with a kernel size of 3×3×3, a step size of 1×1×1, and a padding of 1×1×1 to process the input spatiotemporal features and obtain a feature map of the same size as the input;

[0049] The first residual structure: adds the spatiotemporal features input into the first depthwise separable convolution module to the feature map output by the first depthwise separable convolution module to realize dynamic position encoding of the video features and obtain appearance features;

[0050] The first batch normalization module: performs batch normalization on the appearance features output by the first residual structure, and inputs the obtained appearance features into the first convolution operator;

[0051] The first convolution operator is used to enhance the appearance features of the video frames processed by the first batch of normalization modules, and input the obtained appearance features into the second depth-separable convolution module;

[0052] Second depth-wise separable convolution module: The appearance features output by the first convolution operator are processed through channel-by-channel convolution with a convolution kernel size of 5×1×1, a step size of 1×1×1, and a padding of 2×0×0 to process the spatiotemporal information of the input features, and then through point-by-point convolution with a convolution kernel size of 1×1×5, a step size of 1×1×1, and a padding of 0×2×2 to process the channel information of the input features to obtain a feature map of the same size as the input size;

[0053] The second residual structure: adds the appearance features output by the first batch of normalization modules to the feature maps output by the second depth-separable convolution module to obtain enhanced local spatiotemporal features;

[0054] The second batch normalization module: performs batch normalization on the local spatiotemporal features output by the second residual structure, and inputs the obtained spatiotemporal features into the first fully connected layer;

[0055] The first fully connected layer: maps the spatiotemporal features obtained by the second batch normalization module to the sample label space, performs global feature extraction and nonlinear transformation, and obtains a feature map of the same size as the input size;

[0056] The third residual structure: add the feature map output by the first fully connected layer to the spatiotemporal features output by the second batch normalization module to obtain enhanced local spatiotemporal features.

[0057] The gate control unit specifically includes:

[0058] The third depth-wise separable convolution module processes the input features using a convolution operator with a kernel size of 3×3×3, a step size of 1×1×1, and a padding of 1×1×1 to obtain a feature map of the same size as the input.

[0059] The fourth residual structure: adding the spatiotemporal features input into the third depth-separable convolution module to the feature map output by the third depth-separable convolution module to realize dynamic position encoding of the video features and obtain the appearance features;

[0060] The first global average pooling module: performs an adaptive global average pooling operation on the spatiotemporal features output by the fourth residual structure, and inputs the obtained spatiotemporal features into the second fully connected layer;

[0061] The second fully connected layer maps the spatiotemporal features obtained by the first global average pooling module to the sample tag space, performs global feature extraction and nonlinear transformation, and processes the channel information of the input features;

[0062] Matrix multiplication structure: The appearance features output by the fourth residual structure are matrix multiplied with the output results of the second fully connected layer to obtain enhanced spatiotemporal features;

[0063] Cascade difference structure: The appearance features output by the fourth residual structure are cascaded and subtracted from the spatiotemporal features obtained by the matrix multiplication structure to obtain spatiotemporal features that suppress background information;

[0064] The first layer normalization module: performs layer normalization on the spatiotemporal features obtained by the matrix multiplication structure, and inputs the obtained spatiotemporal features into the first spatiotemporal Transformer module;

[0065] The first spatiotemporal Transformer module: The spatiotemporal features output by the first-layer normalization module are passed through Multi-Head Attention to learn the long-distance temporal dependencies of video actions;

[0066] The fifth residual structure: adds the spatiotemporal features output by the first-layer normalization module to the output results of the first spatiotemporal Transformer module to obtain enhanced motion features;

[0067] The second layer normalization module: performs layer normalization on the motion features output by the fifth residual structure, and inputs the obtained spatiotemporal features into the third fully connected layer;

[0068] The third fully connected layer: maps the spatiotemporal features obtained by the second-layer normalization module to the sample label space, performs global feature extraction and nonlinear transformation, and obtains a feature map of the same size as the input size;

[0069] The sixth residual structure: adds the spatiotemporal features output by the second-layer normalization module to the feature map output by the third fully connected layer to obtain enhanced global spatiotemporal features;

[0070] Cascaded sum structure: The spatiotemporal features output by the cascaded difference structure are cascaded and summed with the global spatiotemporal features output by the sixth residual structure to obtain enhanced motion features.

[0071] The attention module specifically includes:

[0072] Fourth depth-wise separable convolution module: The input features are processed using a convolution operator with a kernel size of 3×3×3, a step size of 1×1×1, and a padding of 1×1×1 to obtain a feature map of the same size as the input size;

[0073] The seventh residual structure: adding the spatiotemporal features input into the fourth depth-separable convolution module to the feature map output by the fourth depth-separable convolution module to realize dynamic position encoding of the video features and obtain the appearance features;

[0074] The third layer normalization module: performs layer normalization on the input spatiotemporal features and inputs the obtained spatiotemporal features into the second spatiotemporal Transformer module;

[0075] The second spatiotemporal Transformer module: The spatiotemporal features output by the third-layer normalization module are passed through Multi-Head attention to learn the long-distance temporal dependencies of video actions;

[0076] The eighth residual structure: the spatiotemporal features output by the third-layer normalization module are added to the output results of the second spatiotemporal Transformer module to obtain enhanced motion features;

[0077] The fourth layer standardization module: performs layer standardization on the motion features output by the eighth residual structure, and inputs the obtained spatiotemporal features into the fourth fully connected layer;

[0078] The fourth fully connected layer: maps the spatiotemporal features obtained by the fourth-layer normalization module to the sample label space, performs global feature extraction and nonlinear transformation, and obtains a feature map of the same size as the input size;

[0079] Ninth residual structure: Add the spatiotemporal features output by the fourth-layer standardization module to the feature map output by the fourth fully connected layer to obtain enhanced global spatiotemporal features.

[0080] The action recognition method based on temporal aggregation and gated Transformer includes the following steps:

[0081] Step 1, feature extraction: extract the spatiotemporal features of the input video using the feature extraction module;

[0082] Step 2, local feature enhancement: the spatiotemporal features extracted in step 1 are sequentially input into the first 3D-ResNet network, the first downsampling module, the second 3D-ResNet network and the second downsampling module, and the spatiotemporal features with enhanced local spatiotemporal feature information are output;

[0083] Step 3: Global feature enhancement: The spatiotemporal features output from step 2 are sequentially input into the first Transformer network, the third downsampling module, and the second Transformer network, and the spatiotemporal features with enhanced global spatiotemporal feature information are output;

[0084] Step 4: Action recognition: Input the spatiotemporal features output from step 3 into the global average pooling module and the fully connected layer for classification, and output the action recognition results.

[0085] Compared with the prior art, the beneficial effects of the present invention are:

[0086] 1. The present invention introduces a deep separable convolution module into a shallow network, and processes the appearance features and channel features respectively through two deep separable convolution operations, thereby enhancing the local spatiotemporal feature information of the video, thereby effectively reducing the information redundancy of the video. Compared with the prior art, it has the characteristics of reducing background information interference and reducing the amount of model calculation.

[0087] 2. The present invention introduces a gating unit into the deep network, suppresses the background information of the video through cascade subtraction, extracts the global information of the video using the time and space attention mechanism, and enhances the motion information of the video through cascade summation to capture the long-term temporal dependency of the video action. Compared with the prior art, the accuracy of action recognition is improved.

[0088] 3. The present invention adopts feature extraction operations and multiple downsampling operations. Compared with the prior art, it can not only effectively compress feature dimensions and reduce model calculation amount, but also ensure the accuracy of the model.

[0089] In summary, compared with the prior art, the present invention introduces a deep separable convolution module into the shallow network, designs a convolution operator, and introduces a gating unit into the deep network, so as to effectively integrate the advantages of the convolutional neural network and the Transformer network, establish an action recognition system that takes into account both local and global spatiotemporal relationships, and realizes the effective extraction of video action appearance features and motion features, reduces background information interference, reduces the model calculation amount, and improves the action recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 It is a structural schematic diagram of the action recognition system of the present invention.

[0091] Figure 2 It is a structural schematic diagram of the local feature aggregation module in the present invention.

[0092] Figure 3 It is a schematic diagram of the structure of the gate control unit in the present invention.

[0093] Figure 4 Schematic diagram of the structure of the attention module in the present invention.

[0094] Figure 5 This is a flow chart of the action recognition method of the present invention. DETAILED DESCRIPTION

[0095] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0096] See also Figure 1 , an action recognition system based on temporal aggregation and gated Transformer, comprising a feature extraction module, a first 3D-ResNet network, a first downsampling module, a second 3D-ResNet network, a second downsampling module, a first Transformer network, a third downsampling module, a second Transformer network, a global average pooling module and a fully connected layer connected in sequence;

[0097] Feature extraction module: Use the convolution operator with a kernel size of 3×4×4, a step size of 2×4×4, and a padding of 1×0×0 to extract the spatiotemporal features of the input video, compress the time and space dimensions of the input data, and then perform layer normalization operations, and input the extracted spatiotemporal features into the first 3D-ResNet network;

[0098] Specifically, it consists of a convolution operator with a convolution kernel size of 3×4×4 and a step size of 2×4×4, a LayerNormalization normalization operation, and a linear rectification function ReLu, which compresses the time and space dimensions of the input data at the same time.

[0099] The calculation formula of the linear rectification function ReLu is as follows:

[0100]

[0101] Among them, x is the element in the input feature map.

[0102] The feature extraction module in the present invention adopts the existing 3D convolution operator, which can effectively extract video features. If the convolution operator is replaced by a decomposed 2D convolution operation and a 1D convolution operation, the purpose of the present invention can also be achieved.

[0103] The first 3D-ResNet network: enhances the local spatiotemporal feature information of the video and inputs the enhanced spatiotemporal features into the first downsampling module;

[0104] The first downsampling module: uses a convolution operator with a kernel size of 1×2×2, a step size of 1×2×2, and a padding of 0 to extract spatiotemporal features, compress the spatial dimension of the input data, and perform layer normalization on the obtained output to reduce the computational complexity of the second 3D-ResNet network and ensure accuracy;

[0105] The second 3D-ResNet network: enhances the local spatiotemporal feature information of the output features of the first downsampling module, and inputs the enhanced spatiotemporal features into the second downsampling module;

[0106] The second downsampling module: uses a convolution operator with a kernel size of 1×2×2, a step size of 1×2×2, and a padding of 0 to extract spatiotemporal features, compress the spatial dimension of the input data, and perform layer normalization on the obtained output to reduce the computational complexity of the first Transformer network and ensure accuracy;

[0107] The first Transformer network: enhances the global spatiotemporal feature information of the output features of the second downsampling module, and inputs the enhanced spatiotemporal features into the third downsampling module;

[0108] The third downsampling module: uses a convolution operator with a kernel size of 1×2×2, a step size of 1×2×2, and a padding of 0 to extract spatiotemporal features, compress the spatial dimension of the input data, and perform layer normalization on the obtained output to reduce the computational complexity of the second Transformer network and ensure accuracy;

[0109] The second Transformer network: enhances the global spatiotemporal feature information of the output features of the third downsampling module, and inputs the enhanced spatiotemporal features into the global average pooling module;

[0110] Global average pooling module and fully connected layer: classify the spatiotemporal features output by the second Transformer network and output action recognition results;

[0111] Among them, the three downsampling operations are achieved by setting a convolution operator with a convolution kernel size of 1×2×2 and a step size of 1×2×2 and LayerNormalization normalization, which only compresses the spatial dimension of the data. These downsampling operations can not only effectively reduce the amount of model calculation, but also ensure the accuracy of the model.

[0112] The present invention combines the advantages of convolutional neural networks and Transformer networks to achieve effective extraction of video action appearance features and motion features, and establishes an action recognition system that takes into account both local and global spatiotemporal relationships. The global dependency relationship is constructed by using a spatiotemporal Transformer attention module, which can achieve effective modeling of long-distance temporal dependencies. If temporal Transformer attention and spatial Transformer attention are used, the purpose of the present invention can also be achieved.

[0113] The first 3D-ResNet network includes three local feature aggregation modules connected in sequence, specifically:

[0114] The first local feature aggregation module is used to enhance the local spatiotemporal feature information of the spatiotemporal features extracted by the feature extraction module, and input the enhanced spatiotemporal features into the second local feature aggregation module;

[0115] The second local feature aggregation module: enhances the local spatiotemporal feature information of the spatiotemporal features output by the first local feature aggregation module, and inputs the enhanced spatiotemporal features into the third local feature aggregation module;

[0116] The third local feature aggregation module: enhances the local spatiotemporal feature information of the spatiotemporal features output by the second local feature aggregation module, and inputs the enhanced spatiotemporal features into the first downsampling module.

[0117] The second 3D-ResNet network includes four local feature aggregation modules connected in sequence, specifically:

[0118] The fourth local feature aggregation module: enhances the local spatiotemporal feature information of the spatiotemporal features output by the first downsampling module, and inputs the enhanced spatiotemporal features into the fourth local feature aggregation module;

[0119] The fifth local feature aggregation module: enhances the local spatiotemporal feature information of the spatiotemporal features output by the fourth downsampling module, and inputs the enhanced spatiotemporal features into the sixth local feature aggregation module;

[0120] The sixth local feature aggregation module: enhances the local spatiotemporal feature information of the spatiotemporal features output by the fifth downsampling module, and inputs the enhanced spatiotemporal features into the seventh local feature aggregation module;

[0121] The seventh local feature aggregation module enhances the local spatiotemporal feature information of the spatiotemporal features output by the sixth downsampling module, and inputs the enhanced spatiotemporal features into the second downsampling module.

[0122] The first Transformer network includes a first gating unit, a first attention module, a second attention module, a third attention module, a second gating unit, a fourth attention module, a fifth attention module and a sixth attention module connected in sequence;

[0123] The first gating unit: enhances the motion feature information of the spatiotemporal features output by the second downsampling module, and inputs the obtained motion features into the first attention module;

[0124] The first attention module: enhances the global spatiotemporal feature information of the motion features output by the first gating unit, and inputs the obtained spatiotemporal features into the second attention module;

[0125] Second attention module: enhance the global spatiotemporal feature information of the spatiotemporal features output by the first attention module, and input the obtained spatiotemporal features into the third attention module;

[0126] The third attention module: enhances the global spatiotemporal feature information of the spatiotemporal features output by the second attention module, and inputs the obtained spatiotemporal features into the second gating unit;

[0127] The second gating unit: enhances the motion feature information of the spatiotemporal features output by the third attention module, and inputs the obtained motion features into the fourth attention module;

[0128] The fourth attention module: enhances the global spatiotemporal feature information of the motion features output by the second gating unit, and inputs the obtained spatiotemporal features into the fifth attention module;

[0129] The fifth attention module: enhances the global spatiotemporal feature information of the spatiotemporal features output by the fourth attention module, and inputs the obtained spatiotemporal features into the sixth attention module;

[0130] The sixth attention module: enhances the global spatiotemporal feature information of the spatiotemporal features output by the fifth attention module, and inputs the obtained spatiotemporal features into the third downsampling module.

[0131] The second Transformer network includes a third gating unit, a seventh attention module and an eighth attention module connected in sequence;

[0132] The third gating unit: enhances the motion feature information of the spatiotemporal features output by the third downsampling module, and inputs the obtained motion features into the seventh attention module;

[0133] The seventh attention module: enhances the global spatiotemporal feature information of the motion features output by the third gating unit, and inputs the obtained spatiotemporal features into the eighth attention module;

[0134] The eighth attention module: enhances the global spatiotemporal feature information of the spatiotemporal features output by the seventh attention module, and inputs the obtained spatiotemporal features into the global average pooling module and the fully connected layer.

[0135] See also Figure 2 , the local feature aggregation module specifically includes:

[0136] The first depth-wise separable convolution module: uses a convolution operator with a kernel size of 3×3×3, a step size of 1×1×1, and a padding of 1×1×1 to process the input spatiotemporal features and obtain a feature map of the same size as the input. This operation uses the convolution operator to encode the spatiotemporal position of the video features and learn the feature representation of the local spatiotemporal neighborhood of the video;

[0137] The first residual structure: using the principle of identity mapping and skip connection, the input of the local feature aggregation module is added to the output of the first depth-separable convolution module to achieve dynamic position encoding of video features and enhance the spatiotemporal feature information of the video;

[0138] The first batch normalization module: performs batch normalization on the appearance features output by the first residual structure, and inputs the obtained appearance features into the convolution operator;

[0139] A first convolution operator: used to enhance the appearance features of the video frames processed by the first batch of normalization modules, and input the obtained appearance features into the second depthwise separable convolution module; the convolution operator has a convolution kernel size of 1×1×1, a step size of 1×1×1, and a padding of 0;

[0140] The second depth-separable convolution module: The appearance features output by the first convolution operator are processed through channel-by-channel convolution with a convolution kernel size of 5×1×1, a step size of 1×1×1, and a padding of 2×0×0 to process the spatiotemporal information of the input features. Then, the channel information of the input features is processed through point-by-point convolution with a convolution kernel size of 1×1×5, a step size of 1×1×1, and a padding of 0×2×2 to obtain a feature map of the same size as the input size. This operation can effectively aggregate local features of the video, reduce the amount of model calculation and parameters, and solve the problem of video redundancy in shallow networks.

[0141] Second residual structure: Using the principle of identity mapping and skip connection, the output of the first batch of normalization modules is added to the output of the second depth-separable convolution module to obtain enhanced local spatiotemporal features;

[0142] The second batch normalization module: performs batch normalization on the spatiotemporal features output by the second residual structure, and inputs the obtained spatiotemporal features into the first fully connected layer;

[0143] The first fully connected layer: maps the spatiotemporal features obtained by the second batch normalization module to the sample tag space, performs global feature extraction and nonlinear transformation, and obtains a feature map of the same size as the input size; specifically, the input features are processed through a linear layer, and the output is passed through a Gaussian error linear unit GELU, and then through a linear layer to obtain a feature map of the same size as the input size;

[0144] The calculation formula of the Gaussian error linear unit GELU is as follows:

[0145] GELU(x)=xP(X≤x)=xΦ(x)

[0146] Where x is an element in the input feature map, and Φ(x) refers to the cumulative distribution function of the Gaussian normal distribution of x, that is:

[0147]

[0148] In the formula, μ refers to the expected value of the input data, and σ refers to the standard deviation of the input data;

[0149] The third residual structure: using the principle of identity mapping and skip connection, the output of the first fully connected layer is added to the output of the second batch of normalization modules to obtain enhanced local spatiotemporal features;

[0150] See also Figure 3 , the gate control unit specifically includes:

[0151] The third depth-wise separable convolution module processes the input features using a convolution operator with a kernel size of 3×3×3, a step size of 1×1×1, and a padding of 1×1×1 to obtain a feature map of the same size as the input.

[0152] The fourth residual structure: using the principle of identity mapping and skip connection, the input of the gated unit is added to the output of the third depth-separable convolution module to achieve dynamic position encoding of video features;

[0153] The first global average pooling module: performs an adaptive global average pooling operation on the spatiotemporal features output by the fourth residual structure to reduce the redundant information of the video, expand the receptive field, and input the obtained spatiotemporal features into the second fully connected layer;

[0154] The second fully connected layer: maps the spatiotemporal features obtained by the first global average pooling module to the sample tag space, performs global feature extraction and nonlinear transformation, and processes the channel information of the input features; specifically, it passes through a linear layer to reduce the channel dimension, passes the obtained output through a linear rectification function ReLu, passes through another linear layer to restore the channel dimension, and passes the output through an S-shaped function Sigmoid;

[0155] The calculation formula of the linear rectification function ReLu is as follows:

[0156]

[0157] Among them, x is the element in the input feature map.

[0158] The calculation formula of the S-type function Sigmoid is as follows:

[0159]

[0160] Among them, x is the element in the input feature map.

[0161] Matrix multiplication structure: Perform matrix multiplication operation on the output of the fourth residual structure and the output result of the second fully connected layer to restore the size of the input features and obtain enhanced spatiotemporal features;

[0162] Cascade difference structure: The output of the fourth residual structure is cascaded and subtracted from the output of the matrix multiplication structure to obtain the spatiotemporal features that suppress background information;

[0163] The first layer normalization module: performs layer normalization on the input spatiotemporal features and inputs the obtained spatiotemporal features into the first spatiotemporal Transformer module;

[0164] The first spatiotemporal Transformer module: The spatiotemporal features output by the first-layer normalization module are passed through multi-head attention to learn the long-distance temporal dependencies of video actions. The calculation process is as follows:

[0165] For multi-head attention, the calculation formula is as follows:

[0166]

[0167]

[0168]

[0169] Among them, H i represents the attention head, n h The number of headers is set to 64. W O Represent the parameters of the linear mappings Q, K, V, and O respectively. k Represents the dimension of the K vector.

[0170] The fifth residual structure: using the principle of identity mapping and skip connection, the output of the first layer normalization module is added to the output of the first spatiotemporal Transformer module to obtain enhanced motion features;

[0171] The second layer normalization module: performs layer normalization on the motion features output by the fifth residual structure, and inputs the obtained spatiotemporal features into the third fully connected layer;

[0172] The third fully connected layer: maps the spatiotemporal features obtained by the second-layer normalization module to the sample tag space, performs global feature extraction and nonlinear transformation, and obtains a feature map of the same size as the input size; specifically, the input features are processed through a linear layer, and the output is passed through a Gaussian error linear unit GELU, and then through another linear layer to obtain a feature map of the same size as the input size;

[0173] The sixth residual structure: using the principle of identity mapping and skip connection, the output of the second layer normalization module is added to the output of the third fully connected layer to obtain the enhanced global spatiotemporal features;

[0174] Cascade sum structure: The spatiotemporal features output by the cascade difference structure are cascaded and summed with the output results of the sixth residual structure to obtain enhanced motion features;

[0175] The gating unit in the present invention adopts the existing 2D convolution operator, which can effectively extract the difference information of the video frames. If other convolution operations are adopted, for example, an improved 2D convolution operator and a 3D convolution operator, the purpose of the present invention can also be achieved.

[0176] See also Figure 4 , the attention module specifically includes:

[0177] Fourth depth-wise separable convolution module: The input features are processed using a convolution operator with a kernel size of 3×3×3, a step size of 1×1×1, and a padding of 1×1×1 to obtain a feature map of the same size as the input size;

[0178] The seventh residual structure: adding the spatiotemporal features input into the fourth depth-separable convolution module to the feature map output by the fourth depth-separable convolution module to realize dynamic position encoding of the video features and obtain the appearance features;

[0179] The third layer normalization module: performs layer normalization on the input spatiotemporal features and inputs the obtained spatiotemporal features into the second spatiotemporal Transformer module;

[0180] The second spatiotemporal Transformer module: The spatiotemporal features output by the third-layer normalization module are passed through multi-head attention to learn the long-distance temporal dependencies of video actions. The calculation process is as follows:

[0181] For multi-head attention, the calculation formula is as follows:

[0182]

[0183] H i =Attention(QW i Q ,KW i K ,VW i V )

[0184]

[0185] Among them, H i represents the attention head, n h is the number of heads, set to 64. i Q , W i K , W i V , W O Represent the parameters of the linear mappings Q, K, V, and O respectively. k Represents the dimension of the K vector.

[0186] The eighth residual structure: using the principle of identity mapping and skip connection, the output of the third-layer normalization module is added to the output of the second spatiotemporal Transformer module to obtain enhanced motion features;

[0187] The fourth layer standardization module: performs layer standardization on the motion features output by the eighth residual structure, and inputs the obtained spatiotemporal features into the fourth fully connected layer;

[0188] The fourth fully connected layer: maps the spatiotemporal features obtained by the fourth-layer normalization module to the sample tag space, performs global feature extraction and nonlinear transformation, and obtains a feature map of the same size as the input size; specifically, the input features are processed through a linear layer, and the output is passed through a Gaussian error linear unit GELU, and then through a linear layer to obtain a feature map of the same size as the input size;

[0189] The ninth residual structure: using the principle of identity mapping and skip connection, the output of the fourth-layer normalization module is added to the output of the fourth fully connected layer to obtain the enhanced global spatiotemporal features;

[0190] See also Figure 5 , the action recognition method based on temporal aggregation and gated Transformer includes the following steps:

[0191] Step 1, feature extraction: extract the spatiotemporal features of the input video using the feature extraction module;

[0192] Step 2, local feature enhancement: the spatiotemporal features extracted in step 1 are sequentially input into the first 3D-ResNet network, the first downsampling module, the second 3D-ResNet network and the second downsampling module, and the spatiotemporal features with enhanced local spatiotemporal feature information are output;

[0193] Step 3: Global feature enhancement: The spatiotemporal features output from step 2 are sequentially input into the first Transformer network, the third downsampling module, and the second Transformer network, and the spatiotemporal features with enhanced global spatiotemporal feature information are output;

[0194] Step 4: Action recognition: Input the spatiotemporal features output from step 3 into the global average pooling module and the fully connected layer for classification, and output the action recognition results.

Claims

1. An action recognition system based on temporal aggregation and gated Transformer, characterized by: It includes a feature extraction module, a first 3D-ResNet network, a first downsampling module, a second 3D-ResNet network, a second downsampling module, a first Transformer network, a third downsampling module, a second Transformer network, a global average pooling module and a fully connected layer connected in sequence; Feature extraction module: extracts the spatiotemporal features of the input video, compresses the temporal and spatial dimensions of the input data, performs layer normalization, and inputs the extracted spatiotemporal features into the first 3D-ResNet network; The first 3D-ResNet network: enhances the local spatiotemporal feature information of the video, and inputs the enhanced local spatiotemporal features into the first downsampling module; The first downsampling module: extracts spatiotemporal features, compresses the spatial dimension of the input data, and performs layer normalization operations to reduce the computational complexity of the second 3D-ResNet network and ensure accuracy; The second 3D-ResNet network: enhances the local spatiotemporal feature information of the output features of the first downsampling module, and inputs the enhanced local spatiotemporal features into the second downsampling module; The second downsampling module: extracts spatiotemporal features, compresses the spatial dimension of the input data, and performs layer normalization operations to reduce the amount of calculation of the first Transformer network and ensure accuracy; The first Transformer network: enhances the global spatiotemporal feature information of the output features of the second downsampling module, and inputs the enhanced global spatiotemporal features into the third downsampling module; The third downsampling module: extracts spatiotemporal features, compresses the spatial dimension of the input data, and performs layer normalization operations to reduce the amount of calculation of the second Transformer network and ensure accuracy; Second Transformer network: enhances the global spatiotemporal feature information of the output features of the third downsampling module, and inputs the enhanced global spatiotemporal features into the global average pooling module and the fully connected layer; Global average pooling module and fully connected layer: classify the spatiotemporal features output by the second Transformer network and output action recognition results.

2. The action recognition system based on temporal aggregation and gated Transformer according to claim 1, characterized in that: The first 3D-ResNet network includes three local feature aggregation modules connected in sequence, specifically: The first local feature aggregation module is used to enhance the local spatiotemporal feature information of the spatiotemporal features extracted by the feature extraction module, and input the enhanced local spatiotemporal features into the second local feature aggregation module; The second local feature aggregation module performs local spatiotemporal feature information enhancement on the local spatiotemporal features output by the first local feature aggregation module, and inputs the enhanced local spatiotemporal features into the third local feature aggregation module; The third local feature aggregation module enhances the local spatiotemporal feature information of the local spatiotemporal features output by the second local feature aggregation module, and inputs the enhanced local spatiotemporal features into the first downsampling module.

3. The action recognition system based on temporal aggregation and gated Transformer according to claim 1, characterized in that: The second 3D-ResNet network includes four local feature aggregation modules connected in sequence, specifically: The fourth local feature aggregation module: enhances the local spatiotemporal feature information of the spatiotemporal features output by the first downsampling module, and inputs the enhanced local spatiotemporal features into the fifth local feature aggregation module; The fifth local feature aggregation module performs local spatiotemporal feature information enhancement on the local spatiotemporal features output by the fourth local feature aggregation module, and inputs the enhanced local spatiotemporal features into the sixth local feature aggregation module; The sixth local feature aggregation module: performs local spatiotemporal feature information enhancement on the local spatiotemporal features output by the fifth local feature aggregation module, and inputs the enhanced local spatiotemporal features into the seventh local feature aggregation module; The seventh local feature aggregation module enhances the local spatiotemporal feature information of the local spatiotemporal features output by the sixth local feature aggregation module, and inputs the enhanced local spatiotemporal features into the second down-sampling module.

4. The action recognition system based on temporal aggregation and gated Transformer according to claim 1, characterized in that: The first Transformer network includes a first gating unit, a first attention module, a second attention module, a third attention module, a second gating unit, a fourth attention module, a fifth attention module and a sixth attention module connected in sequence; The first gating unit: enhances the motion feature information of the spatiotemporal features output by the second downsampling module, and inputs the obtained motion features into the first attention module; The first attention module: enhances the global spatiotemporal feature information of the motion features output by the first gating unit, and inputs the obtained global spatiotemporal features into the second attention module; The second attention module: enhances the global spatiotemporal feature information of the global spatiotemporal feature output by the first attention module, and inputs the obtained global spatiotemporal feature into the third attention module; The third attention module: enhances the global spatiotemporal feature information of the spatiotemporal features output by the second attention module, and inputs the obtained spatiotemporal features into the second gating unit; The second gating unit: enhances the motion feature information of the global spatiotemporal features output by the third attention module, and inputs the obtained motion features into the fourth attention module; The fourth attention module: enhances the global spatiotemporal feature information of the motion features output by the second gating unit, and inputs the obtained global spatiotemporal features into the fifth attention module; The fifth attention module: enhances the global spatiotemporal feature information of the global spatiotemporal feature output by the fourth attention module, and inputs the obtained global spatiotemporal feature into the sixth attention module; The sixth attention module: enhances the global spatiotemporal feature information of the global spatiotemporal features output by the fifth attention module, and inputs the obtained global spatiotemporal features into the third downsampling module.

5. The action recognition system based on temporal aggregation and gated Transformer according to claim 1, characterized in that: The second Transformer network includes a third gating unit, a seventh attention module and an eighth attention module connected in sequence; The third gating unit: enhances the motion feature information of the spatiotemporal features output by the third downsampling module, and inputs the obtained motion features into the seventh attention module; The seventh attention module: enhances the global spatiotemporal feature information of the motion features output by the third gating unit, and inputs the obtained global spatiotemporal features into the eighth attention module; The eighth attention module: enhances the global spatiotemporal feature information of the global spatiotemporal features output by the seventh attention module, and inputs the obtained global spatiotemporal features into the global average pooling module and the fully connected layer.

6. The action recognition system based on temporal aggregation and gated Transformer according to claim 2 or 3, characterized in that: The local feature aggregation module specifically includes: The first depth-wise separable convolution module: uses a convolution operator with a kernel size of 3×3×3, a step size of 1×1×1, and a padding of 1×1×1 to process the input spatiotemporal features and obtain a feature map of the same size as the input; The first residual structure: adds the spatiotemporal features input into the first depthwise separable convolution module to the feature map output by the first depthwise separable convolution module to realize dynamic position encoding of the video features and obtain appearance features; The first batch normalization module: performs batch normalization on the appearance features output by the first residual structure, and inputs the obtained appearance features into the first convolution operator; The first convolution operator is used to enhance the appearance features of the video frames processed by the first batch of normalization modules, and input the obtained appearance features into the second depth-separable convolution module; Second depth-wise separable convolution module: The appearance features output by the first convolution operator are processed through channel-by-channel convolution with a convolution kernel size of 5×1×1, a step size of 1×1×1, and a padding of 2×0×0 to process the spatiotemporal information of the input features, and then through point-by-point convolution with a convolution kernel size of 1×1×5, a step size of 1×1×1, and a padding of 0×2×2 to process the channel information of the input features to obtain a feature map of the same size as the input size; The second residual structure: adds the appearance features output by the first batch of normalization modules to the feature maps output by the second depth-separable convolution module to obtain enhanced local spatiotemporal features; The second batch normalization module: performs batch normalization on the local spatiotemporal features output by the second residual structure, and inputs the obtained spatiotemporal features into the first fully connected layer; The first fully connected layer: maps the spatiotemporal features obtained by the second batch normalization module to the sample label space, performs global feature extraction and nonlinear transformation, and obtains a feature map of the same size as the input size; The third residual structure: add the feature map output by the first fully connected layer to the spatiotemporal features output by the second batch normalization module to obtain enhanced local spatiotemporal features.

7. The action recognition system based on temporal aggregation and gated Transformer according to claim 4 or 5, characterized in that: The gate control unit specifically includes: The third depth-wise separable convolution module processes the input features using a convolution operator with a kernel size of 3×3×3, a step size of 1×1×1, and a padding of 1×1×1 to obtain a feature map of the same size as the input. The fourth residual structure: adding the spatiotemporal features input into the third depth-separable convolution module to the feature map output by the third depth-separable convolution module to realize dynamic position encoding of the video features and obtain the appearance features; The first global average pooling module: performs an adaptive global average pooling operation on the spatiotemporal features output by the fourth residual structure, and inputs the obtained spatiotemporal features into the second fully connected layer; The second fully connected layer maps the spatiotemporal features obtained by the first global average pooling module to the sample tag space, performs global feature extraction and nonlinear transformation, and processes the channel information of the input features; Matrix multiplication structure: The appearance features output by the fourth residual structure are matrix multiplied with the output results of the second fully connected layer to obtain enhanced spatiotemporal features; Cascade difference structure: The appearance features output by the fourth residual structure are cascaded and subtracted from the spatiotemporal features obtained by the matrix multiplication structure to obtain spatiotemporal features that suppress background information; The first layer normalization module: performs layer normalization on the spatiotemporal features obtained by the matrix multiplication structure, and inputs the obtained spatiotemporal features into the first spatiotemporal Transformer module; The first spatiotemporal Transformer module: The spatiotemporal features output by the first-layer normalization module are passed through Multi-Head attention to learn the long-distance temporal dependencies of video actions; The fifth residual structure: adds the spatiotemporal features output by the first-layer normalization module to the output results of the first spatiotemporal Transformer module to obtain enhanced motion features; The second layer normalization module: performs layer normalization on the motion features output by the fifth residual structure, and inputs the obtained spatiotemporal features into the third fully connected layer; The third fully connected layer: maps the spatiotemporal features obtained by the second-layer normalization module to the sample label space, performs global feature extraction and nonlinear transformation, and obtains a feature map of the same size as the input size; The sixth residual structure: adds the spatiotemporal features output by the second-layer normalization module to the feature map output by the third fully connected layer to obtain enhanced global spatiotemporal features; Cascaded sum structure: The spatiotemporal features output by the cascaded difference structure are cascaded and summed with the global spatiotemporal features output by the sixth residual structure to obtain enhanced motion features.

8. The action recognition system based on temporal aggregation and gated Transformer according to claim 4 or 5, characterized in that: The attention module specifically includes: Fourth depth-wise separable convolution module: The input features are processed using a convolution operator with a kernel size of 3×3×3, a step size of 1×1×1, and a padding of 1×1×1 to obtain a feature map of the same size as the input size; The seventh residual structure: adding the spatiotemporal features input into the fourth depth-separable convolution module to the feature map output by the fourth depth-separable convolution module to realize dynamic position encoding of the video features and obtain the appearance features; The third layer normalization module: performs layer normalization on the input spatiotemporal features and inputs the obtained spatiotemporal features into the second spatiotemporal Transformer module; The second spatiotemporal Transformer module: The spatiotemporal features output by the third-layer normalization module are passed through Multi-Head attention to learn the long-distance temporal dependencies of video actions; The eighth residual structure: the spatiotemporal features output by the third-layer normalization module are added to the output results of the second spatiotemporal Transformer module to obtain enhanced motion features; The fourth layer standardization module: performs layer standardization on the motion features output by the eighth residual structure, and inputs the obtained spatiotemporal features into the fourth fully connected layer; The fourth fully connected layer: maps the spatiotemporal features obtained by the fourth-layer normalization module to the sample label space, performs global feature extraction and nonlinear transformation, and obtains a feature map of the same size as the input size; Ninth residual structure: Add the spatiotemporal features output by the fourth-layer standardization module to the feature map output by the fourth fully connected layer to obtain enhanced global spatiotemporal features.

9. The method of the action recognition system based on temporal aggregation and gated Transformer according to claim 1, characterized in that: The following steps are involved: Step 1, feature extraction: extract the spatiotemporal features of the input video using the feature extraction module; Step 2, local feature enhancement: the spatiotemporal features extracted in step 1 are sequentially input into the first 3D-ResNet network, the first downsampling module, the second 3D-ResNet network and the second downsampling module, and the spatiotemporal features with enhanced local spatiotemporal feature information are output; Step 3: Global feature enhancement: The spatiotemporal features output from step 2 are sequentially input into the first Transformer network, the third downsampling module, and the second Transformer network, and the spatiotemporal features with enhanced global spatiotemporal feature information are output; Step 4: Action recognition: Input the spatiotemporal features output from step 3 into the global average pooling module and the fully connected layer for classification, and output the action recognition results.

Images