An action recognition system and method based on spatiotemporal diffusion and parallel Transformer

By constructing a two-branch Transformer network and a spatiotemporal diffusion module, the problem of ignoring temporal relationships in video action recognition is solved, the recognition accuracy is improved and the computational complexity is reduced, and efficient video action recognition is achieved.

CN117975565BActive Publication Date: 2026-11-17XIDIAN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410208172.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-26
Publication Date
2026-11-17
Estimated Expiration
2044-02-26

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively extract spatiotemporal features in video action recognition, especially since they neglect the temporal relationships between video frames, resulting in insufficient recognition accuracy. Furthermore, existing methods are computationally complex and resource-intensive.

Method used

A two-branch Transformer network is constructed, including a parallel Transformer module and a cross Transformer module. Combined with a local motion excitation module and a spatiotemporal diffusion module, it captures the local temporal and global spatiotemporal features of the video through depthwise separable convolution and self-attention mechanism, thereby reducing computational complexity.

Benefits of technology

It improves the accuracy and efficiency of video action recognition, reduces model training costs, and enables better action recognition results in fewer training rounds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117975565B_ABST
    Figure CN117975565B_ABST
Patent Text Reader

Abstract

A motion recognition system and method based on spatiotemporal diffusion and parallel Transformer are disclosed. The system includes a feature extraction module, a two-branch Transformer network, a local motion activation module, a spatiotemporal diffusion module, and a video classifier module. The method includes: first, the input video is fed into the feature extraction module; then, the extracted video features are fed into a two-branch Transformer network structure. The first branch consists of depthwise separable convolutions and parallel Transformer modules, and the second branch consists of cross-Transformer modules. Next, a lightweight local motion activation module is proposed, which can be inserted into any Transformer network. Then, the spatiotemporal features obtained from the two-branch structure are fused through the spatiotemporal diffusion module to enhance the long-distance temporal dependence of the video. Finally, the video action recognition result is output with the help of the video classifier module. This invention has the technical effects of improving model recognition accuracy and reducing model training costs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of video understanding and computer vision technology, specifically relating to an action recognition system and method based on spatiotemporal diffusion and parallel Transformer. Background Technology

[0002] With societal development and the advent of the information age, video has become a primary carrier of information, applied across various aspects of work and life. In recent years, people have accumulated vast amounts of video data in diverse scenarios, necessitating an effective algorithm for video analysis and management. As a popular task in computer vision, action recognition has not only been extensively studied theoretically but also found wide application in areas such as video surveillance, autonomous driving, and anomaly analysis. The goal of action recognition is to extract effective human action features from input videos and predict the category of each video action from a given label set. Due to the rapid progress of deep learning methods in action recognition in recent years, deep learning-based action recognition models have become a popular research area for scholars.

[0003] Based on the different methods used by network models to acquire video features, video action recognition methods can be broadly categorized into traditional machine learning methods and deep learning-based methods. Before deep learning became the mainstream action recognition method, traditional machine learning-based action recognition methods, such as K-means clustering and Gaussian mixture models, were widely used in many fields. In traditional machine learning-based action recognition methods, the video features acquired are often simple spatial features, ignoring temporal dimension features. These methods can achieve relatively good action recognition results when the videos to be recognized are relatively simple. However, when encountering more complex videos such as natural scenes or videos that depend on temporal relationships, the recognition accuracy achieved by traditional machine learning-based action recognition methods is insufficient to support the needs of these projects or tasks. In recent years, with the rapid development of specialized computing chips and the rapid reduction in computing costs, the widespread use of deep learning algorithms has become possible, thereby significantly improving the accuracy of action recognition without increasing costs. Therefore, action recognition methods based on deep learning algorithms have received widespread attention from scholars.

[0004] In deep learning-based action recognition methods, most approaches only consider spatial features, neglecting the temporal relationships between video frames, while actions in a video exhibit sequential relationships. Spatial features contain rich action details, while temporal features convey sequential relationships. If the algorithm model only learns temporal context information and lacks an understanding of fine-grained spatial context, it cannot predict the correct action. Similarly, a model that only understands spatial context but lacks learning of temporal context also cannot predict the correct action category. Therefore, based on spatial modeling, temporal modeling of video features is needed to fuse features from each frame of the video.

[0005] Existing methods directly learn spatial features and temporal features between adjacent frames from video clips, then use the learned high-level features for classification. Popular models include two-stream networks fusing RGB and optical flow, 3D convolutional neural networks, and Transformer models. The additional temporal dimension in videos makes spatial-temporal representation learning computationally expensive, requiring large amounts of training data. Furthermore, the similarity of video data in local spatiotemporal neighborhoods introduces significant inefficient computation into the network model. Accurate action recognition in videos requires dynamically associating features from different regions, modeling long-term dependencies. Therefore, directly introducing temporal modeling into videos is not ideal. Extracting spatiotemporal information from videos and effectively balancing spatiotemporal understanding is crucial.

[0006] In their paper "CAST: Cross-Attention in Space and Time for VideoAction Recognition" (Neural Information Processing Systems, 2023), Dongho Lee et al. proposed a novel two-stream Transformer architecture called Cross-Attention in Space and Time (CAST), achieving balanced spatiotemporal understanding of videos. The method specifically includes: ① Proposing a novel two-stream architecture, namely Cross-Attention in Space and Time, which can effectively learn spatiotemporal features of videos using only RGB input. ② Designing two expert models, including a spatial expert model and a temporal expert model, to jointly achieve effective spatiotemporal feature learning. ③ Proposing a bottleneck cross-attention module, which enables the spatial and temporal expert models to exchange information and perform collaborative predictions, thereby improving performance. However, the proposed method is based on an improvement of the Transformer network, requiring pre-training on massive amounts of video data. This results in high demands on hardware and computing resources during model training, leading to high computational costs.

[0007] In their paper "Gate-Shift-Fuse for Video Action Recognition" (IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023), Sudhakaran S et al. proposed a Gate Shift Fuse (GSF) module for spatiotemporal feature extraction. This module adaptively learns spatiotemporal features in a data-related manner by controlling the interactions in spatiotemporal decomposition. The specific content of this method is as follows: ① It introduces the concepts of spatial gating and channel fusion for spatiotemporal decomposition of 3D convolutional kernels. ② It proposes a gate-shift fusion module that uses learnable spatial gating to decompose the input tensor into gate and residual tensors, and uses channel weights to fuse the two feature groups. ③ The gate-shift fusion module can be inserted into existing 2D convolutional neural networks with almost no parameters and computational overhead. However, the proposed method is based on an improvement of convolutional neural networks. Due to the limitation of the small receptive field of convolutional operators, this method struggles to capture the global spatiotemporal information of videos and performs poorly in recognizing video actions over long time spans.

[0008] In summary, the existing technology still has the following drawbacks:

[0009] First, existing technologies, when extracting video features through convolutional neural networks, have limited receptive fields due to operations such as convolution and pooling. Each operation can only process local windows of adjacent pixels, and the resulting feature maps do not fully consider the overall temporal dependencies, making it difficult to effectively identify video actions.

[0010] Second, although 3D convolution has the advantage of learning video temporal features compared to 2D convolution, 3D convolution can only learn local temporal information in video segments. Although spatiotemporal information can be enhanced through repeated convolution and pooling operations, it suffers from problems such as slow convergence speed and high computational cost.

[0011] Third, when using the Transformer network to capture long-distance temporal dependencies, the presence of many similarities in the video and the fact that the tokens come from global features can lead to redundancy when processing local video information, resulting in a large number of model parameters and high computational complexity. Summary of the Invention

[0012] To overcome the shortcomings of the prior art, the present invention aims to provide an action recognition system and method based on spatiotemporal diffusion and parallel Transformer. By constructing a two-branch network architecture, including a parallel Transformer branch and a cross Transformer branch, it can effectively extract local temporal features and global spatiotemporal features of video. By designing a motion excitation module, it can capture motion information of adjacent frames. By proposing a spatiotemporal diffusion module, it can enhance the temporal dependencies of video actions, fully explore and utilize effective temporal information, and complete model training with fewer training rounds while ensuring inference speed. The present invention has the technical effects of improving model recognition accuracy and reducing model training cost.

[0013] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0014] An action recognition system based on spatiotemporal diffusion and parallel Transformer includes: a feature extraction module, a two-branch Transformer network, a local motion excitation module, a spatiotemporal diffusion module, and a video classifier module;

[0015] The feature extraction module is used to extract video features and send the extracted video features into a two-branch Transformer network;

[0016] The two-branch Transformer network includes a first-branch Transformer network and a second-branch Transformer network. The first-branch Transformer network is composed of depthwise separable convolutions and parallel Transformer modules, and is used to model the local temporal features of the video. The second-branch Transformer network is composed of cross-Transformer modules and is used to model the global spatiotemporal features of the video.

[0017] The local motion excitation module can be inserted at any position in the two-branch Transformer network to convert the spatial features of adjacent video frames into motion features. The local motion excitation module is used twice: First, the output features of the parallel Transformer module are used as input to establish the local correspondence of video frames, and the output result is input into the next layer of the parallel Transformer module; Second, the output features of the cross Transformer module are used as input to establish the local correspondence of video frames, and the result is input into the next layer of the cross Transformer module.

[0018] The spatiotemporal diffusion module is used to supplement the information of the parallel Transformer module by the cross Transformer module. That is, the global spatiotemporal temporal features extracted by the second branch Transformer network are fused into the first branch Transformer network to capture long-distance temporal dependencies in the video, enhance the model's ability to model video actions with a large time span, and effectively improve the performance of video action recognition.

[0019] The video classifier module is used for video action classification. It takes the output features of the last cross-transformer module as input to classify video actions and outputs the video action recognition results.

[0020] The feature extraction module converts the original two-dimensional image into a series of one-dimensional patch embeddings. The feature extraction module consists of a convolution operator with a kernel size of 3×16×16, a stride of 2×16×16, padding of 1×0×0, and a number of kernels of 768. The feature extraction module divides each frame of the input video into 196 segments, each segment being 768 in size. The segments are projected through a linear transformation to flatten the features and obtain a two-dimensional tensor with a dimension of 196×768.

[0021] The structure of the parallel Transformer module is as follows:

[0022] ① A dynamic position encoding module is designed using a depthwise separable convolution operator. This module takes the video features output from the feature extraction module as input. First, it processes the input video features using a convolution operator with a kernel size of 1×1×1, a stride of 1×1×1, and zero padding to obtain a feature map of the same size as the input. Then, it processes the input video features again using a convolution operator with a kernel size of 3×1×1, a stride of 1×1×1, and zero padding to obtain another feature map of the same size as the input. Finally, it processes the input video features again using a convolution operator with a kernel size of 1×1×1, a stride of 1×1×1, and zero padding to obtain yet another feature map of the same size as the input. By using convolution operators to encode the spatiotemporal position of the video features and learning the feature representations of the local neighborhood of the video, a residual structure is then used to process the input video features, enhancing the spatiotemporal feature information of the video.

[0023] ② Using the features obtained after the dynamic positional encoding module as input, the input features are first processed through a layer normalization operation. Then, an attention calculation module is established, and the input features are fed into the attention calculation module. The attention calculation module consists of multi-head self-attention (MHSA), and the calculation process is as follows:

[0024] y1=x+MHSA(Q,K,V)

[0025] Where x represents the input feature, y1 represents the output feature after passing through the spatial self-attention module, and the three matrices Q (Query), K (Key), and V (Value) all come from the same input feature x;

[0026] The formula for calculating multi-head self-attention is as follows:

[0027]

[0028]

[0029]

[0030] Where, n h The number of self-attention heads is set to 196. W O Let d represent the parameters of the linear mappings Q, K, V, and O, respectively. k This represents the dimension of the K-vector;

[0031] ③ Using the features obtained after the dynamic position encoding module as input, the features are first processed through a Layer Normalization operation. Then, a fully connected layer module is established to perform spatial transformation, enhancing the spatiotemporal feature information of the video. This fully connected layer module consists of a feed-forward network (FFN) containing two linear transformation layers: the first linear transformation maps the spatiotemporal features of each location to a larger-dimensional feature space, and the second linear transformation restores the features to their original dimensions, obtaining a feature map of the same size as the input. A Gaussian error linear unit (GELU) activation function is used between the two linear transformation layers to introduce nonlinearity for filtering. The calculation process is as follows:

[0032] y2 = FFN(x)

[0033] Where x represents the input feature and y2 represents the output feature after passing through the fully connected layer structure;

[0034] For feedforward neural networks, the calculation formula is as follows:

[0035] FFN(x) = max(0, xW1+b1)W2+b2

[0036] Where x represents the input feature, W1 and W2 are the weight matrices of the linear layer, b1 and b2 are the basis vectors, and the four parameters W1, W2, b1 and b2 are randomly initialized and adaptively learned.

[0037] The formula for calculating the Gaussian error linear unit (GELU) is as follows:

[0038] GELU(x)=xP(X≤x)=xΦ(x)

[0039] Where x is an element in the input feature map, and Φ(x) refers to the cumulative distribution function of the Gaussian normal distribution of x, i.e.:

[0040]

[0041] Where μ refers to the expected value of the input data, and σ refers to the standard deviation of the input data;

[0042] ④ Using the features obtained after passing through the dynamic position encoding module as input, stacked two-dimensional convolution operators are employed to enhance local video features, i.e., three consecutive two-dimensional convolution operators. First, the input features are processed by a convolution operator with a kernel size of 1×1, a stride of 1×1, and zero padding to obtain a feature map of the same size as the input. Then, a convolution operator with a kernel size of 3×3, a stride of 1×1, and 1×1 padding is used to process the input features again to obtain a feature map of the same size as the input. Finally, a convolution operator with a kernel size of 1×1, a stride of 1×1, and zero padding is used to process the input features again to obtain a feature map of the same size as the input. The stacked two-dimensional convolution calculation process is as follows:

[0043] y3 = Conv(x)

[0044] Where x represents the input feature and y3 represents the output feature after stacked two-dimensional convolution operation;

[0045] ⑤ The outputs of the attention calculation module, the fully connected layer module, and the stacked two-dimensional convolution operators are used as inputs to the feature fusion operation to enhance the local temporal features of the video. The calculation process is as follows:

[0046] y = y1 + αy2 + βy3

[0047] Where y1, y2, and y3 represent three types of features after passing through different modules, y represents the final output feature, and α and β represent the adaptive learning parameters of the input features y2 and y3, respectively. The values ​​of parameters α and β are both between 0 and 1.

[0048] The specific structure of the cross Transformer module is as follows:

[0049] ① A three-dimensional convolution operator is used to design a position encoding module. The position encoding module takes the video features output by the feature extraction module as input. The position encoding module includes a three-dimensional convolution operator and a dimension transformation operation. The three-dimensional convolution operator is implemented by a convolution operator with a kernel size of 3×3×3, a stride of 1×1×1, and padding of 1×1×1. The dimension transformation operation transforms the size of the input feature map from T×H×W×C to N×T×C, where N=H×W. Then, a residual structure is used to process the features of the input video to enhance the spatiotemporal feature information of the video.

[0050] ② For the input features of the parallel Transformer module and the positional encoding module, a Layer Normalization operation is first applied to process the features to reduce redundant information in the video. Then, a CrossAttention module is established, and the input features are fed into the CrossAttention module. Finally, a residual structure for the CrossAttention module is established, and the input features are fused with the obtained module output structure. The CrossAttention module consists of Multi-Head Cross-Attention (MHCA), and its calculation process is as follows:

[0051] y = x + MHCA(Q,K,V)

[0052] Where x represents the input feature, y represents the output feature after the cross-attention module, the Q (Query) matrix comes from the output feature of the parallel Transformer module, and the K (Key) matrix and V (Value) matrix come from the same input feature x;

[0053] The core of multi-head cross-attention is calculating the cross-attention matrix. First, the output features of the parallel Transformer module are linearly projected, and a bias term of the same dimension is added. The result is used as the query matrix (Query). Similarly, the output features of the position encoding module are linearly projected, and a bias term of the same dimension is added. The result is used as the key matrix (Key) and value matrix (Value). Second, the query matrix (Query), key matrix (Key), and value matrix (Value) are mapped, dividing the original channel dimension of 768 into 12 attention heads, each with a dimension of 64. Then, the query matrix (Query) and key matrix (Key) are multiplied, and the output is passed through a sigmoid function to fuse feature information from different modules. Finally, the resulting output is multiplied with the value matrix (Value) to obtain the output feature matrix. The calculation process is as follows:

[0054]

[0055]

[0056]

[0057] Where, n h The number of cross-attention heads is set to 196. W O Let d represent the parameters of the linear mappings Q, K, V, and O, respectively. k This represents the dimension of the K-vector;

[0058] The formula for calculating the sigmoid function is as follows:

[0059]

[0060] Where x is an element in the input feature map;

[0061] ③ The features obtained after passing through the cross-attention module are used as input. First, the features are processed by a layer normalization operation. Then, a fully connected layer module is used to perform spatial transformation to enhance the spatiotemporal feature information of the video. Finally, a residual structure is used to fuse the features to enhance the spatiotemporal information of the video.

[0062] The fully connected layer module consists of a feed-forward network (FFN) with two linear transformation layers: the first linear transformation maps the spatiotemporal features at each location to a larger-dimensional feature space, and the second linear transformation restores the features to their original dimensions, resulting in a feature map of the same size as the input. A non-linearity is introduced between the two linear transformation layers for filtering, and its calculation process is as follows:

[0063] y = FFN(x)

[0064] Where x represents the input feature and y represents the output feature after passing through the fully connected layer structure;

[0065] For feedforward neural networks, the calculation formula is as follows:

[0066] FFN(x) = max(0, xW1+b1)W2+b2

[0067] Where x represents the input feature, W1 and W2 are the weight matrices of the linear layer, b1 and b2 are the basis vectors, and the four parameters W1, W2, b1 and b2 are randomly initialized and adaptively learned.

[0068] The specific structure of the local motion excitation module is as follows:

[0069] ① The video features output by the feature extraction module are used as input for feature extraction. The spatial resolution of the original input features is adjusted to 14×14. First, the video features obtained by the parallel Transformer module or the cross Transformer module are passed through a two-dimensional convolution operator with a kernel size of (3,3), a stride of (1,1), padding of (1,1), and a number of kernels of 768. This two-dimensional convolution operation is used to spatially downsample the input features and extract the spatial information of the local neighborhood of the video. Then, a batch normalization operation is used to process the features to reduce the redundant information of the video. Finally, the output is passed through a linear rectified function ReLU to obtain a feature map of the same size as the input.

[0070] The formula for calculating the linear rectified function ReLU is as follows:

[0071]

[0072] Where x is an element in the input feature map;

[0073] ② The video features obtained after feature extraction are used as input, and correlation calculation is performed to enhance the temporal relationship of the video; the implementation details of this operation are as follows:

[0074] First, the input video features are temporally separated to obtain video features of different time series. Second, the features are L2 normalized to map the input data to the same value range, thereby improving the convergence speed of the model.

[0075] The formula for calculating L2 norm normalization is as follows:

[0076]

[0077] Among them, E in It is the training sample error without a normalization term, initially set to 1e-6, where λ is a learnable normalization parameter, and w j It is the row vector of the input matrix w;

[0078] Then, a dimension reshaping operation is performed to transform the size of the input feature map from B×C×H×W to B×C×N, where N=H×W. Next, a matrix multiplication operation is performed to fuse the feature maps of adjacent time series to enhance the temporal features of the video. Finally, the spatial correlation of video features is captured through local window sampling operations to merge video features from different time series.

[0079] ③ The features obtained after correlation calculation are used as input, and motion estimation is used to enhance the temporal features of the video, i.e., two-dimensional convolution operators at different scales. First, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. Second, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 2×2, and padding of 1×1 to downsample the input features. Then, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 2×2, and padding of 1×1 to downsample the input features a second time. Finally, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 1×1, and padding of 0 to downsample the input features a third time. Batch Normalization is added after each convolution operation, and the output is passed through a linear rectified function ReLU to increase the nonlinear relationship between the layers of the deep network.

[0080] ④ The features obtained after motion estimation are used as input. A feature transformation module is used to restore the dimensionality of the input features, and a residual structure is used to fuse the features after dimensionality restoration to enhance the temporal information of the video. The specific operation of the feature transformation module is as follows:

[0081] First, the input features are processed using a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. Second, the input features are processed using a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. Third, the input features are processed using a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. After each convolutional operation, a batch normalization operation is added, and the output is passed through a linear rectified function ReLU to increase the non-linear relationship between the layers of the deep network.

[0082] Finally, an upsampling operation is used to process the input features, restoring the output features to the same dimension as the original input. Through continuous two-dimensional convolution and upsampling operations, the temporal information of the video is extracted.

[0083] The structure of the spatiotemporal diffusion module is as follows:

[0084] ① The video features output by the cross-Transformer module are used as the input of the spatiotemporal diffusion module to establish a spatial-depth convolution module to extract video spatial information; that is, the input features are processed by a spatial-depth convolution module with a kernel size of 1×3×3, a stride of 1×1×1 and padding of 1×1×1 to obtain a feature map of the same size as the input. The convolutional features are batch normalized and the output is passed through a linear rectified function ReLU to avoid overfitting caused by deep networks.

[0085] ② Using the features obtained from the spatial-depth convolution module as input, a temporal-separation convolution module is established to enhance the interaction of temporal information in the video. Specifically, the input features are processed by a temporal-separation convolution module with a kernel size of 3×1×1, a stride of 1×1×1, and padding of 1×0×0 to obtain a feature map of the same size as the input. The convolutional features are then batch normalized, and the output is passed through a linear rectified function ReLU to avoid overfitting caused by deep networks.

[0086] ③ Using the features obtained after passing through the temporal-separated convolution module as input, a 1×1 convolution module is established to restore the channel dimension of the features and enhance the spatiotemporal information of the video; that is, the input features are processed by a convolution operator with a kernel size of 1×1×1, a stride of 1×1×1 and padding of 0 to obtain a feature map of the same size as the input. The convolutional features are then batch normalized, and the output is passed through a linear rectified function ReLU to avoid overfitting caused by deep networks.

[0087] ④ The features of the input video are processed through the residual structure, and the resulting output is fused with the local video representation.

[0088] The video classifier module takes the output feature map of the last cross-transformer module as its input, performs global average pooling, and then passes it through a fully connected layer to finally output the classification result of the action recognition model.

[0089] The feature extraction module uses 3D convolution operators to extract video features, and can also use a combination of 2D and 1D convolution operators to extract video features;

[0090] The attention calculation module used in the parallel Transformer module is designed based on the self-attention mechanism. It can also be designed according to block attention, step attention, hash attention, or linear attention to achieve effective modeling of global video features.

[0091] The feature extraction, motion estimation, and feature transformation in the local motion excitation module all use 2D convolution operators to extract the difference information of video frames. Alternatively, depth-separable convolution, grouped convolution, transpose convolution, or 3D convolution can be used to extract the difference information of video frames.

[0092] The correlation calculation in the local motion excitation module uses matrix multiplication to realize the information interaction of the local context, or matrix addition or matrix subtraction can be used for correlation calculation.

[0093] This invention also provides an action recognition method based on spatiotemporal diffusion and parallel Transformer, comprising the following steps:

[0094] Step 1: Using a sparse sampling strategy, the input video is sent to the feature extraction module, and video features are extracted through the feature extraction module.

[0095] Step 2: The video features extracted in Step 1 are fed into a two-branch Transformer network structure. The two-branch Transformer network structure is used to learn the spatiotemporal features of the video and to model both local and global video information. The two-branch Transformer network includes a first-branch Transformer network and a second-branch Transformer network. The first-branch Transformer network is composed of depthwise separable convolutions and parallel Transformer modules and is used to extract local temporal features of video actions. The second-branch Transformer network is composed of cross-Transformer modules and is used to extract global spatiotemporal features of video actions.

[0096] Step 3: Construct a local motion excitation module, which can be inserted into any position in the two-branch Transformer network to convert the spatial features of adjacent video frames into motion features. The local motion excitation module is used twice: First, the output features of the parallel Transformer module are used as input to establish the local correspondence of video frames, and the output result is input into the next layer of the parallel Transformer module; Second, the output features of the cross Transformer module are used as input to establish the local correspondence of video frames, and the output result is input into the next layer of the cross Transformer module.

[0097] Step 4: Construct a spatiotemporal diffusion module to effectively learn long-range temporal dependencies in videos. This module takes the global spatiotemporal features extracted by the cross-Transformer module as input, uses a diffusion structure to extract long-range temporal information, and fuses the output into the parallel Transformer module to effectively improve the performance of video action recognition. This module can be inserted into the two-branch Transformer network structure in Step 2 to supplement the information of the parallel Transformer module from the cross-Transformer module.

[0098] Step 5: Construct a video classifier module, which is implemented using a fully connected layer structure. The output features of the last cross-transformer module are used as the input of this module to classify video actions and output the video action recognition results.

[0099] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0100] 1. This invention constructs a two-branch Transformer network, which can capture the local temporal features and global spatiotemporal features of video actions, effectively modeling the main body of video actions. This model can achieve better action recognition results with fewer training rounds, effectively reducing the training cost of the model.

[0101] 2. This invention proposes a parallel Transformer module, which can effectively capture local temporal features of video and improve model accuracy.

[0102] 3. This invention, by designing a lightweight local motion excitation module, can transform the spatial features of adjacent video frames into motion features. This module has the characteristics of being plug-and-play and has the technical effect of improving model accuracy.

[0103] 4. By designing a spatiotemporal diffusion model, this invention can diffuse and fuse the temporal information of the cross-Transformer module to the parallel Transformer module. This module has the technical effect of better capturing long-distance temporal dependencies in video and improving model accuracy.

[0104] In summary, this invention first proposes a parallel Transformer module, which effectively models local features of videos by leveraging self-attention mechanisms, fully connected layer structures, and stacked two-dimensional convolution operators. Secondly, a local motion excitation module is proposed, enabling temporal modeling of adjacent video sequences with low computational cost. Next, a spatiotemporal diffusion module is proposed, fusing local and global video features to enhance the model's long-range temporal modeling capability and effectively improve the accuracy of the action recognition model. Finally, this invention constructs a two-branch Transformer network model that does not require targeted pre-training and achieves better action recognition results with fewer training epochs, effectively reducing the model's training cost. Attached Figure Description

[0105] Figure 1 This is a flowchart illustrating the implementation process of the present invention.

[0106] Figure 2 This is a structural diagram of the parallel Transformer module in the action recognition model of this invention.

[0107] Figure 3 This is a structural diagram of the cross Transformer module in the action recognition model of this invention.

[0108] Figure 4 This is a structural diagram of the local motion excitation module in the action recognition model of this invention.

[0109] Figure 5 This is a structural diagram of the spatiotemporal diffusion module in the action recognition model of this invention. Detailed Implementation

[0110] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0111] Because videos exhibit similarities in their local spatiotemporal neighborhoods, and video actions are continuous and span long spatiotemporal distances, learning effective spatiotemporal features from videos to enhance action recognition is a challenging task. Existing mainstream deep learning models include convolutional neural networks (CNNs) and Transformer networks. CNNs can only aggregate contextual features in local neighborhoods, and their limited receptive field makes it difficult to model long-term temporal features. Transformer networks, on the other hand, can capture large-scale temporal dependencies by calculating the global similarity of features. However, due to the abundance of similar information in videos, they generate significant redundancy when processing local video information, leading to computational inefficiency. This invention aims to address the differences in feature extraction by different networks by designing a two-branch network structure incorporating parallel and cross-transformers. A lightweight local motion excitation module is proposed to convert local information from video frames into motion features. Furthermore, a spatiotemporal diffusion module is designed to learn long-distance temporal dependencies in videos, effectively promoting the utilization of video spatiotemporal information and improving the performance of video action recognition.

[0112] Specifically, an action recognition system based on spatiotemporal diffusion and parallel Transformer includes: a feature extraction module, a two-branch Transformer network, a local motion excitation module, a spatiotemporal diffusion module, and a video classifier module;

[0113] The feature extraction module is used to extract video features and send the extracted video features into a two-branch Transformer network;

[0114] The two-branch Transformer network includes a first-branch Transformer network and a second-branch Transformer network. The first-branch Transformer network is composed of depthwise separable convolutions and parallel Transformer modules, and is used to model the local temporal features of the video. The second-branch Transformer network is composed of cross-Transformer modules and is used to model the global spatiotemporal features of the video.

[0115] A local motion excitation module, which can be inserted at any position in the two-branch Transformer network, converts the spatial features of adjacent video frames into motion features. In this invention, the local motion excitation module is used twice: First, the output features of the parallel Transformer module are used as input to establish the local correspondence of video frames, and the resulting output is input into the next layer of the parallel Transformer module; Second, the output features of the cross Transformer module are used as input to establish the local correspondence of video frames, and the resulting output is input into the next layer of the cross Transformer module.

[0116] The spatiotemporal diffusion module is used to supplement the information of the parallel Transformer module by the cross Transformer module. That is, the global spatiotemporal temporal features extracted by the second branch Transformer network are fused into the first branch Transformer network to capture long-distance temporal dependencies in the video, enhance the model's ability to model video actions with a large time span, and effectively improve the performance of video action recognition.

[0117] The video classifier module is used for video action classification. It takes the output features of the last cross-transformer module as input to classify video actions and outputs the video action recognition results.

[0118] The feature extraction module converts the original two-dimensional image into a series of one-dimensional patch embeddings. The feature extraction module consists of a convolution operator with a kernel size of 3×16×16, a stride of 2×16×16, padding of 1×0×0, and a number of kernels of 768. The feature extraction module divides each frame of the input video into 196 segments, each segment being 768 in size. The segments are projected through a linear transformation to flatten the features and obtain a two-dimensional tensor with a dimension of 196×768.

[0119] The structure of the parallel Transformer module is as follows:

[0120] ① A dynamic position encoding module is designed using a depthwise separable convolution operator. The dynamic position encoding module takes the video features output by the feature extraction module as input. Specifically, firstly, the input video features are processed by a convolution operator with a kernel size of 1×1×1, a stride of 1×1×1, and zero padding to obtain a feature map of the same size as the input. Then, the input video features are processed by a convolution operator with a kernel size of 3×1×1, a stride of 1×1×1, and zero padding to obtain a feature map of the same size as the input. Finally, the input video features are processed again by a convolution operator with a kernel size of 1×1×1, a stride of 1×1×1, and zero padding to obtain a feature map of the same size as the input. These operations use convolution operators to achieve spatiotemporal position encoding of video features. After learning the feature representation of the local neighborhood of the video, a residual structure is used to process the features of the input video to enhance the spatiotemporal feature information of the video.

[0121] ② Using the features obtained after the dynamic positional encoding module as input, the input features are first processed through a layer normalization operation. Then, an attention module is established, and the input features are fed into the attention module. Specifically, the attention module consists of multi-head self-attention (MHSA), and the calculation process is as follows:

[0122] y1=x+MHSA(Q,K,V)

[0123] Where x represents the input feature, y1 represents the output feature after passing through the spatial self-attention module, and the three matrices Q (Query), K (Key), and V (Value) all come from the same input feature x;

[0124] The formula for calculating multi-head self-attention is as follows:

[0125]

[0126]

[0127]

[0128] Where, n h The number of self-attention heads is set to 196 in this invention. W O Let d represent the parameters of the linear mappings Q, K, V, and O, respectively. k This represents the dimension of the K-vector;

[0129] ③ Using the features obtained after the dynamic position encoding module as input, the features are first processed through a Layer Normalization operation. Then, a fully connected layer module is established to perform spatial transformation, enhancing the spatiotemporal feature information of the video. This fully connected layer module consists of a feed-forward network (FFN) containing two linear transformation layers: the first linear transformation maps the spatiotemporal features of each location to a larger-dimensional feature space, and the second linear transformation restores the features to their original dimensions, obtaining a feature map of the same size as the input. A Gaussian error linear unit (GELU) activation function is used between the two linear transformation layers to introduce nonlinearity for filtering. The calculation process is as follows:

[0130] y2 = FFN(x)

[0131] Where x represents the input feature and y2 represents the output feature after passing through the fully connected layer structure;

[0132] For feedforward neural networks, the calculation formula is as follows:

[0133] FFN(x) = max(0, xW1+b1)W2+b2

[0134] Where x represents the input feature, W1 and W2 are the weight matrices of the linear layer, b1 and b2 are the basis vectors, and the four parameters W1, W2, b1 and b2 are randomly initialized and adaptively learned.

[0135] The formula for calculating the Gaussian error linear unit (GELU) is as follows:

[0136] GELU(x)=xP(X≤x)=xΦ(x)

[0137] Where x is an element in the input feature map, and Φ(x) refers to the cumulative distribution function of the Gaussian normal distribution of x, i.e.:

[0138]

[0139] Where μ refers to the expected value of the input data, and σ refers to the standard deviation of the input data;

[0140] ④ Using the features obtained after passing through the dynamic position encoding module as input, stacked two-dimensional convolution operators are employed to enhance local video features, i.e., three consecutive two-dimensional convolution operators. Specifically, first, the input features are processed by a convolution operator with a kernel size of 1×1, a stride of 1×1, and zero padding to obtain a feature map of the same size as the input. Then, a convolution operator with a kernel size of 3×3, a stride of 1×1, and 1×1 padding is used to process the input features again to obtain a feature map of the same size as the input. Finally, a convolution operator with a kernel size of 1×1, a stride of 1×1, and zero padding is used to process the input features again to obtain a feature map of the same size as the input. The stacked two-dimensional convolution calculation process is as follows:

[0141] y3 = Conv(x)

[0142] Where x represents the input feature and y3 represents the output feature after stacked two-dimensional convolution operation;

[0143] These operations achieve the effect of three-dimensional convolution operators using two-dimensional convolution operators, significantly reducing model size and computational cost while improving model modeling capabilities. They effectively aggregate local features from videos, reducing model computation and parameter count, and solving the video redundancy problem in shallow networks.

[0144] ⑤ The outputs of the attention calculation module, the fully connected layer module, and the stacked two-dimensional convolution operators are used as inputs to the feature fusion operation to enhance the local temporal features of the video. The calculation process is as follows:

[0145] y = y1 + αy2 + βy3

[0146] Where y1, y2, and y3 represent three types of features after passing through different modules, y represents the final output feature, and α and β represent the adaptive learning parameters of the input features y2 and y3, respectively. The values ​​of parameters α and β are both between 0 and 1.

[0147] The specific structure of the cross Transformer module is as follows:

[0148] ① A location encoding module is designed using a 3D convolution operator. This module takes the video features output from the feature extraction module as input and includes a 3D convolution operator and a dimension transformation operation. Specifically, the 3D convolution operator is implemented using a 3×3×3 kernel, a 1×1×1 stride, and 1×1×1 padding. The dimension transformation operation transforms the size of the input feature map from T×H×W×C to N×T×C, where N = H×W. Because convolution has the properties of parameter concatenation, local sharing, and position independence, the location encoding module can encode spatiotemporal location information, and the zero-padding effectively determines the location information. Then, a residual structure is used to process the features of the input video, enhancing the spatiotemporal feature information of the video.

[0149] ② For the input features from the parallel Transformer module and the positional encoding module, a Layer Normalization operation is first applied to process the features and reduce redundant information in the video. Then, a CrossAttention module is established, and the input features are fed into the CrossAttention module. Finally, a residual structure for the CrossAttention module is established, and the input features are fused with the obtained module output structure. Specifically, the CrossAttention module consists of Multi-Head Cross-Attention (MHCA), and its calculation process is as follows:

[0150] y = x + MHCA(Q,K,V)

[0151] Where x represents the input feature, y represents the output feature after the cross-attention module, the Q (Query) matrix comes from the output feature of the parallel Transformer module, and the K (Key) matrix and V (Value) matrix come from the same input feature x;

[0152] The core of multi-head cross-attention is calculating the cross-attention matrix. First, the output features of the parallel Transformer module are linearly projected, and a bias term of the same dimension is added. The result is used as the query matrix (Query). Similarly, the output features of the position encoding module are linearly projected, and a bias term of the same dimension is added. The result is used as the key matrix (Key) and value matrix (Value). Second, the query matrix (Query), key matrix (Key), and value matrix (Value) are mapped, dividing the original channel dimension of 768 into 12 attention heads, each with a dimension of 64. Then, the query matrix (Query) and key matrix (Key) are multiplied, and the output is passed through a sigmoid function to fuse feature information from different modules. Finally, the resulting output is multiplied with the value matrix (Value) to obtain the output feature matrix. The calculation process is as follows:

[0153]

[0154]

[0155]

[0156] Where, n h The number of cross-attention heads is set to 196 in this invention. W O Let d represent the parameters of the linear mappings Q, K, V, and O, respectively. k This represents the dimension of the K-vector;

[0157] The formula for calculating the sigmoid function is as follows:

[0158]

[0159] Where x is an element in the input feature map;

[0160] ③ The features obtained after passing through the cross-attention module are used as input. First, the features are processed by a layer normalization operation. Then, a fully connected layer module is used to perform spatial transformation to enhance the spatiotemporal feature information of the video. Finally, a residual structure is used to fuse the features to enhance the spatiotemporal information of the video.

[0161] The fully connected layer module consists of a feed-forward network (FFN) with two linear transformation layers: the first linear transformation maps the spatiotemporal features at each location to a larger-dimensional feature space, and the second linear transformation restores the features to their original dimensions, resulting in a feature map of the same size as the input. A non-linearity is introduced between the two linear transformation layers for filtering, and its calculation process is as follows:

[0162] y = FFN(x)

[0163] Where x represents the input feature and y represents the output feature after passing through the fully connected layer structure;

[0164] For feedforward neural networks, the calculation formula is as follows:

[0165] FFN(x) = max(0, xW1+b1)W2+b2

[0166] Where x represents the input feature, W1 and W2 are the weight matrices of the linear layer, b1 and b2 are the basis vectors, and the four parameters W1, W2, b1 and b2 are randomly initialized and adaptively learned.

[0167] The specific structure of the local motion excitation module is as follows:

[0168] ① The video features output by the feature extraction module are used as input for feature extraction. This operation adjusts the spatial resolution of the original input features to 14×14. First, the video features obtained by the parallel Transformer module or the cross Transformer module are passed through a two-dimensional convolution operator with a kernel size of (3,3), a stride of (1,1), padding of (1,1), and a number of kernels of 768. This two-dimensional convolution operation spatially downsamples the input features to extract the spatial information of the local neighborhood of the video. Then, a batch normalization operation is used to process the features to reduce the redundant information of the video. Finally, the output is passed through a linear rectified function ReLU to obtain a feature map of the same size as the input.

[0169] The formula for calculating the linear rectified function ReLU is as follows:

[0170]

[0171] Where x is an element in the input feature map;

[0172] ② The video features obtained after feature extraction are used as input, and correlation calculation is performed to enhance the temporal relationship of the video; the implementation details of this operation are as follows:

[0173] First, the input video features are temporally separated to obtain video features of different time series. Second, the features are L2 normalized to map the input data to the same value range, thereby improving the convergence speed of the model.

[0174] The formula for calculating L2 norm normalization is as follows:

[0175]

[0176] Among them, E in It is the training sample error without a normalization term, initially set to 1e-6, where λ is a learnable normalization parameter, and w j It is the row vector of the input matrix w;

[0177] Then, a dimension reshaping operation is performed to transform the size of the input feature map from B×C×H×W to B×C×N, where N=H×W. Next, a matrix multiplication operation is performed to fuse the feature maps of adjacent time series to enhance the temporal features of the video. Finally, the spatial correlation of video features is captured through local window sampling operations to merge video features from different time series.

[0178] ③ Using the features obtained after correlation calculation as input, motion estimation is employed to enhance the temporal features of the video, i.e., two-dimensional convolution operators at different scales. Specifically, firstly, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. Secondly, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 2×2, and padding of 1×1 to downsample the input features. Finally, a convolution operator is used to further process the input features. The input features are processed by a convolutional operator with a kernel size of 3×3, a stride of 2×2, and padding of 1×1. The input features are then downsampled a second time. Finally, the input features are processed by a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 0. The input features are then downsampled a third time. Batch normalization is added after each convolutional operation, and the output is passed through a linear rectified function ReLU to increase the nonlinear relationship between the layers of the deep network.

[0179] ④ The features obtained after motion estimation are used as input. A feature transformation module is used to restore the dimensionality of the input features, and a residual structure is used to fuse the features after dimensionality restoration to enhance the temporal information of the video. The specific operation of the feature transformation module is as follows:

[0180] First, the input features are processed using a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. Second, the input features are processed using a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. Third, the input features are processed using a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. After each convolutional operation, a batch normalization operation is added, and the output is passed through a linear rectified function ReLU to increase the non-linear relationship between the layers of the deep network.

[0181] Finally, an upsampling operation is used to process the input features, restoring the output features to the same dimension as the original input. Through continuous two-dimensional convolution and upsampling operations, video temporal information is further extracted, achieving an effect similar to three-dimensional convolution, effectively reducing the number of model parameters and computational overhead.

[0182] The structure of the spatiotemporal diffusion module is as follows:

[0183] ① The video features output by the cross-Transformer module are used as the input of the spatiotemporal diffusion module to establish a spatial-depth convolution module to extract video spatial information. Specifically, the input features are processed by a spatial-depth convolution module with a kernel size of 1×3×3, a stride of 1×1×1, and padding of 1×1×1 to obtain a feature map of the same size as the input. The convolutional features are then batch normalized, and the output is passed through a linear rectified function ReLU to avoid overfitting caused by deep networks.

[0184] ② Using the features obtained from the spatial-depth convolution module as input, a temporal-separation convolution module is established to enhance the interaction of temporal information in the video. Specifically, the input features are processed by a temporal-separation convolution module with a kernel size of 3×1×1, a stride of 1×1×1, and padding of 1×0×0 to obtain a feature map of the same size as the input. The convolutional features are then batch normalized, and the output is passed through a linear rectified function ReLU to avoid overfitting caused by deep networks.

[0185] ③ Using the features obtained after passing through the temporal-separated convolution module as input, a 1×1 convolution module is established to restore the channel dimension of the features and enhance the spatiotemporal information of the video. Specifically, the input features are processed by a convolution operator with a kernel size of 1×1×1, a stride of 1×1×1, and padding of 0 to obtain a feature map of the same size as the input. The convolutional features are then batch normalized, and the output is passed through a linear rectified function ReLU to avoid overfitting caused by deep networks.

[0186] ④ The features of the input video are processed through the residual structure, and the resulting output is fused with the local video representation. These operations use depthwise separable convolution to achieve diffusion modeling of the global video representation, effectively fusing local and global video representations without increasing the number of parameters, thus enhancing the model's long-distance temporal modeling capability.

[0187] The video classifier module takes the output feature map of the last cross-transformer module as its input, performs global average pooling, and then passes it through a fully connected layer to finally output the classification result of the action recognition model.

[0188] The feature extraction module uses 3D convolution operators to extract video features, and can also use a combination of 2D and 1D convolution operators to extract video features;

[0189] The attention calculation module used in the parallel Transformer module is designed based on the self-attention mechanism. It can also be designed according to block attention, step attention, hash attention, or linear attention to achieve effective modeling of global video features.

[0190] The feature extraction, motion estimation, and feature transformation in the local motion excitation module all use 2D convolution operators to extract the difference information of video frames. Alternatively, depth-separable convolution, grouped convolution, transpose convolution, or 3D convolution can be used to extract the difference information of video frames.

[0191] The correlation calculation in the local motion excitation module uses matrix multiplication to realize the information interaction of the local context, or matrix addition or matrix subtraction can be used for correlation calculation.

[0192] This invention also provides an action recognition method based on spatiotemporal diffusion and parallel Transformer, comprising the following steps:

[0193] Step 1: Using a sparse sampling strategy, the input video is fed into the feature extraction module, which extracts video features. This operation is achieved through a three-dimensional convolution operator, which compresses the spatial and temporal dimensions of the data to obtain a three-dimensional tensor containing the spatiotemporal information of the video.

[0194] Step 2: Establish an action recognition model based on a two-branch Transformer network structure. The video features extracted in Step 1 are fed into the two-branch Transformer network structure. The two-branch Transformer network structure is used to learn the spatiotemporal features of the video and to model both local and global video information. The two-branch Transformer network includes a first-branch Transformer network and a second-branch Transformer network. The first-branch Transformer network consists of twelve depthwise separable convolutions and twelve parallel Transformer modules, which are used to extract local temporal features of video actions. The second-branch Transformer network consists of twelve cross-Transformer modules, which are used to extract global spatiotemporal features of video actions.

[0195] Step 3: Construct a lightweight local motion excitation module, which can be inserted into any position in the two-branch Transformer network to convert the spatial features of adjacent video frames into motion features. Specifically, this invention uses the local motion excitation module twice: First, the output features of the parallel Transformer module are used as input to establish the local correspondence of video frames, and the output result is input into the next layer of the parallel Transformer module; Second, the output features of the cross Transformer module are used as input to similarly establish the local correspondence of video frames, and the output result is input into the next layer of the cross Transformer module.

[0196] Step 4: Construct a spatiotemporal diffusion module to effectively learn long-range temporal dependencies in videos. This module takes the global spatiotemporal features extracted by the cross-Transformer module as input, uses a diffusion structure to extract long-range temporal information, and fuses the output into the parallel Transformer module, effectively improving the performance of video action recognition. This module can be inserted into the two-branch Transformer network structure of Step 2 to supplement the information of the parallel Transformer module from the cross-Transformer module. The proposed spatiotemporal diffusion module fuses video features obtained from different receptive fields. By fusing the spatiotemporal features obtained from the two-branch structure through the spatiotemporal diffusion module, the long-range temporal dependencies in videos are enhanced.

[0197] Step 5: Construct a video classifier module, which is implemented using a fully connected layer structure. The output features of the last cross-transformer module are used as the input of this module to classify video actions and output the video action recognition results.

[0198] It should be noted that:

[0199] 1. The feature extraction module used in this invention is an existing 3D convolution operator, which can effectively extract spatiotemporal features from video. Replacing this convolution operator with a combination of 2D and 1D convolution operators can achieve the same objective.

[0200] 2. The attention calculation module used in the parallel Transformer module of this invention is an existing self-attention mechanism design, which can effectively model the global features of the video. If an improved attention mechanism is used, such as block attention, step attention, hash attention, and linear attention, the objectives of this invention can also be achieved.

[0201] 3. The feature extraction, motion estimation, and feature transformation in the local motion excitation module of this invention all employ existing 2D convolution operators, which can effectively extract the difference information of video frames. If other convolution operations are used, such as depthwise separable convolution, grouped convolution, transposed convolution operators, and 3D convolution, the objectives of this invention can also be achieved.

[0202] 4. The correlation calculation in the local motion excitation module of this invention uses matrix multiplication, which enables information interaction within the local context. Matrix addition or subtraction can be used instead, achieving the same objective.

[0203] Although specific embodiments of the invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of the invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims are still within the scope of protection of this invention.

Claims

1. An action recognition system based on spatiotemporal diffusion and parallel Transformer, characterized in that: include: The system includes a feature extraction module, a two-branch Transformer network, a local motion excitation module, a spatiotemporal diffusion module, and a video classifier module. The feature extraction module is used to extract video features and send the extracted video features into a two-branch Transformer network; The two-branch Transformer network includes a first-branch Transformer network and a second-branch Transformer network. The first-branch Transformer network is composed of depthwise separable convolutions and parallel Transformer modules, and is used to model the local temporal features of the video. The second-branch Transformer network is composed of cross-Transformer modules and is used to model the global spatiotemporal features of the video. The local motion excitation module can be inserted at any position in the two-branch Transformer network to convert the spatial features of adjacent video frames into motion features. The local motion excitation module is used twice: First, the output features of the parallel Transformer module are used as input to establish the local correspondence of video frames, and the output result is input into the next layer of the parallel Transformer module; Second, the output features of the cross Transformer module are used as input to establish the local correspondence of video frames, and the result is input into the next layer of the cross Transformer module. The spatiotemporal diffusion module is used to supplement the information of the parallel Transformer module by the cross Transformer module. That is, the global spatiotemporal temporal features extracted by the second branch Transformer network are fused into the first branch Transformer network to capture long-distance temporal dependencies in the video, enhance the model's ability to model video actions with a large time span, and effectively improve the performance of video action recognition. The video classifier module is used for video action classification. It takes the output features of the last cross-transformer module as input to classify video actions and outputs the video action recognition results.

2. The action recognition system based on spatiotemporal diffusion and parallel Transformer according to claim 1, characterized in that: The feature extraction module converts the original two-dimensional image into a series of one-dimensional patch embeddings. The feature extraction module consists of a convolution operator with a kernel size of 3×16×16, a stride of 2×16×16, padding of 1×0×0, and a number of kernels of 768. The feature extraction module divides each frame of the input video into 196 segments, each segment being 768 in size. The segments are projected through a linear transformation to flatten the features and obtain a two-dimensional tensor with a dimension of 196×768.

3. The action recognition system based on spatiotemporal diffusion and parallel Transformer according to claim 1, characterized in that: The structure of the parallel Transformer module is as follows: ① A dynamic position encoding module is designed using a depthwise separable convolution operator. This module takes the video features output from the feature extraction module as input. First, it processes the input video features using a convolution operator with a kernel size of 1×1×1, a stride of 1×1×1, and zero padding to obtain a feature map of the same size as the input. Then, it processes the input video features again using a convolution operator with a kernel size of 3×1×1, a stride of 1×1×1, and zero padding to obtain another feature map of the same size as the input. Finally, it processes the input video features again using a convolution operator with a kernel size of 1×1×1, a stride of 1×1×1, and zero padding to obtain yet another feature map of the same size as the input. By using convolution operators to encode the spatiotemporal position of the video features and learning the feature representations of the local neighborhood of the video, a residual structure is then used to process the input video features, enhancing the spatiotemporal feature information of the video. ② Using the features obtained after the dynamic positional encoding module as input, the input features are first processed through a layer normalization operation. Then, an attention module is established, and the input features are fed into the attention module. The attention module consists of multi-head self-attention (MHSA), and the calculation process is as follows: y1=x+MHSA(Q,K,V) Where x represents the input feature, y1 represents the output feature after passing through the spatial self-attention module, and the three parameter matrices Q (Query), K (Key), and V (Value) all come from the same input feature x; The formula for calculating multi-head self-attention is as follows: Where, n h The number of self-attention heads is set to 196. W O Let d represent the parameters of the linear mappings Q, K, V, and O, respectively. k This represents the dimension of the K-vector; ③ Using the features obtained after the dynamic position encoding module as input, the features are first processed through a Layer Normalization operation. Then, a fully connected layer module is established to perform spatial transformation, enhancing the spatiotemporal feature information of the video. This fully connected layer module consists of a feed-forward network (FFN) containing two linear transformation layers: the first linear transformation maps the spatiotemporal features of each location to a larger-dimensional feature space, and the second linear transformation restores the features to their original dimensions, obtaining a feature map of the same size as the input. A Gaussian error linear unit (GELU) activation function is used between the two linear transformation layers to introduce nonlinearity for filtering. The calculation process is as follows: y2 = FFN(x) Where x represents the input feature and y2 represents the output feature after passing through the fully connected layer structure; For feedforward neural networks, the calculation formula is as follows: FFN(x) = max(0, xW1+b1)W2+b2 Where x represents the input feature, W1 and W2 are the weight matrices of the linear layer, b1 and b2 are the basis vectors, and the four parameters W1, W2, b1 and b2 are randomly initialized and adaptively learned. The formula for calculating the Gaussian error linear unit (GELU) is as follows: GELU(x)=xP(X≤x)=xΦ(x) Where x is an element in the input feature map, and Φ(x) refers to the cumulative distribution function of the Gaussian normal distribution of x, i.e.: Where μ refers to the expected value of the input data, and σ refers to the standard deviation of the input data; ④ Using the features obtained after passing through the dynamic position encoding module as input, stacked two-dimensional convolution operators are employed to enhance local video features, i.e., three consecutive two-dimensional convolution operators. First, the input features are processed by a convolution operator with a kernel size of 1×1, a stride of 1×1, and zero padding to obtain a feature map of the same size as the input. Then, a convolution operator with a kernel size of 3×3, a stride of 1×1, and 1×1 padding is used to process the input features again to obtain a feature map of the same size as the input. Finally, a convolution operator with a kernel size of 1×1, a stride of 1×1, and zero padding is used to process the input features again to obtain a feature map of the same size as the input. The stacked two-dimensional convolution calculation process is as follows: y3 = Conv(x) Where x represents the input feature and y3 represents the output feature after stacked two-dimensional convolution operation; ⑤ The outputs of the attention calculation module, the fully connected layer module, and the stacked two-dimensional convolution operators are used as inputs to the feature fusion operation to enhance the local temporal features of the video. The calculation process is as follows: y = y1 + αy2 + βy3 Where y1, y2, and y3 represent three types of features after passing through different modules, y represents the final output feature, and α and β represent the adaptive learning parameters of the input features y2 and y3, respectively. The values ​​of parameters α and β are both between 0 and 1.

4. The action recognition system based on spatiotemporal diffusion and parallel Transformer according to claim 1, characterized in that: The specific structure of the cross Transformer module is as follows: ① A three-dimensional convolution operator is used to design a position encoding module. The position encoding module takes the video features output by the feature extraction module as input. The position encoding module includes a three-dimensional convolution operator and a dimension transformation operation. The three-dimensional convolution operator is implemented by a convolution operator with a kernel size of 3×3×3, a stride of 1×1×1, and padding of 1×1×1. The dimension transformation operation transforms the size of the input feature map from T×H×W×C to N×T×C, where N=H×W. Then, a residual structure is used to process the features of the input video to enhance the spatiotemporal feature information of the video. ② For the input features of the parallel Transformer module and the positional encoding module, a layer normalization operation is first applied to process the features to reduce redundant information in the video. Then, a cross-attention module is established, and the input features are fed into the cross-attention module. Finally, a residual structure for the cross-attention module is established, and the input features are fused with the obtained module output structure. The cross-attention module consists of multi-head cross-attention (MHCA), and its calculation process is as follows: y = x + MHCA(Q,K,V) Where x represents the input feature, y represents the output feature after the cross-attention module, the Q (Query) matrix comes from the output feature of the parallel Transformer module, and the K (Key) matrix and V (Value) matrix come from the same input feature x; The core of multi-head cross-attention is calculating the cross-attention matrix. First, the output features of the parallel Transformer module are linearly projected, and a bias term of the same dimension is added. The result is used as the query matrix (Query). Similarly, the output features of the position encoding module are linearly projected, and a bias term of the same dimension is added. The result is used as the key matrix (Key) and value matrix (Value). Second, the query matrix (Query), key matrix (Key), and value matrix (Value) are mapped, dividing the original channel dimension of 768 into 12 attention heads, each with a dimension of 64. Then, the query matrix (Query) and key matrix (Key) are multiplied, and the output is passed through a sigmoid function to fuse feature information from different modules. Finally, the resulting output is multiplied with the value matrix (Value) to obtain the output feature matrix. The calculation process is as follows: Where, n h The number of cross-attention heads is set to 196. W O Let d represent the parameters of the linear mappings Q, K, V, and O, respectively. k This represents the dimension of the K-vector; The formula for calculating the sigmoid function is as follows: Where x is an element in the input feature map; ③ The features obtained after passing through the cross-attention module are used as input. First, the features are processed by a layer normalization operation. Then, a fully connected layer module is used to perform spatial transformation to enhance the spatiotemporal feature information of the video. Finally, a residual structure is used to fuse the features to enhance the spatiotemporal information of the video. The fully connected layer module consists of a feed-forward network (FFN) with two linear transformation layers: the first linear transformation maps the spatiotemporal features at each location to a larger-dimensional feature space, and the second linear transformation restores the features to their original dimensions, resulting in a feature map of the same size as the input. A non-linearity is introduced between the two linear transformation layers for filtering, and its calculation process is as follows: y = FFN(x) Where x represents the input feature and y represents the output feature after passing through the fully connected layer structure; For feedforward neural networks, the calculation formula is as follows: FFN(x) = max(0, xW1+b1)W2+b2 Where x represents the input feature, W1 and W2 are the weight matrices of the linear layer, b1 and b2 are the basis vectors, and the four parameters W1, W2, b1 and b2 are randomly initialized and adaptively learned.

5. The action recognition system based on spatiotemporal diffusion and parallel Transformer according to claim 1, characterized in that: The specific structure of the local motion excitation module is as follows: ① The video features output by the feature extraction module are used as input for feature extraction. The spatial resolution of the original input features is adjusted to 14×14. First, the video features obtained by the parallel Transformer module or the cross Transformer module are passed through a two-dimensional convolution operator with a kernel size of (3,3), a stride of (1,1), padding of (1,1), and a number of kernels of 768. This two-dimensional convolution operation is used to spatially downsample the input features and extract the spatial information of the local neighborhood of the video. Then, a batch normalization operation is used to process the features to reduce the redundant information of the video. Finally, the output is passed through a linear rectified function ReLU to obtain a feature map of the same size as the input. The formula for calculating the linear rectified function ReLU is as follows: Where x is an element in the input feature map; ② The video features obtained after feature extraction are used as input, and correlation calculation is performed to enhance the temporal relationship of the video; the implementation details of this operation are as follows: First, the input video features are temporally separated to obtain video features of different time series. Second, the features are L2 normalized to map the input data to the same value range, thereby improving the convergence speed of the model. The formula for calculating L2 norm normalization is as follows: Among them, E in It is the training sample error without a normalization term, initially set to 1e-6, where λ is a learnable normalization parameter, and w j It is the row vector of the input matrix w; Then, a dimension reshaping operation is performed to transform the size of the input feature map from B×C×H×W to B×C×N, where N=H×W. Next, a matrix multiplication operation is performed to fuse the feature maps of adjacent time series to enhance the temporal features of the video. Finally, the spatial correlation of video features is captured through local window sampling operations to merge video features from different time series. ③ The features obtained after correlation calculation are used as input, and motion estimation is used to enhance the temporal features of the video, i.e., two-dimensional convolution operators at different scales. First, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. Second, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 2×2, and padding of 1×1 to downsample the input features. Then, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 2×2, and padding of 1×1 to downsample the input features a second time. Finally, the input features are processed by a convolution operator with a kernel size of 3×3, a stride of 1×1, and padding of 0 to downsample the input features a third time. Batch Normalization is added after each convolution operation, and the output is passed through a linear rectified function ReLU to increase the nonlinear relationship between the layers of the deep network. ④ The features obtained after motion estimation are used as input. A feature transformation module is used to restore the dimensionality of the input features, and a residual structure is used to fuse the features after dimensionality restoration to enhance the temporal information of the video. The specific operation of the feature transformation module is as follows: First, the input features are processed using a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. Second, the input features are processed using a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. Third, the input features are processed using a convolutional operator with a kernel size of 3×3, a stride of 1×1, and padding of 1×1 to obtain features of the same size as the input. After each convolutional operation, a batch normalization operation is added, and the output is passed through a linear rectified function ReLU to increase the non-linear relationship between the layers of the deep network. Finally, an upsampling operation is used to process the input features, restoring the output features to the same dimension as the original input. Through continuous two-dimensional convolution and upsampling operations, the temporal information of the video is extracted.

6. The action recognition system based on spatiotemporal diffusion and parallel Transformer according to claim 1, characterized in that: The structure of the spatiotemporal diffusion module is as follows: ① The video features output by the cross-Transformer module are used as the input of the spatiotemporal diffusion module to establish a spatial-depth convolution module to extract video spatial information; that is, the input features are processed by a spatial-depth convolution module with a kernel size of 1×3×3, a stride of 1×1×1 and padding of 1×1×1 to obtain a feature map of the same size as the input. The convolutional features are batch normalized and the output is passed through a linear rectified function ReLU to avoid overfitting caused by deep networks. ② Using the features obtained from the spatial-depth convolution module as input, a temporal-separation convolution module is established to enhance the interaction of temporal information in the video. Specifically, the input features are processed by a temporal-separation convolution module with a kernel size of 3×1×1, a stride of 1×1×1, and padding of 1×0×0 to obtain a feature map of the same size as the input. The convolutional features are then batch normalized, and the output is passed through a linear rectified function ReLU to avoid overfitting caused by deep networks. ③ Using the features obtained after passing through the temporal-separated convolution module as input, a 1×1 convolution module is established to restore the channel dimension of the features and enhance the spatiotemporal information of the video; that is, the input features are processed by a convolution operator with a kernel size of 1×1×1, a stride of 1×1×1 and padding of 0 to obtain a feature map of the same size as the input. The convolutional features are then batch normalized, and the output is passed through a linear rectified function ReLU to avoid overfitting caused by deep networks. ④ The features of the input video are processed through the residual structure, and the resulting output is fused with the local video representation.

7. The action recognition system based on spatiotemporal diffusion and parallel Transformer according to claim 1, characterized in that: The video classifier module takes the output feature map of the last cross-transformer module as its input, performs global average pooling, and then passes it through a fully connected layer to finally output the classification result of the action recognition model.

8. An action recognition system based on spatiotemporal diffusion and parallel Transformer as described in claim 2 or 3, characterized in that: The feature extraction module uses 3D convolution operators to extract video features, and can also use a combination of 2D and 1D convolution operators to extract video features; The attention calculation module used in the parallel Transformer module is designed based on the self-attention mechanism. It can also be designed according to block attention, step attention, hash attention, or linear attention to achieve effective modeling of global video features.

9. The action recognition system based on spatiotemporal diffusion and parallel Transformer according to claim 5, characterized in that: The feature extraction, motion estimation, and feature transformation in the local motion excitation module all use 2D convolution operators to extract the difference information of video frames. Alternatively, depth-separable convolution, grouped convolution, transpose convolution, or 3D convolution can be used to extract the difference information of video frames. The correlation calculation in the local motion excitation module uses matrix multiplication to achieve information interaction in the local context, or matrix addition or subtraction can be used for correlation calculation.

10. An action recognition method based on spatiotemporal diffusion and parallel Transformer, characterized in that: Includes the following steps: Step 1: Using a sparse sampling strategy, the input video is sent to the feature extraction module, and video features are extracted through the feature extraction module. Step 2: The video features extracted in Step 1 are fed into a two-branch Transformer network structure. The two-branch Transformer network structure is used to learn the spatiotemporal features of the video and to model both local and global video information. The two-branch Transformer network includes a first-branch Transformer network and a second-branch Transformer network. The first-branch Transformer network is composed of depthwise separable convolutions and parallel Transformer modules and is used to extract local temporal features of video actions. The second-branch Transformer network is composed of cross-Transformer modules and is used to extract global spatiotemporal features of video actions. Step 3: Construct a local motion excitation module, which can be inserted into any position in the two-branch Transformer network to convert the spatial features of adjacent video frames into motion features. The local motion excitation module is used twice: First, the output features of the parallel Transformer module are used as input to establish the local correspondence of video frames, and the output result is input into the next layer of the parallel Transformer module; Second, the output features of the cross Transformer module are used as input to establish the local correspondence of video frames, and the output result is input into the next layer of the cross Transformer module. Step 4: Construct a spatiotemporal diffusion module to effectively learn long-range temporal dependencies in videos. This module takes the global spatiotemporal features extracted by the cross-Transformer module as input, uses a diffusion structure to extract long-range temporal information, and fuses the output into the parallel Transformer module to effectively improve the performance of video action recognition. This module can be inserted into the two-branch Transformer network structure in Step 2 to supplement the information of the parallel Transformer module from the cross-Transformer module. Step 5: Construct a video classifier module, which is implemented using a fully connected layer structure. The output features of the last cross-transformer module are used as the input of this module to classify video actions and output the video action recognition results.

Citation Information

Patent Citations

  • Multi-modal fusion detection method for deeply-forged audio and video

    CN116797896A

  • Behavior detection method and system based on global spatial-temporal feature learning

    CN117197727A