Video behavior recognition method based on multi-scale time feature fusion

Abstract

The invention discloses a video behavior recognition method based on multi-scale time feature fusion, and the method comprises the steps: taking a dual-path network as a basic structure, carrying out the normalization and uniform frame extraction of an input video, and respectively inputting a Slow path and a Fast path for feature extraction. The Slow path focuses on low-frequency background semantic modeling, the Fast path integrates a dynamic Token sparse module (VDTS) constructed based on one-dimensional convolution, and a key action area is adaptively selected through a Gumbel Top-K mechanism. Furthermore, according to the method, a gating bidirectional cross attention module (Bi-CAM) is designed between a fast path and a slow path, dynamic complementary fusion of bidirectional information streams is achieved by introducing a Query-Key-Value fusion structure and a channel attention gating mechanism, and the coupling efficiency and expression ability between multi-scale features are effectively improved. And finally, accurate recognition of behavior categories is completed through a classifier. The method gives consideration to the recognition precision and the calculation efficiency, is suitable for scenes such as behavior monitoring, man-machine interaction and intelligent security and protection, and has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video behavior recognition method based on multi-scale temporal feature fusion. Technical Background

[0002] With the rapid development of artificial intelligence (AI) technology, deep learning-based video action recognition methods have gradually become a key research area in computer vision. Their primary goal is to accurately identify specific human behaviors, such as running, jumping, and fighting, from videos. These methods are widely used in real-world scenarios such as security surveillance, intelligent transportation, human-computer interaction, autonomous driving, and motion analysis. Research in the field of video action recognition can be categorized into two main groups: traditional methods and deep learning methods, primarily distinguished by their feature extraction methods. Traditional methods rely primarily on manually designed feature extraction techniques, extracting handcrafted features to classify video content. In contrast, deep learning methods can leverage large amounts of data to learn features, demonstrating greater accuracy and robustness in complex visual tasks.

[0003] Traditional video action recognition methods in the field of computer vision primarily rely on manually designed features for action modeling and recognition. After years of development, this type of method has developed into several representative technical approaches. Early on, Bobick et al. proposed motion history graphs and motion energy graphs (Bobick AF, Davis JW. The recognition of human movement using temporal templates [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(3): 257-267.) to capture the temporal dynamics of object motion in videos, effectively characterizing the temporal characteristics of actions. Subsequently, Chaudhry et al. introduced directional optical flow histograms (Chaudhry R, ​​Ravichandran A, Hager G, et al. Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions [C]. 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009: 1932-1939.). By extracting optical flow information to model the target's motion direction and speed, this method improved action recognition accuracy to a certain extent. Such methods are generally highly efficient with limited computing resources and can quickly complete feature extraction and classification tasks. However, due to their limited feature expression capabilities, they often struggle to fully model spatiotemporal relationships in complex scenes, and recognition accuracy still needs to be improved.

[0004] Among deep learning methods, the literature (Ji S, Xu W, Yang M, et al. 3D convolutional neural networks for human action recognition [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35 (1): 221-231) uses a three-dimensional convolutional network to extract video features for video recognition, but the computational overhead is still high. Feichtenhofer et al. (Feichtenhofer C., Fan H., Malik J., et al. Slowfast networks for video recognition [C]. Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 6202-6211.) proposed a SlowFast network based on 3DResnet that uses two branches at different speeds to process the spatiotemporal information in the video. The slow path is responsible for capturing spatial information at low frame rates, while the fast path is responsible for extracting dynamic information at high frame rates. The two achieve efficient spatiotemporal feature learning through information fusion. While ensuring high accuracy, the SlowFast network significantly reduces computational overhead through a slow path with a low frame rate. However, it suffers from the problems of low attention to the target area and lack of information interaction between the fast and slow paths.

[0005] Although the above-mentioned video action recognition methods based on convolutional neural networks can also achieve video action recognition tasks well, they are challenging for long-term videos with large spans. In 2017, Vaswani et al. proposed the Transformer method for natural language processing (Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [J]. Advances in Neural Information Processing Systems, 2017, 30.), which can effectively process long-term data. This technology has now been extended to the fields of images and videos. Liu et al. proposed the VideoSwim Transformer model (Ze Liu, Jia Ning, et al. Video Swin Transformer [A]. 2021.arXiv: 2106.13230.), which extends the application of the transformer architecture from two-dimensional images to three-dimensional videos. With the help of the self-attention mechanism, this model can effectively capture long-range dependencies and handle complex dynamics in video sequences. The proposed method for exchanging features between different regions uses a self-attention mechanism with a variable window size to achieve feature interaction between different windows, reducing computational effort while ensuring accuracy. This type of method helps address the difficulty of extracting long-term temporal information in video understanding tasks, but it is insufficient for extracting high- and low-level features.

[0006] Although current action recognition technology has achieved high accuracy on multiple public datasets, it still faces numerous challenges in practical application and has yet to reach the maturity of image recognition technology. This is primarily due to the greater complexity of video understanding tasks compared to image processing: video data introduces a temporal dimension (T) in addition to the two-dimensional spatial dimensions (H×W), requiring the model to simultaneously process a large number of sequential image frames, significantly increasing the computational burden and the difficulty of model design. Therefore, existing action recognition methods still have considerable room for improvement in terms of model structure, computational efficiency, and application feasibility. Current mainstream deep learning-based video behavior recognition methods generally have the following shortcomings: First, there is serious redundant modeling. Traditional methods usually treat all frames or spatial regions equally, failing to effectively identify key action information, resulting in repeated modeling of non-key frames and background areas, causing a waste of computing resources; second, the fusion mechanism is single. For example, the lateral connection of SlowFast is a static convolution, which cannot adaptively adjust the information flow between different paths; third, the coupling between spatial and temporal feature modeling is weak. Existing methods often decouple spatiotemporal modeling or focus on a single dimension, lacking the ability to jointly model spatial static background and temporal dynamic changes, thereby limiting recognition accuracy in complex behavior scenarios. Summary of the Invention

[0007] To address the challenges of video action recognition due to video frame redundancy, inadequate action representation, and inefficient path feature fusion, a video action recognition method based on multi-scale temporal feature fusion is proposed. This method builds a dual-channel model based on the SlowFast framework. A Conv1D-based dynamic token sparse module (VDTS) is introduced in the Fast path to adaptively filter the most discriminative motion segments in space and time, effectively reducing redundant computation. A gated bidirectional cross-attention module (Bi-CAM) is designed between the Slow and Fast paths to enhance the multi-scale feature interaction between the paths and improve the overall model's ability to jointly model spatial context and temporal dynamics.

[0008] 1. A video behavior recognition method based on multi-scale temporal feature fusion, characterized by mainly comprising the following steps:

[0009] S1. Normalize the input video (5-10s), extract frames evenly according to the length of the video, extract one frame every two frames, and then evenly extract 32 frames, and divide them into 4 stages in chronological order;

[0010] S2: Input the first frame of each of the four stages obtained in S1 into the spatial background information extraction branch network built on the basis of ResNet3D, perform feature extraction on these four frames, and obtain a set of spatial background features;

[0011] S3: Input the four-stage video frames obtained in S1 into a dynamic token sparse module (VDTS) containing one-dimensional convolution to extract important motion features of the input video, and then input it into the time information extraction branch network built on the basis of ResNet3D to obtain a motion time feature set;

[0012] S4, the spatial information feature set and the temporal information feature set are subjected to feature interaction through the improved gated bidirectional cross attention module to finally obtain the classification result;

[0013] 2. The method for video behavior recognition based on multi-scale temporal feature fusion according to claim 1, wherein step S2 comprises the following steps:

[0014] S21. The spatial background information extraction branch backbone network is constructed based on an improved three-dimensional convolutional residual network ResNet3D structure. The network input is a three-channel RGB video sequence tensor. The first stage convolution operation is performed on the input video sequence. A three-dimensional convolution kernel of size (1×7×7) is used to preliminarily extract the spatial edge structure. After being processed by the ReLU activation function, the kernel is input into a 3D maximum pooling with a pooling window size of (1×3×3) and a stride of (1×2×2). The integrity of the time dimension is retained while downsampling in the spatial dimension.

[0015] S22, the network backbone consists of four residual module stages (Layer 1 to Layer 4), each module contains multiple 3DBottleneck residual blocks; each residual block contains a three-layer convolution structure: (1×1×1) dimensionality increase convolution → (3×3×3) spatial convolution → (1×1×1) dimensionality reduction convolution, and all are equipped with BatchNorm and ReLU;

[0016] S23, Slow path performs 3D convolution residual encoding on the input video sequence and obtains the output feature tensor with the shape of [B, C, T, H, W], where B is the batch size, C = 2048, T = 4, H = W = 7; this feature tensor will retain the spatial and temporal dimensions for subsequent attention fusion operations;

[0017] S24. The spatial background feature tensor after the above 3D encoding is retained and input into the gated bidirectional cross attention module (Bi-CAM) together with the temporal motion feature tensor from the Fast path, and interactively fused in multi-scale spatiotemporal dimensions to form the final spatiotemporal semantic features.

[0018] 3. The video action recognition method based on multi-scale temporal feature fusion according to claim 1, wherein step S3 comprises the following steps:

[0019] S31, use a lightweight 3D convolution to extract local spatiotemporal features from the 32 frames obtained in S1, and flatten the spatial and temporal dimensions of the obtained feature tensor into a one-dimensional token sequence. The calculation formula is as follows:

[0020]

[0021] Where T s Represents the transformed token sequence, F represents the output feature tensor from 3D convolution, C represents the number of channels, p represents the patch size, T represents the number of time frames, H represents the spatial height of the feature map, and W represents the spatial width of the feature map;

[0022] S32. A one-dimensional convolution with a kernel size of 1 is input to the token sequence to obtain the importance score of each token. The calculation formula is as follows:

[0023]

[0024] Conv1D represents a one-dimensional convolution with a kernel size of 1, and s represents the score of each token;

[0025] S33, adding Gumbel distribution noise g to the score s i , calculate the score after perturbation The calculation formula is as follows:

[0026]

[0027] The noise scoring formula is as follows:

[0028]

[0029] Where τ is the temperature parameter that controls the degree of sampling smoothness, and ∈ is a very small positive constant used to prevent the logarithmic operation in numerical calculations from being infinite or undefined;

[0030] S34, score from disturbance The top-K tokens with the largest scores are dynamically selected and retained, and the others are set as masks. The formula is as follows:

[0031]

[0032] The position where the mask value is 1 represents the reserved token;

[0033] S35, the obtained two-dimensional sparse mask mask∈{0,1} B×N Map back to the corresponding spatiotemporal dimension structure and reshape into a three-dimensional mask map The mask image is upsampled by the nearest neighbor interpolation method to restore it to a spatiotemporal shape mask consistent with the input tensor full ∈{0,1} B×1×T×H×W , the unretained area is filled with the channel mean μ(x), and the output feature tensor after adding the residual connection fusion is x′, the formula is as follows:

[0034]

[0035] x′=x+(x×mask full +μ(x)×(1-mask full ))

[0036] S36: Input the obtained output x′ into the ResNet3D backbone (ResLayer1-4) as one of the inputs of the subsequent gated bidirectional cross attention module and fuse it with the spatial features of the Slow path;

[0037] 4. The behavior recognition method based on multi-stage spatiotemporal feature fusion according to claim 1 is characterized in that the feature interaction step S4 mainly includes the following steps:

[0038] S41, the tensor F output by the S3 path fast Perform linear transformation to obtain query vector Q f , and generate key-value pairs K for the tensors output by the S2 path s 、V s , calculate the attention to get the cross attention output F for the fast path f , introduce channel attention gating to the fused features to obtain the module output F fout , the formula is as follows:

[0039]

[0040] F fout =F fast +γf·f f

[0041] S42 and S41 are relative, the tensor output by the Fast path is used as K f 、V f , the tensor output by the Slow path is Q s , calculated using the same formula:

[0042] f sout =f slow +γs·F s

[0043] S43, the output of the final Fast path and the output of the Slow path will be sent to the classification head respectively, and after global average pooling, Dropout and fully connected layers, the final video behavior category prediction result will be output. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a diagram showing the overall structure of a video behavior recognition method based on multi-scale temporal feature fusion used in an embodiment of the invention;

[0045] Figure 2 The VDTS module structure diagram of the video behavior recognition method based on multi-scale temporal feature fusion used in the embodiment of the invention;

[0046] Figure 3A block diagram of the gated bidirectional cross attention module of the video behavior recognition method based on multi-scale temporal feature fusion used in an embodiment of the invention;

[0047] Figure 4 The training method of the video behavior recognition method based on multi-scale temporal feature fusion used in the embodiment of the invention is implemented;

[0048] Figure 5 This is a diagram showing the implementation effect of the video behavior recognition method based on multi-scale temporal feature fusion used in an embodiment of the invention; DETAILED DESCRIPTION

[0049] To provide a clearer understanding of the technical features, objectives, and beneficial effects of the present invention, an embodiment of the present invention is further described below with reference to the accompanying drawings. The embodiment is intended only to further illustrate the present invention and is not to be construed as limiting the scope of protection of the present invention. Non-essential improvements and adjustments made by those skilled in the art based on the present invention also fall within the scope of protection of the present invention.

[0050] 1. A video behavior recognition method based on multi-scale temporal feature fusion, such as Figure 1 The overall structure of the method is shown, which is characterized by mainly including the following steps:

[0051] S1. Normalize the input video (5-10s), extract frames evenly according to the length of the video, extract one frame every two frames, and then evenly extract 32 frames, and divide them into 4 stages in chronological order;

[0052] S2: Input the first frame of each of the four stages obtained in S1 into the spatial background information extraction branch network built on the basis of ResNet3D, perform feature extraction on these four frames, and obtain a set of spatial background features;

[0053] S3, input the 4-stage video frames obtained in S1 into Figure 2 The dynamic token sparse module (VDTS) containing one-dimensional convolution extracts important motion features from the input video, which are then input into the time information extraction branch network built on the basis of ResNet3D to obtain a set of motion time features.

[0054] S4, the spatial information feature set and the temporal information feature set are combined as follows: Figure 3 The improved gated bidirectional cross attention module shown in the figure performs feature interaction and finally obtains the classification result;

[0055] 2. The method for video behavior recognition based on multi-scale temporal feature fusion according to claim 1, wherein step S2 comprises the following steps:

[0056] S21. The spatial background information extraction branch backbone network is constructed based on an improved three-dimensional convolutional residual network ResNet3D structure. The network input is a three-channel RGB video sequence tensor. The first stage convolution operation is performed on the input video sequence. A three-dimensional convolution kernel of size (1×7×7) is used to preliminarily extract the spatial edge structure. After being processed by the ReLU activation function, the kernel is input into a 3D maximum pooling with a pooling window size of (1×3×3) and a stride of (1×2×2). The integrity of the time dimension is retained while downsampling in the spatial dimension.

[0057] S22, the network backbone consists of four residual module stages (Layer 1 to Layer 4), each module contains multiple 3DBottleneck residual blocks; each residual block contains a three-layer convolution structure: (1×1×1) dimensionality increase convolution → (3×3×3) spatial convolution → (1×1×1) dimensionality reduction convolution, and all are equipped with BatchNorm and ReLU;

[0058] S23, Slow path performs 3D convolution residual encoding on the input video sequence and obtains the output feature tensor with the shape of [B, C, T, H, W], where B is the batch size, C = 2048, T = 4, H = W = 7; this feature tensor will retain the spatial and temporal dimensions for subsequent attention fusion operations;

[0059] S24. The spatial background feature tensor after the above 3D encoding is retained and input into the gated bidirectional cross attention module (Bi-CAM) together with the temporal motion feature tensor from the Fast path, and interactively fused in multi-scale spatiotemporal dimensions to form the final spatiotemporal semantic features.

[0060] 3. The video action recognition method based on multi-scale temporal feature fusion according to claim 1, wherein step S3 comprises the following steps:

[0061] S31, use a lightweight 3D convolution to extract local spatiotemporal features from the 32 frames obtained in S1, and flatten the spatial and temporal dimensions of the obtained feature tensor into a one-dimensional token sequence. The calculation formula is as follows:

[0062]

[0063] Where T s Represents the transformed token sequence, F represents the output feature tensor from 3D convolution, C represents the number of channels, p represents the patch size, T represents the number of time frames, H represents the spatial height of the feature map, and W represents the spatial width of the feature map;

[0064] S32. A one-dimensional convolution with a kernel size of 1 is input to the token sequence to obtain the importance score of each token. The calculation formula is as follows:

[0065]

[0066] Conv1D represents a one-dimensional convolution with a kernel size of 1, and s represents the score of each token;

[0067] S33, adding Gumbel distribution noise g to the score s i , calculate the score after perturbation , the calculation formula is as follows:

[0068]

[0069] The noise scoring formula is as follows:

[0070]

[0071] Where τ is the temperature parameter that controls the degree of sampling smoothness, and ∈ is a very small positive constant used to prevent the logarithmic operation in numerical calculations from being infinite or undefined;

[0072] S34, score from disturbance The top-K tokens with the largest scores are dynamically selected and retained, and the others are set as masks. The formula is as follows:

[0073]

[0074] The position where the mask value is 1 represents the reserved token;

[0075] S35, the obtained two-dimensional sparse mask mask∈{0,1} B×N Map back to the corresponding spatiotemporal dimension structure and reshape into a three-dimensional mask map The mask image is upsampled by the nearest neighbor interpolation method to restore it to a spatiotemporal shape mask consistent with the input tensor full ∈{0,1} B×1×T×H×W , the unretained area is filled with the channel mean μ(x), and the output feature tensor after adding the residual connection fusion is x′, the formula is as follows:

[0076]

[0077] x′=x+(x×mask full +μ(x)×(1-mask full ))

[0078] S36: Input the obtained output x′ into the ResNet3D backbone (ResLayer1-4) as one of the inputs of the subsequent gated bidirectional cross attention module and fuse it with the spatial features of the Slow path;

[0079] 4. The behavior recognition method based on multi-stage spatiotemporal feature fusion according to claim 1 is characterized in that the feature interaction step S4 mainly includes the following steps:

[0080] S41, the tensor F output by the S3 path fast Perform linear transformation to obtain query vector Q f , and generate key-value pairs K for the tensors output by the S2 path s 、V s , calculate the attention to get the cross attention output F for the fast path f , introduce channel attention gating to the fused features to obtain the module output F fout , the formula is as follows:

[0081] γf=σ(FC2(ReLU(FC1(AvgPool3D(Ff f )))))

[0082] F fout =F fast +γf·F f

[0083] S42 and S41 are relative, the tensor output by the Fast path is used as K f 、V f , the tensor output by the Slow path is Q s , calculated using the same formula:

[0084] F sout =F slow +γs·F s

[0085] S43, the output of the final Fast path and the output of the Slow path will be sent to the classification head respectively, and after global average pooling, Dropout and fully connected layers, the final video behavior category prediction result will be output.

[0086] Simulation experiment: By using Pycharm development tools, the NVIDIA GeForceRTX 4060 graphics card is used for network training in the PyTorch framework. The training process is as follows Figure 4 The experimental results are shown in Figure 5 .

[0087] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the claims of the present invention.

Claims

1. A video behavior recognition method based on multi-scale temporal feature fusion, characterized in that: The main steps include: S1. Normalize the input video (5-10s), extract frames evenly according to the length of the video, extract one frame every two frames, and then evenly extract 32 frames, and divide them into 4 stages in chronological order; S2: Input the first frame of each of the four stages obtained in S1 into the spatial background information extraction branch network built on the basis of ResNet3D, perform feature extraction on these four frames, and obtain a set of spatial background features; S3: Input the four-stage video frames obtained in S1 into a dynamic token sparse module (VDTS) containing one-dimensional convolution to extract important motion features of the input video, and then input it into the time information extraction branch network built on the basis of ResNet3D to obtain a motion time feature set; S4. The spatial information feature set and the temporal information feature set are interacted through an improved gated bidirectional cross attention module to finally obtain the classification result.

2. The video behavior recognition method based on multi-scale temporal feature fusion according to claim 1 is characterized in that: The step S2 comprises the following steps: S21. The spatial background information extraction branch backbone network is constructed based on an improved three-dimensional convolutional residual network ResNet3D structure. The network input is a 3-channel RGB video sequence tensor. The first stage convolution operation is performed on the input video sequence. A three-dimensional convolution kernel of size (1×7×7) is used to preliminarily extract the spatial edge structure. After being processed by the ReLU activation function, the kernel is input into a 3D maximum pooling with a pooling window size of (1×3×3) and a stride of (1×2×2). The integrity of the time dimension is retained while downsampling in the spatial dimension. S22, the network backbone consists of four residual module stages (Layer 1 to Layer 4), each module contains multiple 3DBottleneck residual blocks; each residual block contains a three-layer convolution structure: (1×1×1) dimensionality increase convolution → (3×3×3) spatial convolution → (1×1×1) dimensionality reduction convolution, and all are equipped with BatchNorm and ReLU; S23, Slow path performs 3D convolution residual encoding on the input video sequence and obtains the output feature tensor with the shape of [B, C, T, H, W], where B is the batch size, C = 2048, T = 4, H = W = 7; This feature tensor will retain the spatial and temporal dimensions for use in subsequent attention fusion operations; S24. The spatial background feature tensor after the above 3D encoding is retained and input into the gated bidirectional cross attention module (Bi-CAM) together with the temporal motion feature tensor from the Fast path, and interactively fused in multi-scale spatiotemporal dimensions to form the final spatiotemporal semantic features.

3. The video behavior recognition method based on multi-scale temporal feature fusion according to claim 1 is characterized in that: The step S3 comprises the following steps: S31, use a lightweight 3D convolution to extract local spatiotemporal features from the 32 frames obtained in S1, and flatten the spatial and temporal dimensions of the obtained feature tensor into a one-dimensional token sequence. The calculation formula is as follows: Where T s Represents the transformed token sequence, F represents the output feature tensor from 3D convolution, C represents the number of channels, p represents the patch size, T represents the number of time frames, H represents the spatial height of the feature map, and W represents the spatial width of the feature map; S32. A one-dimensional convolution with a kernel size of 1 is input to the token sequence to obtain the importance score of each token. The calculation formula is as follows: Conv1D represents a one-dimensional convolution with a kernel size of 1, and s represents the score of each token; S33, adding Gumbel distribution noise g to the score s i , calculate the score after perturbation The calculation formula is as follows: The noise scoring formula is as follows: Where τ is the temperature parameter that controls the degree of sampling smoothness, and ∈ is a very small positive constant used to prevent the logarithmic operation in numerical calculations from being infinite or undefined; S34, score from disturbance The top-K tokens with the largest scores are dynamically selected and retained, and the others are set as masks. The formula is as follows: The position where the mask value is 1 represents the reserved token; S35, the obtained two-dimensional sparse mask mask∈{0,1} B×N Map back to the corresponding spatiotemporal dimension structure and reshape into a three-dimensional mask map The mask image is upsampled by the nearest neighbor interpolation method to restore it to a spatiotemporal shape mask consistent with the input tensor full ∈{0,1} B×1×T×H×W , the unretained area is filled with the channel mean μ(x), and the output feature tensor after adding the residual connection fusion is x′, the formula is as follows: x′=x+(x×mask full +μ(x)×(1-mask full )) S36. Input the obtained output x′ into the ResNet3D backbone (ResLayer1~4) as one of the inputs of the subsequent gated bidirectional cross attention module and fuse it with the spatial features of the Slow path.

4. The behavior recognition method based on multi-stage spatiotemporal feature fusion according to claim 1 is characterized in that: The feature interaction step S4 mainly includes the following steps: S41, the tensor F output by the S3 path fast Perform linear transformation to obtain query vector Q f , and generate key-value pairs V for the tensors output by the S2 path s 、V s , calculate the attention to get the cross attention output F for the fast path f , introduce channel attention gating to the fused features to obtain the module output F fout , the formula is as follows: γf=σ(FC2(ReLU(FC1(AvgPool3D(F f ))))) F fout =F fast +γf·F f S42 and S41 are relative, the tensor output by the Fast path is used as K f 、V f , the tensor output by the Slow path is Q s , calculated using the same formula: F sout =F slow +γs·F s S43, the output of the final Fast path and the output of the Slow path will be sent to the classification head respectively, and after global average pooling, Dropout and fully connected layers, the final video behavior category prediction result will be output.

Images