A video action detection method based on lightweight spatiotemporal joint network

Abstract

The present invention discloses a video action detection method based on a lightweight spatiotemporal joint network, which belongs to the field of video behavior recognition technology. The present invention uses a 3D backbone network and a 2D backbone network to respectively extract spatial features and temporal features on temporal information, and further fully fuses and extracts spatiotemporal features by designing a lightweight spatiotemporal feature fusion module by deeply fusing convolution with a self-attention mechanism; in addition, a spatiotemporal translation module TSM containing only 2D convolution is used as the backbone branch of the spatiotemporal feature extraction module. The YOWOv3 model of the present invention is not only lighter than the latest high-efficiency spatiotemporal action detection model, but also has the best performance in terms of accuracy at the same level; the further optimized YOWOv3‑TSM not only reduces the difficulty of training, but also maintains excellent performance in both accuracy and speed, achieving the best balance between accuracy and speed in video action detection.

Description

Technical Field

[0001] The present invention relates to a video action detection method based on a lightweight spatiotemporal joint network, and belongs to the technical field of video behavior recognition. Background Art

[0002] Spatiotemporal action detection has broad application prospects in security monitoring, driver and passenger behavior recognition, and short video review. Compared to standard visual object detection, spatiotemporal action detection not only detects the subject but also analyzes the subject's behavior. This requires the help of temporal information to understand the subject's specific behavior. For example, a single frame of image information alone cannot help us complete the classification of the subject's behavior during door opening and closing. Therefore, we need to extract temporal features. However, once a temporal feature extraction network is introduced, the model often becomes bloated, making it difficult to deploy and use directly.

[0003] Chen S et al. proposed an end-to-end video action detection framework called "Watch Once Only (WOO)", which uses a unified video backbone to simultaneously extract the location of the action and the features of action classification (Chen S, Sun P, Xie E, et al. Watch only once: An end-to-end video action detection framework [C]. Proceedings of the IEEE / CVF International Conference on Computer Vision, On line. 2021: 8178-8187.). However, due to the use of its full 3D convolutional structure, the model has problems such as large parameters, bloated model, and low computational efficiency, which affects the accuracy and efficiency of action recognition.

[0004] Therefore, how to design an efficient and lightweight spatiotemporal action detection network becomes crucial. Summary of the Invention

[0005] To address the lightweight problem of joint detection of temporal behavior recognition and spatiotemporal behavior subjects and improve the accuracy and efficiency of video behavior recognition, the present invention provides a video action detection method based on a lightweight spatiotemporal joint network and designs a lightweight spatiotemporal action detection model called YOWOv3. This model integrates an efficient 3D temporal feature extraction model with a lightweight 2D object detection model, injects temporal information into a multi-level 2D spatiotemporal feature extraction backbone model, deeply integrates convolution with the self-attention mechanism, further decomposes the window partitioning of the self-attention mechanism calculation branch and utilizes the cross-sparse self-attention mechanism to design a lightweight and efficient spatiotemporal feature fusion module to fuse spatial features and temporal features. The fully fused output features are then decoupled to obtain a lightweight and efficient spatiotemporal action detection model.

[0006] At the same time, to mitigate the special situation where the spatiotemporal action detection model may not support 3D convolution operators when deployed at the edge, a spatiotemporal translation module (TSM) containing only 2D convolution is used to extract temporal feature information. The spatiotemporal translation module enables the model to obtain temporal information, and then injects the obtained temporal feature information into the multi-level spatiotemporal feature extraction model. This not only frees the model from the limitations of 3D convolution operations, but also further improves the model's ability to balance accuracy and speed. The implementation details of each step are described in detail below.

[0007] In order to enable the network to extract temporal information and ensure the lightweight of the network model, an efficient 3D CNN method is used. By expanding the classic lightweight network into three-dimensional space, converting 2D convolution into 3D convolution, and converting 2D pooling operations into 3D pooling operations, the 3D branch can extract the required spatiotemporal information. To ensure that the 3D branch can fully extract temporal features, the present invention does not use a pyramid multi-scale feature extraction method similar to YOLO to extract multi-level features at different stages, but instead upsampling the final feature layer, thereby achieving the purpose of splicing the multi-level features extracted by the 2D backbone and fusing the spatiotemporal features.

[0008] F stemi =f conv1×1 (F leveli )

[0009] F headi =f conv1×1 (f conv3×3 (f conv3×3 (F stemi ))) (1)

[0010] In order to extract multi-scale and multi-level spatial features, the most advanced YOLOv8 in the YOLO series is used as the backbone network for spatial feature extraction. In order to speed up the inference speed, in the enhanced feature extraction part, the feature layer extracted by the backbone network is not convolved but only up-sampled.

[0011] After the backbone feature extraction network and the enhanced feature extraction network are completed, the present invention uses 1x1 convolution to integrate the channels so that the F stem i The number of channels is compressed to 256, and then each channel is decoupled. Two parallel 3×3 standard convolutions (two times each) are used to extract the decoupled features. Then, a standard 1×1 convolution is used to re-integrate the decoupled feature channels to 64. Instead of integrating classification and regression features into a single feature matrix, this accelerates model convergence. The overall decoupling process is shown in Equation 1. To further accelerate the convergence of the spatiotemporal action detection model, during training, a YOLOv8 pre-trained model on the COCO dataset is used, and only the pre-trained features on Backbone are loaded into the spatial feature detection branch of the spatiotemporal action detector for training.

[0012] In order to fuse the spatial features and temporal features extracted by the 2D backbone and the 3D backbone, the two are first spliced ​​in the channel dimension. Because after extracting the temporal features, the present invention performs an upsampling operation on the final feature layer, so the final temporal feature layers and the multi-level features extracted by the 2D backbone are consistent in width and height dimensions, so that the feature layers of the two can be directly spliced ​​in the channel dimension.

[0013] In order to fully integrate spatial features and temporal features, the convolution and self-attention mechanisms are combined. First, the concatenated features are extracted twice using two standard 3x3 convolutions (CBR). Then, three 1x1 convolutions are used to map the input spatiotemporal features. The features are then reshaped into N pieces, thereby obtaining a set of rich intermediate features containing 3×N feature maps. The intermediate features are then divided into two branches. The two branches share the intermediate features mapped by three 1x1 convolutions. Then, one branch performs self-attention calculations: the intermediate features are grouped into N groups, each containing 3 features. The three features correspond to the query, key, and value of the self-attention mechanism, and follow the traditional multi-head self-attention module. The other branch performs convolution calculations and generates K by connecting the intermediate features to the fully connected layer. 2 By shifting and aggregating the generated features, it is possible to collect information from the local receptive field like traditional convolution.

[0014] By dividing the window and performing cross sparse self-attention mechanism calculation, the runtime consumption is reduced. Specifically, we first divide the Q, K, and V of the projection mapping into P based on the window. L groups, each containing Q L Positions, get P L local groups The characteristics of attention weight calculation and computational enhancement are shown in formula (2):

[0015]

[0016]

[0017] Among them, A p represents the attention weight of the first stage, Represents the query block Query divided by the window, K p represents the key after window partitioning, d represents the hyperparameter for scaling similarity, X p Represents the feature map of the input after window division, Z p Represents the final output after self-attention calculation, V p Represents the value after window division, and C′ represents the number of feature channels after projection.

[0018] By dividing the window and then performing local self-attention mechanism calculation, although the amount of calculation is reduced, it also hinders the communication of global information. Therefore, in the second stage, the window information is first mixed by the Permute method, and then the self-attention calculation within the window is performed again, so that it can perceive the long-distance features of other windows. The attention mechanism calculation in the second stage is shown in formula (3):

[0019]

[0020] Among them, A q represents the attention weight of the second stage, Represents the query block Query after the second window division, K q represents the key after the second window division, d represents the hyperparameter used to scale the similarity, V q Represents the value after the second window division, Q S represents the number of input feature channels for the second-stage attention calculation, and C′ represents the number of output feature channels after projection.

[0021] Finally, the features of the two branches are spliced ​​in the channel dimension, and the spliced ​​features are added together to obtain the final fused features.

[0022] The beneficial effects of the present invention are:

[0023] The video action detection method of the present invention, by fusing an efficient 3D temporal feature extraction model and a lightweight 2D target detection model, injects temporal information into a multi-level 2D spatiotemporal feature extraction backbone model, deeply fuses convolution with the self-attention mechanism, and designs a lightweight and efficient spatiotemporal feature fusion module to fuse spatial features and temporal features by further decomposing the window division of the self-attention mechanism calculation branch and utilizing the cross-sparse self-attention mechanism. The output features after the final full fusion are decoupled to obtain a lightweight and efficient spatiotemporal action detection model.

[0024] At the same time, in order to reduce the special situation that the spatiotemporal action detection model may encounter when it is deployed on the edge and does not support the 3D convolution operator, the spatiotemporal translation module TSM containing only 2D convolution is used to extract the temporal feature information. The spatiotemporal translation module TSM enables the model to obtain temporal information, and then injects the obtained temporal feature information into the multi-level spatiotemporal feature extraction model respectively. This not only makes the model free from the restrictions of 3D convolution operations, but also further improves the model's ability to balance accuracy and speed, thereby effectively improving the accuracy and efficiency of video behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 This is a diagram of the YOWOv3 network structure of the second embodiment of the present invention.

[0027] Figure 2 This is a diagram of the decoupled feature fusion structure of the second embodiment of the present invention.

[0028] Figure 3 This is the CFCAM module structure of the second embodiment of the present invention.

[0029] Figure 4 This is a structural diagram of the ACISA module according to the second embodiment of the present invention.

[0030] Figure 5 This is a specific structural diagram of the decoupled feature fusion module of the second embodiment of the present invention. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0032] Example 1:

[0033] This embodiment provides a video action detection method, including: constructing a lightweight spatiotemporal action detection deep neural network, which is specifically implemented according to the following steps:

[0034] Step 1: Use the spatiotemporal feature extraction network to extract the temporal information of the video sequence to be detected;

[0035] Step 2: Use the YOLOv8 network as the backbone network for spatial feature extraction and perform upsampling on the feature layer extracted by the backbone network;

[0036] Step 3: Use 1×1 convolution to integrate the feature channels output in step 2, so that the number of channels at each level is compressed to 256. Then use two parallel 3×3 standard convolutions to extract the decoupled features, and then use a standard 1×1 convolution to re-integrate the decoupled feature channels to 64.

[0037] Step 4: Upsample the final feature layer output from step 1 so that it can be concatenated with the multi-level features extracted from step 2.

[0038] Step 5: First, the spatiotemporal features and temporal features extracted in steps 3 and 4 are spliced ​​in the channel dimension, and the convolution and self-attention mechanisms are deeply integrated. The two branches are run in parallel. Finally, the features of the two branches are spliced ​​in the channel dimension and the spliced ​​features are added to obtain the final fused features.

[0039] Step 6: Use three 1×1 convolutions to perform decoupling operations on the features output in step 5, and output classification, regression, and confidence information respectively to obtain the video action detection results.

[0040] Example 2:

[0041] This embodiment provides a video action detection method. First, construct Figure 1 The lightweight spatiotemporal action detection model YOWOv3 network shown is implemented in the following steps:

[0042] Step 1: Expand the efficient classic lightweight network ShuffleNetV2 to three dimensions by expanding it, that is, converting 2D convolution into 3D convolution and 2D pooling into 3D pooling, thus obtaining 3D-ShuffleNetV2 for extracting temporal information of video sequences. Figure 1 Spatio-temporal trunk branches.

[0043] In order to avoid the risk that some devices may not support 3D convolution operations, a spatiotemporal translation module (TSM) containing only 2D convolution is used as Figure 1 The backbone branch of the spatiotemporal feature extraction module.

[0044] Step 2: Use the most advanced YOLOv8 in the YOLO series as the backbone network for spatial feature extraction. In order to speed up the inference, in the enhanced feature extraction part, no convolution operation is performed on the feature layer extracted by the backbone network, but only upsampling operation is performed. Figure 1 Spatial backbone branch.

[0045] Step 3: Use 1×1 convolution to integrate the feature channels output in step 2, so that the number of channels at each level is compressed to 256. Then use two parallel 3×3 standard convolutions (2 times each) to extract the decoupled features, and then use a standard 1×1 convolution to re-integrate the decoupled feature channels to 64. The structure is as follows: Figure 2 shown.

[0046] Step 4: Upsample the final feature layer output in step 1 so that it can be concatenated with the multi-level features extracted in step 2.

[0047] Step 5: Construct the CFCAM module. First, the spatiotemporal features and temporal features extracted in steps 3 and 4 are spliced ​​in the channel dimension, and the convolution and self-attention mechanisms are deeply integrated. The two branches are run in parallel. Finally, the features of the two branches are spliced ​​in the channel dimension and the spliced ​​features are added to obtain the final fused features. The overall structure of the CFCAM module is as follows: Figure 3 As shown in Figure 2, the specific steps of feature fusion are as follows:

[0048] Step 5.1: First, the concatenated features are subjected to secondary feature extraction using two standard 3x3 convolutions (CBR), and then three 1x1 convolutions are used to perform feature mapping on the input spatiotemporal features, and then reshaped into N pieces, thereby obtaining a rich set of intermediate features containing 3×N feature maps.

[0049] Step 5.2: The intermediate features are then split into two branches. The two branches share the intermediate features after three 1x1 convolutions. Then one branch performs self-attention calculation: the intermediate features are grouped into N groups, each containing 3 features. The three features correspond to the query, key, and value of the self-attention mechanism, and follow the traditional multi-head self-attention module.

[0050] Step 5.3: The other branch performs convolution calculations and generates K by connecting the intermediate features to the fully connected layer. 2By shifting and aggregating the generated features, it is possible to collect information from the local receptive field like traditional convolution.

[0051] Step 5.4: In the self-attention mechanism calculation of step 5.2, the window is divided and the cross sparse self-attention mechanism calculation is performed to reduce the runtime time consumption. Specifically, we first divide the Q, K, V through the projection mapping into P based on the window L groups, each containing Q L Positions, get P L local groups The characteristics of attention weight calculation and computational enhancement are shown in formula (1):

[0052]

[0053]

[0054] Among them, A p represents the attention weight of the first stage, Represents the query block Query divided by the window, K p represents the key after window partitioning, d represents the hyperparameter for scaling similarity, X p Represents the feature map of the input after window division, Z p Represents the final output after self-attention calculation, V p Represents the value after window division, and C′ represents the number of feature channels after projection.

[0055] By dividing the window and then performing local self-attention mechanism calculation, although the amount of calculation is reduced, it also hinders the communication of global information. Therefore, in the second stage, the window information is first mixed by the Permute method, and then the self-attention calculation within the window is performed again, so that it can perceive the long-distance features of other windows. The attention mechanism calculation in the second stage is shown in formula (2):

[0056]

[0057] Among them, A q represents the attention weight of the second stage, Represents the query block Query after the second window division, K q represents the key after the second window division, d represents the hyperparameter used to scale the similarity, V q Represents the value after the second window division, Q S represents the number of input feature channels for the second-stage attention calculation, and C′ represents the number of output feature channels after projection.

[0058] Step 5.5: Finally, the features of the two branches are spliced ​​in the channel dimension, and the spliced ​​features are added to obtain the final fused features. The ACISA module structure that implements this splicing process is as follows: Figure 4 shown.

[0059] Step 6: Decouple the features output in step 5 using three 1×1 convolutions to output classification, regression, and confidence information respectively. The decoupled feature fusion module structure is as follows: Figure 5 shown.

[0060] Step 7: Train and test the lightweight network model for spatiotemporal action detection.

[0061] Step 7.1: Use the UCF101-24 and JHMDB51-21 video action detection datasets to test the effectiveness of the improved model and extract video frames from the video datasets.

[0062] Step 7.2: Perform preprocessing operations on the video frames, such as normalization, random cropping, etc. The resolution of the input image sequence is controlled at 224x224.

[0063] Step 7.3: Input the preprocessed 16-frame video sequence into the network model for model training. The network's forward computation and loss function are used to calculate the network error. This error is then backpropagated to calculate the error gradient for each layer of the network weights. The weights are then updated to gradually reduce the network error. This process is repeated repeatedly to find the most effective network training parameters. Minimizing the network loss completes the network training process and yields the network model. This process can be simply summarized as parameter optimization.

[0064] Step 7.4: Test and calculate the network model's parameter count, floating-point computational effort, accuracy, and inference latency on the GPU, CPU, and Mac's M1 pro chip.

[0065] In order to further illustrate the beneficial effects of the present invention, a comparative experiment with the existing model was conducted, and the experimental comparative data are shown in Table 1:

[0066] Table 1: Quantitative comparison results of the proposed method with other existing models on the UCF101-24 dataset

[0067]

[0068] As can be seen from Table 1, the present invention significantly outperforms other models in terms of parameter quantity and computational complexity on large-scale video action detection datasets, and demonstrates its competitiveness in accuracy. Compared with models of the same magnitude, the present invention achieves the best accuracy performance, effectively reducing the difficulty of deploying spatiotemporal action detection models to edge devices. The further designed lightweight YOWOv3-TSM model containing only 2D convolutions has shown excellent potential in both inference speed and accuracy, achieving the best balance between accuracy and speed. The CFACM module designed by the present invention significantly further improves the accuracy of the model while maintaining its lightweight.

[0069] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video action detection method, characterized in that: The video action detection method includes: constructing a lightweight spatiotemporal action detection deep neural network, which is specifically implemented according to the following steps: Step 1: Use the spatiotemporal feature extraction network to extract the temporal information of the video sequence to be detected; Step 2: Use the YOLOv8 network as the backbone network for spatial feature extraction and perform upsampling on the feature layer extracted by the backbone network; Step 3: Use 1×1 convolution to integrate the feature channels output in step 2, so that the number of channels at each level is compressed to 256. Then use two parallel 3×3 standard convolutions to extract the decoupled features, and then use a standard 1×1 convolution to re-integrate the decoupled feature channels to 64. Step 4: Perform an upsampling operation on the final feature layer outputted in step 1 so that it can be spliced ​​with the multi-level features extracted in step 2; Step 5: First, the spatiotemporal features and temporal features extracted in steps 3 and 4 are spliced ​​in the channel dimension, and the convolution and self-attention mechanisms are deeply integrated. The two branches are run in parallel. Finally, the features of the two branches are spliced ​​in the channel dimension, and the spliced ​​features are added to obtain the final fused features. Step 6: Use three 1×1 convolutions to perform decoupling operations on the features output in step 5, and output classification, regression, and confidence information respectively to obtain the video action detection results; The spatiotemporal feature extraction network uses a 3D-ShuffleNetV2 network, and the construction method includes: expanding the lightweight network ShuffleNetV2 to three dimensions, converting 2D convolution into 3D convolution, and converting 2D pooling into 3D pooling; at the same time, in order to avoid the risk that some devices do not support 3D convolution operations, a spatiotemporal translation module TSM containing only 2D convolution is used as the backbone branch of the spatiotemporal feature extraction module.

2. The video motion detection method according to claim 1, wherein: The step 5 comprises: Step 5.1: First, the concatenated features are extracted twice using two standard 3x3 convolutions. Then, three 1x1 convolutions are used to map the input spatiotemporal features. The features are then resized into N feature blocks, thereby obtaining a set of intermediate features containing 3xN feature maps. Step 5.2: Split the intermediate features into two branches, which share the intermediate features after three 1x1 convolutions. Then one branch performs self-attention calculation: the intermediate features are grouped into N groups, each containing 3 features, where the three features correspond to the query, key, and value of the self-attention mechanism, and follow the multi-head self-attention module. Step 5.3: Another branch performs convolution calculations and generates K by connecting the intermediate features to the fully connected layer. 2 feature maps, K represents the size of the convolution kernel, and information is collected from the local receptive field by shifting and aggregating the generated features; Step 5.4: Concatenate the features of the two branches in the channel dimension and add the concatenated features to obtain the final fused features.

3. The video motion detection method according to claim 2, wherein: The two standard 3x3 convolutions in step 5.1 are CBR modules.

4. The video motion detection method according to claim 2, wherein: In the self-attention mechanism calculation of step 5.2, the runtime consumption is reduced by dividing the window and performing cross-sparse self-attention mechanism calculation, specifically including: In the first stage, Q, K, and V obtained by projection mapping are first divided into P using a window-based partitioning method. L groups, each containing Q L Positions, get P L local groups The characteristics of attention weight calculation and computational enhancement are shown in formula (1): Among them, A p represents the attention weight of the first stage, Represents the query block Query divided by the window, K p represents the key after window partitioning, d represents the hyperparameter for scaling similarity, X p Represents the feature map of the input after window division, Z p Represents the final output after self-attention calculation, V p Represents the value after window division, C′ represents the number of feature channels after projection; In the second stage, the window information is first mixed by Permute, and then the self-attention calculation within the window is performed again, so that it can perceive the long-distance features of other windows. The attention mechanism calculation of the second stage is shown in formula (2): Among them, A q represents the attention weight of the second stage, Represents the query block Query after the second window division, K q represents the key after the second window division, d represents the hyperparameter used to scale the similarity, V q Represents the value after the second window division, Q S represents the number of input feature channels for the second-stage attention calculation, and C′ represents the number of output feature channels after projection.

5. The video motion detection method according to claim 1, wherein: The lightweight spatiotemporal action detection deep neural network is trained and tested using the UCF101-24 and JHMDB51-21 datasets.

6. The video motion detection method according to claim 5, characterized in that: During training and testing, video frames are extracted from the video dataset and preprocessed, including normalization and random cropping.

7. The video motion detection method according to claim 6, characterized in that: The resolution of the input image sequence is controlled to 224x 224.

8. The video motion detection method according to claim 6, wherein: The 16-frame continuous video frame sequence that has undergone preprocessing operations is input into the network model for model training.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the video action detection method according to any one of claims 1 to 8.

Images