A method and system for action recognition based on dual branches

By combining 2D and 3D action recognition methods and utilizing feature fusion and attention mechanisms, the problems of high complexity and low accuracy in existing action recognition technologies are solved, achieving efficient and accurate action recognition in complex scenarios.

CN119152569BActive Publication Date: 2026-11-10SOUTH CENTRAL UNIVERSITY FOR NATIONALITIES
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411173624.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2026-11-10
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

In existing technologies, the two-stage network architecture generates complex and time-consuming action pipelines containing bounding boxes across frames during action recognition. The classification performance depends on the proposal boxes and ignores the relationship between people and the background, resulting in insufficient action recognition accuracy and the ability to adapt to complex scenes.

Method used

A dual-branch action recognition method is adopted, which combines 2D and 3D branches. Through feature fusion and attention mechanisms, spatial and spatiotemporal features are extracted. The SCConv layer and CloAtt attention module are used to reduce feature redundancy and enhance feature representation capabilities. Full-dimensional dynamic convolution and non-local blocks are introduced to improve feature extraction capabilities.

Benefits of technology

It significantly improves the accuracy and efficiency of action recognition, enabling more precise action recognition in complex scenarios, reducing model parameters and computational complexity, and enhancing the ability to understand complex action scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119152569B_ABST
    Figure CN119152569B_ABST
Patent Text Reader

Abstract

This invention proposes a dual-branch action recognition method and system, belonging to the field of action recognition. The method includes: S1, acquiring an action video and preprocessing it to obtain a preprocessed image; S2, using 2D and 3D branches to extract features from the preprocessed image, obtaining spatial and spatiotemporal features; S3, using a channel fusion and attention mechanism module to fuse the spatial and spatiotemporal features, obtaining fused features; and S4, detecting the fused features to obtain the action category prediction result for each spatial location. This application effectively extracts spatial and spatiotemporal features by combining 2D and 3D branches, and significantly improves the accuracy and efficiency of action recognition through feature fusion and attention mechanisms. It can not only capture spatial information in static images but also effectively process the temporal dynamic features in video sequences, thereby achieving more accurate action recognition in various complex scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of action recognition technology, and in particular to a method and system for action recognition based on a dual-branch approach. Background Technology

[0002] Action recognition is an important research area in computer vision and pattern recognition, with wide applications in intelligent surveillance, sports analysis, virtual reality, and human-computer interaction. Traditional action recognition methods mainly rely on manually designed features, such as optical flow, motion history images, and spatiotemporal points of interest. With the development of deep learning, techniques such as convolutional neural networks (CNNs), 3D convolutional neural networks, and recurrent neural networks (RNNs) have made significant progress in action recognition tasks.

[0003] Inspired by Faster R-CNN, many advanced action recognition methods employ a two-stage network architecture: first generating action proposal boxes, and then classifying them. For example, the YOWO (You Only Watch Once) model uses a 2D CNN to extract spatial features of keyframes, while simultaneously using a 3D CNN to extract spatiotemporal features of the first few frames, achieving good action recognition results.

[0004] However, in existing technologies, generating action pipelines containing bounding boxes across frames when using a two-stage network architecture for action recognition is much more complex and time-consuming than in the two-dimensional case. Classification performance heavily depends on these proposal boxes, and the detected bounding boxes may not be optimal for subsequent classification tasks. Furthermore, action proposal boxes only focus on human features in the video, ignoring the relationship between people and certain attributes in the background, which can provide important contextual information for behavior prediction.

[0005] Therefore, finding a recognition method that can both improve the accuracy of action recognition and adapt to complex scenarios is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention proposes a dual-branch action recognition method and system, which can combine 2D branches and 3D branches to effectively extract spatial and spatiotemporal features, and significantly improve the accuracy and efficiency of action recognition through feature fusion and attention mechanisms.

[0007] The technical solution of this invention is implemented as follows:

[0008] In a first aspect, the present invention provides a dual-branch-based action recognition method, comprising the following steps:

[0009] S1. Acquire motion video and preprocess the motion video to obtain preprocessed images;

[0010] S2. Use 2D and 3D branches to extract features from the preprocessed image to obtain spatial and spatiotemporal features. The 2D branch includes a backbone network and an FPN. The backbone network includes a C2F-SCCloATT module. The 3D branch includes one first feature extraction layer, one first max pooling layer, and four second feature extraction layers. The second feature extraction layers include an OD_Bottleneck module.

[0011] S3. Use the channel fusion and attention mechanism module to fuse spatial features and spatiotemporal features to obtain fused features;

[0012] S4. Detect the fused features to obtain the action category prediction results for each spatial location.

[0013] Based on the above technical solutions, the preferred network structure of the 2D branches is as follows:

[0014] The backbone network consists of layers 1, 2, 3, 4, and 5. Layer 1 includes a first convolutional unit, a BN unit, and a SiLU; layer 2 includes a second convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module; layer 3 includes a third convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module; layer 4 includes a fourth convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module; and layer 5 includes a fifth convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module. The first convolutional unit has 64 channels, the second convolutional unit has 128 channels, the third convolutional unit has 256 channels, the fourth convolutional unit has 512 channels, and the fifth convolutional unit has 1024 channels.

[0015] The FPN consists of four parts: the first part includes an up-application layer, a Concat unit, and a first C2F module; the second part includes an up-application layer, a Concat unit, and a second C2F module; the third part includes a sixth convolutional unit, a Concat unit, and a third C2F module; and the fourth part includes a seventh convolutional unit, a Concat unit, a fourth C2F module, and an SPP_Block_CSP. The sixth convolutional unit has a 3x3 kernel and 256 channels; the seventh convolutional unit has a 3x3 kernel and 512 channels; the first C2F module has 512 channels; the second C2F module has 256 channels; the third C2F module has 512 channels; and the fourth C2F module has 1024 channels.

[0016] Based on the above technical solution, preferably, in step S2, a 2D branch is used to extract features from the preprocessed image, specifically including:

[0017] The backbone network is used to extract features from the preprocessed image to obtain the first multi-scale features;

[0018] The first multi-scale feature is fused and enhanced using FPN to obtain the second multi-scale feature.

[0019] The second multi-scale features are processed and transformed to obtain spatial features, which include Fcls and Freg.

[0020] Based on the above technical solutions, preferably, the C2F-SCCloATT module includes an SCConv layer and a CloAtt attention module, wherein the SCConv layer includes a spatial reconstruction unit and a channel reconstruction unit, and the CloAtt attention module includes high-frequency attention, low-frequency attention, and feature fusion.

[0021] Based on the above technical solutions, preferably, the C2F-SCCloATT module further includes an efficient attention mechanism, wherein the efficient attention mechanism has the following characteristics:

[0022] E(Q,K,V)=ρ q (Q)(ρ k (K) T V);

[0023] Where, ρ q ρ represents the normalization function for the query. k Let Q represent the normalization function for the key features, Q represent the query vector, K represent the key vector, and T represent the normalization function for ρ. k (K) matrix transpose, V represents the value vector.

[0024] Based on the above technical solutions, the preferred network structure of the 3D branch is as follows:

[0025] The first feature extraction layer includes a convolutional layer with a kernel size of 3×3×3 and 64 channels, a BN layer, a ReLU layer, and a 3×3×3 second max pooling layer;

[0026] The first max pooling layer is 3×1×1;

[0027] The four second feature extraction layers are layer1, layer2, layer3 and layer4. Layer1 includes three consecutive OD_Bottleneck modules, layer2 includes four consecutive OD_Bottleneck modules, layer3 includes five consecutive OD_Bottleneck modules and one OD_Bottleneck with added non-local blocks, and layer4 includes three consecutive OD_Bottleneck modules.

[0028] Based on the above technical solutions, preferably, the OD_Bottleneck module specifically includes:

[0029] The input is grouped into a set of 3x3 first convolutional branches and a set of 3x3 full-dimensional dynamic convolutional branches;

[0030] The fusion unit is configured to add the output of the first convolutional branch to the output of the full-dimensional dynamic convolutional branch;

[0031] The dimension restoration unit includes a 1*1 convolutional layer, which is used to restore the output of the fusion unit to the input feature dimension to obtain the first feature;

[0032] The residual connection unit is used to add the first feature to the input to obtain the residual feature.

[0033] Based on the above technical solutions, preferably, the four different attention mechanisms of the full-dimensional dynamic convolution include kernel attention, spatial dimension attention, input channel dimension attention, and output channel attention. The expression for the full-dimensional dynamic convolution is as follows:

[0034] y=(α ω1 ⊙α f1 ⊙α c1 ⊙α s1 W1+…+α ωn ⊙α fn ⊙α cn ⊙α sn ⊙W n )×x;

[0035] Where, α ω1 Let α represent the attention scalar of the first convolutional kernel W1. f1 The spatial dimension representing attention in spatial dimension, α c1 The input channel dimension of attention is represented by α. s1 The output channel dimension, α, represents the output channel attention. ωn Let W represent the attention scalar of the nth convolutional kernel. n Let x represent the nth convolutional kernel, x represent the input feature map, and α represent the input feature map. fnLet α represent the spatial dimension of the nth convolutional kernel and the spatial dimension of the attention. cn Let α represent the input channel dimension of the nth convolutional kernel and the input channel dimension of the attention. sn This represents the output channel dimension of the attention function of the nth convolutional kernel.

[0036] Furthermore, step S3 specifically includes:

[0037] S31. Input the spatiotemporal features and spatial features into the channel fusion module, and fuse the spatiotemporal features and spatial features according to the channel dimension of the channel fusion module to obtain the first feature map;

[0038] S32. Compress the number of channels using a 1*1 convolution on the first feature and extract the features using a 3*3 convolution to obtain the second feature map;

[0039] S33. Reshape the second feature into a two-dimensional feature map, including feature map F1 and feature map F2;

[0040] S34. Perform element-wise dot product on feature maps F1 and F2, and process them using the softmax function to obtain the attention matrix;

[0041] S35. Perform element-wise multiplication of the attention matrix with the feature map F1 to obtain the third feature map. Reshape the third feature map to the size of the second feature map to obtain the fused feature.

[0042] Secondly, the present invention provides a dual-branch-based action recognition system, employing the action recognition method described above, including:

[0043] The image processing module is configured to acquire motion video, preprocess the motion video, and obtain preprocessed images.

[0044] The feature extraction module is configured to use 2D and 3D branches to extract features from the preprocessed image to obtain spatial and spatiotemporal features. The 2D branch includes a backbone network and an FPN, and the backbone network includes a C2F-SCCloATT module. The 3D branch includes one first feature extraction layer, one first max pooling layer, and four second feature extraction layers, and the second feature extraction layers include an OD_Bottleneck module.

[0045] The feature fusion module is configured to use the channel fusion and attention mechanism module to fuse spatial features and spatiotemporal features to obtain fused features;

[0046] The action prediction module is configured to detect fused features and obtain action category prediction results for each spatial location.

[0047] The action recognition method of the present invention has the following advantages over the prior art:

[0048] (1) By combining 2D branches and 3D branches, spatial and spatiotemporal features are effectively extracted. Through feature fusion and attention mechanisms, the accuracy and efficiency of action recognition are significantly improved. It can not only capture spatial information in static images, but also effectively process the temporal dynamic features in video sequences, thereby achieving more accurate action recognition in various complex scenarios.

[0049] (2) By introducing the SCConv layer and the CloAtt attention module, the C2F-SCCloATT module effectively reduces feature redundancy and enhances feature representation capability. It not only reduces model parameters and computational complexity, but also focuses on local details and global information at the same time. Thus, while maintaining computational efficiency, it significantly improves the model's feature extraction and expression capabilities, enabling the model to better handle complex action scenarios.

[0050] (3) By introducing full-dimensional dynamic convolution and non-local blocks in the OD_Bottleneck module, the feature extraction capability of 3D branches is greatly enhanced. Full-dimensional dynamic convolution is used to achieve adaptive adjustment of features, while non-local blocks effectively capture long-distance dependencies, which can better adapt to different types of action features and improve the ability to understand complex action sequences.

[0051] (4) By using feature fusion and attention mechanism, spatial and spatiotemporal features are effectively combined, which can intelligently highlight important features and suppress irrelevant information, thereby generating more refined and information-rich fusion features, improving the ability to capture key information of actions, and effectively improving the accuracy and robustness of action recognition. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of the action recognition method of the present invention;

[0054] Figure 2 This is a flowchart of the action recognition method of the present invention;

[0055] Figure 3 This is a schematic diagram of the overall framework of the action recognition method of the present invention;

[0056] Figure 4 This is a schematic diagram of the 2D branch architecture of the action recognition method of the present invention;

[0057] Figure 5 This is a schematic diagram of the C2F-CloATT module of the action recognition method of the present invention;

[0058] Figure 6 This is a schematic diagram of the CloATT attention module in the action recognition method of the present invention;

[0059] Figure 7 This is a schematic diagram of the 3D branch architecture of the action recognition method of the present invention;

[0060] Figure 8 This is a schematic diagram of the OD_Bottleneck module in the action recognition method of the present invention;

[0061] Figure 9 This is a schematic diagram of the channel fusion and attention mechanism of the action recognition method of the present invention. Detailed Implementation

[0062] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0063] like Figure 1 and Figure 2 As shown, the present invention provides a dual-branch action recognition method, comprising the following steps:

[0064] S1. Acquire the motion video and preprocess it to obtain the preprocessed image.

[0065] Specifically, the preprocessing includes data augmentation of the motion video images and random cropping, resizing, random flipping, and image distortion of each RGB image, with each image cropped to a size of 224*224.

[0066] S2. Use 2D and 3D branches to extract features from the preprocessed image to obtain spatial and spatiotemporal features. The 2D branch includes a backbone network and an FPN. The backbone network includes a C2F-SCCloATT module. The 3D branch includes one first feature extraction layer, one first max pooling layer, and four second feature extraction layers. The second feature extraction layers include an OD_Bottleneck module.

[0067] In one embodiment of this application, the input parameters of the 2D branch are (1,3,224,224), which represent batch size, number of channels, length of image, and width of image, respectively.

[0068] In one embodiment of this application, the input parameters of the 3D branch are (1,3,16,224,224), which represent batch size, number of channels, number of image frames, length of image, and width of image, respectively.

[0069] S3. Use the channel fusion and attention mechanism module to fuse spatial features and spatiotemporal features to obtain fused features;

[0070] S4. Detect the fused features to obtain the action category prediction results for each spatial location.

[0071] This application effectively extracts spatial and spatiotemporal features by combining 2D and 3D branches, and significantly improves the accuracy and efficiency of action recognition through feature fusion and attention mechanisms. It can not only capture spatial information in static images, but also effectively process the temporal dynamic features in video sequences, thereby achieving more accurate action recognition in various complex scenarios.

[0072] like Figure 3 As shown, this application first uses Backone to recognize and extract features from the image, then uses FPN for feature processing, and finally uses Head to detect the processed feature map to obtain the action recognition result.

[0073] like Figure 4 As shown, in one embodiment of this application, the network structure of the 2D branch is as follows:

[0074] The backbone network consists of layers 1, 2, 3, 4, and 5. Layer 1 includes a first convolutional unit, a BN unit, and a SiLU; layer 2 includes a second convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module; layer 3 includes a third convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module; layer 4 includes a fourth convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module; and layer 5 includes a fifth convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module. The first convolutional unit has 64 channels, the second convolutional unit has 128 channels, the third convolutional unit has 256 channels, the fourth convolutional unit has 512 channels, and the fifth convolutional unit has 1024 channels.

[0075] The FPN consists of four parts: the first part includes an up-application layer, a Concat unit, and a first C2F module; the second part includes an up-application layer, a Concat unit, and a second C2F module; the third part includes a sixth convolutional unit, a Concat unit, and a third C2F module; and the fourth part includes a seventh convolutional unit, a Concat unit, a fourth C2F module, and an SPP_Block_CSP. The sixth convolutional unit has a 3x3 kernel and 256 channels; the seventh convolutional unit has a 3x3 kernel and 512 channels; the first C2F module has 512 channels; the second C2F module has 256 channels; the third C2F module has 512 channels; and the fourth C2F module has 1024 channels.

[0076] like Figure 5 and Figure 6 As shown, the C2F-SCCloATT module further includes an SCConv layer and a CloAtt attention module, wherein the SCConv layer includes a spatial reconstruction unit (SRU) and a channel reconstruction unit (CRU), and the CloAtt attention module includes high-frequency attention, low-frequency attention, and feature fusion.

[0077] As will be understood by those skilled in the art, the input is fed into the SCConv layer through the Conv+BN+SiLU layer, and then the convolutional layer is connected to the SCConv through skip connections, so that each SCConv can obtain additional information from the previous layer, directly passing the low-level features to the high-level layer and retaining more detailed information.

[0078] SRU separates redundant features based on weights and reconstructs them to suppress redundancy in the spatial dimension and enhance the representation of features. SRU includes separation and reconstruction operations.

[0079] Specifically, the separation operation will be illustrated with a concrete example, given an intermediate feature mapping X∈R N ×C×H×W:

[0080] The input feature X is standardized by subtracting the mean μ and dividing by the standard deviation σ, as shown in the following formula:

[0081]

[0082] Where μ represents the mean of the intermediate feature X, σ represents the standard deviation of the intermediate feature X, ε represents the positive constant, ε can increase the stability of the division calculation, and γ and β are trainable affine transformations.

[0083] Trainable parameters γ∈R in the GN layer C Used to measure the spatial pixel variance for each batch and channel. Richer spatial information reflects more variation in spatial pixels, resulting in a larger γ.

[0084] Normalized correlation weights W γ ∈R C The following formula represents the importance of different feature maps:

[0085]

[0086] Will pass W γ The reweighted feature map weights are mapped to the (0,1) range using the sigmoid function and then gated using a threshold. Weights above the threshold are set to 1 to obtain the information weight W1, and those above the threshold are set to 0 to obtain the non-information weight W2. The entire process of obtaining W can be represented by the following formula:

[0087] W = Gate(Sigmoid(W) γ (GN(X))))

[0088] Multiply the input feature X by W1 and W2 respectively to obtain two weighted features: one with higher information content. Features and features with less information This successfully splits the input features into two parts: Spatial content that is informative and expressive, and There is almost no information; it is considered redundant.

[0089] Reconstruction operations add information-rich features to less information-rich features to generate more information-rich features, thus saving space. Cross-reconstruction operations are used to fully combine two different informational features after weighting, strengthening the information flow between them. Then, the cross-reconstructed features... and By concatenating the features, we obtain the spatial fine-grained feature map X. W The process is as follows:

[0090]

[0091]

[0092] X W1 ∪X W2 =X W

[0093] in It is element-wise multiplication. ∪ represents element-wise summation, while ∪ represents concatenation. and express and Further processing results, and express and Further processing results, X W1 express and Cross-reconstruction features, X W2 express and Cross-reconstruction features, X W Represents a fine-grained feature mapping in the representation space.

[0094] Applying SRU to the intermediate input feature X not only separates informative features from less informative features but also reconstructs them, enhancing representative features and suppressing redundant features in the spatial dimension. However, the spatially refined feature mapping X... W It remains redundant in the channel dimension.

[0095] Understandably, CRU employs split transformation and fusion strategies to reduce redundancy in the channel dimension, as well as computational and storage costs.

[0096] The segmentation operation divides the input spatial refinement features into two parts: one part has αC channels, and the other part has (1-α)C channels. Then, a 1*1 convolution kernel is used to compress the channels of the two sets of features, yielding X. up and X low ;

[0097] The conversion operation will convert the input X up As input to "rich feature extraction", GWC and PWC are performed separately, and then the results are added together to obtain the output Y1. The input is used as a supplement to "rich feature extraction", PWC is performed, and the result is combined with the original input to obtain Y2.

[0098] The fusion operation uses a simplified SKNet approach to adaptively merge Y1 and Y2.

[0099] Specifically, the fusion operation will be described using a specific embodiment:

[0100] Global average pooling is used to combine global spatial information and channel statistics to obtain pooled S1 and S2.

[0101] Softmax is applied to S1 and S2 to obtain the feature weight vectors β1 and β2;

[0102] The output Y = β1Y1 + β2Y2 is obtained by using the feature weight vector, where Y is the feature extracted from the channel.

[0103] In one embodiment of this application, the result of the SCConv layer is input into the CloAtt attention module. The CloAtt attention module extracts fine-grained and global information from the input feature map through high-frequency and low-frequency attention mechanisms. Specifically, the high-frequency attention mechanism uses convolution operations to focus on local features, while the low-frequency attention mechanism uses pooling operations to focus on global features.

[0104] Feature fusion is achieved by concatenating features from different branches along the channel dimension. The concatenated features contain information from different branches, which helps increase the network's nonlinearity and representation capabilities, thereby improving the network's ability to model complex data. It enhances the model's feature extraction capabilities while maintaining computational efficiency. SCConv effectively limits feature redundancy, reducing not only the number of model parameters and FLOPs but also enhancing feature representation capabilities. It improves model performance while reducing redundant features.

[0105] This application introduces the SCConv layer and the CloAtt attention module, and utilizes the C2F-SCCloATT module to effectively reduce feature redundancy while enhancing feature representation capabilities. This not only reduces model parameters and computational complexity, but also allows the model to focus on both local details and global information. Thus, while maintaining computational efficiency, it significantly improves the model's feature extraction and expressive capabilities, enabling the model to better handle complex action scenarios.

[0106] In one embodiment of this application, in step S2, feature extraction is performed on the preprocessed image using a 2D branch, specifically including:

[0107] The backbone network is used to extract features from the preprocessed image to obtain the first multi-scale features;

[0108] The first multi-scale feature is fused and enhanced using FPN to obtain the second multi-scale feature.

[0109] The second multi-scale features are processed and transformed to obtain spatial features, which include Fcls and Freg.

[0110] In one embodiment of this application, the C2F-SCCloATT module further includes an efficient attention mechanism, wherein the efficient attention mechanism is characterized as follows:

[0111] E(Q, K, V) = ρ q (Q)(ρ k (K) T V)

[0112] Where, ρ q ρ represents the normalization function for the query. kLet Q represent the normalization function for the key features, Q represent the query vector, K represent the key vector, and T represent the normalization function for ρ. k (K) matrix transpose, V represents the value vector.

[0113] Specifically, for ρ q and ρ k Perform scaling operations and process them using the softmax function:

[0114] Scaling:

[0115] Softmax: ρ q (Y)=σ row (Y)

[0116] ρ k (Y)=σ col (Y)

[0117] Where Scaling represents the scaling operation, σ row and σ col This indicates that the softmax function is applied along each row or column of matrix Y.

[0118] For the input feature map x∈R h×w×d The module flattens it into a matrix x∈R hw×d We apply an efficient attention mechanism to it and reshape the result into h×w×d v If d v If the dimension is not equal to d, then a 1x1 convolution is applied to restore the dimension to d. Finally, the obtained features are added to the input features to form the residual structure.

[0119] like Figure 7 As shown, in one embodiment of this application, the network structure of the 3D branch is as follows:

[0120] The first feature extraction layer includes a convolutional layer with a kernel size of 3×3×3 and 64 channels, a BN layer, a ReLU layer, and a 3×3×3 second max pooling layer;

[0121] The first max pooling layer is 3×1×1;

[0122] The four second feature extraction layers are layer1, layer2, layer3 and layer4. Layer1 includes three consecutive OD_Bottleneck modules, layer2 includes four consecutive OD_Bottleneck modules, layer3 includes five consecutive OD_Bottleneck modules and one OD_Bottleneck with added non-local blocks, and layer4 includes three consecutive OD_Bottleneck modules.

[0123] like Figure 8 As shown, the OD_Bottleneck module further includes:

[0124] The input is grouped into a set of 3x3 first convolutional branches and a set of 3x3 full-dimensional dynamic convolutional branches;

[0125] The fusion unit is configured to add the output of the first convolutional branch to the output of the full-dimensional dynamic convolutional branch;

[0126] The dimension restoration unit includes a 1*1 convolutional layer, which is used to restore the output of the fusion unit to the input feature dimension to obtain the first feature;

[0127] The residual connection unit is used to add the first feature to the input to obtain the residual feature.

[0128] This application significantly enhances the feature extraction capability of 3D branches by introducing full-dimensional dynamic convolution and non-local blocks in the OD_Bottleneck module. Full-dimensional dynamic convolution enables adaptive adjustment of features, while non-local blocks effectively capture long-distance dependencies, better adapting to different types of action features and improving the understanding of complex action sequences.

[0129] Specifically, a non-local block is a module that embeds non-local operations into a neural network, that is:

[0130] z i =W z y i +x i ;

[0131] Among them, z i W represents the output feature of a nonlocal block. z Let y represent the learnable weight matrix. i x represents the result of a nonlocal operation. i This indicates a residual connection.

[0132] Understandably, a non-local block receives feature map a, performs a non-local operation on the feature map, and the weight layer W... z The output of the Non-local operation is weighted, and the feature map a is added to the weighted Non-local output and then residually connected to obtain the output of the non-local block.

[0133] Specifically, the formula for non-local operations is as follows:

[0134]

[0135] Where y represents the output of the non-local operation, which has the same shape as the input feature map, and x represents the input of the non-local operation, which has the same shape as the output y. i Similarly, 'i' represents the coordinates (spatial, temporal, or spatiotemporal coordinates) of a point on the output 'y' or input 'x', and the response value at position 'i' (the output of a non-local operation) is 'y'. i j sequentially enumerates (traverses) the coordinates of all points on x or y, x j This represents the value at position j in the input data, and the function f(x) i x j ) Calculate the relationship between two quantities, position i and position j, in the input data using the function g(x). j ) Calculate a representation of the j-th position of the input data, which can be understood here as f(x) i x j A coefficient or weight of ), the final response result of the non-local position i is the relationship f(x) of each position j relative to the current position i. i x j ) weighted f(x) i x j )g(x j Summing, and finally using the normalization factor C (X) Perform a normalization operation.

[0136] Furthermore, the four different attention mechanisms of the full-dimensional dynamic convolution include kernel attention, spatial dimension attention, input channel dimension attention, and output channel attention. The expression for the full-dimensional dynamic convolution is as follows:

[0137] y=(α ω1 ⊙α f1 ⊙α c1 ⊙α s1 W1+…+α ωn ⊙α fn ⊙α cn ⊙α sn ⊙W n )×x;

[0138] Where, α ω1 Let α represent the attention scalar of the first convolutional kernel W1. f1 The spatial dimension representing attention in spatial dimension, α c1 The input channel dimension of attention is represented by α. s1 The output channel dimension, α, represents the output channel attention. ωn Let W represent the attention scalar of the nth convolutional kernel. n Let x represent the nth convolutional kernel, x represent the input feature map, and α represent the input feature map. fnLet α represent the spatial dimension of the nth convolutional kernel and the spatial dimension of the attention. cn Let α represent the input channel dimension of the nth convolutional kernel and the input channel dimension of the attention. sn This represents the output channel dimension of the attention function of the nth convolutional kernel.

[0139] like Figure 9 As shown, in one embodiment of this application, step S3 specifically includes:

[0140] S31. Input the spatiotemporal features and spatial features into the channel fusion module, and fuse the spatiotemporal features and spatial features according to the channel dimension of the channel fusion module to obtain the first feature map; wherein the first feature map includes motion information and appearance information;

[0141] S32. Compress the number of channels using a 1*1 convolution on the first feature and extract the features using a 3*3 convolution to obtain the second feature map;

[0142] S33. Reshape the second feature into a two-dimensional feature map, including feature map F1 and feature map F2;

[0143] S34. Perform element-wise dot product on feature maps F1 and F2, and process them using the softmax function to obtain the attention matrix;

[0144] S35. Perform element-wise multiplication of the attention matrix with the feature map F1 to obtain the third feature map. Reshape the third feature map to the size of the second feature map to obtain the fused feature.

[0145] By combining spatial and spatiotemporal features through feature fusion and attention mechanisms, the system can intelligently highlight important features and suppress irrelevant information, thereby generating more refined and information-rich fused features. This improves the ability to capture key information about actions and effectively enhances the accuracy and robustness of action recognition.

[0146] This application provides a dual-branch-based action recognition system, employing the action recognition method described above, including:

[0147] The image processing module is configured to acquire motion video, preprocess the motion video, and obtain preprocessed images.

[0148] The feature extraction module is configured to use 2D and 3D branches to extract features from the preprocessed image to obtain spatial and spatiotemporal features. The 2D branch includes a backbone network and an FPN, and the backbone network includes a C2F-SCCloATT module. The 3D branch includes one first feature extraction layer, one first max pooling layer, and four second feature extraction layers, and the second feature extraction layers include an OD_Bottleneck module.

[0149] The feature fusion module is configured to use the channel fusion and attention mechanism module to fuse spatial features and spatiotemporal features to obtain fused features;

[0150] The action prediction module is configured to detect fused features and obtain action category prediction results for each spatial location.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dual-branch action recognition method, characterized in that, Includes the following steps: S1. Acquire motion video and preprocess the motion video to obtain preprocessed images; S2. Use 2D and 3D branches to extract features from the preprocessed image to obtain spatial and spatiotemporal features. The 2D branch includes a backbone network and an FPN. The backbone network includes a C2F-SCCloATT module. The 3D branch includes one first feature extraction layer, one first max pooling layer, and four second feature extraction layers. The second feature extraction layers include an OD_Bottleneck module. S3. Use the channel fusion and attention mechanism module to fuse spatial features and spatiotemporal features to obtain fused features; S4. Detect the fused features to obtain the action category prediction results for each spatial location; The network structure of the 2D branch is as follows: The backbone network consists of layers 1, 2, 3, 4, and 5. Layer 1 includes a first convolutional unit, a BN unit, and a SiLU; layer 2 includes a second convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module; layer 3 includes a third convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module; layer 4 includes a fourth convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module; and layer 5 includes a fifth convolutional unit, a BN unit, a SiLU, and a C2F-SCCloATT module. The first convolutional unit has 64 channels, the second convolutional unit has 128 channels, the third convolutional unit has 256 channels, the fourth convolutional unit has 512 channels, and the fifth convolutional unit has 1024 channels. The FPN consists of four parts: the first part includes an upsampling layer, a Concat unit, and a first C2F module; the second part includes an upsampling layer, a Concat unit, and a second C2F module; the third part includes a sixth convolutional unit, a Concat unit, and a third C2F module; and the fourth part includes a seventh convolutional unit, a Concat unit, a fourth C2F module, and an SPP_Block_CSP. The sixth convolutional unit has a 3x3 kernel and 256 channels; the seventh convolutional unit has a 3x3 kernel and 512 channels; the first C2F module has 512 channels; the second C2F module has 256 channels; the third C2F module has 512 channels; and the fourth C2F module has 1024 channels. The network structure of the 3D branch is as follows: The first feature extraction layer includes a convolutional layer with a kernel size of 3×3×3 and 64 channels, a BN layer, a ReLU layer, and a 3×3×3 second max pooling layer; The first max pooling layer is 3×1×1; The four second feature extraction layers are layer1, layer2, layer3 and layer4. Layer1 includes three consecutive OD_Bottleneck modules, layer2 includes four consecutive OD_Bottleneck modules, layer3 includes five consecutive OD_Bottleneck modules and one OD_Bottleneck with added non-local blocks, and layer4 includes three consecutive OD_Bottleneck modules.

2. The action recognition method based on a dual-branch approach as described in claim 1, characterized in that, In step S2, a 2D branch is used to extract features from the preprocessed image, specifically including: The backbone network is used to extract features from the preprocessed image to obtain the first multi-scale features; The first multi-scale feature is fused and enhanced using FPN to obtain the second multi-scale feature. The second multi-scale features are processed and transformed to obtain spatial features, which include Fcls and Freg.

3. The action recognition method based on a dual-branch approach as described in claim 1, characterized in that, The C2F-SCCloATT module includes an SCConv layer and a CloAtt attention module. The SCConv layer includes a spatial reconstruction unit and a channel reconstruction unit, and the CloAtt attention module includes high-frequency attention, low-frequency attention, and feature fusion.

4. The action recognition method based on a dual-branch approach as described in claim 3, characterized in that, The C2F-SCCloATT module also includes an efficient attention mechanism, which is characterized by the following: ; in, This represents the normalization function for the query. The normalization function representing the key characteristics. Represents the query vector. Represents the key vector. Indicates to Matrix transpose Represents a value vector.

5. The action recognition method based on a dual-branch approach as described in claim 1, characterized in that, The OD_Bottleneck module specifically includes: The input is grouped into a set of 3x3 first convolutional branches and a set of 3x3 full-dimensional dynamic convolutional branches; The fusion unit is configured to add the output of the first convolutional branch to the output of the full-dimensional dynamic convolutional branch; The dimension restoration unit includes a 1*1 convolutional layer, which is used to restore the output of the fusion unit to the input feature dimension to obtain the first feature; The residual connection unit is used to add the first feature to the input to obtain the residual feature.

6. The action recognition method based on a dual-branch approach as described in claim 5, characterized in that, The full-dimensional dynamic convolution employs four different attention mechanisms: kernel attention, spatial dimension attention, input channel dimension attention, and output channel attention. The expression for full-dimensional dynamic convolution is as follows: ; in, Indicates the first convolution kernel Attention scalar The spatial dimension representing attention in spatial dimensions. The input channel dimension represents the input channel dimension of attention. The output channel dimension represents the output channel attention. This represents the attention scalar of the nth convolutional kernel. This represents the nth convolutional kernel. Indicates the input feature map, The spatial dimension of the attention domain represents the spatial dimension of the nth convolutional kernel. The input channel dimension of the nth convolutional kernel and the input channel dimension of the attention are represented. This represents the output channel dimension of the attention function of the nth convolutional kernel.

7. The action recognition method based on a dual-branch approach as described in claim 1, characterized in that, Step S3 specifically includes: S31. Input the spatiotemporal features and spatial features into the channel fusion module, and fuse the spatiotemporal features and spatial features according to the channel dimension of the channel fusion module to obtain the first feature map; S32. Compress the number of channels using a 1*1 convolution on the first feature and extract the features using a 3*3 convolution to obtain the second feature map; S33. Reshape the second feature into a two-dimensional feature map, including feature map F1 and feature map F2; S34. Perform element-wise dot product on feature maps F1 and F2, and process them using the softmax function to obtain the attention matrix; S35. Perform element-wise multiplication of the attention matrix with the feature map F1 to obtain the third feature map. Reshape the third feature map to the size of the second feature map to obtain the fused feature.

8. A dual-branch-based action recognition system, characterized in that, The action recognition method as described in any one of claims 1-7 includes: The image processing module is configured to acquire motion video, preprocess the motion video, and obtain preprocessed images. The feature extraction module is configured to use 2D and 3D branches to extract features from the preprocessed image to obtain spatial and spatiotemporal features. The 2D branch includes a backbone network and an FPN, and the backbone network includes a C2F-SCCloATT module. The 3D branch includes one first feature extraction layer, one first max pooling layer, and four second feature extraction layers, and the second feature extraction layers include an OD_Bottleneck module. The feature fusion module is configured to use the channel fusion and attention mechanism modules to fuse spatial and spatiotemporal features to obtain fused features. The action prediction module is configured to detect fused features and obtain action category prediction results for each spatial location.

Citation Information

Patent Citations

  • Action recognition method based on double-flow convolution attention

    CN112926396A

  • Motion detection method based on 2DCNN and 3DCNN

    CN117409481A