Action recognition method and system based on double branches

Abstract

The invention provides an action recognition method and system based on double branches, and belongs to the field of action recognition, and the method comprises the steps: S1, obtaining an action video, and carrying out the preprocessing of the action video, and obtaining a preprocessed picture; s2, using a 2D branch and a 3D branch to perform feature extraction on the preprocessed picture to obtain spatial features and spatial-temporal features; s3, performing feature fusion on the spatial features and the spatial-temporal features by using a channel fusion and attention mechanism module to obtain fused features; and S4, detecting the fusion features to obtain an action category prediction result of each spatial position. According to the method, the spatial features and the spatial-temporal features are effectively extracted by combining the 2D branch and the 3D branch, the accuracy and efficiency of action recognition are remarkably improved through feature fusion and an attention mechanism, spatial information in a static image can be captured, the temporal dynamic features in a video sequence can be effectively processed, and the accuracy and efficiency of action recognition are improved. Therefore, more accurate action recognition can be realized in various complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of action recognition, and in particular to a dual-branch based action recognition method and system. Background Art

[0002] Action recognition is an important research direction in the field of computer vision and pattern recognition, and is widely used in intelligent monitoring, sports analysis, virtual reality, and human-computer interaction. Traditional action recognition methods mainly rely on manually designed features such as optical flow, motion history images, and spatiotemporal points of interest. With the development of deep learning, technologies such as convolutional neural networks (CNNs), three-dimensional convolutional neural networks, and recurrent neural networks (RNNs) have made significant progress in action recognition tasks.

[0003] Inspired by Faster R-CNN, many advanced action recognition methods use a two-stage network architecture to generate action proposals first and then classify them. For example, the YOWO (You Only Watch Once) model uses 2D CNN to extract spatial features of key frames and 3D CNN to extract spatiotemporal features of the first few frames, achieving better action recognition results.

[0004] However, in the existing technology, generating an action pipeline containing bounding boxes across frames when using a two-stage network architecture for action recognition is much more complicated and time-consuming than the two-dimensional case. The classification performance depends greatly on these proposal boxes, and the detected bounding boxes may not be optimal for subsequent classification tasks. Moreover, the action proposal boxes only focus on the features of humans in the video, ignoring the relationship between certain attributes in the background, which can provide important contextual information for behavior prediction.

[0005] Therefore, finding a recognition method that can both improve the accuracy of action recognition and adapt to complex scenarios is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0006] In view of this, the present invention proposes a dual-branch based action recognition method and system, which can combine 2D branch and 3D branch, effectively extract spatial features and spatiotemporal features, and significantly improve the accuracy and efficiency of action recognition through feature fusion and attention mechanism.

[0007] The technical solution of the present invention is achieved in this way:

[0008] In a first aspect, the present invention provides a dual-branch action recognition method, comprising the following steps:

[0009] S1, obtaining an action video, and preprocessing the action video to obtain a preprocessed image;

[0010] S2. Use the 2D branch and the 3D branch to extract features from the preprocessed images respectively to obtain spatial features and spatiotemporal features; the 2D branch includes a backbone network and FPN, the backbone network includes a C2F-SCCloATT module, the 3D branch includes a first feature extraction layer, a first maximum pooling layer and four second feature extraction layers, and the second feature extraction layer includes an OD_Bottleneck module;

[0011] S3, use channel fusion and attention mechanism modules to fuse spatial features and spatiotemporal features to obtain fused features;

[0012] S4. Detect the fused features to obtain the action category prediction results for each spatial position.

[0013] On the basis of the above technical solution, preferably, the network structure of the 2D branch is:

[0014] The backbone network includes layer_1, layer_2, layer_3, layer_4 and layer_5, layer_1 includes the first convolution unit, BN unit and SiLU, layer_2 includes the second convolution unit, BN unit, SiLU and C2F-SCCloATT module, layer_3 includes the third convolution unit, BN unit, SiLU and C2F-SCCloATT module, layer_4 includes the fourth convolution unit, BN unit, SiLU and C2F-SCCloATT module, layer_5 includes the fifth convolution unit, BN unit, SiLU and C2F-SCCloATT module; wherein the number of channels of the first convolution unit is 64, the number of channels of the second convolution unit is 128, the number of channels of the third convolution unit is 256, the number of channels of the fourth convolution unit is 512, and the number of channels of the fifth convolution unit is 1024;

[0015] FPN consists of four parts. The first part includes the upper adopting layer, the Concat unit and the first C2F module. The second part includes the upper adopting layer, the Concat unit and the second C2F module. The third part includes the sixth convolution unit, the Concat unit and the third C2F module. The fourth part includes the seventh convolution unit, the Concat unit, the fourth C2F module and SPP_Block_CSP; the convolution kernel of the sixth convolution unit is 3*3 and the number of channels is 256. The convolution kernel of the seventh convolution unit is 3*3 and the number of channels is 512. The number of channels of the first C2F module is 512, the number of channels of the second C2F module is 256, the number of channels of the third C2F module is 512, and the number of channels of the fourth C2F module is 1024.

[0016] Based on the above technical solution, preferably, in step S2, the 2D branch is used to extract features from the preprocessed image, specifically including:

[0017] Use the backbone network to extract features from the preprocessed image to obtain the first multi-scale features;

[0018] Use FPN to fuse and enhance the first multi-scale features to obtain second multi-scale features;

[0019] The second multi-scale features are processed and transformed to obtain spatial features, where the spatial features include Fcls and Freg.

[0020] Based on the above technical solution, preferably, the C2F-SCCloATT module includes an SCConv layer and a CloAtt attention module, wherein the SCConv layer includes a spatial reconstruction unit and a channel reconstruction unit, and the CloAtt attention module includes high-frequency attention, low-frequency attention and feature fusion.

[0021] On the basis of the above technical solution, preferably, the C2F-SCCloATT module also includes an efficient attention mechanism, wherein the characteristics of the efficient attention mechanism are as follows:

[0022] E(Q,K,V)=ρ q (Q)(ρ k (K) T V);

[0023] Among them, ρ q represents the normalization function of the query, ρ k represents the normalization function of the key feature, Q represents the query vector, K represents the key vector, and T represents the normalization function of ρ k (K) matrix transpose, V represents the value vector.

[0024] On the basis of the above technical solution, preferably, the network structure of the 3D branch is:

[0025] The first feature extraction layer includes a convolution layer with a convolution kernel of 3×3×3 and 64 channels, a BN layer, a ReLU and a 3×3×3 second maximum pooling layer;

[0026] The first maximum pooling layer is 3×1×1;

[0027] The four second feature extraction layers include layer1, layer2, layer3 and layer4, wherein layer1 includes three consecutive OD_Bottleneck modules, layer2 includes four consecutive OD_Bottleneck modules, layer3 includes five consecutive OD_Bottleneck modules and one OD_Bottleneck with a non-local block added, and layer4 includes three consecutive OD_Bottleneck modules.

[0028] On the basis of the above technical solution, preferably, the OD_Bottleneck module specifically includes:

[0029] Input grouping, including a group of 3*3 first convolution branches and a group of 3*3 full-dimensional dynamic convolution branches;

[0030] a fusion unit configured to add an output of the first convolution branch and an output of the full-dimensional dynamic convolution branch;

[0031] The dimension recovery unit includes a 1*1 convolution layer, which is used to restore the output of the fusion unit to the input feature dimension to obtain the first feature;

[0032] The residual connection unit is used to add the first feature to the input to obtain a residual feature.

[0033] On the basis of the above technical solution, preferably, the full-dimensional dynamic convolution has four different attentions, including convolution kernel attention, spatial dimension attention, input channel dimension attention and output channel attention. The expression of the full-dimensional dynamic convolution is as follows:

[0034] y=(α ω1 ⊙α f1 ⊙α c1 ⊙α s1 W 1 +…+α ωn ⊙α fn ⊙α cn ⊙α sn ⊙W n )×x;

[0035] Among them, α ω1 Represents the first convolution kernel W 1 The attention scalar, α f1 represents the spatial dimension of attention in the spatial dimension, α c1 represents the input channel dimension of the attention, α s1 The output channel dimension, α, represents the output channel attention ωn represents the attention scalar of the nth convolution kernel, W nrepresents the nth convolution kernel, x represents the input feature map, α fn Represents the spatial dimension of the attention of the nth convolution kernel, α cn represents the input channel dimension of the nth convolution kernel, α represents the input channel dimension of the attention, sn The output channel dimension representing the output channel attention of the nth convolution kernel.

[0036] More preferably, step S3 specifically includes:

[0037] S31, inputting the spatiotemporal features and the spatial features into a channel fusion module, and fusing the spatiotemporal features and the spatial features according to the channel dimension of the channel fusion module to obtain a first feature map;

[0038] S32, using 1*1 convolution to compress the number of channels of the first feature, and using 3*3 convolution to extract features to obtain a second feature map;

[0039] S33, reshaping the second feature into feature maps of two dimensions, including a feature map F1 and a feature map F2;

[0040] S34, perform element-by-element dot multiplication on the feature map F1 and the feature map F2, and process them through the softmax function to obtain an attention matrix;

[0041] S35. Perform element-by-element dot multiplication of the attention matrix and the feature map F1 to obtain a third feature map, and reshape the third feature map to the size of the second feature map to obtain a fused feature.

[0042] In a second aspect, the present invention provides a dual-branch based action recognition system, which adopts the action recognition method as described above, and comprises:

[0043] An image processing module is configured to obtain an action video and preprocess the action video to obtain a preprocessed image;

[0044] A feature extraction module is configured to use a 2D branch and a 3D branch to extract features from the preprocessed image respectively to obtain spatial features and spatiotemporal features; wherein the 2D branch includes a backbone network and an FPN, the backbone network includes a C2F-SCCloATT module, the 3D branch includes a first feature extraction layer, a first maximum pooling layer and four second feature extraction layers, and the second feature extraction layer includes an OD_Bottleneck module;

[0045] A feature fusion module, which is configured to use a channel fusion and attention mechanism module to fuse spatial features and spatiotemporal features to obtain fused features;

[0046] The action prediction module is configured to detect the fused features and obtain the action category prediction result of each spatial position.

[0047] Compared with the prior art, the motion recognition method of the present invention has the following beneficial effects:

[0048] (1) By combining the 2D branch and the 3D branch, the spatial features and spatiotemporal features are effectively extracted. Through feature fusion and attention mechanism, the accuracy and efficiency of action recognition are significantly improved. It can not only capture the spatial information in static images, but also effectively process the temporal dynamic features in video sequences, thereby achieving more accurate action recognition in various complex scenarios.

[0049] (2) By introducing the SCConv layer and the CloAtt attention module, the C2F-SCCloATT module is used to effectively reduce feature redundancy and enhance feature representation capabilities. This not only reduces model parameters and computational complexity, but also allows attention to both local details and global information. This significantly improves the model’s feature extraction and expression capabilities while maintaining computational efficiency, enabling the model to better handle complex action scenes.

[0050] (3) By introducing full-dimensional dynamic convolution and non-local blocks in the OD_Bottleneck module, the feature extraction capability of the 3D branch is greatly enhanced. The full-dimensional dynamic convolution is used to achieve adaptive adjustment of features, and the non-local block effectively captures long-distance dependencies, which can better adapt to different types of action features and improve the ability to understand complex action sequences.

[0051] (4) Through feature fusion and attention mechanism, the effective combination of spatial features and spatiotemporal features is achieved, which can intelligently highlight important features and suppress irrelevant information, thereby generating more refined and information-rich fusion features, improving the ability to capture key information of actions, and effectively improving the accuracy and robustness of action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a flow chart of the action recognition method of the present invention;

[0054] Figure 2 is a flowchart of the action recognition method of the present invention;

[0055] Figure 3 Schematic diagram of the overall framework of the action recognition method of the present invention;

[0056] Figure 4 Schematic diagram of a 2D branch architecture of the action recognition method of the present invention;

[0057] Figure 5 Schematic diagram of the C2F-CloATT module of the action recognition method of the present invention;

[0058] Figure 6 Schematic diagram of CloATT attention module of the action recognition method of the present invention;

[0059] Figure 7 Schematic diagram of a 3D branch architecture of the action recognition method of the present invention;

[0060] Figure 8 It is a schematic diagram of the OD_Bottleneck module of the action recognition method of the present invention;

[0061] Fig. 9 Schematic diagram of channel fusion and attention mechanism of the action recognition method of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] like Figure 1 and Figure 2 As shown, the present invention provides an action recognition method based on dual branches, comprising the following steps:

[0064] S1. Obtain an action video, and preprocess the action video to obtain a preprocessed image.

[0065] Specifically, the preprocessing includes data enhancement of action video images, and random cropping, resizing, random flipping, and image distortion of each RGB image. Each image is cropped to a size of 224*224.

[0066] S2. Use the 2D branch and the 3D branch to extract features from the preprocessed images respectively to obtain spatial features and spatiotemporal features; the 2D branch includes a backbone network and FPN, the backbone network includes a C2F-SCCloATT module, the 3D branch includes a first feature extraction layer, a first maximum pooling layer and four second feature extraction layers, and the second feature extraction layer includes an OD_Bottleneck module;

[0067] In one embodiment of the present application, the input parameters of the 2D branch are (1, 3, 224, 224), which respectively represent batch size, number of channels, length of the image, and width of the image.

[0068] In one embodiment of the present application, the input parameters of the 3D branch are (1, 3, 16, 224, 224), which respectively represent batch size, number of channels, number of picture frames, length of the picture, and width of the picture.

[0069] S3, use channel fusion and attention mechanism modules to fuse spatial features and spatiotemporal features to obtain fused features;

[0070] S4. Detect the fused features to obtain the action category prediction results for each spatial position.

[0071] This application effectively extracts spatial features and spatiotemporal features by combining 2D branches and 3D branches, and significantly improves the accuracy and efficiency of action recognition through feature fusion and attention mechanisms. It can not only capture spatial information in static images, but also effectively process temporal dynamic features in video sequences, thereby achieving more accurate action recognition in various complex scenarios.

[0072] like Figure 3 As shown, this application first tries to use Backone to recognize the image and extract features, then uses FPN for feature processing, and uses Head to detect the processed feature map to obtain the result of action recognition.

[0073] like Figure 4 As shown, in one embodiment of the present application, the network structure of the 2D branch is:

[0074] The backbone network includes layer_1, layer_2, layer_3, layer_4 and layer_5, layer_1 includes the first convolution unit, BN unit and SiLU, layer_2 includes the second convolution unit, BN unit, SiLU and C2F-SCCloATT module, layer_3 includes the third convolution unit, BN unit, SiLU and C2F-SCCloATT module, layer_4 includes the fourth convolution unit, BN unit, SiLU and C2F-SCCloATT module, layer_5 includes the fifth convolution unit, BN unit, SiLU and C2F-SCCloATT module; wherein the number of channels of the first convolution unit is 64, the number of channels of the second convolution unit is 128, the number of channels of the third convolution unit is 256, the number of channels of the fourth convolution unit is 512, and the number of channels of the fifth convolution unit is 1024;

[0075] FPN consists of four parts. The first part includes the upper adopting layer, the Concat unit and the first C2F module. The second part includes the upper adopting layer, the Concat unit and the second C2F module. The third part includes the sixth convolution unit, the Concat unit and the third C2F module. The fourth part includes the seventh convolution unit, the Concat unit, the fourth C2F module and SPP_Block_CSP; the convolution kernel of the sixth convolution unit is 3*3 and the number of channels is 256. The convolution kernel of the seventh convolution unit is 3*3 and the number of channels is 512. The number of channels of the first C2F module is 512, the number of channels of the second C2F module is 256, the number of channels of the third C2F module is 512, and the number of channels of the fourth C2F module is 1024.

[0076] like Figure 5 and Figure 6 As shown, further, the C2F-SCCloATT module includes an SCConv layer and a CloAtt attention module, wherein the SCConv layer includes a spatial reconstruction unit (SRU) and a channel reconstruction unit (CRU), and the CloAtt attention module includes high-frequency attention, low-frequency attention and feature fusion.

[0077] As those skilled in the art can understand, the input is input into the SCConv layer via the Conv+BN+SiLU layer, and then the convolution layer is connected to the SCConv through a skip connection, so that each SCConv can obtain additional information from the previous layer, pass the low-level features directly to the high-level, and retain more detailed information.

[0078] SRU separates redundant features according to weights and reconstructs them to suppress redundancy in spatial dimensions and enhance feature representation, where SRU includes separation and reconstruction operations.

[0079] Specifically, the separation operation is described with a specific embodiment. Given an intermediate feature map X∈R N ×C×H×W:

[0080] The input feature X is normalized by subtracting the mean μ and dividing by the standard deviation σ, as follows:

[0081]

[0082] Among them, μ represents the mean of the intermediate feature X, σ represents the standard deviation of the intermediate feature X, ε represents a positive constant, ε can increase the stability of the division calculation, γ and β are trainable affine transformations;

[0083] Trainable parameters γ∈R in the GN layer C Used to measure the spatial pixel variance of each batch and channel. Richer spatial information reflects more variations in spatial pixels, resulting in larger γ.

[0084] Normalized correlation weight W γ ∈R C It is obtained from the following formula, which indicates the importance of different feature maps:

[0085]

[0086] Will be W γ The weights of the re-weighted feature map are mapped to the (0, 1) range through the sigmoid function and gated by the threshold. The weights above the threshold are set to 1, and the information weight W is obtained. 1 , set it to 0, and get the non-information weight W 2 ,The whole process of obtaining W can be expressed by the following formula:

[0087] W=Gate(Sigmoid(W γ (GN(X))))

[0088] Multiply the input feature X by W 1 and W 2 , we get two weighted features: Features and less informative features This successfully divides the input features into two parts: Informative and expressive spatial content, while There is little information and it is considered redundant.

[0089] The reconstruction operation is to add the information-rich features to the less informative features to generate more informative features, thereby saving space. The cross reconstruction operation is used to fully combine the two weighted different information features and strengthen the information flow between them. Then the cross-reconstructed features and Splice to get the spatial fine feature map X W The process is shown as follows:

[0090]

[0091]

[0092] X W1 ∪X W2 =X W

[0093] in is an element-wise multiplication, is the element-by-element summation and ∪ is the concatenation. and express and The further processing results, and express and The further processing result, X W1 express and The cross-reconstruction features, X W2 express and The cross-reconstruction features, X W Represents a spatially fine feature map.

[0094] After applying SRU to the intermediate input features X, not only the informative features are separated from the less informative features, but also they are reconstructed to enhance the representative features and suppress the redundant features in the spatial dimension. However, the spatial fine feature map X W There is still redundancy in the channel dimension.

[0095] It can be understood that CRU adopts split transformation and fusion strategy to reduce the redundancy of channel dimension as well as computational cost and storage, where

[0096] The segmentation operation divides the input spatial refinement features into two parts, one with αC channels and the other with (1-α)C channels. The channels of the two sets of features are then compressed using a 1*1 convolution kernel to obtain X up and X low ;

[0097] The conversion operation converts the input X up As the input of "Rich Feature Extraction", GWC and PWC are performed respectively, and then added to get the output Y1. The input is used as the supplement of "Rich Feature Extraction", PWC is performed, and the obtained demerits are taken as the union with the original input to get Y2;

[0098] The fusion operation uses a simplified SKNet method to adaptively merge Y1 and Y2.

[0099] Specifically, the fusion operation is described with a specific embodiment:

[0100] Use global average pooling to combine global spatial information and channel statistics to obtain pooled S1 and S2;

[0101] Perform Softmax on S1 and S2 to obtain the feature weight vector β 1 and β 2 ;

[0102] Use the feature weight vector to get the output Y = β 1 Y 1 +β 2 Y2 , Y is the feature extracted from the channel.

[0103] In one embodiment of the present application, the result of the SCConv layer is input into the CloAtt attention module, which extracts fine-grained and global information from the input feature map through a high-frequency attention mechanism and a low-frequency attention mechanism. The high-frequency attention mechanism uses a convolution operation to focus on local features, while the low-frequency attention mechanism uses a pooling operation to focus on global features.

[0104] By splicing the features of different branches in the channel dimension, feature fusion is achieved. The spliced ​​features will contain information from different branches, which will help increase the nonlinear ability and representation ability of the network, thereby improving the network's ability to model complex data. It can improve the model's feature extraction ability while maintaining computational efficiency. SCConv effectively limits feature redundancy, which not only reduces the number of model parameters and FLOPs, but also enhances the ability of feature representation. It improves model performance while reducing redundant features.

[0105] This application introduces the SCConv layer and the CloAtt attention module, and uses the C2F-SCCloATT module to effectively reduce feature redundancy and enhance feature representation capabilities. It not only reduces model parameters and computational complexity, but also can focus on local details and global information at the same time. While maintaining computational efficiency, it significantly improves the model's feature extraction and expression capabilities, allowing the model to better handle complex action scenes.

[0106] In an embodiment of the present application, in step S2, a 2D branch is used to extract features from the preprocessed image, specifically including:

[0107] Use the backbone network to extract features from the preprocessed image to obtain the first multi-scale features;

[0108] Use FPN to fuse and enhance the first multi-scale features to obtain second multi-scale features;

[0109] The second multi-scale features are processed and transformed to obtain spatial features, where the spatial features include Fcls and Freg.

[0110] In one embodiment of the present application, the C2F-SCCloATT module also includes an efficient attention mechanism, wherein the characteristics of the efficient attention mechanism are as follows:

[0111] E(Q, K, V) = ρ q (Q)(ρ k (K) T V)

[0112] Among them, ρq represents the normalization function of the query, ρ k represents the normalization function of the key feature, Q represents the query vector, K represents the key vector, and T represents the normalization function of ρ k (K) matrix transpose, V represents the value vector.

[0113] Specifically, for q and ρ k Perform scaling operations and use the softmax function for processing:

[0114] Scaling:

[0115] Softmax: ρ q (Y) = σ row (Y)

[0116] ρ k (Y) = σ col (Y)

[0117] Among them, Scaling represents the scaling operation, σ row and σ col denotes applying the softmax function along each row or column of the matrix Y, respectively.

[0118] For the input feature map x∈R h×w×d , the module flattens it into a matrix x∈R hw×d , apply an efficient attention mechanism to it, and reshape the result into h×w×d v If d v ≠d, a 1x1 convolution is further applied to restore the dimension to d. Finally, the obtained features are added to the input features to form a residual structure.

[0119] like Figure 7 As shown, in one embodiment of the present application, the network structure of the 3D branch is:

[0120] The first feature extraction layer includes a convolution layer with a convolution kernel of 3×3×3 and 64 channels, a BN layer, a ReLU and a 3×3×3 second maximum pooling layer;

[0121] The first maximum pooling layer is 3×1×1;

[0122] The four second feature extraction layers include layer1, layer2, layer3 and layer4, wherein layer1 includes three consecutive OD_Bottleneck modules, layer2 includes four consecutive OD_Bottleneck modules, layer3 includes five consecutive OD_Bottleneck modules and one OD_Bottleneck with a non-local block added, and layer4 includes three consecutive OD_Bottleneck modules.

[0123] like Figure 8 As shown, further, the OD_Bottleneck module specifically includes:

[0124] Input grouping, including a group of 3*3 first convolution branches and a group of 3*3 full-dimensional dynamic convolution branches;

[0125] a fusion unit configured to add an output of the first convolution branch and an output of the full-dimensional dynamic convolution branch;

[0126] The dimension recovery unit includes a 1*1 convolution layer, which is used to restore the output of the fusion unit to the input feature dimension to obtain the first feature;

[0127] The residual connection unit is used to add the first feature to the input to obtain a residual feature.

[0128] This application greatly enhances the feature extraction capability of the 3D branch by introducing full-dimensional dynamic convolution and non-local blocks in the OD_Bottleneck module, and realizes adaptive adjustment of features by using full-dimensional dynamic convolution. The non-local blocks effectively capture long-distance dependencies, can better adapt to different types of action features, and improve the ability to understand complex action sequences.

[0129] Specifically, the non-local block is a module that embeds non-local operations into the neural network, namely:

[0130] z i =W z y i +x i ;

[0131] Among them, z i represents the output features of the non-local block, W z represents the learnable weight matrix, y i represents the result of a non-local operation, x i represents a residual connection.

[0132] It can be understood that the non-local block receives the feature map a, performs non-local operations on the feature map, and the weight layer W zThe output of the Non-local operation is weighted, and the feature map a is added to the weighted Non-local output for residual connection to obtain the output of the non-local block.

[0133] Specifically, the non-local operations are publicized as follows:

[0134]

[0135] Among them, y represents the output of the non-local operation, which has the same shape as the input feature map, and x represents the input of the non-local operation, which has the same shape as the output y i Similarly, i represents the coordinates of a point on the output y or input x (can be spatial coordinates, time coordinates, or spatiotemporal coordinates), and the response value of position i (the output of the non-local operation) is y i , j enumerates (traverses) the coordinate positions of all points on x or y in turn, x j Represents the value of position j on the input data, function f(x i , x j ) calculates the relationship between the two quantities at position i and position j of the input data, and the function g(x j ) calculates a representation of the j position of the input data, which can be understood here as f(x i , x j ), the final response of non-local position i is the relationship f(x i , x j ) to weight f(x i , x j )g(x j ) and finally through the normalization factor C (X) Perform normalization operation.

[0136] Furthermore, the four different attentions of the full-dimensional dynamic convolution include convolution kernel attention, spatial dimension attention, input channel dimension attention and output channel attention. The expression of the full-dimensional dynamic convolution is as follows:

[0137] y=(α ω1 ⊙α f1 ⊙α c1 ⊙α s1 W 1 +…+α ωn ⊙α fn ⊙α cn ⊙α sn ⊙W n )×x;

[0138] Among them, α ω1Represents the first convolution kernel W 1 The attention scalar, α f1 represents the spatial dimension of attention in the spatial dimension, α c1 represents the input channel dimension of the attention, α s1 The output channel dimension, α, represents the output channel attention ωn represents the attention scalar of the nth convolution kernel, W n represents the nth convolution kernel, x represents the input feature map, α fn Represents the spatial dimension of the attention of the nth convolution kernel, α cn represents the input channel dimension of the nth convolution kernel, α represents the input channel dimension of the attention, sn The output channel dimension representing the output channel attention of the nth convolution kernel.

[0139] like Fig. 9 As shown, in one embodiment of the present application, step S3 specifically includes:

[0140] S31, inputting the spatiotemporal features and the spatial features into a channel fusion module, and fusing the spatiotemporal features and the spatial features according to the channel dimension of the channel fusion module to obtain a first feature map; wherein the first feature map includes motion information and appearance information;

[0141] S32, using 1*1 convolution to compress the number of channels of the first feature, and using 3*3 convolution to extract features to obtain a second feature map;

[0142] S33, reshaping the second feature into feature maps of two dimensions, including a feature map F1 and a feature map F2;

[0143] S34, perform element-by-element dot multiplication on the feature map F1 and the feature map F2, and process them through the softmax function to obtain an attention matrix;

[0144] S35. Perform element-by-element dot multiplication of the attention matrix and the feature map F1 to obtain a third feature map, and reshape the third feature map to the size of the second feature map to obtain a fused feature.

[0145] Through feature fusion and attention mechanism, an effective combination of spatial features and spatiotemporal features is achieved, which can intelligently highlight important features and suppress irrelevant information, thereby generating more refined and information-rich fusion features, improving the ability to capture key action information, and effectively improving the accuracy and robustness of action recognition.

[0146] The present application provides a dual-branch based action recognition system, which adopts the action recognition method as described above, including:

[0147] An image processing module is configured to obtain an action video and preprocess the action video to obtain a preprocessed image;

[0148] A feature extraction module is configured to use a 2D branch and a 3D branch to extract features from the preprocessed image respectively to obtain spatial features and spatiotemporal features; wherein the 2D branch includes a backbone network and an FPN, the backbone network includes a C2F-SCCloATT module, the 3D branch includes a first feature extraction layer, a first maximum pooling layer and four second feature extraction layers, and the second feature extraction layer includes an OD_Bottleneck module;

[0149] A feature fusion module, which is configured to use a channel fusion and attention mechanism module to fuse spatial features and spatiotemporal features to obtain fused features;

[0150] The action prediction module is configured to detect the fused features and obtain the action category prediction result of each spatial position.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A dual-branch action recognition method, characterized in that: The following steps are involved: S1, obtaining an action video, and preprocessing the action video to obtain a preprocessed image; S2. Use the 2D branch and the 3D branch to extract features from the preprocessed images respectively to obtain spatial features and spatiotemporal features; the 2D branch includes a backbone network and FPN, the backbone network includes a C2F-SCCloATT module, the 3D branch includes a first feature extraction layer, a first maximum pooling layer and four second feature extraction layers, and the second feature extraction layer includes an OD_Bottleneck module; S3, use channel fusion and attention mechanism modules to fuse spatial features and spatiotemporal features to obtain fused features; S4. Detect the fused features to obtain the action category prediction results for each spatial position.

2. A dual-branch action recognition method as claimed in claim 1, characterized in that: The network structure of the 2D branch is: The backbone network includes layer_1, layer_2, layer_3, layer_4 and layer_5, layer_1 includes the first convolution unit, BN unit and SiLU, layer_2 includes the second convolution unit, BN unit, SiLU and C2F-SCCloATT module, layer_3 includes the third convolution unit, BN unit, SiLU and C2F-SCCloATT module, layer_4 includes the fourth convolution unit, BN unit, SiLU and C2F-SCCloATT module, layer_5 includes the fifth convolution unit, BN unit, SiLU and C2F-SCCloATT module; wherein the number of channels of the first convolution unit is 64, the number of channels of the second convolution unit is 128, the number of channels of the third convolution unit is 256, the number of channels of the fourth convolution unit is 512, and the number of channels of the fifth convolution unit is 1024; FPN consists of four parts. The first part includes the upper adopting layer, the Concat unit and the first C2F module. The second part includes the upper adopting layer, the Concat unit and the second C2F module. The third part includes the sixth convolution unit, the Concat unit and the third C2F module. The fourth part includes the seventh convolution unit, the Concat unit, the fourth C2F module and SPP_Block_CSP; the convolution kernel of the sixth convolution unit is 3*3 and the number of channels is 256. The convolution kernel of the seventh convolution unit is 3*3 and the number of channels is 512. The number of channels of the first C2F module is 512, the number of channels of the second C2F module is 256, the number of channels of the third C2F module is 512, and the number of channels of the fourth C2F module is 1024.

3. A dual-branch action recognition method as claimed in claim 2, characterized in that: In step S2, the 2D branch is used to extract features from the preprocessed image, specifically including: Use the backbone network to extract features from the preprocessed image to obtain the first multi-scale features; Use FPN to fuse and enhance the first multi-scale features to obtain second multi-scale features; The second multi-scale features are processed and transformed to obtain spatial features, where the spatial features include Fcls and Freg.

4. A dual-branch action recognition method as claimed in claim 2, characterized in that: The C2F-SCCloATT module includes an SCConv layer and a CloAtt attention module, wherein the SCConv layer includes a spatial reconstruction unit and a channel reconstruction unit, and the CloAtt attention module includes high-frequency attention, low-frequency attention and feature fusion.

5. A dual-branch action recognition method as claimed in claim 4, characterized in that: The C2F-SCCloATT module also includes an efficient attention mechanism, wherein the characteristics of the efficient attention mechanism are as follows: E(Q,K,V)=ρ q (Q)(ρ k (K) T V); Among them, ρ q represents the normalization function of the query, ρ k represents the normalization function of the key feature, Q represents the query vector, K represents the key vector, and T represents the normalization function of ρ k (K) matrix transpose, V represents the value vector.

6. The dual-branch action recognition method according to claim 1, characterized in that: The network structure of the 3D branch is: The first feature extraction layer includes a convolution layer with a convolution kernel of 3×3×3 and 64 channels, a BN layer, a ReLU and a 3×3×3 second maximum pooling layer; The first maximum pooling layer is 3×1×1; The four second feature extraction layers include layer1, layer2, layer3 and layer4, wherein layer1 includes three consecutive OD_Bottleneck modules, layer2 includes four consecutive OD_Bottleneck modules, layer3 includes five consecutive OD_Bottleneck modules and one OD_Bottleneck with a non-local block added, and layer4 includes three consecutive OD_Bottleneck modules.

7. A dual-branch action recognition method as claimed in claim 6, characterized in that: The OD_Bottleneck module specifically includes: Input grouping, including a group of 3*3 first convolution branches and a group of 3*3 full-dimensional dynamic convolution branches; a fusion unit configured to add an output of the first convolution branch and an output of the full-dimensional dynamic convolution branch; The dimension recovery unit includes a 1*1 convolution layer, which is used to restore the output of the fusion unit to the input feature dimension to obtain the first feature; The residual connection unit is used to add the first feature to the input to obtain a residual feature.

8. A dual-branch action recognition method as claimed in claim 7, characterized in that: The four different attentions of the full-dimensional dynamic convolution include convolution kernel attention, spatial dimension attention, input channel dimension attention and output channel attention. The expression of the full-dimensional dynamic convolution is as follows: y=(α ω1 ⊙α f1 ⊙α c1 ⊙α s1 W1+…+α ωn ⊙α fn ⊙α cn ⊙α sn ⊙W n )×x; Among them, α ω1 represents the attention scalar of the first convolution kernel W1, α f1 represents the spatial dimension of attention in the spatial dimension, α c1 represents the input channel dimension of attention, α s1 The output channel dimension, α, represents the output channel attention ωn represents the attention scalar of the nth convolution kernel, W n represents the nth convolution kernel, x represents the input feature map, α fn Represents the spatial dimension of the attention of the nth convolution kernel, α cn represents the input channel dimension of the nth convolution kernel, α represents the input channel dimension of the attention, sn The output channel dimension representing the output channel attention of the nth convolution kernel.

9. The dual-branch action recognition method according to claim 1, characterized in that: Step S3 specifically includes: S31, inputting the spatiotemporal features and the spatial features into a channel fusion module, and fusing the spatiotemporal features and the spatial features according to the channel dimension of the channel fusion module to obtain a first feature map; S32, using 1*1 convolution to compress the number of channels of the first feature, and using 3*3 convolution to extract features to obtain a second feature map; S33, reshaping the second feature into feature maps of two dimensions, including a feature map F1 and a feature map F2; S34, perform element-by-element dot multiplication on the feature map F1 and the feature map F2, and process them through the softmax function to obtain an attention matrix; S35. Perform element-by-element dot multiplication of the attention matrix and the feature map F1 to obtain a third feature map, and reshape the third feature map to the size of the second feature map to obtain a fused feature.

10. A dual-branch action recognition system, characterized in that: The action recognition method according to any one of claims 1 to 9 comprises: An image processing module is configured to obtain an action video and preprocess the action video to obtain a preprocessed image; A feature extraction module is configured to use a 2D branch and a 3D branch to extract features from the preprocessed image respectively to obtain spatial features and spatiotemporal features; wherein the 2D branch includes a backbone network and an FPN, the backbone network includes a C2F-SCCloATT module, the 3D branch includes a first feature extraction layer, a first maximum pooling layer and four second feature extraction layers, and the second feature extraction layer includes an OD_Bottleneck module; A feature fusion module, which is configured to use a channel fusion and attention mechanism module to fuse spatial features and spatiotemporal features to obtain fused features; The action prediction module is configured to detect the fused features and obtain the action category prediction result of each spatial position.

Images