A video action recognition method based on a multi-dimensional feature excitation network

Abstract

The application provides a video action recognition method based on a multi-dimensional feature excitation network, and relates to the fields of computer vision and images. The multi-dimensional feature excitation network method comprises three parts: S1. modeling time sequence motion features through a motion complement excitation module to excite time channel motion information; S2. through a joint feature excitation module, learning better space-time feature expression by means of space information attention excitation time sequence dimension and channel information attention excitation time sequence dimension; and S3. fusing the motion complement excitation module and the joint feature excitation module to construct a final multi-dimensional feature excitation fusion network for action recognition. The model can effectively express space-time features by using the multi-dimensional feature excitation fusion network, and obtain a better balance between complexity and classification accuracy. The experimental results on the HMDB51 and UCF101 data sets show that the algorithm has a high recognition accuracy, and is worthy of promotion.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to the improvement of video action recognition feature extraction models, video classification, and simulation implementation. Background Technology

[0002] Video behavior recognition refers to the use of computer vision to automatically analyze human behavior in videos, judging, analyzing, and understanding human actions and movements. Video behavior recognition is widely used in intelligent manufacturing and safety monitoring, providing intelligent early warnings for abnormal and non-standard behaviors, thereby improving production efficiency and reducing safety accidents.

[0003] Currently, deep learning-based video behavior recognition methods mainly fall into two categories: those based on 3D convolutional networks and those based on 2D two-stream networks. Two-stream convolutional networks decompose the video into spatial and temporal components, training them independently and fusing them only in the last layer, lacking sufficient spatiotemporal interaction. Furthermore, extracting optical flow information in the temporal component requires additional time and memory consumption. On the other hand, while 3D convolutional networks do not need to extract optical flow information, the video itself contains a large amount of redundant information. Directly extending 2D convolution to the temporal dimension not only multiplies the number of parameters and computational cost, slowing down network inference speed, but also fails to utilize the redundancy of the video. To address these issues, this application proposes a motion completion activation module to replace optical flow information on top of 2D convolution, modeling short-range temporal dependencies and learning motion information in the video. Simultaneously, a joint feature activation module is proposed, combining channel, temporal, and spatial features to model global contextual information, injecting the learned motion information into the temporal and spatial dimensions to achieve information interaction across different dimensions and effectively learn spatiotemporal features. Summary of the Invention

[0004] 1. Technical problems to be solved:

[0005] To address the shortcomings of existing technologies, this invention provides a video action recognition method based on a multi-dimensional feature-excited network. By building upon 2D convolution, the network can learn sufficient temporal features, solving the problems of excessive computation and parameter count in existing network structures, and achieving a better balance between complexity and classification accuracy.

[0006] 2. Technical Solution:

[0007] This invention proposes a video action recognition method based on a multidimensional feature activation network. This method captures short-term and long-term spatiotemporal features of video data through the motion completion activation module and the multidimensional feature activation module designed in this invention, respectively. The backbone of the network is composed of stacked multidimensional feature activation residual blocks. These multidimensional feature activation residual blocks are constructed by connecting the proposed motion completion activation module and joint feature activation module in parallel, enabling them to fully learn short- and long-term motion information. The network first samples the input video using a sparse temporal sampling strategy to obtain T frames; then, a 7×7 convolutional layer is used to perform preliminary feature extraction on the images; next, the images are input into the backbone network to extract spatiotemporal features; finally, the images are input into a classification network to obtain the final classification result. The method includes the following steps:

[0008] S1. First, the input features are fed into the Motion Excitation Enrichment (MSE) module. The input features are sliced ​​in the time dimension to obtain time frame features. Then, two-level frame subtraction is performed in the time dimension to capture motion information. After that, the obtained motion information is concatenated in the time dimension. Spatial pooling and the sigmoid function are used to obtain the weights of motion attention information. Finally, the motion attention excitation information is multiplied with the original input feature map to complete the learning and enhancement of temporal motion features.

[0009] S2. Similarly, the input features are simultaneously input into the Joint Feature Activation Module (UIE) to perform spatial information attention activation temporal dimension and channel information attention activation temporal dimension operations on the features respectively. By modeling the interdependence between temporal, channel and spatial information, long-distance spatiotemporal features are learned.

[0010] S3. Then, the feature maps obtained from S1 and S2 are concatenated in parallel. 1×1 convolutions are added before and after the concatenated module to construct multidimensional feature activation residual blocks (MFARs). Finally, a multidimensional feature activation network (MFARs) is composed of multiple sets of multidimensional feature activation residual blocks with different resolutions and a classification module. The network is trained to obtain the final classification result.

[0011] 3. Beneficial effects:

[0012] This invention provides a video action recognition method based on a multi-dimensional feature activation network. Building upon 2D convolution, this application proposes a motion completion activation module to replace optical flow information, modeling short-range temporal dependencies and learning motion information in the video. Simultaneously, it proposes a joint feature activation module that combines channel, temporal, and spatial features to model global contextual information, injecting the learned motion information into the temporal and spatial dimensions to achieve information interaction across different dimensions and effectively learn spatiotemporal features. The method demonstrates good recognition performance on the UCF101 and HMDB51 datasets. Attached Figure Description

[0013] Figure 1 This is the overall process framework of the present invention;

[0014] Figure 2 This is a structural diagram of the motion compensation excitation module of the present invention;

[0015] Figure 3 This is a structural diagram of the joint feature excitation module of the present invention;

[0016] Figure 4 This is a visualization of the invention on the UCF101 dataset;

[0017] Figure 5 This is a visualization of the intermediate layers of the network in the UCF101 dataset according to the present invention. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. A video action recognition method based on a multi-dimensional feature excitation network, the specific implementation steps of which are as follows:

[0019] S1: Design of the Motion Compensation Excitation Module (MSE)

[0020] The structure of the motion compensation stimulus module is as follows: Figure 2 As shown, the input feature map is first processed by a 2D 1×1 convolution to reduce the number of channels, and then the input feature map is subtracted from the feature maps of adjacent frames along the time dimension. Considering that the subject of the action will shift over time, before the subtraction, a 2D 3×3 convolution is used in the next frame to learn the action shift deviation. This process is shown in Equation (1):

[0021] M t =conv 3×3 (X t )-X t-1 ,……1≤t<T(1)

[0022] Among them, M t Let conv represent the feature map of frame t. 3×3 This represents a 3×3 convolution. To ensure that the number of frames in the time dimension remains unchanged after subtraction, the input feature map is padded in the time dimension by adding a feature map filled with zeros as frame 0 (X0).

[0023] In obtaining motion information M at the level of movement t Subsequently, in order to remove the influence of background motion noise and learn more reliable motion features, the obtained motion features are subtracted from adjacent frames again to learn more reliable and sufficient motion information. This process is shown in formula (2), and the obtained feature map M is... t+1 With M t Subtraction yields V tTo further learn higher-level sports information.

[0024] V t =M t+1 -M t ,1≤t<T-1(2)

[0025] Similarly, to maintain the time dimension, a feature map with all values ​​of 0 was used as padding for the (T+1)th frame. The resulting motion information feature map V for frame T... t V is obtained by splicing the data along the time dimension. c .

[0026] Next, a spatial global average pooling is used to globally quantize the motion information in each frame to eliminate the influence of spatial details on motion information learning. Then, a 2D 1×1 convolution is used to recover the number of channels, and then the sigmoid function is used to obtain motion attention stimulus information. In addition, in order to enhance the motion stimulus information, the output range of the sigmoid function is extended from [0,1] to [-1,1], as shown in Equation (3):

[0027] A = 2δ(conv) 1×1 (V c ))-1(3)

[0028] Where δ represents the sigmoid function, conv 1×1 This represents a 1×1 convolution.

[0029] Finally, the motion attention activation information is multiplied by the original input feature map to complete the learning and enhancement of motion features. Residual connections are also added to reduce the gradient vanishing problem. The process is shown in equation (4):

[0030] X out =X + X⊙A(4)

[0031] Where ⊙ represents channel-level element-wise multiplication.

[0032] S2: Design the Joint Feature Activation Module (UIE)

[0033] The motion completion activation module aims to supplement short-range motion information into spatial information to learn spatiotemporal features. The joint feature activation module aims to supplement channel information into the motion information stream (channel attention activation of the temporal dimension) and spatial information into the motion information stream (spatial attention activation of the temporal dimension). By modeling the interdependencies between temporal, channel, and spatial information, long-range spatiotemporal features are learned. The structure of the joint feature activation module is as follows: Figure 3As shown, it consists of two parts: long-short term motion-activated channel attention (MACA) and long-short term motion-activated spatial attention (MASA).

[0034] like Figure 3 As shown on the left, the MASA structure has two branches. The left branch uses only a 1D temporal convolution with a kernel size of 3 to learn short-term motion information, while the right branch is used to obtain long-term motion information. The long-term motion information branch first learns short-term motion information by passing the input features through a 1D temporal convolution with a kernel size of 3, and then uses a 1D temporal convolution with a kernel size of T to achieve temporal dimension pooling to learn global temporal information. After the features containing global temporal information are subjected to channel dimension softmax, they are multiplied with short-term motion information to obtain the feature map after channel information pooling. Traditional methods simply pool the channels, which will cause a large loss of channel information. MASA uses the matrix multiplication method to integrate channel information into the spatial and temporal dimensions, and can learn better spatial attention without losing channel features. After the feature map after channel information pooling is further learned by a 3×3 convolution, it is activated by the sigmoid function to obtain the spatial attention map. The operation process is shown in Equation (5).

[0035]

[0036] Where, conv 1D This represents a temporal 1D convolution with a kernel of 3. RS stands for ReShape operation, and TP stands for TemporalPooling operation. This represents matrix multiplication.

[0037] like Figure 3 As shown on the right, MACA also has two branches: the left branch uses a 1D temporal convolution with a kernel size of 3 and a reshape operation to model short-range spatiotemporal features; the right branch, after a 1D temporal convolution with a kernel size of 3, uses a 1D temporal convolution with a kernel size of T to pool the temporal dimension, and then reshapes it to obtain global spatiotemporal features; then, the global spatiotemporal features are multiplied by the short-range spatiotemporal features after a softmax operation on the spatial dimension, so that the spatial information is integrated into the channel and temporal dimensions, and better channel attention is learned without losing spatial features. The operation process is shown in Equation (6).

[0038]

[0039] After obtaining the spatial attention and channel attention of the long and short-term motion information stimulation, the joint feature stimulation module multiplies them to obtain attention-based multidimensional feature stimulation information. The complete operation process of the joint feature stimulation module is shown in Equation (7).

[0040] Y=X⊙(MASA(X)⊙MACA(X))+X(7)

[0041] Where X represents the input feature map, ⊙ represents element-wise multiplication, and MASA() and MACA() represent spatial attention and channel attention functional blocks excited by long and short time motion information, respectively.

[0042] S3: Design a multi-dimensional feature activation network

[0043] like Figure 1 As shown, the backbone of the Multi-Dimensional Feature Activation Residual (MFAR) network consists of multiple MFAR modules with different resolutions. Each MFAR module is composed of two parallel modules: a Motion Complementary Activation (MSE) module and a Joint Feature Activation (UIE) module. The MSE learns short-term motion features, while the UIE learns comprehensive spatiotemporal features across channels and spatial dimensions, stimulating both short-term and long-term motion information. Input features are first reduced in number by two parallel 1×1 convolutions, then input to the MSE and UIE modules respectively, achieving the goal of modeling motion information and joint feature information. Each module then recovers its channel count through 1×1 convolutions, and the feature maps output by the two modules are summed to obtain the final spatiotemporal features. The network backbone is composed of stacked Multi-dimensional Feature Activation Residual (MFAR) blocks. The MFAR, constructed by the proposed Motion Complementary Activation (MSE) module and Joint Feature Activation (UIE) module connected in parallel, is capable of fully learning both short-term and long-term motion information. The network first uses a sparse temporal sampling strategy to sample the input video, obtaining T frames of images; then a 7×7 convolutional layer is used to perform preliminary feature extraction on the images; next, the images are input into the backbone network to extract spatiotemporal features; finally, the images are input into the classification network to obtain the final classification result.

[0044] Table 1 compares the recognition accuracy of the proposed method (hereinafter referred to as MFARs) with other methods on the UCF101 and HMDB51 datasets. The experimental results in Table 1 show that the algorithm of this invention has significant advantages over other algorithms on both datasets. In particular, compared with two-stream convolutional networks and 3D networks, the algorithm of this invention not only reduces the number of parameters and computational cost, but also achieves a significant improvement in accuracy. This indicates that the two modules proposed in this invention can effectively extract spatiotemporal features from videos, which is beneficial for improving the accuracy of action recognition in videos.

[0045] Table 1 compares the network structure with existing state-of-the-art methods on UCF101 and HBDB51 (Kinetics* indicates that only part of the network structure was pre-trained on Kinetics).

[0046]

[0047] S4: Design a visualization module for multi-dimensional feature-stimulated networks

[0048] To verify that the proposed MFAR module can effectively extract spatiotemporal features, we visualized the network using the grad-CAM method. For example... Figure 4 As shown, the first row contains the input video: 8 frames sampled from the "ApplyEyeMakeup" action video in the UCF101 dataset; the second row shows the heatmap of features extracted by the last layer of conv5_x when using ResNet50 as the backbone; the third row shows the heatmap of features extracted by the last layer of MFAR-4. The red parts represent the parts of the image that receive attention, while the blue parts represent the parts that do not receive attention from the network. Figure 4 It can be seen that ResNet50 focuses on the entire face, meaning that the network cannot accurately capture specific action information and its area of ​​focus is relatively broad; while MFARs focus on the area around the eyebrows, indicating that the MFAR module can help the network pay more attention to the motion area, reduce the influence of background information, and learn more expressive spatiotemporal features. Figure 5 We further compared the feature heatmaps of ResNet50 and MFARs networks at different layers. Figure 5 The first column shows a keyframe from the “Blow Dry Hair” class in the UCF101 dataset. Figure 5 Columns two through five show the feature heatmaps extracted by the ResNet50 network and the MFARs network at four levels, respectively. Figure 5It can be seen that ResNet50 is not sensitive to motion capture in shallow layers, causing each layer's features to focus on non-moving regions. In contrast, MFARs can accurately focus on key moving regions in shallow layers, and their action localization is also very accurate in the last layer. This proves that the MFAR module plays a role in different layers of the network.

[0049] This application proposes a Multidimensional Joint Feature Activation Network (MFARs) based on an attention mechanism for video action recognition. The MFARs network consists of multiple MFAR modules at different resolutions, including an MSE module for modeling local spatiotemporal information and a UIE module that uses multidimensional attention to learn spatiotemporal features and model global spatiotemporal information. The two modules complement and reinforce each other's spatiotemporal information in parallel, enabling the network to learn more comprehensive and richer spatiotemporal features. Experimental comparisons with two-stream convolutional networks and 3D convolutional networks demonstrate that MFARs can effectively represent video spatiotemporal information, retaining the low parameter count advantage of two-stream convolutional networks while possessing the same effective spatiotemporal feature modeling capability as 3D convolutional networks, achieving a balance between performance and efficiency.

[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A video action recognition method based on a multidimensional feature excitation network, comprising the following steps: S1. First, the input feature map is input into the motion stimulus completion module (MSE). The input feature map is sliced ​​in the time dimension to obtain time frame features. Then, two-level frame subtraction is performed in the time dimension to capture motion information. After that, the obtained motion information is concatenated in the time dimension and then spatial pooling and sigmoid function are used to obtain motion attention stimulus information. Finally, the motion attention stimulus information is multiplied with the input feature map to complete the learning and enhancement of temporal motion features, and the feature map of S1 is obtained. S2. Similarly, the input feature map is simultaneously input into the Joint Feature Activation Module (UIE). The input feature map is subjected to spatial information attention activation in the temporal dimension and channel information attention activation in the temporal dimension to enhance motion information and suppress background interference. The spatial information attention activation in the temporal dimension includes two branches. One branch uses a 1D temporal convolution with a kernel size of 3 to learn short-term motion information. The other branch is used to obtain long-term motion information. The long-term motion information branch first uses a 1D temporal convolution with a kernel size of 3 to learn short-term motion information from the input feature map. Then, it uses a 1D temporal convolution with a kernel size of T to achieve temporal dimension pooling to learn global temporal information. The features containing global temporal information are multiplied with short-term motion information after channel dimension softmax to obtain channel information pooled feature map. The channel information pooled feature map is further processed by a 3×3 convolution to learn spatial information and then activated by the sigmoid function to obtain spatial attention. In the channel information attention-excited temporal dimension, there are also two branches. One branch uses a 1D temporal convolution with a kernel size of 3 and a reshape operation to model short-range spatiotemporal features. The other branch uses a 1D temporal convolution with a kernel size of T to pool the temporal dimension after the 1D temporal convolution with a kernel size of 3 and then reshapes it to obtain global spatiotemporal features. Then, the global spatiotemporal features are multiplied by the short-range spatiotemporal features after passing the softmax of the spatial dimension, so that the spatial information is integrated into the channel and time dimensions, and channel attention is learned without losing spatial features. After obtaining the spatial attention and channel attention of the long and short time motion information stimulation, the joint feature stimulation module multiplies them to obtain attention-based multidimensional feature stimulation information, resulting in the feature map of S2. S3. Then, the feature maps obtained from S1 and S2 are concatenated in parallel. 1×1 convolutions are added before and after the concatenated module to construct a multidimensional feature-stimulated residual block (MFAR). Finally, a multidimensional feature-stimulated network (MFARs) is composed of multiple sets of multidimensional feature-stimulated residual blocks with different resolutions and a classification module. The network is trained to obtain the final classification result.

2. The video action recognition method based on a multidimensional feature excitation network according to claim 1, characterized in that: The joint feature activation module consists of two parts: the temporal dimension of spatial information attention activation and the temporal dimension of channel information attention activation.

3. The video action recognition method based on a multidimensional feature excitation network according to claim 1, characterized in that: The joint feature activation module fuses the feature information of the two parts through element-wise multiplication, and then multiplies it with the input feature map after passing it through the sigmoid function.

Images