A Method for Separating and Extracting Spatiotemporal Features Based on Channel Grouping in Action Recognition

Abstract

The present invention discloses a method for separating and extracting spatiotemporal features based on channel grouping in action recognition. First, the video is sparsely sampled, and the ResNet network is used as the backbone network to extract basic features from the sampled frame sequence to obtain feature maps of each frame. Then, by grouping the channels of the feature maps, the channels of each frame feature map are divided into two groups. One group uses a learnable channel shift operation to give 2D convolution the ability to learn temporal features; the other group focuses on learning spatial features by reducing temporal features and increasing spatial channels. Finally, the features obtained from the temporal feature extraction branch and the spatial feature extraction branch are temporally aligned and stacked and fused to obtain an aggregated expression of spatiotemporal features. The present invention solves the problem in the prior art that 2D convolutional neural networks are difficult to extract temporal features of videos in video action recognition, thereby achieving a balance between action recognition efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to a method for separating and extracting spatio-temporal features, and particularly to a method for separating and extracting spatio-temporal features based on channel grouping in action recognition. Background Art

[0002] With the development of Internet technology and the popularization of video acquisition devices, videos have become the main carrier of information. The quantity of video data has increased explosively, and how to analyze and understand the content of videos has become increasingly important. Action recognition is one of the important topics in the field of video understanding and has wide applications in video surveillance, human-computer interaction, motion analysis, video information retrieval, etc. The actions in videos contain a time dimension to be processed, so capturing the time information contained between video frames has become the focus of action recognition. Action recognition also therefore requires more processing time, so how to balance performance and speed is crucial.

[0003] With the development of deep learning, algorithms based on deep learning and convolutional neural networks have made breakthrough progress in action recognition tasks. Algorithms based on 2D convolutional neural networks are difficult to capture the features of the video time dimension due to their limitations. Although algorithms based on 3D convolutional neural networks solve the problem that 2D convolutional neural networks can only be applied to 2D feature maps and are difficult to process video data containing temporal features, they require high computational complexity, have high requirements for machine performance, have high limitations in deployment and application, and at the same time, their number of parameters is too large, resulting in difficulty in training and high requirements for the scale of the dataset and training techniques. Summary of the Invention

[0004] Object of the Invention: The object of the present invention is to provide a method for separating and extracting spatio-temporal features based on channel grouping in action recognition that can extract video time features and improve the efficiency and accuracy of action recognition.

[0005] Technical Solution: The spatio-temporal feature separation and extraction method of the present invention includes the following steps:

[0006] S1, sparsely sample the video, and use the ResNet network as the backbone network to extract basic features from the sampled frame sequence to obtain the feature maps of each frame;

[0007] S2, in the residual branch of each residual block of the ResNet network, the feature map of each frame passes through the channel attention module to obtain the weights of each channel of the feature map; then, according to the weight values, the channels of the feature map of each frame are divided into two groups. The channels with weights not less than the threshold are assigned to the first group, and the channels with weights less than the threshold are assigned to the second group; the channel order within the group follows the channel order of the original feature map;

[0008] S3: The feature map composed of the first set of channels is sent to the temporal feature extraction branch. The learnable channel translation operation is applied to all channels of the feature map, so that the feature map of each frame has the ability to dynamically interact with the feature maps of adjacent frames to learn temporal features.

[0009] The feature map composed of the second group of channels is sent to the spatial feature extraction branch. Under the premise of keeping the total number of features unchanged, the number of spatial channels of the feature map of each frame is increased through convolution operation while downsampling the time dimension of the feature map.

[0010] S4, the features obtained by the temporal feature extraction branch and the spatial feature extraction branch are temporally aligned and stacked and fused to obtain an aggregated expression of the spatiotemporal features.

[0011] Furthermore, in step S1, the action video is divided into T parts on average, a frame of image is randomly selected from each part, and then basic features are extracted through the ResNet network to obtain the corresponding feature map group:

[0012] X={x1,x2,…,x T}

[0013] Where T is the number of video frames input to the network, is the feature map of the tth frame, C is the number of channels, and H and W are the spatial dimensions.

[0014] Furthermore, in step S2, the feature map of each frame passes through the channel attention module to obtain the weight of each channel of the feature map; then the channels of the feature map of each frame are divided into two groups according to the weight value. The implementation steps are as follows:

[0015] S21, the feature map x of each frame in the spatial dimension t Perform average pooling to obtain

[0016] S22, for all temporally adjacent feature maps after pooling and After performing 2D convolution operations, the elements are subtracted one by one. The result of the subtraction is s t :

[0017]

[0018] Among them, W θ and is the convolution parameter, Conv() is the convolution operation;

[0019] S23, for s t Perform 2D convolution operation to obtain Then the sigmoid function is used, which is recorded as

[0020]

[0021] Where σ is the sigmoid function, is the convolution parameter, Conv() is the convolution operation; Each element in the range is between 0 and 1;

[0022] S24, according to The feature map x t The channels are divided into two groups:

[0023] The first group selection The corresponding x whose median is not less than the threshold ζ t Channel, denoted as The number of its channels is denoted as ρ(ζ);

[0024] The second group selection The corresponding x whose median is less than the threshold ζ t Channel, denoted as The number of channels is recorded as C-ρ(ζ).

[0025] Furthermore, in step S3, the learnable channel shift operation is applied to all channels of the feature map so that each channel has the ability to interact with adjacent channels. The implementation steps are as follows:

[0026] For the first set of feature maps Using channel-wise 1D convolution on each channel, we get Where T is the size of the time dimension, H and W are the sizes of the spatial dimensions, and ρ(ζ) is the number of channel dimensions;

[0027] For the second set of feature maps Temporal-wise 1D convolution is used in the time dimension to reduce the time dimension of the feature map group from T to Then 2D convolution is used to increase the channels of each feature map in the feature map group to α(C-ρ(ζ)), and finally we get

[0028] Furthermore, in step S5, the steps for fusing the features obtained by the temporal feature extraction branch and the spatial feature extraction branch are as follows:

[0029] The second set of output feature maps Perform Reshape operation to get Align the two sets of features in the time dimension; then use the Concatenate operation and the output of the first set to integrate.

[0030] Compared with the prior art, the remarkable effects of the present invention are as follows:

[0031] 1. The present invention sparsely and randomly samples video frames. While obtaining video frames, it discards most redundant video frame information and can basically cover the entire video with the sampled video frame information, reducing the computational amount in subsequent feature extraction while maintaining good video information generalization ability;

[0032] 2. According to the different functions of each channel of the feature map, the channels of each feature map are dynamically and evenly divided into two groups. One group focuses on the change of the action itself, and uses 1D convolution with less computational amount to enable the interaction of features at different times, thereby efficiently extracting temporal features; the other group focuses on the change of video background information, reduces the time dimension, reduces the computational amount on the premise of effectively capturing spatial features, and at the same time uses convolution to increase spatial channels, enrich the network spatial features, and improve the learning ability of spatial features; it solves the problem that it is difficult for 2D convolutional neural networks in video action recognition to extract video temporal features in the prior art, making both the action recognition efficiency and accuracy taken into account. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a structural flowchart of the present invention;

[0034] Figure 2 is a schematic diagram of an example of channel grouping of the present invention;

[0035] Figure 3 is a schematic diagram of an example of temporal feature extraction of the present invention;

[0036] Figure 4 is a schematic diagram of an example of spatial feature extraction of the present invention;

[0037] Figure 5 is a schematic diagram of the overall effect of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0038] The present invention will be further described in detail below in conjunction with the accompanying drawings of the specification and the specific embodiments. The following description is only a part of the implementation cases. For those of ordinary skill in the art, without creative efforts, the technical solutions described in the present invention can also be applied to other similar scenarios according to these.

[0039] The present invention's spatiotemporal feature separation and extraction method is a lightweight action recognition method. By grouping the channels of the feature map, the channels of each frame's feature map are divided into two parts. One part is more conducive to temporal feature extraction, using a learnable channel shift operation to give 2D convolution the ability to learn temporal features; the other part is more conducive to spatial feature extraction, focusing on learning spatial features by reducing temporal features and increasing spatial channels. This invention solves the problem of 2D convolutional neural networks' difficulty in extracting temporal features in video action recognition, achieving a balance between efficiency and accuracy in action recognition.

[0040] The flow chart of the present invention is as follows Figure 1 As shown, specifically including:

[0041] Step 1: Basic feature extraction

[0042] The video is sparsely sampled, and the ResNet series network is used as the backbone network to extract basic features from the sampled frame sequence as the input of the subsequent network.

[0043] During the inference phase of the action recognition network, a 720p camera was used to capture a video of a person running. The video was captured at a frame rate of 25 fps, with the camera angle parallel to the person, capturing the entire range of motion. The captured video was converted into video frames using the ffmpeg tool. The entire video frame was divided into T = 16 equal parts. One video frame was randomly selected from each part and sequentially assembled into a video frame group. This video frame sequence served as the input to the action recognition network.

[0044] This example uses the ResNet series residual network as the basic feature extraction network. As part of the classic backbone network for computer vision tasks, the ResNet series network uses four residual blocks to solve the problems of gradient diffusion and explosion when the neural network is deepened. The video frame group is input into the ResNet series network to extract the basic features of the video frame; the corresponding feature map is obtained as follows:

[0045] X={x1,x2,…,x T} (1)

[0046] Among them, T is the number of video frames input to the network, is the feature map of the tth frame, C is the number of channels, and H and W are the spatial dimensions.

[0047] Step 2: Channel grouping

[0048] In the residual branch of each residual block in the ResNet series of networks, for the feature map of each frame, it passes through a channel attention module. The channel attention module quantifies the importance of each channel for temporal feature extraction by comparing the differences in channels at the same position in adjacent frames, and obtains the weights of each channel in the feature map. Then, according to the weight values, the channels of this frame's feature map are divided into two groups. The channels with weights not less than the threshold are assigned to the first group, and the channels with weights less than the threshold are assigned to the second group. The channel order within the group follows the channel order of the original feature map. The schematic diagram of channel grouping is as shown in Figure 2 shown

[0049] Step 2.1, perform average pooling on the feature map x of each frame in the spatial dimension t to obtain where C is the number of channels

[0050] Step 2.2, perform 2D convolution operations on all temporally adjacent feature maps after pooling and respectively, and then subtract element by element, denoted as

[0051]

[0052] where W θ and are convolution parameters, and Conv() is the convolution operation; s t is the result obtained by subtracting the corresponding channels of adjacent temporal feature maps, indicating the degree of difference between the two feature maps

[0053] Step 2.3, perform a 2D convolution operation on s t to obtain and then use the sigmoid function operation, denoted as

[0054]

[0055] where σ is the sigmoid function is the convolution parameter, and Conv() is the convolution operation Each element in is in the range of 0 to 1, indicating the importance of the channel corresponding to this element's position for the network to learn temporal features. The closer to 1, the more beneficial it is for temporal feature learning

[0056] Step 2.4, according to divide the channels of the feature map x t into two groups: The first group selects the corresponding x channels in with values not less than the threshold ζ, denoted as t and its number of channels is denoted as p(ζ); The second group selects the corresponding x channels in with values less than the threshold ζ, denoted as and the corresponding x t channels, denoted as The number of its channels is denoted as C-ρ(ζ). The channels selected in the first group are all more conducive to learning time features, which helps subsequent time feature extraction. The channels selected in the second group are more conducive to learning spatial features, which helps subsequent spatial feature extraction.

[0057] Step 3, spatio-temporal feature separation and extraction

[0058] The feature map composed of the first group of channels is fed into the time feature extraction branch. The learnable channel translation operation is applied to all channels of the feature map, enabling each frame of features to have the ability to dynamically interact with adjacent frame features, thereby learning time features.

[0059] As Figure 3 shown, the first group of feature maps is where T is the size of the time dimension, H and W are the sizes of the spatial dimensions, ρ(ζ) is the number of channel dimensions. Using channel-wise 1D convolution on each channel of this group of feature maps, we get which enables interaction between adjacent feature channels and enables each frame to dynamically adjust the channels in a learned manner to obtain the feature information of adjacent frames.

[0060] The second group of channels is fed into the spatial feature extraction branch. On the premise of keeping the total number of features unchanged, while downsampling the time dimension of the feature map, the number of spatial channels is increased through convolution operations, achieving the effect of removing redundant time features and increasing spatial features.

[0061] As Figure 4 shown, the second group of feature maps is where T is the size of the time dimension, H and W are the sizes of the spatial dimensions, C-ρ(ζ) is the number of channel dimensions. Using Temporal-wise 1D convolution in the time dimension, the time dimension of the feature map is reduced from T to α is the change amplitude. Then using 2D convolution, the number of channels of the feature map is increased from C-ρ(ζ) to α(C-ρ(ζ)), and finally we get For action recognition, the changes in spatial features are relatively slow. Reducing the time dimension of the feature map can remove redundant spatial features, while using 2D convolution to increase channels can enhance the richness of spatial features, which is conducive to the network's learning of spatial features.

[0062] Step 4, spatio-temporal feature fusion

[0063] After time-aligning the features obtained from the two branches and stacking and fusing them, an aggregated expression of spatio-temporal features is obtained.

[0064] Since the time dimensions of the feature maps output by the two branches that separately extract temporal and spatial features are inconsistent, the spatial feature map is reshaped to facilitate the fusion of the two, specifically as follows:

[0065] Reshape the feature map of the second group of outputs to so that the time dimension is the same as that of the output of the first group, aligning the two groups of features in the time dimension, and then use the Concatenate operation to fuse with the output of the first group for fusion.

[0066] The overall effect schematic diagram of the spatio-temporal feature separation extraction method of this embodiment is as shown in Figure 5 the figure.

[0067] Obviously, the above embodiments are only examples clearly described and not limitations on the implementation manners. It is not necessary and impossible to enumerate all the implementation manners here. And the obvious changes or variations derived therefrom are still within the protection scope of the present invention.

Claims

1. A method for separating and extracting spatio-temporal features based on channel grouping in action recognition, characterized in that, It includes the following steps: S1, sparsely sample the video, use the ResNet network as the backbone network to extract basic features from the sampled frame sequence, and obtain the feature maps of each frame; S2, in the residual branch of each residual block of the ResNet network, the feature map of each frame passes through the channel attention module to obtain the weights of each channel of the feature map; then, according to the weight values, the channels of the feature map of each frame are divided into two groups, the channels with weights not less than the threshold are assigned to the first group, and the channels with weights less than the threshold are assigned to the second group; The channel order within the group follows the channel order of the original feature map; S3, the feature map composed of the channels in the first group is fed into the temporal feature extraction branch, and the learnable channel translation operation is applied to all channels of the feature map, so that each frame feature map has the ability to dynamically interact with the adjacent frame feature map and learn temporal features; The feature map composed of the channels in the second group is fed into the spatial feature extraction branch. On the premise of keeping the total number of features unchanged, while downsampling in the temporal dimension of the feature map, the number of spatial channels of each frame feature map is increased through convolution operations; S4, perform temporal alignment on the features obtained from the temporal feature extraction branch and the spatial feature extraction branch, and then stack and fuse them to obtain the aggregated expression of spatio-temporal features; In step S1, the action video is evenly divided into T parts, one frame picture is randomly selected from each part, and then the basic features are extracted through the ResNet network to obtain the corresponding feature map group X as follows: X = {x1, x2, …, x T} Among them, T is the number of video frames input to the network, is the feature map of the t-th frame, where t = 1, 2, …, T; C is the number of channels, and H and W are the spatial dimension sizes; In step S2, the feature map of each frame passes through the channel attention module to obtain the weights of each channel of the feature map; then, the implementation steps of dividing the channels of the feature map of each frame into two groups according to the weight values are as follows: S21, perform average pooling on the feature maps x of each frame in the spatial dimension to obtain t ​ S22, perform 2D convolution operations on all feature maps that are adjacent in time after pooling and respectively, and then subtract element by element. The result of the subtraction is s t : Among them, W θ and are convolution parameters, and Conv() is a convolution operation; S23, perform a 2D convolution operation on s t to obtain Then perform a sigmoid function operation, denoted as: where σ is the sigmoid function, are convolution parameters, and Conv() is the convolution operation; each element in is in the range of 0 to 1; S24, according to divide the channels of the feature map x t into two groups: The first group is selected The corresponding x where the median is not less than the threshold is selected t The channels are denoted as The number of channels is denoted as The second group selects with a median value less than the threshold for the corresponding x t channels, denoted as The number of its channels is denoted as 2. The method for separating and extracting spatio-temporal features based on channel grouping in action recognition according to claim 1, wherein, In the said step S3, the implementation steps of applying the learnable channel shift operation to all channels of the feature map so that each channel has the ability to interact with adjacent channels are as follows: For the first set of feature maps Apply channel-wise 1D convolution on each channel to obtain where T is the size of the time dimension, H and W are the sizes of the spatial dimensions, and is the number of channel dimensions; For the second group of feature maps Apply Temporal-wise 1D convolution in the time dimension, reducing the time dimension of the feature map group from T to where α is the change amplitude; then apply 2D convolution to increase the channels of each feature map in the feature map group to Finally, obtain 3. The method for separating and extracting spatio-temporal features based on channel grouping in action recognition according to claim 1, wherein In the said step S4, the implementation steps of fusing the features obtained from the temporal feature extraction branch and the spatial feature extraction branch are as follows: The feature maps output by the second group are subjected to a Reshape operation to obtain so that the two groups of features are aligned in the time dimension; then a Concatenate operation is used with the output of the first group for fusion.

Images