A method and system for action recognition based on enhanced spatio-temporal features
By employing sparse sampling, channel-level motion information enhancement, and spatiotemporal feature aggregation, combined with multi-branch output, the problems of high computational resource consumption and insufficient flexibility in existing technologies are solved, achieving efficient video action recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2021-06-10
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies struggle to effectively utilize temporal motion features in video action recognition, resulting in high computational resource requirements and a lack of end-to-end network architecture. Furthermore, existing methods suffer from high computational costs or insufficient flexibility when extracting spatiotemporal features.
A sparse sampling strategy is adopted to acquire video frames. Through a channel-level motion information enhancement module and a spatiotemporal feature aggregation module, combined with multi-branch output, features in the temporal dimension are adaptively extracted and fused to reduce redundant information and save computational costs.
It effectively extracts temporal features from videos, reduces computational resource requirements, and improves the accuracy and efficiency of action recognition. By supplementing global features through multi-branch output, it enhances the comprehensiveness and fine-grainedness of action recognition.
Smart Images
Figure CN115546885B_ABST
Abstract
Description
Invention Field
[0001] This invention belongs to the field of computer vision and video classification technology, and in particular relates to an action recognition method and system based on enhanced spatiotemporal features. Background Technology
[0002] Action recognition in videos is one of the most important problems in computer vision. It can extract semantic information from videos, which makes it promising for a wide range of applications, such as patient monitoring, motion analysis, intelligent video surveillance, and human-computer interaction. At the same time, the semantic information extracted by action recognition can also provide video features for other computer vision tasks (such as action detection and localization). Because videos are essentially formed by stacking multiple images over time, there is a natural connection between videos and images. Therefore, early action recognition methods were often modified and extended from image recognition methods, such as shallow high-dimensional coding based on local spatiotemporal features, including HOG3D and SIFT-3D.
[0003] With the rapid development of deep learning, CNN has achieved great results in the field of image recognition. However, the results of directly applying 2D CNN to the field of action recognition in video are not good. When using 2D CNN to perform video recognition directly, it can only classify each frame of the input video and integrate the classification results of all frames, but ignores the temporal information. However, the motion features contained between frames in the temporal dimension are crucial for video classification.
[0004] To extract motion features in the temporal dimension, 2D networks typically employ a two-stream structure, with two streams extracting spatial and temporal features respectively. The spatial branch takes RGB frames as input, while the temporal branch takes optical flow as input, since optical flow is defined by the instantaneous displacement vector of a pixel and contains motion information. Finally, the classification scores from the two network branches are fused to obtain the final classification result. However, using optical flow presents additional challenges in terms of offline computation and storage space, and the network structure cannot achieve end-to-end processing, which are also drawbacks of the two-stream structure.
[0005] To achieve end-to-end extraction of temporal features, some researchers have proposed 3D convolution methods. Unlike 2D convolution, 3D convolution requires stacked video frames as input, and then uses a 3D convolution kernel to directly extract spatiotemporal features from this input. Although using 3D convolution to extract spatiotemporal features is quite natural and easy to understand, the expansion of the convolution kernel from 2D to 3D significantly increases the computational load, demanding higher computational resources compared to 2D convolution.
[0006] Besides convolution, motion features can also be obtained using motion information extraction modules. The shift operation is frequently used in motion information extraction modules. Specifically, this operation moves a portion of the feature map along the time dimension to exchange information between adjacent frames and extract spatiotemporal features at low cost. However, some studies regard the shift operation as a channel-level temporal convolution with fixed kernel parameters because the parameters of the shift operation are manually set and do not update adaptively, so it lacks flexibility. Summary of the Invention
[0007] Based on the advantages and disadvantages of the main algorithms in the current field of action recognition, the main objective of this invention is to provide an action recognition method and system based on enhanced spatiotemporal features. By adopting a strategy of sparse sampling of video, redundant information can be removed while information from different time spans can be obtained. Through channel-level motion information enhancement, weights are assigned to each feature channel of the feature map, promoting beneficial information and suppressing interference information, laying the foundation for subsequent feature extraction. Spatiotemporal context information is modeled with very low computational cost through spatiotemporal feature aggregation, and features of adjacent frames are fused in an adaptive manner. Through multi-branch output, high-level local features are further extracted from the video, supplementing the global features extracted by the single output layer of the original backbone network.
[0008] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The present invention discloses an action recognition method based on enhanced spatiotemporal features, comprising the following steps:
[0009] S1. Determine the dataset; Video datasets for action recognition are mainly divided into two categories: The first category contains a large amount of spatial information, such as UCF101, HMDB51, Sports-1M, and Kinetics-400. In this type of dataset, some actions can be recognized with just one frame, such as "playing golf," "getting a haircut," and "weightlifting." Therefore, recognition of this type of dataset mainly relies on spatial information and does not fully utilize the motion information contained between frames. The second category has very limited appearance information, including datasets such as Jester, Something-Something V1 & V2. In this case, relying on a single frame is insufficient for action recognition, and adjusting the frame order will affect the recognition results. For example, for the action "place something in front of something," if the frame sequence is reversed, it becomes the action "remove something and reveal something behind it." Therefore, when using the second type of dataset, extracting the temporal relationship between frames is crucial, and the focus must be on extracting motion features. The goal of this method is to effectively extract features in the time dimension, focusing on identifying the characteristics of the motion itself and reducing the reliance on appearance information in the background. Therefore, the Something-Something V1 dataset was chosen.
[0010] S2. Obtain the input data, use sparse sampling to divide the video V into T segments, randomly sample one frame from each of the T segments and crop it to a uniform size, denoted as T. t The final input obtained from a video is represented as X. in = [V1, V2, ... V T ], X in ∈R T*C*H*W Where T is the value in the time dimension, C is the number of channels, and H and W are the height and width of the cropped frame, respectively. Existing technologies sample consecutive frames in the video as input data, but there is a lot of redundancy between consecutive frames, and dense sampling increases the computational cost. Therefore, this invention uses coefficient sampling instead of dense sampling, which can remove redundant information and sample frames in different time spans.
[0011] S3. Implement channel-level motion information enhancement, the specific steps of which are as follows:
[0012] S31. Pass the input data through a 1*1 2D convolutional layer Conv sp The feature map X = Conv is obtained. sp (X in ), X∈R T*C / r*H*W , where r = 16;
[0013] S32. Segment X along the T dimension to obtain the feature map x corresponding to the t-th frame. t ∈R 1*C / r*H*W ;
[0014] S33, x t+1 (t < T) A 3*3 2D convolutional layer Conv1 is fed in, and the difference between the feature maps of two adjacent frames is calculated as the motion feature at time t: Δx t =Conv1(x t+1 )-x t Δx t ∈R 1*C / r*H*W Δx corresponding to time T T From the original feature map x of frame T T Directly copy to obtain all Δx t (1≤t≤T) concatenated along dimension T yields feature map M, M∈R T*C / r*H*W This ensures that the time dimension remains consistent with the dimension before the difference is calculated;
[0015] S34. Able to calculate the weights of each channel and use a global average pooling layer to compress global spatial information into channel descriptors: Where M S ∈R T*C / r*1*1 ;
[0016] S35, M S Features are passed through another 1*1 2D convolutional layer Conv ex The number of channels is restored to C, and sigmoid(δ) is used as the activation function to calculate the weight of each channel. Here, the original range of weights is extended from [0, 1] to [-1, 1].
[0017] S=2δ*Conv ex (M S )-1, S∈R T*C*1*1
[0018] S36. The weights are multiplied by the input feature map channel by channel to adjust the response values of each channel's feature values. A residual structure is used to enhance motion information without losing a large amount of background information, ultimately yielding the output of the channel-level motion information enhancement module: in Multiply by channel;
[0019] S4. Implement spatiotemporal feature aggregation. After enhancing motion-related features through the channel-level motion information enhancement module, the spatiotemporal feature aggregation module is then used to achieve the goal of modeling spatiotemporal context information with very low computational cost. The specific steps are as follows:
[0020] S41. Change feature map X O ∈R T*C*H*W The shape is X O →X re ∈R HW*C*T ;
[0021] S42, in X re Apply a 1D convolution with a kernel size of 3 channels along the time T dimension: in Let be the kernel weights belonging to channel c, i∈{-1, 0, 1}. This means that the features of frame 1 interact with the features of frames 2 and 3. This approach has two advantages. First, compared to manually fixing parameters with a shift operation, using a one-dimensional convolution in the T direction at the channel level achieves the same effect as the current frame's feature map containing information from adjacent frames. Furthermore, it adaptively learns how to fuse features to achieve the best results, reducing the randomness of manually setting parameters. Second, compared to using optical flow or 3D convolution to obtain spatiotemporal features, using 1D convolution saves significant computational costs. Finally, the changed shape is converted to a regular feature map, resulting in the final output of the spatiotemporal feature aggregation module:
[0022] S5. Perform multi-branch output;
[0023] S6. Design model training details: During the model training phase, sample 8 frames for each video in the training set. For each frame, first adjust the length of its shorter side to 256, and then use center cropping to obtain the final input size of 224*224. Train the network for a total of 50 epochs. The initial learning rate is set to 0.0025. At epochs 30, 40, and 45, change the learning rate to one-tenth of the previous learning rate. Use SGD with momentum of 0.9 and weight decay of 1e-4 on a Tesla K80 GPU with a minimum batch size of 16 to train the network model. Validate the model with a validation set after each epoch.
[0024] Furthermore, in S1, the Something-Something V1 dataset was selected, which includes everyday actions involving interactions with common objects. V1 includes 108,499 video clips with 174 action categories.
[0025] Furthermore, in S5, the multi-branch output uses three classification branches, and the three classification branches are trained together to obtain the final classification result.
[0026] Furthermore, the outputs of the three classification branches include:
[0027] The first output is Out mean This is a common output layer structure in the ResNet50 network. It uses a global average pooling layer before the fully connected layer to aggregate the spatiotemporal information of all input frames of each video and capture the global information of the video. However, this also limits the classifier's ability to learn local features.
[0028] The second output is Out max , by F max Obtained through Output Layers;
[0029] The third output is Out channel Considering that although the fine-grained feature extractor captures N*C local features, it cannot guarantee that N local features are extracted for each category, a separate branch is opened after the global max pooling layer Max, using a cross-channel pooling layer. After averaging every N responses, they are naturally divided into C categories; Out max and Out channel The design aims to learn local features containing key information in videos, with the goal of... mean The information is complementary, and the two outputs share the network in the first half of the computation. This allows the N convolutional kernels to pay more attention to local features specific to this category during network training.
[0030] Among them, Fmax A global max pooling layer Max extracts the part with the largest response among all features, N is the number of local features extracted from each class, C is the total number of classes in the video, and the output layers consist of a 1*1 2D convolutional layer, a global average pooling layer, a fully connected layer, and a loss function layer.
[0031] Furthermore, F max =Max(Conv L (F)), F max The specific steps are as follows:
[0032] S51. Convert the output feature map identity of the fourth EST block in the network. y4 Perform bilinear upsampling and compare it with the output feature map of the third EST block. y3 The feature map F is obtained by concatenating along the channel dimension. This can provide a feature map with more fine details, which provides conditions for further extraction of local information from the feature map.
[0033] S52, Using a 1*1 2D convolutional layer Conv L As a fine-grained feature extractor, the number of convolution kernels is set to N*C, where N is the number of local features extracted from each class and C is the total number of classes in the video. Therefore, the goal of the fine-grained feature extractor is to detect N specific local features for each of the C classes.
[0034] S53. A global max pooling layer Max is used to obtain the part with the largest response among all features as the local feature F extracted by the network. max .
[0035] Furthermore, regarding Out mean Out max and Out channel Both use the standard cross-entropy loss function, defined as loss respectively. mean loss max and loss channel This allows each output to be learned independently, and the final training loss is obtained by directly summing these three losses: loss = loss mean +loss max +loss channel .
[0036] This invention also discloses an action recognition system based on enhanced spatiotemporal features, comprising:
[0037] The input module is used to process video data by sparsely sampling the video data and sampling frames from different time spans.
[0038] The channel-level motion information enhancement module is used to calculate the motion features of the feature map and uses an attention network to assign weights to each channel according to the richness of motion information.
[0039] The spatiotemporal feature aggregation module is used to model the spatiotemporal context after enhancing motion information and to fuse features from adjacent frames in an adaptive manner.
[0040] The multi-branch output module extracts advanced local features from the video, supplementing the global features extracted by the single output layer of the original backbone network.
[0041] Existing technologies sample consecutive frames in a video as input data, but there is a lot of redundancy between consecutive frames, and dense sampling increases computational costs. Therefore, this invention uses coefficient sampling instead of dense sampling in the input module, which can remove redundant information and sample frames from different time spans.
[0042] This invention utilizes a channel-level motion information enhancement module. Firstly, it integrates the process of calculating differences into network modeling, enabling end-to-end network training. Secondly, feature maps convey more semantic information than RGB frames, and operations on feature maps can further extract high-level features. Generally, different channels of a feature map extract different features. To selectively enhance useful features and suppress useless features, this invention calculates motion features at the feature map level and uses them as input to an attention module, adaptively adjusting the feature response values of each channel.
[0043] This invention utilizes a spatiotemporal feature aggregation module. First, compared to the manually set parameter shift operation in TSM, it employs a channel-level one-dimensional convolution in the temporal dimension, achieving the same effect of the current frame's feature map containing information from adjacent frames. Furthermore, this is an adaptive learning process, reducing the randomness of manually set parameters. Second, compared to using optical flow or 3D convolution to obtain spatiotemporal features, our use of 1D convolution saves significant computational costs.
[0044] The multi-branch output module of this invention uses a 1*1 2D convolutional kernel as a fine-grained feature extractor. The number of convolutional kernels is set to N*C, which is used to detect N specific local features in each of the C categories. Then, a global max pooling layer is used to obtain the part with the largest response among all features as the second output. However, this output cannot guarantee that every N local features belong to the same category. To solve this problem, a 1*1 convolution is used after the global max pooling layer to calculate the average of the N*C responses for every N features. The result is naturally divided into C categories, and a third output is obtained.
[0045] Compared to existing technologies, this invention offers at least the following advantages: This invention employs deep learning to extract features from actions in videos for classification. First, by using a sparse sampling strategy for the video, redundant information can be removed while information from different time spans can be obtained. Second, channel-level motion enhancement assigns weights to each feature channel of the feature map, promoting beneficial information and suppressing interfering information, laying the foundation for subsequent feature extraction. Third, spatiotemporal feature aggregation models spatiotemporal context information with low computational cost, adaptively fusing features from adjacent frames. Finally, multi-branch output further extracts high-level local features from the video, supplementing the global features extracted by the single output layer of the original backbone network. Attached Figure Description
[0046] Figure 1 This is a flowchart of the action recognition method based on enhanced spatiotemporal features of the present invention;
[0047] Figure 2 This is a diagram illustrating the overall framework of the action recognition method based on enhanced spatiotemporal features of the present invention. Detailed Implementation
[0048] To make the present invention easier to understand, the present invention will be further described below with reference to the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein, nor does it constitute any limitation on the present invention.
[0049] like Figures 1-2 As shown, this embodiment uses the ResNet50 network as the basic backbone network, and inserts the channel-level motion information enhancement module and the spatiotemporal feature aggregation module into each ResNet-Block to form the basic module ESTblock of this method.
[0050] Step 1: Determine the dataset.
[0051] Video datasets for action recognition are mainly divided into two categories: The first category contains a large amount of spatial information, such as UCF101, HMDB51, Sports-1M, and Kinetics-400. In this type of dataset, some actions can be recognized with just one frame, such as "playing golf," "getting a haircut," and "weightlifting." Therefore, recognition of this type of dataset mainly relies on spatial information and does not fully utilize the motion information contained between frames. The second category has very limited appearance information, including datasets such as Jester, Something-Something V1 & V2. In this case, relying solely on a single frame is insufficient for action recognition, and adjusting the frame order can affect the recognition results. For example, for the action "placing something in front of something," reversing the frame sequence would result in the action "removing something and revealing something behind it." Therefore, when using the second type of dataset, extracting the temporal relationship between frames is crucial, and the focus must be on extracting motion features. The goal of this embodiment is to effectively extract features in the temporal dimension, emphasizing the characteristics of the motion itself and reducing reliance on appearance information in the background. Therefore, the Something-Something V1 dataset is selected. Specifically, the Something-Something V1 dataset contains everyday actions involving interactions with common objects, focusing on the movement itself without distinguishing between the manipulated objects. V1 includes 108,499 video clips with 174 action categories.
[0052] Step 2: Obtaining input data.
[0053] Compared to existing technologies that sample consecutive frames from a video as input data, which introduce significant redundancy between frames and increase computational costs through dense sampling, this embodiment uses sparse sampling instead of dense sampling. This removes redundant information while still sampling frames across different time spans. Specifically, the video V is divided into T segments, and a frame is randomly sampled from each of the T segments and cropped to a uniform size, denoted as T. t The final input obtained from a video is represented as X. in = [V1, V2, ... V T ], X in ∈R T*C*H*W , where T is the value in the time dimension, C is the number of channels, and H and W are the height and width of the cropped frame, respectively.
[0054] Step 3: Implement channel-level motion information enhancement (CME).
[0055] First, the input data is passed through a 1x1 2D convolutional layer. sp Initially, spatial features are extracted and the feature map is reduced in dimensionality to obtain the feature map X = Conv.sp (X in ), X∈R T*C / r*H*W , where r = 16.
[0056] Then X is segmented along the T dimension to obtain the feature map x corresponding to the t-th frame. t ∈R 1*C / r*H*W .
[0057] Next, x t+1 (t < T) A 3*3 2D convolutional layer Conv1 is fed in, and the difference between the feature maps of two adjacent frames is calculated as the motion feature at time t:
[0058] Δx t =Conv1(x t+1 )-x t Δx t ∈R 1*C / r*H*W
[0059] In this embodiment, Δx is defined as the time T. T From the original feature map x of frame T T Obtained by direct copying. All Δx... t (1≤t≤T) concatenated along dimension T yields feature map M, M∈R T*C / r*H*W This ensures that the time dimension remains consistent with the dimension before the difference is calculated.
[0060] To determine the weights for each channel, this embodiment first uses a global average pooling layer to compress global spatial information into channel descriptors:
[0061]
[0062] Where M S ∈R T*C / r*1*1 Then feature M S Through another 1*1 2D convolutional layer Conv ex The number of channels is restored to C, and sigmoid(δ) is used as the activation function to calculate the weight S of each channel. Here, the original range of weights is extended from [0, 1] to [-1, 1].
[0063] S=2δ*Conv ex (M S )-1, S∈R T*C*1*1
[0064] The final step of this module is to multiply the weights by the input feature map channel by channel, thereby adjusting the response values of each channel's feature values. This method uses a residual structure to enhance motion information without losing a significant amount of background information. The final output of the channel-level motion information enhancement module is obtained as follows:
[0065]
[0066] in Defined as multiplication by channel.
[0067] Step 4: Implement Spatiotemporal Feature Aggregation (STA).
[0068] After enhancing motion-related features through the channel-level motion information enhancement module, this method then uses a spatiotemporal feature aggregation module to achieve the goal of modeling spatiotemporal context information with very low computational cost.
[0069] For feature map X O ∈R T*C*H*W First, change its shape to X. O →X re ∈R HW*C*T Then in X re Apply a 1D convolution with a kernel size of 3 channels along the time T dimension:
[0070]
[0071] in Let represent the kernel weights belonging to channel c. Since the kernel size is 3, i ∈ {-1, 0, 1}, meaning that the features of frame t interact with the features of frames t-1 and t+1. The above method has two advantages. First, compared to manually fixing parameters with a shift operation, using a one-dimensional convolution in the T direction at the channel level can achieve the same effect as the current frame's feature map containing information from adjacent frames. Furthermore, it adaptively learns how to fuse features to achieve the best results, reducing the randomness of manually setting parameters. Second, compared to using optical flow or 3D convolution to obtain spatiotemporal features, using 1D convolution saves a significant amount of computational cost. Finally, it will change... The shape is a conventional feature map, resulting in the final output of the spatiotemporal feature aggregation module:
[0072] Step 5: Multi-branch output. This embodiment uses three classification branches in the final classification stage of the network and trains them together to obtain the final classification result. The Output Layers of this invention consist of a 1*1 2D convolutional layer, a global average pooling layer, a fully connected layer, and a loss function layer.
[0073] (1) The first output is denoted as Out. mean This is a common output layer structure in the ResNet50 network. It uses a global average pooling layer before the fully connected layer to aggregate the spatiotemporal information of all input frames of each video and capture the global information of the video. However, this also limits the classifier's ability to learn local features.
[0074] (2) Second output Out max and the third output Out channel The design aims to learn local features containing key information from the video, complementing the information from the first output. These two outputs share the network in the first half of the computation. In this embodiment, the output feature map of the fourth EST block in the network is used for identity... y4 Perform bilinear upsampling and compare it with the output feature map of the third EST block. y3 The feature map F is obtained by concatenating the features along the channel dimension. This provides a feature map with more refined details, which is beneficial for further extraction of local information from the feature map. Next, this embodiment uses a 1*1 2D convolutional layer Conv... L As a fine-grained feature extractor, the number of convolutional kernels is set to N*C, where N represents the number of local features to be extracted from each class. In this embodiment, this hyperparameter is set to 5, and C represents the total number of classes in the video. Therefore, the fine-grained feature extractor aims to detect N specific local features for each of the C classes. Then, a global max-pooling layer Max is used to obtain the part with the largest response among all features as the local feature F extracted by the network. max The above steps can be represented as:
[0075] F max =Max(Conv L (F))
[0076] The second output is Out. max It is F max This is obtained through Output Layers.
[0077] (3) Considering that although the fine-grained feature extractor Conv L A total of N*C local features were captured, but it cannot be guaranteed that N local features were extracted for each category. To address this issue, this embodiment branches out after the global max pooling layer Max, using a cross-channel pooling layer. After averaging every N responses, they are naturally divided into C categories, thus obtaining the third output Out. channel This approach allows the N convolutional kernels to focus more on local features specific to this category during network training.
[0078] For each output, this embodiment uses the standard cross-entropy loss function, defined as loss. mean loss max and loss channelThis encourages each output to be learned independently. The final loss used for training is obtained by directly summing these three losses:
[0079] loss = loss mean +loss max +loss channel
[0080] Step Six: Design Model Training Details. The videos are divided into training and validation sets. During model training, 8 frames are sampled from each video in the training set. For each frame, the shorter side length is first adjusted to 256, and then centered to obtain a final input size of 224*224. In this embodiment, the network is trained for 50 epochs with an initial learning rate of 0.0025. The learning rate is then changed to one-tenth of the previous learning rate at epochs 30, 40, and 45. This embodiment uses SGD (Stochastic Gradient Descent) with a momentum of 0.9 and a weight decay of 1e-4 on a Tesla K80 GPU with a minimum batch size of 16 to train the network model. This embodiment uses a validation set to validate the model after each epoch. The accuracy shown in the final experimental results is the highest among all validation results, where each epoch represents a complete training run using all the data from the training set.
[0081] This embodiment employs deep learning to extract features from actions in a video for classification. First, a sparse sampling strategy is used to remove redundant information while capturing information across different time spans. Second, channel-level motion enhancement assigns weights to each feature channel of the feature map, promoting useful information and suppressing interference, laying the foundation for subsequent feature extraction. Third, spatiotemporal feature aggregation models the spatiotemporal context information with low computational cost, adaptively fusing features from adjacent frames. Finally, multi-branch output further extracts high-level local features from the video, supplementing the global features extracted by the original backbone network's single output layer.
[0082] The above description, in conjunction with specific embodiments, provides a further detailed explanation of this application and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, several simple deductions or substitutions can be made without departing from the inventive concept of this application.
Claims
1. An action recognition method based on enhanced spatiotemporal features, characterized in that, Includes the following steps: S1. Determine the dataset; S2. Obtain the input data, use sparse sampling to divide the video V into T segments, randomly sample one frame from each of the T segments and crop it to a uniform size, denoted as . The final input representation obtained from a video is as follows: , where T is the value in the time dimension, C is the number of channels, and H and W are the height and width of the cropped frame, respectively; S3. Implement channel-level motion information enhancement, the specific steps of which are as follows: S31. Pass the input data through a 1*1 2D convolutional layer. , to obtain feature map , where r=16; S32, will Segmenting along the T-dimensional axis yields the feature map corresponding to the t-th frame. ; S33, will Feed a 3x3 2D convolutional layer And calculate the difference between the feature maps of two adjacent frames as the motion feature at time t: The time T corresponds to From the original feature map of frame T Copy directly to get all of them The feature map is obtained by concatenating the data along the T-dimensional axis. ; S34. Use a global average pooling layer to compress global spatial information into channel descriptors: ,in ; S35, will Features are passed through another 1x1 2D convolutional layer. Restore the number of channels to C, and As the activation function, the weights for each channel are calculated. Here, the original range of weights is extended from [0, 1] to [-1, 1]. ; S36. Multiply the weights by the input feature map by channel to obtain the output of the channel-level motion information enhancement module: ,in Multiply by channel; S4. Implement spatiotemporal feature aggregation, the specific steps are as follows: S41. Change the feature map The shape is ; S42, in Apply a 1D convolution with a kernel size of 3 channels along the time T dimension: ,in Belongs to the channel The convolution kernel weights, ; S5. Perform multi-branch output; S6, Design Model Training Details.
2. The action recognition method based on enhanced spatiotemporal features according to claim 1, characterized in that, In S1, the Something-Something V1 dataset is selected, which includes everyday actions involving interactions with common objects. V1 contains 108,499 video clips with 174 action categories.
3. The action recognition method based on enhanced spatiotemporal features according to claim 1, characterized in that, In S5, the multi-branch output uses three classification branches, and the three classification branches are trained together to obtain the final classification result.
4. The action recognition method based on enhanced spatiotemporal features according to claim 3, characterized in that, The output of the three classification branches includes: The first output is This is a common output layer structure in residual neural networks; the second output is... ,Depend on Obtained after passing through the output layer; The third output is After the global max pooling layer Max, a new branch is opened, using a cross-channel pooling layer. After averaging every N responses, they are naturally divided into C classes. in, A global max pooling layer Max extracts the part with the largest response among all features, N is the number of local features extracted from each class, C is the total number of classes in the video, and the output layer consists of a 1*1 2D convolutional layer, a global average pooling layer, a fully connected layer, and a loss function layer.
5. The action recognition method based on enhanced spatiotemporal features according to claim 4, characterized in that, The specific steps are as follows: S51. Perform bilinear upsampling on the output feature map of the fourth EST block in the ResNet50 network, and concatenate it with the output feature map of the third EST block in the channel dimension to obtain feature map F. S52, Using a 1*1 2D convolutional layer As a fine-grained feature extractor, the number of convolution kernels is set to N*C, where N is the number of local features extracted from each class and C is the total number of classes in the video; S53. A global max pooling layer Max is used to obtain the part with the largest response among all features as the local features extracted by the network. , .
6. The action recognition method based on enhanced spatiotemporal features according to claim 4, characterized in that, for , and Both use the standard cross-entropy loss function, defined as follows: , and This allows each output to be learned independently, and the final training loss is obtained by directly summing these three losses: .
7. An action recognition system based on enhanced spatiotemporal features, executing the action recognition method based on enhanced spatiotemporal features as described in claims 1-6, characterized in that, include: The input module is used to process video data by sparsely sampling the video data and sampling frames from different time spans. The channel-level motion information enhancement module is used to calculate the motion features of the feature map and uses an attention network to assign weights to each channel according to the richness of motion information. The spatiotemporal feature aggregation module is used to model the spatiotemporal context after enhancing motion information and to fuse features from adjacent frames in an adaptive manner. The multi-branch output module extracts advanced local features from the video, supplementing the global features extracted by the single output layer of the original backbone network.