A Weakly Supervised Temporal Action Localization Method Based on Adaptive Temporal Feature Enhancement
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-02
- Publication Date
- 2026-08-14
AI Technical Summary
然而，由于感受野是固定的，膨胀卷积网络仍然难以充分学习一个动作内或动作和背景之间的多种不同的时间依赖关系
[0028]本发明提供的一种基于自适应时序特征增强的弱监督时序动作定位方法，改进了现有方法中存在的时序信息利用不充分的问题。具体来说，本申请提出了一种自适应的时序特征增强的方法，通过对原始RGB和光流特征进行时序增强，实现更好的动作分类与定位性能。值得注意的是，与以前的方法相比，由于可变形卷积动态的感受野，本申请方法可以自适应地学习动作内部以及动作与背景间复杂多变的时序关系。此外，本申请引入了教师模型来更好地利用学习到时序信息，并指导模态共识的训练。本申请提高了弱监督时序动作定位任务的精度，在两个流行的数据集上比最近的方法取得了领先的性能。
Smart Images
Figure CN117372932B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of temporal action localization technology, specifically relating to a weakly supervised temporal action localization method based on adaptive temporal feature enhancement. Background Technology
[0002] Temporal action localization aims to simultaneously classify and locate action instances within an untrimmed long video, aiding many downstream video understanding tasks, including video editing and video surveillance. While fully supervised methods have achieved impressive performance, frame-level manual annotation is extremely expensive and laborious. In contrast, weakly supervised temporal action localization requires only video-level annotation and has gained increasing popularity in recent years.
[0003] Weakly supervised temporal action localization is more challenging due to the lack of frame-level annotations to aid training. Most previous methods have introduced multiple instance learning (MIL) approaches to build their frameworks. These methods first generate class activation sequences (CAS) and then leverage a top-k strategy to obtain video-level predictions. During inference, a series of action proposals are generated through thresholding and merging of CAS.
[0004] Furthermore, previous methods mostly used RGB and optical flow features extracted by the I3D model. Since this I3D model is designed for action recognition tasks, it is trained on carefully cropped short videos, resulting in features that are not well applied to weakly supervised temporal action localization tasks in uncropped long videos. Therefore, some existing methods have attempted to make these features more compatible with action localization tasks, but their performance remains limited. Recent research suggests that these performance limitations are attributed to insufficient utilization of temporal information. To overcome this limitation, recent methods employ dilated convolutional networks to expand the receptive field. This method has proven simple and effective. However, because the receptive field is fixed, dilated convolutional networks still struggle to adequately learn multiple different temporal dependencies within an action or between an action and the background. Summary of the Invention
[0005] The purpose of this invention is to provide a weakly supervised temporal action localization method based on adaptive temporal feature enhancement, so as to overcome the problems mentioned in the background art. In this application, this method is also referred to as ATFE.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] A weakly supervised temporal action localization method based on adaptive temporal feature enhancement, comprising:
[0008] A weakly supervised temporal action localization network is constructed, which includes a feature extractor, an adaptive temporal dependency capture module, a teacher-guided modality consensus module, and a classifier.
[0009] Input a video frame sequence into the feature extractor to obtain optical flow features and RGB features;
[0010] The optical flow features are input into the adaptive temporal-dependent capture module to obtain enhanced optical flow features and corresponding optical flow attention sequences;
[0011] The adaptive temporal dependency capture module is used as the student model to update the teacher model in the modal consensus module of the teacher guidance.
[0012] Optical flow features and RGB features are input into the teacher-guided modal consensus module to obtain enhanced RGB features and corresponding RGB attention sequences;
[0013] The enhanced optical flow features and the enhanced RGB features are concatenated and then input into a classifier to obtain the class activation sequence output by the classifier. The weakly supervised temporal action localization network is then trained based on the class activation sequence using a multiple instance learning method.
[0014] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.
[0015] Preferably, the adaptive temporal dependency capture module includes several temporally deformable convolutional layers and attention units corresponding to each temporally deformable convolutional layer. The step of inputting optical flow features into the adaptive temporal dependency capture module to obtain enhanced optical flow features and corresponding optical flow attention sequences includes:
[0016] After the input optical flow features pass through several temporally deformable convolutional layers, the output of the last temporally deformable convolutional layer is used as the enhanced optical flow features.
[0017] The input optical flow features are passed through several temporally deformable convolutional layers and attention units of each temporally deformable convolutional layer. The average value of the attention sequences output by each attention unit is taken as the optical flow attention sequence.
[0018] Preferably, the temporally deformable convolutional layer consists of a temporally deformable convolution, a ReLU function, and a Dropout function, wherein the temporally deformable convolution includes two one-dimensional convolutions.
[0019] Preferably, when the adaptive temporal dependency capture module includes two temporally deformable convolutional layers and two attention units, the operation of the adaptive temporal dependency capture module is as follows:
[0020] The first temporally deformable convolutional layer receives the input optical flow features. The output of the first temporally deformable convolutional layer is connected to the second temporally deformable convolutional layer. The output of the first temporally deformable convolutional layer is multiplied element-wise with the optical flow features after passing through a sigmoid function. The result of this multiplication is then input into the first attention unit to obtain the first attention sequence. The output of the second temporally deformable convolutional layer is also multiplied element-wise with the optical flow features after passing through a sigmoid function. The result of this multiplication is then used as an enhanced optical flow feature and simultaneously input into the second attention unit to obtain the second attention sequence. The first and second attention sequences are then averaged to obtain the optical flow attention sequence.
[0021] Preferably, the step of using the adaptive temporal dependency capture module as the student model to update the teacher model in the modal consensus module of the teacher guidance includes:
[0022] The weights in the teacher model and the student model correspond one-to-one. For each weight, the weight value applied to the teacher model is obtained by exponential weighted averaging based on the weight value in the student model.
[0023] Preferably, the step of inputting optical flow features and RGB features into the teacher-guided modal consensus module to obtain enhanced RGB features and corresponding RGB attention sequences includes:
[0024] The optical flow features are input into the teacher module of the teacher-guided modal consensus module to obtain the teacher-guided enhanced optical flow features;
[0025] Teacher-guided enhanced optical flow features and RGB features are input into a shared convolutional layer. The teacher-guided enhanced optical flow features and RGB features output from the shared convolutional layer are then multiplied element-wise with the original RGB features to obtain the enhanced RGB features.
[0026] The enhanced RGB features are input into the attention unit to obtain the corresponding RGB attention sequence.
[0027] As a preferred approach, a mutual learning operation is introduced to align the attention weights of the optical flow attention sequence and the RGB attention sequence, enabling them to learn from each other.
[0028] This invention provides a weakly supervised temporal action localization method based on adaptive temporal feature enhancement, which improves upon the problem of insufficient utilization of temporal information in existing methods. Specifically, this application proposes an adaptive temporal feature enhancement method that enhances the original RGB and optical flow features temporally, achieving better action classification and localization performance. Notably, compared to previous methods, due to the dynamic receptive field of deformable convolution, this application's method can adaptively learn complex and varied temporal relationships within actions and between actions and the background. Furthermore, this application introduces a teacher model to better utilize the learned temporal information and guide the training of modality consensus. This application improves the accuracy of weakly supervised temporal action localization tasks, achieving state-of-the-art performance on two popular datasets compared to recent methods. Attached Figure Description
[0029] Figure 1 The flowchart of the weakly supervised temporal action localization method based on adaptive temporal feature enhancement in this application is shown below.
[0030] Figure 2 This is a schematic diagram of the temporally deformable convolution effect in this application. Detailed Implementation
[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.
[0033] In one embodiment, such as Figure 1 As shown, a weakly supervised temporal action localization method based on adaptive temporal feature enhancement is proposed, including:
[0034] Step S1: Construct a weakly supervised temporal action localization network, which includes a feature extractor, an adaptive temporal dependency capture module, a teacher-guided modality consensus module, and a classifier.
[0035] The feature extractor is used to extract RGB features and optical flow features; the adaptive temporal dependency capture module is used to capture the complex and ever-changing temporal relationships within actions and between actions and the background; the teacher-guided modal consensus module guides the training of modal consistency by introducing a teacher model; and the classifier is used to generate class activation sequences (CAS) for training based on the multi-instance learning (MIL) method.
[0036] This application employs the popular I3D (Inflated 3D ConvNet) network as a feature extractor to process the input video frame sequence and output RGB features F. RGB and optical flow characteristics F flow All meet Where T represents the number of snippets, and each snippet is composed of 16 non-overlapping frames; D represents the number of feature dimensions, which is 1024 in this embodiment.
[0037] Step S2: Apply optical flow feature F flow The enhanced optical flow features are obtained by inputting them into the adaptive temporal-dependent capture module. And the corresponding optical flow attention sequence A flow .
[0038] The adaptive temporal dependency capture module utilizes temporal deformable convolution (TDC) to capture the temporal dependencies required for actions, thereby enhancing optical flow features. The input optical flow feature F... flow After several temporally deformable convolutional layers, the output of the last temporally deformable convolutional layer is used as the enhanced optical flow feature.
[0039] The optical flow feature F flow The input is fed into several temporally deformable convolutional layers and the attention units of each layer to generate corresponding attention sequences. The average of the attention sequences from each layer is taken as the optical flow attention sequence A. flow .
[0040] The temporally deformable convolutional layer consists of a temporally deformable convolution, a ReLU function, and a Dropout function. Temporally deformable convolutions mainly consist of two one-dimensional convolutions, such as... Figure 2 As shown, the standard sampling position centered on the i-th snippet is denoted as . Where K is the kernel length, and then a one-dimensional convolution conv1 (with the video length as the input dimension and the kernel length as the output dimension) is used to measure the temporal offset Δ of the segment. r :
[0041] Δ r =f conv1 (P i )
[0042] Compared with the original sampling position P i Adding them together gives Then it is important to note that due to the learned Δ r For decimals, linear interpolation is used to re-represent the sampling positions of integers. Finally, another one-dimensional convolution, conv2 (input dimension is the video length, output dimension is the video length), aggregates information from the offset sampling segments, ultimately yielding a temporally deformable convolution for P. i The output F i as follows:
[0043]
[0044] This embodiment uses temporally deformable convolution. At the original sampling position, a one-dimensional convolution (input dimension is the video length, output dimension is the kernel length) is used to measure the temporal offset of the sampled segment, and this offset is added to the original sampling position. Then, linear interpolation is used to re-represent the sampling position. Finally, another one-dimensional convolution (input dimension is the video length, output dimension is the video length) is used to aggregate information from the offset sampled segment. Additionally, the attention unit is a common technique in this field, typically consisting of two convolutional layers, and will not be elaborated upon in this embodiment.
[0045] For ease of understanding, this embodiment assumes that the adaptive temporal dependency capture module includes two temporally deformable convolutional layers and two attention units. The operation of the adaptive temporal dependency capture module is as follows:
[0046] The first temporally deformable convolutional layer receives the input optical flow features. The output of the first temporally deformable convolutional layer is connected to the second temporally deformable convolutional layer. The output of the first temporally deformable convolutional layer is multiplied element-wise with the optical flow features after passing through a sigmoid function. The result of this multiplication is then input into the first attention unit to obtain the first attention sequence. The output of the second temporally deformable convolutional layer is also multiplied element-wise with the optical flow features after passing through a sigmoid function. The result of this multiplication is then used as an enhanced optical flow feature and simultaneously input into the second attention unit to obtain the second attention sequence. The first and second attention sequences are then averaged to obtain the optical flow attention sequence.
[0047] Step S3: Transfer RGB features F RGB The modal consensus module input to the teacher guidance obtains enhanced RGB features. and the corresponding RGB attention sequence A RGB .
[0048] This application updates network parameters through backpropagation during network training, but does not directly update the teacher model. After each update of the weakly supervised temporal action localization network, the teacher model is updated based on the student model. The mean-teacher method is used to generate the teacher model, which is updated from the weights of the student model using an exponentially weighted average (EMA). Specifically, the adaptive temporal dependency capture module is used as the student model, and the weights in the teacher model and student model correspond one-to-one. For each weight, the weight value applied to the teacher model is obtained based on the weight value in the student model through an exponentially weighted average operation.
[0049] The updated teacher model targets the output optical flow feature F flow Enhanced optical flow characteristics under the guidance of teachers Then, under the guidance of teachers, the optical flow characteristics were enhanced. and RGB features F RGB The two features are input into a shared convolutional layer to align their weight distributions. The enhanced RGB features are obtained by element-wise multiplying the teacher-guided enhanced optical flow features and RGB features output from the shared convolutional layer with the original RGB features.
[0050] Next, the RGB features will be enhanced. The input is fed into the attention unit to obtain the corresponding attention sequence A. RGB For optical flow attention sequence A flow and RGB attention sequence A RGB Introducing mutual learning operations allows A RGB and A flow Align attention weights to learn each other's knowledge. The mutual learning loss is calculated as follows:
[0051]
[0052] In the formula, L ml For mutual learning losses, among which, Let δ(·) denote a function that truncates the input gradient, and let δ(·) denote a similarity metric function. α is a hyperparameter. In the equation, A... RGB and A flow These are considered as pseudo-labels to each other so that they can learn from each other and align attention weights. In this embodiment, the function δ(·) in the equation uses mean squared error (MSE).
[0053] Step S4: Enhance RGB features and enhanced optical flow characteristics Features are obtained by splicing. and features The class activation sequence CAS is obtained by inputting into the classifier. The network is trained by a training method based on multiple instance learning. The training method based on multiple instance learning is a relatively mature technology in this field, and will not be elaborated here.
[0054] Experimental data from this application demonstrate that the method described herein achieves better positioning accuracy compared to other existing methods. The experimental data are shown in Tables 1 and 2 below:
[0055] Table 1 compares the performance of our method with other methods on the Thumos14 dataset.
[0056]
[0057]
[0058] Table 1 shows the performance comparison between the proposed method and other methods on the Thumos14 dataset. mAP@IoU represents the average accuracy at different IoU thresholds, and AVG represents the average mAP at multiple IoU thresholds of 0.1:0.1:0.5 and 0.1:0.1:0.7. It can be seen that the proposed method significantly outperforms previous methods in almost all metrics.
[0059] Table 2 compares the performance of our method with other methods on the ActivityNet 1.2 dataset.
[0060]
[0061] As shown in Table 2, on the ActivityNet 1.2 dataset, mAP@IoU represents the average accuracy at different IoU thresholds, and AVG represents the average mAP at IoU thresholds of 0.5:0.05:0.95. Our method still achieves state-of-the-art performance in terms of average mAP.
[0062] The full texts and source articles for each of the other existing methods listed in Tables 1 and 2 are as follows:
[0063] The full name of CoLA(2021) is Contrastive learning to Localize Actions, which originates from the article Can Zhang, Meng Cao, Dongming Yang, Jie Chen, and Yuexian Zou, "Cola: Weakly-supervised temporal action localization with snippet contrastive learning," inCVPR, 2021, pp.16010–16019.;
[0064] FAC-Net (2021) stands for Foreground-action consistency network, which originates from the paper Linjiang Huang, Liang Wang, and Hongsheng Li, “Foreground-action consistency network for weakly supervised temporal action localization,” in ICCV, 2021, pp. 8002–8011.
[0065] ACGNET (2022) stands for Action Complement Graph Network, derived from the article Zichen Yang, Jie Qin, and Di Huang, “Acgnet: Action Complement Graph Network for Weakly-Supervised Temporal Action Localization,” in AAAI, 2022, vol. 36, pp. 3090–3098.
[0066] The full name of FTCL(2022) is Fine-grained temporal contrastive learning, which originates from the article Junyu Gao, Mengyuan Chen, and Changsheng Xu, "Fine-grained temporal contrastive learning for weaklysupervised temporal action localization," in CVPR, 2022, pp.19999–20009.;
[0067] DCC (2022) stands for denoised cross-video contrast, which originates from the article Jingjing Li, Tianyu Yang, Wei Ji, Jue Wang, and Li Cheng, “Exploring denoised cross-video contrast for weakly-supervised temporal action localization,” in CVPR, 2022, pp. 19914–19924.
[0068] RSKP (2022) stands for representative snippet knowledge propagation, which originated from Linjiang Huang, Liang Wang, and Hongsheng Li, “Weakly supervised temporal action localization via representative snippet knowledge propagation,” in CVPR, 2022, pp. 3272–3281.
[0069] The full name of DFE (2023) is Dual-feature enhancement, which originates from the article Siying Liu, Qiankun Liu, Qi Chu, Bin Liu, and Nenghai Yu, “Dual-feature enhancement for weakly supervised temporal action localization,” in ICASSP.IEEE, 2023, pp. 1–5.
[0070] TFE-DCN (2023) stands for Temporal feature enhancement dilated convolution network, which originates from the paper Jianxiong Zhou and Ying Wu, “Temporal feature enhancement dilated convolution network for weakly supervised temporal action localization,” in WACV, 2023, pp. 6028–6037.
[0071] P-MIL (2023) stands for Proposal-based multiple instance learning, which originates from the article Huan Ren, Wenfei Yang, Tianzhu Zhang, and Yongdong Zhang, “Proposal-based multiple instance learning for weakly-supervised temporal action localization,” in CVPR, 2023, pp. 2394–2404.
[0072] TS-Net (2023) stands for Two-stream networks, derived from the paper Yu Wang, Yadong Li, and Hongbin Wang, “Two-stream networks for weakly-supervised temporal action localization with semantic-aware mechanisms,” in CVPR, 2023, pp. 18878–18887.
[0073] Bas-Net (2020) stands for Background suppression network, which originates from the article Pilhyeon Lee, Youngjung Uh, and Hyeran Byun, “Background suppression network for weakly-supervised temporal action localization,” in AAAI, 2020, vol. 34, pp. 11320–11327.
[0074] CO2-Net (2021) stands for Cross-modal consensus network, derived from the paper Fa-TingHong, Jia-Chang Feng, Dan Xu, Ying Shan, and Wei-Shi Zheng, “Cross-modal consensus network for weakly supervised temporal action localization,” in ACM MM, 2021, pp. 1591–1599.
[0075] The full name of D2-Net (2021) is discriminative embeddings and denoisedactivations, derived from the article Sanath Narayan, Hisham Cholakkal, Munawar Hayat, FahadShahbaz Khan, Ming-Hsuan Yang, and Ling Shao, "D2-net: Weakly-supervised action localization via discriminative embeddings and denoised activations," in ICCV,2021,pp.13608–13617.;
[0076] P-MIL (2023) stands for Proposal-based multiple instance learning, which originates from the article Huan Ren, Wenfei Yang, Tianzhu Zhang, and Yongdong Zhang, “Proposal-based multiple instance learning for weakly-supervised temporal action localization,” in CVPR, 2023, pp. 2394–2404.
[0077] This invention improves upon existing methods by temporally enhancing the original RGB and optical flow features, addressing the issue of insufficient utilization of temporal information and achieving better action classification and localization performance. Notably, compared to previous methods, due to the dynamic receptive field of deformable convolution, this invention can adaptively learn complex and varied temporal relationships within actions and between actions and the background. Furthermore, this invention introduces a teacher model to better utilize the learned temporal information and guide the training of modality consensus. This application improves the accuracy of weakly supervised temporal action localization tasks, achieving state-of-the-art performance on two popular datasets.
[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0079] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.
Claims
1. A weakly supervised temporal action localization method based on adaptive temporal feature enhancement, characterized in that, The weakly supervised temporal action localization method based on adaptive temporal feature enhancement includes: A weakly supervised temporal action localization network is constructed, which includes a feature extractor, an adaptive temporal dependency capture module, a teacher-guided modality consensus module, and a classifier. Input a video frame sequence into the feature extractor to obtain optical flow features and RGB features; The optical flow features are input into the adaptive temporal-dependent capture module to obtain enhanced optical flow features and corresponding optical flow attention sequences; The adaptive temporal dependency capture module is used as the student model to update the teacher model in the modal consensus module of the teacher guidance. Optical flow features and RGB features are input into the teacher-guided modal consensus module to obtain enhanced RGB features and corresponding RGB attention sequences; The enhanced optical flow features and the enhanced RGB features are concatenated and then input into the classifier to obtain the class activation sequence output by the classifier. The weakly supervised temporal action localization network is trained based on the class activation sequence using a multiple instance learning method. The adaptive temporal dependency capture module includes several temporally deformable convolutional layers and attention units corresponding to each temporally deformable convolutional layer. The step of inputting optical flow features into the adaptive temporal dependency capture module to obtain enhanced optical flow features and corresponding optical flow attention sequences includes: After the input optical flow features pass through several temporally deformable convolutional layers, the output of the last temporally deformable convolutional layer is used as the enhanced optical flow features. The input optical flow features are passed through several temporally deformable convolutional layers and attention units of each temporally deformable convolutional layer. The average value of the attention sequences output by each attention unit is taken as the optical flow attention sequence. The step of inputting optical flow features and RGB features into the modal consensus module for teacher guidance to obtain enhanced RGB features and corresponding RGB attention sequences includes: The optical flow features are input into the teacher module of the teacher-guided modal consensus module to obtain the teacher-guided enhanced optical flow features; Teacher-guided enhanced optical flow features and RGB features are input into a shared convolutional layer. The teacher-guided enhanced optical flow features and RGB features output from the shared convolutional layer are then multiplied element-wise with the original RGB features to obtain the enhanced RGB features. The enhanced RGB features are input into the attention unit to obtain the corresponding RGB attention sequence.
2. The weakly supervised temporal action localization method based on adaptive temporal feature enhancement according to claim 1, characterized in that, The temporally deformable convolutional layer consists of a temporally deformable convolution, a ReLU function, and a Dropout function. The temporally deformable convolution includes two one-dimensional convolutions.
3. The weakly supervised temporal action localization method based on adaptive temporal feature enhancement according to claim 1, characterized in that, When the adaptive temporal dependency capture module includes two temporally deformable convolutional layers and two attention units, the operation of the adaptive temporal dependency capture module is as follows: The first temporally deformable convolutional layer receives the input optical flow features. The output of the first temporally deformable convolutional layer is connected to the second temporally deformable convolutional layer. The output of the first temporally deformable convolutional layer is multiplied element-wise with the optical flow features after passing through a sigmoid function. The result of this multiplication is then input into the first attention unit to obtain the first attention sequence. The output of the second temporally deformable convolutional layer is also multiplied element-wise with the optical flow features after passing through a sigmoid function. The result of this multiplication is then used as an enhanced optical flow feature and simultaneously input into the second attention unit to obtain the second attention sequence. The first and second attention sequences are then averaged to obtain the optical flow attention sequence.
4. The weakly supervised temporal action localization method based on adaptive temporal feature enhancement according to claim 1, characterized in that, The step of using the adaptive temporal dependency capture module as the student model to update the teacher model in the modal consensus module of the teacher guidance includes: The weights in the teacher model and the student model correspond one-to-one. For each weight, the weight value applied to the teacher model is obtained by exponential weighted averaging based on the weight value in the student model.
5. The weakly supervised temporal action localization method based on adaptive temporal feature enhancement according to claim 1, characterized in that, The mutual learning operation is introduced to align the attention weights of the optical flow attention sequence and the RGB attention sequence, enabling them to learn from each other.