Weakly supervised temporal action localization method based on mask pyramid enhancement

Abstract

The application discloses a weakly supervised temporal action localization method based on mask pyramid enhancement, and belongs to the fields of computer vision and action localization. System modules for realizing the method include a feature extraction module, a mask pyramid classification module MPCM for action integrity learning, and an attention module FDAM for feature discriminative learning. The feature extraction module is used for feature extraction and serves as input data of subsequent links; the MPCM constructs a refined enhanced mask pyramid by using a temporal adaptive enhancement mechanism TAEM, and introduces a mask strategy driven model in hierarchical processing to mine new feature regions complementary to a target action. Perception fusion is performed on classification results of different levels to generate an enhanced class activation sequence CAS. The MPENet introduces an exclusive loss in the FDAM, helps the model focus on key parts of the target action, thereby guiding the model to learn appropriate attention distribution, and improving the understanding and distinguishing ability of the model to features. Finally, the CAS is weighted with attention to obtain a prediction result of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision and action localization, and specifically relates to a weakly supervised temporal action localization method based on deep learning technology, combined with a time sequence adaptive enhancement mechanism and a mask strategy, and formed by a mask pyramid enhancement of joint visual features, which is suitable for intelligent monitoring, abnormal detection, visual question answering and the like. BACKGROUND

[0002] With the popularity of electronic shooting equipment and the rapid development of multimedia networks, video data on network platforms is growing explosively. For video temporal action localization, traditional fully supervised learning methods rely on precise temporal boundary labeling for training and learning, which is time-consuming and laborious in labeling, limiting its promotion in practical applications. Therefore, many studies have turned to weakly supervised learning, which only needs video-level labels in the training process, but can locate the temporal boundary and class attribute of the action in the test stage. Weakly supervised learning greatly reduces human resources, time cost and labeling errors, and has good flexibility.

[0003] In the weakly supervised temporal action localization (WTAL) task, the action instances in the untrimmed video usually have different time spans, from several seconds to tens of minutes. The current mainstream method is to use the classification-localization paradigm, which relies on the highly abstract semantic information of single-scale feature maps to analyze actions of different lengths, and generates a segment-level class activation score (CAS) through classification supervision. Then, the action proposal is generated by thresholding, and the CAS represents the probability of action occurrence. It can be seen that the quality of CAS determines the upper limit of the model performance. However, single-scale feature modeling often ignores some action details, and there is a deviation in the optimization goal between the classification and localization tasks. The former emphasizes promoting the distinguishability of action features, and the latter focuses on ensuring the integrity of the action region. Since the model must be balanced between the two, it leads to fluctuations in CAS, which in turn causes inaccurate prediction of action boundaries, resulting in incomplete or over-complete action localization generated by the model. How to completely locate the action instance is an important challenge faced by weakly supervised learning.

[0004] The application considers that constructing a pyramid structure along the time dimension of the extracted feature sequence can extract action information of different scales from different levels, and the low-level pyramid contains more detailed and rapidly changing information of the action, while the high-level pyramid can capture more extensive context or slower action changes. This helps to more comprehensively describe the video content and enrich the expression of action features. However, in WTAL, although the pyramid structure has advantages in providing multi-scale expression and rich action features, the classification supervision makes the model tend to learn only the high-discriminative feature regions and ignore the less distinctive parts. The experiments in the latter part of the application also verify that simply using the pyramid structure does not bring any gain to the model, further supporting the conclusion that the simple pyramid structure is not suitable for the WTAL task. Therefore, in order to fully utilize the characteristics of the pyramid structure and make it better suitable for WTAL, the application combines the mask strategy with the temporal pyramid to propose a mask pyramid (MP). SUMMARY

[0005] The application proposes a mask pyramid enhanced network (MPENet) to explore the feature distribution of action instances in the video and understand the structure and evolution process of the action, and to realize further accurate positioning. MPENet first uses a temporal adaptive enhancement mechanism (TAEM) to construct a refined enhanced mask pyramid, and introduces a mask strategy to drive the model to excavate new feature regions complementary to the target action during hierarchical processing. MPNet generates an enhanced class activation sequence (CAS) by perceptual fusion of classification results of different levels, thereby ensuring the discovery of the complete time interval of the action. To further suppress the interference of context noise in the CAS, MPENet introduces an exclusive loss to help the model focus on the key part of the target action, thereby guiding the model to learn appropriate attention distribution and improve the understanding and discrimination ability of the model to features. Finally, the CAS and attention weighting are used to obtain the prediction result of the model.

[0006] To achieve the above purpose, the application adopts the following technical solution as a weakly supervised temporal action localization method based on mask pyramid enhancement. The implementation steps of the method are as follows:

[0007] Step 1: Construct a feature extraction module. First, divide the input uncut video into non-overlapping segments, and apply a dual-stream network I3D to extract scene features RGB and motion features Flow, then stack along the time dimension to construct a video feature representation;

[0008] Step 2: Masked Pyramid Classification Module (MPCM) for action integrity learning. According to the input video features, first downsample along the temporal dimension, and use the Temporal Adaptive Enhancement Mechanism (TEAM) to fine-tune the hierarchical features, forming a fine-tuned pyramid; then introduce a masking strategy during hierarchical processing to help the model focus on the feature area where the target action is not prominent, thereby ensuring the discovery of the complete time interval of the action and outputting high-quality class activation scores.

[0009] Step 3: Feature Discrimination Attention Module (FDAM) for feature discrimination learning. To further suppress noise interference in CAS, a mutually exclusive loss constraint hybrid attention head is designed to learn the importance of segment features, improve the discrimination ability of action, background, and context features, and output segment attention scores, which are then weighted with CAS to obtain the model's prediction results.

[0010] Step 4: Joint optimization and model positioning inference process. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 The network overall framework diagram of the present application.

[0012] Figure 2 The temporal adaptive enhancement mechanism diagram for hierarchical feature context modeling.

[0013] Figure 3 The influence of different erasing thresholds θ and masked pyramid levels L on the overall performance of the model.

[0014] Figure 4 The result diagram for qualitative visual analysis. DETAILED DESCRIPTION

[0015] The present application will be described in detail below in conjunction with the drawings and examples.

[0016] The technical scheme of the present application is a weakly supervised temporal action localization method based on mask pyramid enhancement, which explores the feature distribution of action instances in the video and understands the structure and evolution process of the action, and realizes accurate positioning. The system module for realizing the method includes a feature extraction module, a mask pyramid classification module MPCM for action integrity learning, and an attention module FDAM for feature discriminative learning. The feature extraction module is used for feature extraction and serves as the input data of the subsequent link; the MPCM first constructs a refined enhanced mask pyramid using a temporal adaptive enhancement mechanism TAEM, and introduces a mask strategy driven model during hierarchical processing to mine new feature regions complementary to the target action. Then, the classification results of different levels are perceptually fused to generate an enhanced class activation sequence CAS, thereby ensuring the discovery of the complete time interval of the action. To further suppress the context noise interference in the CAS, the MPENet introduces an exclusive loss in the FDAM to help the model focus on the key part of the target action, thereby guiding the model to learn appropriate attention distribution and improving the model's understanding and discrimination ability of features. Finally, the CAS and attention weighting are used to obtain the prediction result of the model.

[0017] The overall framework diagram of the technical scheme of the present application is shown in Figure 1 Further, the feature extraction module is trained on a dataset containing daily activities and sports events. Given N untrimmed videos and class labels where Y i ∈{y1,y2...,y C+1}∈R C+1 , y c ∈{0,1} indicates whether a c th category action appears in the video V i , C is the total number of action categories, and the C+1th category represents the background category. Each video V i contains T segments, denoted as where T is the total number of segments. The I3D feature extractor is applied to the video segment v t to extract D-dimensional features x i ∈R D . Then, the segment features are spliced along the time dimension to form the video feature representation X=[x1,...,x T ] T ∈R T×D , which serves as the input data of the subsequent link.

[0018] Further, the implementation process of the mask pyramid classification module is as follows: first, input the given feature X into the embedding layer Φ to generate the embedded feature Z1=[Φ(x1),Φ(x2),...,Φ(x T )] T ,Φ(x i )∈RD For x i The embedded features are derived by a 1D convolution and a ReLU activation function. Then, Z1 is used as input and downsampled using a 1D convolutional layer with a stride depth of 2 to generate a pyramid Z = [Z1, Z2, ..., Z]. L ],in The input of the layer is The calculation formula is as follows:

[0019]

[0020] While this multi-resolution stacked feature construction is effective, the lack of complete annotation information often leads models to rely on high-discrimination action features while ignoring less obvious but equally important regions, resulting in incomplete action localization. Therefore, this invention proposes a masked pyramid, combining temporal pyramids and masking strategies. First, a temporal adaptive enhancement mechanism (TAEM) is introduced into the pyramid levels for refined modeling, improving the representation of feature details and forming a "refined pyramid." Based on this, MPCM introduces two branches to detect features in different regions. The refined pyramid classification branch mainly predicts high-discrimination feature regions, while the mask-driven refined pyramid classification branch masks the detected high-discrimination regions using a masking strategy, guiding the model to discover low-discrimination feature parts complementary to the target action. The outputs of these two branches are concatenated to generate a complete action prediction, effectively mitigating the differences in optimization objectives between classification and localization tasks. The specific implementation of these two modules will be described in detail below.

[0021] 1) Refined pyramid classification: First, this invention classifies each layer of Z... Applying time-adaptive enhancement mechanisms for refined context modeling (e.g.) Figure 2 As shown), and introduces a base classifier (C). BC To detect the high-resolution part of the action.

[0022] for if This invention uses the features of the next layer. Perform context enhancement; conversely, if Then use the features of the previous layer. Then, adaptive max pooling, sigmoid activation, and residual connection operations are applied to obtain refined and enhanced features. Through the above steps, the features of each layer in the pyramid have undergone refined contextual enhancement. The calculation is as follows:

[0023]

[0024] where, sigma(·) denotes the sigmoid function.

[0025] Compared with using the Transformer long sequence relationship modeling, the information redundancy and over-smoothing problem caused by global modeling are avoided while reducing the calculation consumption, as shown in the ablation experiment.

[0026] After the fine modeling of the features, C BC The feature sequence of each layer is projected into the category space to generate the class activation score (CAS).

[0027]

[0028] where, C BC is realized by a linear function, and η cas represents a learnable parameter.

[0029] 2) Mask-driven fine-grained pyramid classification: Although the classification result S BC can effectively capture the high-discriminative action region, this module still has certain limitations in positioning. Since the classification network often focuses on high-discriminative local features and ignores low-discriminative regions related to the target action. This tendency leads the model to achieve higher classification accuracy, but it is difficult to provide high-quality action positioning results. Therefore, the text introduces a C BC with the same structure and shared parameters (C MC ), to better capture low-discriminative features related to the target action and improve the integrity of action positioning.

[0030] Specifically, for each action class c, the classification result S is first filtered by applying a threshold θ, which involves masking the high-discriminative action feature region in S to obtain S The calculation method is as follows:

[0031]

[0032] where, c is an element of [1, C+1], and Ψ[·] is an indicator function, and represents element-wise multiplication. Then, it is taken as the input of C MC to drive C MC to mine new feature regions complementary to the high-discriminative regions.

[0033]

[0034] Finally, the present application will​​ Cascade is performed to form a complete Thus, the integrity of action localization is ensured.

[0035]

[0036] In order to further improve the detection ability of the model, the present application perceives and fuses the classification results of different levels. First, the video-level class attention map is generated by using Then, the is weighted and applied to CAS1, and an enhanced CAS with fused multi-scale information is obtained in a residual connection manner, denoted as The calculation is as follows:

[0037]

[0038] This fusion process not only combines the advantages of features of each level, but also effectively smooths the fluctuations between adjacent segments, reduces misjudgments caused by too drastic changes in short action features, and improves the detection ability of the model for actions of different time lengths.

[0039] Further, the implementation process of the feature discrimination attention module (FDAM) is as follows:

[0040] In addition to high-discriminative actions and non-action backgrounds, there are also some semantically ambiguous context segments in uncut videos, and their existence is one of the reasons for the increase in action localization difficulty. Therefore, this module starts from reducing context confusion caused by the lack of boundary label information and enhancing the separability of features to improve discrimination accuracy.

[0041] Specifically, the present application inputs the features in the fine pyramid to the mixed attention head to measure the possibility of the video segment belonging to action, context and background, and assigns a corresponding attention value A th (t) to each segment, A act (t) and A con (t). bak

[0042]

[0043] Wherein, A * (t)∈[0,1], *∈{act,con,bak}, t∈[1,T] and θ att represent the learnable parameters.

[0044] Then, based on the obtained attention weight A * , three-branch attention-guided class activation sequences are constructed respectively ​​and which can suppress the interference of irrelevant information. The calculation is as follows:

[0045]

[0046] wherein, The activation of the context segment with ambiguous semantics can be suppressed, and a high-probability action segment is obtained. Similarly, The high-discrimination action segment can be ignored, and the context segment related to the action is focused on. For , the weighted class activation pays more attention to the non-action background segment.

[0047] After that, the model predicts the CAS * Apply the top-k aggregation strategy, and obtain the video-level prediction

[0048] Finally, the model overall optimization function and model inference are implemented as follows:

[0049] Mutual exclusion loss. The present application proposes a mutual exclusion loss function L rg to encourage the difference between the weights of each branch when predicting at the same time, so as to obtain a proper attention distribution.

[0050]

[0051] wherein, *∈{act,con,bak}.

[0052] Video-level classification loss. In addition to L rg , the present application also applies a cross-entropy loss between the video-level action probability distribution and the real label Y i to constrain the video action class prediction. As shown below:

[0053]

[0054] wherein, L cls,* is constrained by the pre-defined label Y act =[y c =1,y C+1 =0], Y con =[y c =1,y C+1 =1], and Y bak =[y c =0,y C+1 =1] respectively, and the classification training guided by the action attention, the context attention and the background attention.

[0055] Objective function. The entire learning process of the model consists of classification loss L. cls Consistency loss L cs Feature separation loss L feat and mutual exclusion loss L rg Common constraints. The overall loss function is expressed as follows:

[0056] L total =L cls +λ1L rg +λ2L feat +λ3L cs (12)

[0057] Where λ1, λ2, λ3 are learnable parameters, and L feat and L cs The specific details refer to the previous ACMNet. The model localizes inference. During testing, the model first computes attention-weighted video-level predictions. As mentioned earlier. Secondly, based on the predefined classification threshold... By selecting from the video predictions, a set of possible actions is obtained, namely Subsequently, for each action in this set, the present invention applies multiple positioning thresholds. exist To locate all time segments of two or more consecutive time steps, use {(s j ,e j ,c j ,q j The confidence score q indicates the detected action instance. j It is obtained through an internal-external comparison function. Finally, overlapping action proposals are removed using non-maximum suppression (NMS).

[0058] Experimental Section

[0059] Experimental Datasets: This invention evaluates its method using the publicly available and highly challenging THUMOS14 and ActivityNet1.3 datasets. THUMOS14 comprises 413 unedited videos across 20 action classes. The model is trained on 200 validation videos and evaluated on 213 test videos. Each video contains approximately 15 action instances. ActivityNet1.3 contains 19,994 unedited videos from 200 action classes, split into training, validation, and test sets in a 2:1:1 ratio. On average, each video contains 1.6 action instances, with approximately 36% of the content consisting of blurred context or non-action background frames. This invention is trained on the training set and tested on the validation set.

[0060] Evaluation metrics: Following the standard evaluation protocol, the mAP under different overlap thresholds (IoU) is used as the measurement to evaluate the performance of weakly supervised temporal action localization, denoted as mAP@IoU. Specifically, in the ablation experiments, the detection evaluation metrics are initialized as follows: IoU is set to [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7] on the THUMOS14 dataset, and IoU is set to [0.5, 0.75, 0.95] on the ActivityNet1.3 dataset.

[0061] Experimental setup: First, the untrimmed videos are divided into T groups of non-overlapping 16-frame segments, and the I3D network pre-trained based on the Kinetics dataset is used to extract features. Please note that, for fairness, the model does not introduce any fine-tuning operation to the I3D network.

[0062] During the training process, the training parameters of the model on THOUMS14 and ActivityNet1.3 are as follows: the training batch is set to 16 and 64 respectively, the epoch is 800, and the Adam optimizer with learning rate 0.0001 and weight decay rate 0.001 is applied to optimize the model parameters. The number of pyramid layers L is 3, the feature dimension D is set to 1024, the mask θ is 0.7, and the hyperparameters λ1, λ2, λ3 are 5e-5, 5e-5, and 2e-3 respectively. The classification threshold is set to 0.25 on THUMOS14 and 0.1 on ActivityNet1.3. The localization threshold is set to [0.1, 1, 0.1] and [0.5, 1, 0.05] respectively using the multi-threshold method. Finally, in order to remove the high-overlapping action proposals, non-maximum suppression (NMS) is used, and its IoU is set to 0.55 and 0.9 respectively. The entire model is implemented by PyTorch1.7.

[0063] The proposed method is evaluated on the THUMOS14 and ActivityNet1.3 datasets, and compared with the most advanced localization method, as shown in Tables 1 and 2.

[0064] Table 1: The proposed MPENet method is evaluated on the THUMOS14 dataset, and the detection performance (mAP) of the most advanced localization method under different IoU thresholds.

[0065]

[0066] Table 2: The performance of the proposed method and the previous method in temporal action localization under different IoU thresholds (mAP) is compared on the ActivityNet1.3 dataset.

[0067]

[0068] Ablation experiments: Table 3 shows the ablation of the components of the proposed MPENet, and Table 4 compares the mask pyramid generation using two methods: Transformer global relation modeling and temporal adaptive enhancement mechanism, while also evaluating computational efficiency. Figure 3 The paper demonstrates the impact of different erasure thresholds θ and mask pyramid levels L on the overall model performance; finally, the invention... Figure 4 The local regions and CAS results for the three actions are qualitatively visualized in the figure. In the figure, orange bars represent action instances, green bars indicate the context, and gray bars represent the background. The line chart below shows the CAS values ​​predicted by the proposed method, where the horizontal axis represents the time step and the vertical axis represents the activation intensity, ranging from 0 to 1.

[0069] Table 3: This invention performs ablation on the loss function and mask pyramid strategy on the THUMOS14 dataset, where "PS" represents the pyramid structure, "MM" indicates the masking mechanism, "AMP" represents the temporal adaptive enhancement mechanism, and "L" represents the time-adaptive enhancement mechanism. rg "Represents the mutually exclusive loss used for feature separation, with the loss without the above components serving as the baseline (L) base ). + Represents each setting with L base The gain compared to the previous one.

[0070]

[0071] Table 4: Evaluation of the effects of using Transformer global relation modeling and temporal adaptive enhancement mechanism for local relation modeling on model parameter size (Params), computational complexity (GFLOPs), and inference speed (Time / )

[0072] The impact of video" and detection performance on AVG@[0.1:0.7].

[0073]

[0074] Qualitative visualization analysis. To more intuitively verify the effectiveness of the proposed method, this invention... Figure 4 The local regions and CAS results of the three actions are visualized. Compared with the baseline method, the MPENet algorithm proposed in this invention has richer CAS distribution information and can accurately locate action instances at different time scales in the video.

[0075] Figure 4(a) depicts a typical archery video action, containing two segments, one long action and one short action. Since the baseline does not model the features of actions with different durations, it misses some details and action parts when analyzing actions of different lengths, resulting in low-quality predictions for short actions. MPENet introduces a mask pyramid to model multi-temporal scale action information, ensuring the preservation of key information and minimizing missed information. Figure 4 (b) shows a swimming video example. Since the high discriminative regions of the action dominate the gradient, the baseline ignores the learning of action un-distinctive features. MPENet introduces a mask mechanism to the pyramid structure, driving the model to mine new feature regions complementary to the target action. Similarly, due to the lack of accurate time labeling, the baseline model incorrectly merges the action context into the GT instance, producing false positives in the context area (e.g., close to the start of the dunk). MPENet uses mutual exclusion constraints to help the model correctly distinguish between actions and context, improving the accuracy of action localization, as shown in Figure 4 (c) shows a basketball dunk video action.

Claims

1. A weakly supervised temporal action localization method based on mask pyramid enhancement, characterized in that, The system modules implementing this method include a feature extraction module, a mask pyramid classification module MPCM for action integrity learning, and an attention module FDM for feature discriminative learning. The feature extraction module is used for feature extraction, which serves as input data for subsequent steps. MPCM first uses the temporal adaptive enhancement mechanism TAEM to construct a refined and enhanced mask pyramid, and introduces a masking strategy during hierarchical processing to drive the model to mine new feature regions that are complementary to the target action. Then, the classification results at different levels are perceptually fused to generate enhanced class activation sequences (CAS), thereby ensuring the complete time interval of the action detection. To suppress the interference of contextual noise in CAS, the Masked Pyramid Enhancement Network (MPENet) introduces a mutual exclusion loss in FDM, focusing on the key parts of the target action, thereby guiding the learning of an appropriate attention distribution and improving the understanding and discrimination of features. Finally, the CAS and attention are weighted to obtain the model's prediction results. The feature extraction module is trained on a dataset containing daily activities and sports events; Given N unedited videos and category labels Where Y i ∈{y1,y2,...,y C+1 }∈R C+1 y c ∈{0,1} indicates whether c exists. th Category actions appear in video V i C represents the total number of action categories, with the (C+1)th category representing the background category; each video V i Each contains T segments, represented as Where T is the total number of segments; for video segment v t Extracting D-dimensional features x using the I3D feature extractor i ∈R D ; The segment features are concatenated along the time dimension to form the video feature representation X = [x1,...,x2]. T ] T ∈R T×D It serves as the input for subsequent steps; The implementation process of the masked pyramid classification module is as follows: Given feature X, input it into the embedding layer Φ to generate embedded features Z1=[Φ(x1),Φ(x2),...,Φ(x...] . T )] T ,Φ(x i )∈R D For x i The embedded features are obtained by taking a 1D convolution and a ReLU activation function; then, Z1 is used as input and downsampled using a 1D convolutional layer with a stride depth of 2 to generate a pyramid Z = [Z1, Z2, ..., Z]. L ],in The input of the layer is The calculation formula is as follows: A masked pyramid is proposed by combining temporal pyramid and masking strategy; a temporal adaptive enhancement mechanism TAEM is introduced into the pyramid level to refine the modeling and improve the performance of feature details, forming a "refined pyramid"; MPCM introduces two branches to detect features in different regions respectively. The refined pyramid classification branch is used to predict high-discriminative feature regions, while the mask-driven refined pyramid classification branch masks the detected high-discriminative regions through a masking strategy, guiding the model to mine low-discriminative feature parts that are complementary to the target action; the outputs of these two branches are concatenated to generate a complete action prediction, effectively mitigating the differences between classification and localization tasks in terms of optimization objectives. Refined pyramid classification: for each level in Z A temporal adaptive enhancement mechanism is applied to perform refined context modeling, and a base classifier C is introduced. BC To detect the high-resolution portion of the action; for if Use the features of the next layer Perform context enhancement; if Then use the features of the previous layer. Refined and enhanced features are obtained by applying adaptive max pooling, sigmoid activation, and residual connection operations. The features of each layer in the pyramid have undergone refined context enhancement; the calculation is as follows: Where σ(·) represents the sigmoid function; After feature refinement modeling, C BC Each layer of feature sequences Projected onto the category space to generate class activation scores (CAS); in, C BC Implemented by a linear function, η cas This represents the learnable parameters.

2. The weakly supervised temporal action localization method based on mask pyramid enhancement according to claim 1, characterized in that, Mask-Driven Refined Pyramid Classification: Text Introduction and C BC Mining classifier C with the same structure and shared parameters MC This is to better capture low-resolution features related to the target action and improve the completeness of action localization; against First, let's look at the classification results. Apply a threshold θ to filter, Masking high-resolution action feature regions in the image to obtain... The calculation method is as follows: in, c∈[1,C+1], Ψ[·] is an indicator function and This represents element-wise multiplication; then, it is used as C. MC Input to drive C MC Discover new feature regions that complement high-resolution regions; Finally, Cascade to form a complete This ensures the integrity of the motion positioning; The classification results at different levels are perceptually fused and utilized. To generate video-level class attention graphs Will Weighting is applied to CAS1, and an enhanced CAS fusing multi-scale information is obtained through residual connections, denoted as: The calculation is as follows:

3. The weakly supervised temporal action localization method based on mask pyramid enhancement according to claim 1, characterized in that, The implementation process of the attention module FDAM for feature discriminative learning is as follows: In the fine pyramid Features are input into a hybrid attention head to measure t. th The video clips are considered to belong to the possibilities of action, context, and background, and each clip is assigned a corresponding attention value A. act (t), A con (t) and A bak (t); Among them, A * (t)∈[0,1], *∈{act,con,bak}, t∈[1,T] and θ att Represents learnable parameters; Based on the obtained attention weight A * Construct three-branch attention-guided class activation sequences respectively as well as Suppressing interference from irrelevant information; the calculation is as follows: in, Suppressing the activation of semantically ambiguous context fragments yields high-probability action fragments; Ignore high-discrimination action segments and focus on contextual segments related to the action; for For example, weighted class activation focuses more on non-action background fragments; for CAS... * A top-k aggregation strategy is applied, and video-level predictions are obtained through Softmax normalization.

4. The weakly supervised temporal action localization method based on mask pyramid enhancement according to claim 1, characterized in that, This method also includes optimization functions and model inference, the specific process of which is as follows: Mutual exclusion loss; proposing a mutual exclusion loss function L rg This encourages differences in weights among branches when predicting at the same time, thereby achieving an appropriate attention distribution. Where *∈{act,con,bak}; Video-level classification loss; except L rg Beyond the probability distribution of video-level actions Compared with the real label Y i Cross-entropy loss is applied to constrain the prediction of video action categories; as shown below: Among them, L cls,* By predefined label Y act =[y c =1,y C+1 =0], Y con =[y c =1,y C+1 =1], and Y bak =[y c =0,y C+1 =1] to constrain action attention, context attention, and background attention respectively for classification training; The objective function; the entire learning process of the model consists of the classification loss L. cls,* Consistency loss L cs Feature separation loss L feat and mutual exclusion loss L rg Common constraints; the overall loss function is expressed as follows: L total =L cls,* +λ1L rg +λ2L feat +λ3L cs (12) Where λ1, λ2, and λ3 are learnable parameters; The model localizes inference; during testing, the model first computes attention-weighted video-level predictions. As mentioned earlier; secondly, based on predefined classification thresholds... By selecting from the video predictions, a set of possible actions is obtained, namely For each action in this set, multiple location thresholds are applied. exist To locate all time segments of two or more consecutive time steps, use {(s j ,e j ,c j ,q j The confidence score q indicates the detected action instance. j It is obtained through the inner and outer comparison function; finally, overlapping action proposals are removed by non-maximum suppression (NMS).

Images