Weak supervision time sequence action positioning method based on mask pyramid enhancement

Abstract

The invention discloses a weak supervision time sequence action positioning method based on mask pyramid enhancement, and belongs to the field of computer vision and action positioning. A system module for realizing the method comprises a feature extraction module, a mask pyramid classification module MPCM for motion integrity learning, and an attention module FDAM for feature discrimination learning. The feature extraction module is used for extracting features as input data of subsequent links; the MPCM constructs a refined and enhanced mask pyramid by using a time sequence adaptive enhancement mechanism (TAEM), and introduces a mask strategy to drive a model to mine a new feature region complementary with a target action during hierarchical processing. And performing perceptual fusion on classification results of different levels to generate an enhanced class activation sequence (CAS). The MPENet introduces mutual exclusion loss in FDAM to help the model focus on the key part of the target action, so that the model is guided to learn proper attention distribution, and the understanding and distinguishing ability of the model to features is improved. And finally, weighting the CAS and the attention to obtain a prediction result of the model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and action localization, and specifically is a weakly supervised temporal action localization method based on deep learning technology, combined with a temporal adaptive enhancement mechanism and a mask strategy, and a mask pyramid enhanced by joint visual features. The method is suitable for tasks such as intelligent monitoring, anomaly detection, and visual question and answer. Background Art

[0002] With the popularization of electronic shooting equipment and the rapid development of multimedia networks, the video data on network platforms has exploded. For video temporal action localization, the traditional fully supervised learning method relies on accurate temporal boundary annotation for training and learning. The annotation cost is time-consuming and labor-intensive, which limits its promotion in practical applications. Therefore, many studies have begun to turn to weakly supervised learning, which only requires video-level labels during training and can locate the temporal boundaries and category attributes of actions during the test phase. Weakly supervised learning greatly reduces human resources, time costs, and annotation errors, and has good flexibility.

[0003] In the weakly supervised temporal action localization (WTAL) task, action instances in uncut videos usually have different time spans, ranging from a few seconds to tens of minutes. The current mainstream method is to adopt a classification-localization paradigm, relying on the highly abstract semantic information of single-scale feature maps to analyze actions of different lengths, and generate segment-level class activation scores (CAS) through classification supervision, and then threshold selection to generate action proposals, whose CAS represents the probability of action occurrence. It can be seen that the quality of CAS determines the upper limit of model performance. However, single-scale feature modeling often ignores some action details, and there is a deviation in the optimization objectives of classification and localization tasks. The former emphasizes promoting the distinguishability of action features, while the latter focuses on ensuring the integrity of the action area. Since the model must make a trade-off between the two, CAS fluctuates, which in turn causes inaccurate prediction of action boundaries, resulting in incomplete or over-complete action localization generated by the model. How to completely locate action instances is an important challenge that weakly supervised learning needs to face.

[0004] The present invention believes that constructing a pyramid structure along the time dimension of the extracted feature sequence can extract action information of different scales from different levels. The low-level pyramid contains more details and rapidly changing information of the action, while the high-level pyramid can capture a wider context or slower action changes. This helps to describe the video content more comprehensively and enrich the expression of action features. However, in WTAL, although the pyramid structure has advantages in providing multi-scale expression and enriching action features, the classification supervision makes the model often only learn highly discriminative feature areas and ignore the parts with unclear features. The present invention also verifies in the subsequent experimental part that the simple use of the pyramid structure does not bring gains to the model, further proving the conclusion that the simple pyramid structure is not suitable for WTAL tasks. Therefore, in order to make full use of the characteristics of the pyramid structure and make it better suitable for WTAL, the present invention combines the mask strategy with the temporal pyramid to propose a mask pyramid (MP). Summary of the invention

[0005] The present invention proposes a mask pyramid enhancement network (MPENet) to explore the feature distribution of action instances in videos and understand the structure and evolution of actions, so as to achieve further precise positioning. MPENet first uses the temporal adaptive enhancement mechanism (TAEM) to construct a refined and enhanced mask pyramid, and introduces a mask strategy to drive the model to mine new feature areas complementary to the target action during hierarchical processing. MPNet performs perceptual fusion on the classification results of different levels to generate an enhanced class activation sequence (CAS), thereby ensuring the complete time interval of the action. In order to further suppress the interference of contextual noise in CAS, MPENet introduces mutual exclusion loss to help the model focus on the key parts of the target action, thereby guiding the model to learn the appropriate attention distribution and improve the model's understanding and distinction of features. Finally, CAS and attention are weighted to obtain the prediction result of the model.

[0006] To achieve the above purpose, the present invention adopts the following technical solution as a weakly supervised temporal action localization method based on mask pyramid enhancement, and the implementation steps of the method are as follows:

[0007] Step 1: Construct a feature extraction module. First, divide the input uncut video into non-overlapping segments, and apply the two-stream network I3D to extract the scene feature RGB and motion feature Flow, and then stack them along the temporal dimension to construct a video feature representation;

[0008] Step 2: Mask pyramid classification module MPCM for action completeness learning. According to the input video features, it first downsamples along the temporal dimension, and uses the temporal adaptive enhancement mechanism TEAM to refine the hierarchical features to form a refined enhanced pyramid; then, during hierarchical processing, a mask strategy is introduced to help the model focus on the feature areas where the target action is not clear, thereby ensuring the discovery of the complete time interval of the action and outputting high-quality class activation scores.

[0009] Step 3: Attention module FDAM for feature discriminative learning. In order to further suppress noise interference in CAS, a mutually exclusive loss constrained hybrid attention head is designed to learn the importance of segment features, improve the ability to distinguish action, background, and context features, output segment attention scores, and then weighted with CAS to obtain the prediction results of the model.

[0010] Step 4: Give the joint optimization and model positioning reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is the overall network framework diagram of the present invention.

[0012] Figure 2 Schematic diagram of the temporal adaptive enhancement mechanism for modeling hierarchical feature context.

[0013] Figure 3 The impact of different erasing thresholds θ and mask pyramid levels L on the overall performance of the model.

[0014] Figure 4 This is a schematic diagram of the results of qualitative visual analysis. DETAILED DESCRIPTION

[0015] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0016] The technical solution adopted by the present invention is a weakly supervised temporal action localization method based on mask pyramid enhancement, which explores the feature distribution of action instances in videos and understands the structure and evolution process of actions to achieve precise positioning. The system modules implementing the method include a feature extraction module, a mask pyramid classification module MPCM for action integrity learning, and an attention module FDAM for feature discriminative learning. The feature extraction module is used for feature extraction as input data for subsequent links; MPCM first uses the temporal adaptive enhancement mechanism TAEM to construct a refined enhanced mask pyramid, and introduces a mask strategy to drive the model to mine new feature areas complementary to the target action during hierarchical processing. Then, the classification results of different levels are perceptually fused to generate an enhanced class activation sequence CAS, thereby ensuring the complete time interval of the action. In order to further suppress the interference of contextual noise in CAS, MPENet introduces mutually exclusive loss in FDAM to help the model focus on the key parts of the target action, thereby guiding the model to learn appropriate attention distribution and improve the model's ability to understand and distinguish features. Finally, CAS and attention are weighted to obtain the prediction result of the model.

[0017] The overall framework diagram of the technical solution of the present invention is as follows Figure 1 As shown, further, the feature extraction module is trained on a dataset containing daily activities and sports events. Given N uncut videos and category labels where Y i ∈{y1,y2...,y C+1}∈R C+1 ,y c ∈{0,1} indicates whether there is c th Category action appears in video V i , C is the total number of action categories, and the C+1th category represents the background category. i contains T fragments, expressed as Where T is the total number of clips. t Apply I3D feature extractor to extract D-dimensional features x i ∈R D Then, the segment features are concatenated along the time dimension to form a video feature representation X = [x1,...,x T ] T ∈R T×D , which serves as input for subsequent steps.

[0018] Furthermore, the implementation process of the mask pyramid classification module is as follows: First, the given feature X is input into the embedding layer Φ to generate the embedded feature Z1 = [Φ(x1), Φ(x2), ..., Φ(x T )] T ,Φ(x i )∈RD For x i The embedded features of Z1 include a 1D convolution and RELU activation function. Then, Z1 is used as input and downsampled using a 1D convolution layer with a step depth of 2 to generate a pyramid Z = [Z1, Z2, .., Z L ],in The input of the layer is The calculation formula is as follows:

[0019]

[0020] Although this multi-resolution stacked feature construction is effective, due to the lack of complete annotation information, the model often relies on highly discriminative action features and ignores inconspicuous but equally important areas, resulting in incomplete action positioning. Therefore, the present invention combines the temporal pyramid with the mask strategy to propose a mask pyramid. First, the temporal adaptive enhancement mechanism (TAEM) is introduced in the pyramid level for refined modeling to improve the performance of feature details and form a "refined pyramid". On this basis, MPCM introduces two branches to detect features in different areas respectively. The refined pyramid classification branch is mainly used to predict highly discriminative feature areas, while the mask-driven refined pyramid classification branch masks the detected high-discriminative areas through a mask strategy, guiding the model to mine low-discriminative feature parts that are complementary to the target action. The output results of these two branches generate complete action predictions in a cascade manner, effectively alleviating the differences in optimization objectives between classification and positioning tasks. The specific implementation of these two modules will be introduced in detail below.

[0021] 1) Refined pyramid classification: First, the present invention classifies each layer in Z Applying the temporal adaptive enhancement mechanism for refined context modeling (e.g. Figure 2 As shown), and introduce the basic classifier (C BC ) to detect the high-resolution part of the action.

[0022] for if The present invention uses the following features Perform context enhancement; otherwise, if Then use the previous layer features Then, adaptive maximum pooling, Sigmoid activation, and residual connection operations are applied to obtain refined and enhanced features. Through the above steps, the features of each layer in the pyramid are refined and contextually enhanced. The calculation is as follows:

[0023]

[0024] Here, σ(·) represents the sigmoid function.

[0025] Compared with using Transformer to model long temporal relationships, it reduces computational consumption while avoiding the information redundancy and over-smoothing problems caused by global modeling, as shown in the ablation experiment.

[0026] After the feature refinement modeling, C BC Each layer of feature sequence Projected into the class space to generate a class activation score (CAS).

[0027]

[0028] in, C BC Implemented by a linear function, η cas represents learnable parameters.

[0029] 2) Mask-driven refined pyramid classification: Although the classification result S BC It can effectively capture high-resolution action areas, but the module still has certain limitations in positioning. The classification network often only focuses on high-resolution local features and ignores low-resolution areas related to the target action. This tendency leads to the model being able to achieve higher classification accuracy but difficult to provide high-quality action positioning results. Therefore, the text introduces the C BC Mining classifiers with the same structure and shared parameters (C MC ) to better capture the low-discrimination features related to the target action and improve the integrity of action localization.

[0030] Specifically, for First, the classification results Applying the threshold θ for screening involves The high-discrimination action feature area in is masked to obtain The calculation is as follows:

[0031]

[0032] in, c∈[1,C+1], Ψ[·] is an indicator function and represents element-wise multiplication. Then, it is used as C MC The input to drive C MC Mining new feature regions that are complementary to high-discrimination regions.

[0033]

[0034] Finally, the present invention will Cascade to form a complete Thereby ensuring the integrity of action positioning.

[0035]

[0036] In order to further improve the detection capability of the model, the present invention performs perceptual fusion on the classification results at different levels. To generate video-level class attention maps Then The weighted CAS is applied to CAS1, and the enhanced CAS that integrates multi-scale information is obtained in a residual connection manner, which is expressed as The calculation is as follows:

[0037]

[0038] This fusion process not only combines the feature advantages of each level, but also effectively smooths the fluctuations between adjacent segments, reduces misjudgments caused by drastic changes in short action features, and improves the model's ability to detect actions of different time series lengths.

[0039] Furthermore, the implementation process of the feature discrimination attention module (FDAM) is as follows:

[0040] In addition to highly identifiable actions and non-action backgrounds, uncut videos also contain some semantically ambiguous context segments, which are one of the reasons why action localization becomes more difficult. Therefore, this module reduces the context confusion caused by the lack of boundary annotation information and enhances the separability of features to improve the discrimination accuracy.

[0041] Specifically, the present invention will be fine pyramid The features are fed into the hybrid attention head to measure t th The probability that a video segment belongs to an action, context, or background, and each segment is assigned a corresponding attention value A act (t), A con (t) and A bak (t).

[0042]

[0043] Among them, A * (t)∈[0,1],*∈{act,con,bak},t∈[1,T]and θ att represents learnable parameters.

[0044] Next, based on the obtained attention weight A * Construct three-branch attention-guided class activation sequences separately as well as It can suppress the interference of irrelevant information. The calculation is as follows:

[0045]

[0046] in, The activation of semantically ambiguous context fragments can be suppressed to obtain high-probability action fragments. We can ignore highly discriminative action segments and focus on action-related context segments. For example, the weighted class activations focus more on non-action background segments.

[0047] Afterwards, the model was used to evaluate the CAS * Apply top-k aggregation strategy and get video-level prediction through Softmax normalization

[0048] Finally, the overall model optimization function and model inference are implemented as follows:

[0049] Mutually exclusive loss. The present invention proposes a mutually exclusive loss function L rg , to encourage the differences in weights between branches when predicting at the same time, so as to obtain appropriate attention distribution.

[0050]

[0051] Among them, *∈{act,con,bak}.

[0052] Video-level classification loss. In addition to L rg In addition, the present invention also analyzes the video-level action probability distribution and the true label Y i The cross entropy loss is applied to constrain the video action category prediction. As shown below:

[0053]

[0054] Among them, L cls,* By predefined label Y act =[y c =1,y C+1 =0],Y con =[y c =1,y C+1 =1], and Y bak =[y c =0,y C+1 = 1] to constrain the classification training guided by action attention, context attention, and background attention respectively.

[0055] Objective function. The entire learning process of the model is determined by the classification loss L cls , consistency loss L cs , feature separation loss L feat and the mutual exclusion loss L rg Common constraints. The overall loss function is expressed as follows:

[0056] L total =L cls +λ1L rg +λ2L feat +λ3L cs , (12)

[0057] Among them, λ1, λ2, λ3 are learnable parameters, L feat and L cs The details refer to the previous ACMNet. Model localization reasoning. During testing, the model first calculates the attention-weighted video-level prediction As mentioned above. Secondly, according to the predefined classification threshold Select the video prediction to get a set of possible actions, namely Next, for each action in the set, the present invention applies multiple positioning thresholds exist To locate all time segments of two or more consecutive time steps, use {(s j ,e j ,c j ,q j )} to indicate the detected action instance, and its confidence score q j It is obtained by the inside-outside contrast function. Finally, non-maximum suppression (NMS) is used to remove overlapping action proposals.

[0058] Experimental Section

[0059] Experimental data set: The present invention uses the public and highly challenging THUMOS14 and ActivityNet1.3 data sets to evaluate the method of the present invention. THUMOS14 covers 413 uncut videos involving 20 action classes. The model is trained on 200 validation videos and evaluated on 213 test videos. There are about 15 action instances in each video. ActivityNet1.3 contains 19,994 uncut videos from 200 action categories, divided into training set, validation set and test set in a ratio of 2:1:1. There are an average of 1.6 action instances per video, and about 36% of the content is blurred context or non-action background frames. The present invention is trained on the training set and tested on the validation set.

[0060] Evaluation indicators: Following the standard evaluation protocol, mAP under different overlap thresholds (IoU) is used as the metric to evaluate the performance of weakly supervised temporal action localization, expressed as mAP@IoU. Specifically, in the ablation experiment, the detection evaluation indicator is uniformly initialized as follows, that is, IoU is set to [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7] on the THUMOS14 dataset, and IoU is set to [0.5, 0.75, 0.95] on the ActivityNet1.3 dataset.

[0061] Experimental setup: First, the uncut video is divided into T groups of non-overlapping 16-frame segments, and features are extracted using an I3D network pre-trained on the Kinetics dataset. Please note that for fairness, this model does not introduce any fine-tuning operations on the I3D network.

[0062] During the training process, the training parameters of this model on THOUMS14 and ActivityNet1.3 are as follows: the training batch is set to 16 and 64 respectively, the epoch is 800, and the Adam optimizer with a learning rate of 0.0001 and a weight decay rate of 0.001 is applied to optimize the model parameters. The number of pyramid layers L is 3, the feature dimension D is set to 1024, the mask θ is 0.7, and the hyperparameters λ1, λ2, and λ3 are 5e-5, 5e-5, and 2e-3 respectively. The classification threshold It is set to 0.25 on THUMOS14 and 0.1 on ActivityNet1.3. The multi-threshold method is used and set to [0.1, 1, 0.1] and [0.5, 1, 0.05] respectively. Finally, in order to remove highly overlapping action proposals, non-maximum suppression (NMS) is used, and its IoU is set to 0.55 and 0.9 respectively. The entire model is implemented in PyTorch1.7.

[0063] The proposed method is evaluated on THUMOS14 and ActivityNet1.3 datasets and compared with the state-of-the-art localization methods, as shown in Tables 1 and 2.

[0064] Table 1: The detection performance (mAP) of the proposed MPENet method and the most advanced localization method at different IoU thresholds is evaluated on the THUMOS14 dataset.

[0065]

[0066] Table 2: Performance comparison (mAP) of the proposed method and previous methods for temporal action localization at different IoU thresholds on the ActivityNet1.3 dataset.

[0067]

[0068] Ablation experiment: In Table 3, the present invention ablates the components of the proposed MPENet. In Table 4, a comparative experiment is conducted on the generation of mask pyramids using Transformer global relationship modeling and temporal adaptive enhancement mechanism, and the computational efficiency is evaluated. Figure 3 The influence of different erasing thresholds θ and mask pyramid levels L on the overall performance of the model is shown in Figure 4 The local regions and CAS results of the three actions are qualitatively visualized in the figure. The orange barcode in the figure is the action instance, the green barcode indicates the context, and the gray barcode indicates the background. The line graph below shows the CAS value predicted by the proposed method, where the horizontal axis is the time step and the vertical axis is the activation intensity, ranging from 0 to 1.

[0069] Table 3: The present invention performs ablation on the loss function and mask pyramid strategy on the THUMOS14 dataset, where “PS” represents the pyramid structure, “MM” indicates the mask mechanism, “AMP” represents the timing adaptive enhancement mechanism, and “L rg ” represents the mutually exclusive loss for feature separation, with the baseline without the above components (L base ). + Represents each setting with L base Compared to the gain.

[0070]

[0071] Table 4: The effects of Transformer global relationship modeling and temporal adaptive enhancement mechanism local relationship modeling on model parameter quantity “Params”, computational complexity “GFLOPs”, and inference speed “Time /

[0072] video” and the impact of detection performance on AVG@[0.1:0.7].

[0073]

[0074] Qualitative visualization analysis. In order to more intuitively verify the effectiveness of the method proposed in this paper, the present invention Figure 4 The local regions and CAS results of the three actions are visualized in Figure 2. Compared with the baseline method, the MPENet algorithm proposed in this paper has richer CAS distribution information and can accurately locate action instances at different time scales in the video.

[0075] Figure 4(a) depicts a typical archery video action, which contains two clips, one long and one short. Since there is no targeted feature modeling for actions of different durations, the baseline misses some details and action parts when analyzing actions of different lengths, resulting in low-quality predictions for short actions. MPENet introduces mask pyramid modeling of action information at multiple temporal scales, ensuring the retention of key information and minimization of missed detections. Figure 4 (b) shows an example of a swimming video. Since the highly discriminative regions of the action dominate the gradient, the baseline ignores the learning of unclear features of the action. MPENet introduces a mask mechanism into the pyramid structure, driving the model to mine new feature regions that are complementary to the target action. Similarly, due to the lack of accurate time annotations, the baseline model incorrectly merges the action context into the GT instance and produces false positives in the context region (e.g., near the start of a dunk). MPENet uses mutually exclusive constraints to help the model correctly distinguish between action and context, improving the accuracy of action localization, such as Figure 4 (c) As shown in the basketball dunk video action.

Claims

1. A weakly supervised temporal action localization method based on mask pyramid enhancement, characterized in that: The system modules for implementing this method include a feature extraction module, a mask pyramid classification module MPCM for action integrity learning, and an attention module FDAM for feature discriminative learning; the feature extraction module is used for feature extraction as input data for subsequent links; MPCM first uses the temporal adaptive enhancement mechanism TAEM to construct a refined and enhanced mask pyramid, and introduces a mask strategy to drive the model to mine new feature areas that are complementary to the target action during hierarchical processing; Then, the classification results at different levels are perceptually fused to generate enhanced class activation sequences (CAS), thereby ensuring the complete time interval of the action is discovered; In order to suppress the interference of contextual noise in CAS, MPENet introduces mutual exclusion loss in FDAM, focusing on the key parts of the target action, thereby guiding the learning of appropriate attention distribution and improving the ability to understand and distinguish features; finally, CAS and attention are weighted to obtain the prediction results of the model.

2. According to claim 1, a weakly supervised temporal action localization method based on mask pyramid enhancement is characterized in that: The feature extraction module is trained on a dataset containing daily activities and sports events; given N uncut videos and category labels where Y i ∈{y1,y2...,y C+1 }∈R C+1 ,y c ∈{0,1} indicates whether there is c th Category action appears in video V i , C is the total number of action categories, and the C+1th category represents the background category; each video V i contains T fragments, expressed as Where T is the total number of clips; for video clip v t Apply I3D feature extractor to extract D-dimensional features x i ∈R D ; The segment features are concatenated along the time dimension to form a video feature representation X = [x1,...,x T ] T ∈R T×D , which serves as input for subsequent steps.

3. According to claim 2, a weakly supervised temporal action localization method based on mask pyramid enhancement is characterized in that: The implementation process of the mask pyramid classification module is as follows: the given feature X is input into the embedding layer Φ to generate the embedded feature Z1 = [Φ(x1), Φ(x2), ..., Φ(x T )] T ,Φ(x i )∈R D For x i The embedded features of Z1 include a 1D convolution and RELU activation function; then, Z1 is used as input and downsampled using a 1D convolution layer with a step depth of 2 to generate a pyramid Z = [Z1, Z2, .., Z L ], where the input of layer l is Z l-1 ; The calculation formula is as follows: Z l =Conv(Z l-1 ,stride=2),l=2,...,L. (1) The mask pyramid is proposed by combining the temporal pyramid and the mask strategy; the temporal adaptive enhancement mechanism TAEM is introduced in the pyramid level for refined modeling to improve the ability to express feature details and form a "refined pyramid"; MPCM introduces two branches to detect features in different regions respectively; The refined pyramid classification branch is used to predict high-discriminative feature areas, while the mask-driven refined pyramid classification branch masks the detected high-discriminative areas through a mask strategy, guiding the model to mine low-discriminative feature parts that are complementary to the target action; the output results of these two branches are cascaded to generate complete action predictions, effectively alleviating the differences in optimization objectives between classification and localization tasks.

4. According to claim 3, a weakly supervised temporal action localization method based on mask pyramid enhancement is characterized in that: Refined pyramid classification: for each layer Z in Z e , l∈[1,L], T l =T / 2 l-1 , apply the temporal adaptive enhancement mechanism to perform refined context modeling, and introduce the basic classifier C BC To detect the high-resolution part of the action; For Z l , if 1≤l<L, use the next layer of features Z l+1 Perform context enhancement; if l = L, use the previous layer feature Z e-1 ; Adaptive max pooling, sigmoid activation and residual connection operations are used to obtain refined and enhanced features The features at each level in the pyramid are enhanced with refined context; the calculation is as follows: Where σ(·) represents the sigmoid function; After the feature refinement modeling, C BC Each layer of feature sequence Project to category space to generate class activation score CAS; in, C BC Implemented by a linear function, η cas represents learnable parameters.

5. According to claim 3, a weakly supervised temporal action localization method based on mask pyramid enhancement is characterized in that: Mask-driven refined pyramid classification: text introduction and C BC Mining classifiers C with the same structure and shared parameters MC , in order to better capture the low-discrimination features related to the target action and improve the integrity of action positioning; against First, the classification result S BC (l) Apply threshold θ to filter S BC The high-discrimination action feature area in (l) is masked to obtain The calculation is as follows: Among them, t∈[1,T e ], c∈[1,C+1], Ψ[·] is an indicator function and represents element-wise multiplication; then, it is used as C MC The input to drive C MC Mining new feature regions that complement high-discrimination regions; Finally, S BC (e) S MC (e) Cascade to form a complete CAS e , Thereby ensuring the integrity of action positioning; CAS l =max(S BC (l),S MC (l)). (6) The classification results at different levels are fused and perceived using CAS l To generate the video-level class attention map M l , M l ∈[0,1], l∈[2,..,L]; M l The weighted CAS is applied to CAS1, and the enhanced CAS that integrates multi-scale information is obtained in a residual connection manner, which is expressed as The calculation is as follows:

6. The method for weakly supervised temporal action localization based on mask pyramid enhancement according to claim 2, characterized in that: The implementation process of the attention module FDAM for feature discriminative learning is as follows: In the fine pyramid The features are fed into the hybrid attention head to measure t th The probability that a video segment belongs to an action, context, or background, and each segment is assigned a corresponding attention value A act (t), A con (t) and A bak (t); Among them, A * (t)∈[0,1],*∈{act,con,bak},t∈[1,T]and θ att represents learnable parameters; Based on the obtained attention weight A * Construct three-branch attention-guided class activation sequences separately as well as Suppress the interference of irrelevant information; the calculation is as follows: in, Suppress the activation of semantically ambiguous context fragments and obtain high-probability action fragments; Ignore high-discrimination action fragments and focus on context fragments related to the action; For CAS, the weighted class activation focuses more on non-action background fragments; * Apply top-k aggregation strategy and get video-level prediction through Softmax normalization 7. The method for weakly supervised temporal action localization based on mask pyramid enhancement according to claim 1, characterized in that: The method also includes optimization function and model reasoning. The specific process is as follows: Mutually exclusive loss; propose a mutually exclusive loss function L rg , to encourage the differences in weights between branches when predicting at the same time, so as to obtain appropriate attention distribution; Among them, *∈{act,con,bak}; Video-level classification loss; except L rg In addition, the video-level action probability distribution and the true label Y i A cross entropy loss is applied to constrain the video action category prediction; as shown below: Among them, L cls,* By predefined label Y act =[y c =1,y C+1 =0],Y con =[y c =1,y C+1 =1], and Y bak =[y c =0,y C+1 = 1] to constrain the classification training guided by action attention, context attention, and background attention respectively; Objective function: The entire learning process of the model is determined by the classification loss L cls , consistency loss L cs , feature separation loss L feat and the mutual exclusion loss L rg Common constraints; the overall loss function is expressed as follows: L total =L cls +λ1L rg +λ2L feat +λ3L cs , (12) Among them, λ1, λ2, λ3 are learnable parameters, L feat and L cs The specific details refer to the previous ACMNet; Model localization inference; during testing, the model first computes attention-weighted video-level predictions As mentioned above; secondly, according to the predefined classification threshold Select the video prediction to get a set of possible actions, namely For each action in the set, multiple localization thresholds are applied exist To locate all time segments of two or more consecutive time steps, use {(s j ,e j ,c j ,q j )} to indicate the detected action instance, and its confidence score q j It is obtained by the internal and external contrast function; finally, non-maximum suppression NMS is used to remove overlapping action proposals.

Images