A Weakly Supervised Temporal Behavior Localization Method and System Based on Fragment Feature Propagation

Abstract

This invention proposes a weakly supervised temporal action localization method and system based on segment feature propagation, belonging to the field of computer vision technology. The method includes extracting video segment features; obtaining representative segments for each video segment to obtain intra-segment video features; modeling the relationships between multiple video segments to obtain inter-segment video features; iteratively propagating the intra-segment and inter-segment video features among multiple video segments to obtain updated intra-segment and inter-segment video features; constructing pseudo-labels; obtaining uncertainty scores based on the updated inter-segment video features; using the uncertainty scores to weight the pseudo-label loss to guide the detector to be trained to learn from the pseudo-labels, resulting in a trained detector for action detection in the video to be processed. This invention can filter background frames and compensate for short-term action information, update representative action features, propagate representative information within and between videos, generate better pseudo-labels, and improve localization performance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a weakly supervised temporal behavior localization method and system based on fragment feature propagation. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of computer technology, internet videos are being generated and disseminated at an unprecedented pace. Compared to previous forms of expression such as pure text and images, video content is richer, more diverse, and more eye-catching, making it an important medium for information dissemination and expression. Against this backdrop, there is an urgent need for tools to classify, identify, and detect massive amounts of video data, leading to the emergence of video content understanding algorithms. Temporal behavior localization, as a subtask within the field of video content understanding, can analyze unedited long videos, predict the start and end times of behaviors within the video, and determine the category of those behaviors. Its application scenarios are more extensive, providing crucial support and impetus for the intelligent automation of video analysis technology.

[0004] Currently, the successful application of deep learning technology in image processing provides a solution for this task. Existing temporal behavior localization models mainly employ two learning methods: fully supervised and weakly supervised. The difference lies in the training method: the former uses frame-level behavior instance annotations as labels, while the latter uses video-level coarse-grained labels. In comparison, fully supervised models require precise temporal boundary annotation information, and considering the subjectivity of annotators, they incur significant annotation costs and effort. Therefore, weakly supervised temporal behavior localization has gained widespread attention due to the ease of obtaining labels and the objectivity of annotation.

[0005] Because real-life videos are long and have complex and varied backgrounds and actions, resulting in complex video structures, current research is still in its early stages. There are still problems such as the easy confusion between background, context, and actions, as well as the difficulty in modeling temporal relationships. This makes it impossible for weakly supervised temporal behavior localization algorithms to meet real-world needs. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a weakly supervised temporal behavior localization method and system based on fragment feature propagation. This method can filter out useless background frames and compensate for short-term action information, thereby improving the accuracy of short action localization. At the same time, it updates the representative features of the action and propagates representative information within and between videos to generate better pseudo-labels to supervise behavior suggestions, further refine action boundaries, and improve localization performance.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of this invention provides a weakly supervised temporal behavior localization method based on fragment feature propagation.

[0009] The weakly supervised temporal behavior localization method based on fragment feature propagation includes the following steps:

[0010] The video to be processed is obtained, divided into multiple video segments, and the features of each video segment are extracted and input into the detector to be trained.

[0011] We use expectation-maximization attention to obtain representative segments of each video clip to obtain intra-segment video features; and we use a self-attention mechanism to model the relationships between multiple video clips to obtain inter-segment video features.

[0012] The intra-segment and inter-segment video features are iteratively propagated among multiple video segments to obtain updated intra-segment and inter-segment video features.

[0013] Pseudo-labels are constructed, and uncertainty scores, temporal activation sequences, and foreground-background attention are obtained based on the updated inter-segment video features. The uncertainty score-weighted pseudo-label loss is used to guide the detector to be trained to learn from the pseudo-labels, resulting in the final trained detector.

[0014] Based on the finally trained detector, attention-weighted video-level action probabilities are obtained to achieve action detection in the video to be processed.

[0015] A second aspect of the present invention provides a weakly supervised temporal behavior localization system based on fragment feature propagation.

[0016] A weakly supervised temporal behavior localization system based on fragment feature propagation includes:

[0017] The video segment feature acquisition module is configured to: acquire the video to be processed, divide the video to be processed into multiple video segments, extract the features of each video segment and input them into the detector to be trained;

[0018] The intra-segment video feature and inter-segment video feature acquisition modules are configured to: use expectation-maximization attention to acquire representative segments of each video segment to obtain intra-segment video features; and use a self-attention mechanism to model the relationship between multiple video segments to obtain inter-segment video features.

[0019] The iterative propagation module is configured to iteratively propagate the intra-segment video features and inter-segment video features among multiple video segments to obtain updated intra-segment video features and inter-segment video features.

[0020] The pseudo-label training module is configured to: construct pseudo-labels, obtain uncertainty scores, temporal activation sequences and foreground-background attention based on the updated inter-segment video features, use uncertainty scores to weight pseudo-label loss, guide the detector to be trained to learn from the pseudo-labels, and obtain the finally trained detector;

[0021] The detection module is configured to: obtain attention-weighted video-level action probabilities based on the finally trained detector, and realize the detection of actions in the video to be processed.

[0022] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the weakly supervised temporal behavior localization method based on fragment feature propagation as described in the first aspect of the present invention.

[0023] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the weakly supervised temporal behavior localization method based on fragment feature propagation as described in the first aspect of the present invention.

[0024] The above one or more technical solutions have the following beneficial effects:

[0025] This invention provides a weakly supervised temporal behavior localization method and system based on fragment feature propagation. It can perform background filtering, filter out useless background frames and compensate for short-term action information, thereby improving the accuracy of short action localization. At the same time, it propagates representative information within and between videos, thereby generating better pseudo-labels to supervise behavior suggestions, further refining action boundaries and improving localization performance.

[0026] To address the impact of pseudo-label noise, this invention uses an FC layer to output an uncertainty score. This score is then used to reweight the pseudo-label loss at each time step, guiding the model to learn better from noisy pseudo-labels.

[0027] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0028] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0029] Figure 1 This is a flowchart of the method in the first embodiment.

[0030] Figure 2 This is a flowchart of the filtering sampling process for the first embodiment. Detailed Implementation

[0031] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0032] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0033] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0034] Research Background:

[0035] (1) Temporal behavior localization:

[0036] Fully supervised temporal action localization algorithms are divided into two categories: single-stage and two-stage. Single-stage algorithms utilize a unified framework to generate action nominations with high confidence scores; representative models include SSAD and SMS. Two-stage algorithms, on the other hand, divide action localization into candidate nomination generation and action classification. Action nomination generation is mainly based on sliding windows, action scores, and anchor point techniques; representative models include SCNN, BSN, and R-C3D. All these methods require precise temporal annotation for each action instance, consuming significant time and effort.

[0037] In addition, weakly supervised temporal behavior localization tasks have become a research hotspot in recent years, with major representative methods including STPN, HAM-Net, and CoLA.

[0038] (2) Weakly supervised temporal behavior positioning:

[0039] The weakly supervised temporal action localization task was formally proposed in the 2017 paper *UntrimmedNets*, becoming a benchmark for subsequent research. With the deepening of research, numerous methods and research results focusing on weakly supervised temporal action localization have emerged, making it a hot research topic at computer vision conferences such as CVPR, ECCV, and ICCV, with the number of papers published in this area increasing year by year, attracting more and more attention from researchers. In early research on weakly supervised temporal action localization, most researchers borrowed action recognition methods to complete the task. Typically, the video was first evenly divided into several segments, and then each segment was identified to distinguish between action and background. As research progressed, researchers combined this segment-level feature representation-dependent localization method with the characteristics of weakly supervised temporal action localization itself, designing algorithms more suitable for this task.

[0040] Specifically, UntrimmedNet was the first to introduce a multiple instance learning (MIL) framework to address this problem, selecting foreground segments and grouping them into action segments. STPN improves UntrimmedNet by enhancing the sparsity of the selected segments through a sparsity loss. CoLA utilizes contrastive learning to distinguish foreground and background segments. UGCT proposes an online pseudo-label generation method with an uncertainty-aware learning mechanism, providing pseudo-label supervision for attention weights. All these MIL-based methods process each segment in the video individually, ignoring the rich temporal information at the segment level. In contrast, our method focuses on modeling segment-level temporal structure for weakly supervised temporal action localization, which has been rarely explored in previous work.

[0041] (3) Pseudo-label supervision:

[0042] In weakly or limitedly supervised vision tasks, the use of pseudo-labels to guide model training has been widely adopted. In weakly supervised object detection, an important direction is self-training, where a base model is first trained, and then the predictions with high confidence are used as instance-level pseudo-labels to train the final detector. Similarly, in semi-supervised learning and domain adaptation, the model is first trained on labeled source datasets and then used to generate pseudo-labels for unlabeled target datasets to guide the training process.

[0043] Overall concept of the invention:

[0044] The task of weakly supervised temporal behavior localization is to automatically identify the categories of human behaviors contained in a long, undisturbed video sequence and pinpoint the start and end times of these behaviors. Its training model only requires video-level labels. Compared to strongly supervised temporal behavior localization, which incurs extremely high annotation costs, weakly supervised temporal behavior localization can complete behavior identification and localization without incurring significant costs, making it more practically significant and a research hotspot in the field of video understanding. Currently, due to the large scale of video data, complex backgrounds, and lack of precise action boundary information, existing weakly supervised temporal behavior localization methods still have shortcomings, such as the inability to effectively distinguish between foreground actions and background regions, the omission of short-term actions, and difficulties in modeling temporal relationships.

[0045] To address the above issues, this paper designs a weakly supervised temporal behavior localization method based on fragment feature propagation, which mainly includes the following three parts: (1) Filter sampling, used to filter redundant background information and compensate for the contribution of short-term actions; (2) Fragment feature propagation, which realizes the propagation learning of representative features by mining the most representative fragments in each video; (3) Pseudo-label supervision, used to generate better pseudo-labels to effectively alleviate the difference between classification and localization.

[0046] Experiments on the THUMOS14 and ActivityNet1.3 datasets demonstrate that our method can achieve relatively accurate temporal behavior localization.

[0047] Example 1

[0048] This embodiment discloses a weakly supervised temporal behavior localization method based on fragment feature propagation.

[0049] like Figure 1 As shown, the weakly supervised temporal behavior localization method based on fragment feature propagation includes the following steps:

[0050] The video to be processed is obtained, divided into multiple video segments, and the features of each video segment are extracted and input into the detector to be trained.

[0051] We use expectation-maximization attention to obtain representative segments of each video clip to obtain intra-segment video features; and we use a self-attention mechanism to model the relationships between multiple video clips to obtain inter-segment video features.

[0052] The intra-segment and inter-segment video features are iteratively propagated among multiple video segments to obtain updated intra-segment and inter-segment video features.

[0053] Pseudo-labels are constructed, and uncertainty scores, temporal activation sequences, and foreground-background attention are obtained based on the updated inter-segment video features. The uncertainty score-weighted pseudo-label loss is used to guide the detector to be trained to learn from the pseudo-labels, resulting in the final trained detector.

[0054] Based on the finally trained detector, attention-weighted video-level action probabilities are obtained to achieve action detection in the video to be processed.

[0055] The method described in this paper will be explained in detail below:

[0056] First, use the FAC-Net method to generate behavior suggestions. We perform preliminary estimations of the location and duration of actions in the untrimmed video. Based on the generated behavior suggestions, we mainly designed three modules, which correspond to the three stages of the weakly supervised temporal behavior localization model: feature extraction, feature modeling, and prediction.

[0057] Feature extraction stage: Given a video, it is divided into a series of non-overlapping segments. Then, a Kinetics-400 pre-trained I3D model is applied to extract RGB and optical flow features. Next, the RGB and optical flow features are concatenated along the channel dimension to form segment features F∈R. T×DWhere T is the number of video segments and D = 2048 is the feature dimension. Then, the features are fed into a temporal convolutional layer, activating ReLU to perform feature modeling as input to the model.

[0058] X = ReLU(conv(F)) (1)

[0059] Feature Modeling and Prediction Stages: A representative segment feature propagation framework is designed. To facilitate information propagation, we mine representative segments, which mitigate the impact of outliers and act as bridges for information propagation between segments. Specifically, we utilize Expectation-Maximization (EM) attention to handle variations caused by different camera views, sub-action differences, and confusing background contexts, and capture the important semantics of each video, which are segmented into representative segments in our method. Subsequently, to propagate information from the representative segments, we propose an iterative propagation module to update the features of the input video with representative segments within and between videos. The TCAM of the updated features is used as an online refined pseudo-label to correct the prediction of the main branch.

[0060] (I) W-TAL Basic Model

[0061] Most existing weakly supervised temporal behavior localization methods employ the Multiple Instance Learning (MIL) formula. A typical pipeline for MIL-based methods includes three main stages: (1) Feature extraction stage: Untrimmed RGB video and optical flow are used as inputs to extract segment-level features using a pre-trained backbone network. (2) Feature modeling stage: The extracted features are transformed into task-oriented features by performing temporal modeling. (3) Prediction stage: Class probabilities and attention weights are generated for each time step, and video-level loss is calculated according to the MIL formula during training.

[0062] Given embedded features X, a fully connected (FC) layer is applied to predict temporal class activation sequences (CAS). Here, C+1 represents the number of action categories plus the number of background categories. To better distinguish between foreground and background segments, a common strategy is to introduce an additional attention module that outputs attention weights for each time step of the undone video. Next, we use a fully connected (FC) layer to generate the attention weights. The two weights at each time step are normalized using a softmax operation to obtain the attention weights for the foreground and background. Finally, the CAS (Attention Component Analysis) is combined with the attention weights to obtain the attention-weighted CAS.

[0063]

[0064] Where 'c' represents the category and '⊙' represents element-wise multiplication.

[0065] According to the MIL formula, video-level classification scores are generated using a top-k mean strategy. For each class c, we take the k maximum values ​​of attention-weighted CAS and calculate their average:

[0066]

[0067] We then perform Softmax normalization across all classes to obtain attention-weighted video-level action probabilities. In this weakly supervised environment, we employ three video-level losses.

[0068] Foreground Loss: To guide training for video-level action classification, we apply foreground-attention-weighted action probabilities. With video-level action tags y fg =[y;0] Calculate the foreground loss:

[0069]

[0070] Background Loss: To ensure that negative instances in the undressed video are predicted as background, we use an additional background loss to regularize the background attention-weighted action probabilities. Specifically, we calculated With background category label y bg Cross-entropy between:

[0071]

[0072] Among them, y bg (C+1)=1, y bg (C) = 0.

[0073] Action-Aware Background Loss: Although no action occurs in the background fragment, we believe that rich contextual information can still be used to reflect the actual action category label. Because background instances are related not only to background class labels but also to action class labels, we describe the action-aware background loss as a background attention-weighted action probability. With video-level action tags y fg Cross-entropy loss between:

[0074]

[0075] The total loss of the base model is:

[0076] L tol =λ fg L fg +λ bg L bg +λ abg L abg

[0077] Where, λfg , λ bg and λ abg It is a trade-off parameter.

[0078] (II) Filtered Sampling

[0079] In this embodiment, each untrimmed video is first divided into 16 non-overlapping frames. Then, a Kinetics-400 pre-trained I3D model is applied to extract RGB and optical flow features. Afterward, the RGB and optical flow features are concatenated along the channel dimension to form the segment feature F∈R. T×D Next, the features are fed into a temporal convolutional layer, activating ReLU for feature modeling:

[0080] X = ReLU(conv(F)) (8)

[0081] Then input it into the filtering module, such as Figure 2 As shown in (a), the foreground weights W1∈R are output via two 1d convolutional layers and a sigmoid function. T (Values ​​range from 0 to 1), and the foreground weights are multiplied by the embedded features:

[0082]

[0083] in This represents element-wise multiplication over time.

[0084] To make the weights more biased towards 0 or 1, where 0 and 1 represent background and action frames respectively, thus making background frames more biased towards 0 and thus suppressed, while action frames remain unaffected, we regularize the filtering weights, where n is the nth video segment:

[0085]

[0086] Unedited video clips can vary in duration, from less than 2 seconds to over 1 minute, making their information prone to loss or distortion during the feature modeling phase. To address this issue, we designed a sampling module to amplify short-duration action clips, such as... Figure 2 As shown in (b) above, this module makes behavioral suggestions. The estimated duration is dynamically upsampled. First, the sampling weights are initialized so that their values ​​are 1 at all time steps. Then, the sampling weights are updated according to a predetermined threshold γ.

[0087]

[0088] Where s n ,e nThese represent the start and end times of the suggestion for the nth action, respectively; then, the cumulative distribution function is calculated for the sampling weights: f W =cdf(W2), which uses the inverse function of each suggested estimated duration to uniformly sample and amplify short action segments:

[0089]

[0090] (III) Fragment Feature Propagation Module

[0091] Different videos may contain the same action but have different backgrounds, camera angles, and action amplitudes. Therefore, we extract and learn representative segments of the same category from different videos to obtain intra-segment video features, helping the network to identify difficult action instances. Furthermore, we utilize a self-attention mechanism to model the relationships between multiple action proposals, obtaining inter-segment video features and capturing the temporal dependencies between action segments. Then, we iteratively propagate and learn the information from representative segments to further improve recognition and localization performance.

[0092] Based on the features obtained from the feature extraction section, we employ Expectation-Maximization (EM) attention to extract representative segments from each video. EM attention uses a specific EM algorithm based on Gaussian Mixture Model (GMM), employing a separate GMM to capture features for each video and... ( The distribution of the i-th segment feature is modeled as a linear combination of Gaussian distributions, as shown below:

[0093]

[0094] Where n is the number of Gaussian distributions, and z ik Let represent the mean, covariance, and weights of the k-th Gaussian distribution, respectively. In the following formulas, we will replace the covariance with the identity matrix I.

[0095] EM attention starts from the randomly initialized mean To begin the calculation, in the t-th iteration, first calculate the new weights of the Gaussian function.

[0096] Z (t) =softmax(λNorm2(X) fs Norm2(μ (t-1) ) T (14)

[0097] Where λ represents the hyperparameter controlling the smoothness of the distribution; Norm2(X fs ) indicates along X fsThe l2 norm of each row; the softmax function is executed along each row of Z, therefore Representing fragment features x i The probability generated by the k-th Gaussian function. Then, the mean μ is iteratively updated:

[0098] μ (t) =Norm1(Z (t) ) T X fs (15)

[0099] Norm1(Z) (t) ) represents Z (t) The column-by-column l1 normalization. Therefore, the above formula applies to feature X. fs Perform a weighted summation to update the mean, Norm1(Z) (t) The i-th row and k-th column of ) represent feature x i The k-th Gaussian membership value. Meanwhile, normalization also ensures that the updated μ and X... fs In the same embedding space, therefore, alternately executing the two formulas (13) and (14) above can capture the global context of the video in a non-local but more efficient way, thereby obtaining more representative intra-segment feature fragments.

[0100] Action segments in undone videos often exhibit temporal dependencies, with different actions occurring consecutively, and the same type of action may appear repeatedly within a single video. Therefore, capturing these dependencies and interactions between action segments can improve recognition and localization performance. Firstly, in the temporal dimension, video segment X... fs The aggregated video segment features X are obtained by performing average pooling. n Then, a self-attention mechanism is used to model the relationships between multiple video segments:

[0101] Q = X n W q K = X n W k V = X n W v (16)

[0102]

[0103] X a =X fs +BN(AVW o (18)

[0104] Among them, W q W k W v W o ∈RD × D To generate a linear projection matrix for queries, keys, values, and output.

[0105] We then propagate representative segments to the segment features X of the current video. fs The propagation process is as follows:

[0106]

[0107] Here, ω represents a parameter that controls the trade-off between feature propagation and the original features, ensuring that the updated features do not deviate too much from the original features, and its value ranges from [0,1]. However, even if the representative segment has a high similarity to most segments of the same class, it is impractical to fully propagate the information of the representative segment through a single propagation. Therefore, we propose an iterative propagation module to achieve multiple propagations, fully integrating the information of the representative segment into the video segment features.

[0108] The propagation process is performed through multiple iterations. In the t-th iteration, the propagation process can be represented as follows:

[0109]

[0110]

[0111] In the formula and μ *(0) These are inter-segment video features and representative segments (intra-segment video features), respectively. This process will... * Fixed, alternating updates for X fs and μ * Therefore, representative segments are not only used to disseminate representative information, but also serve as X. fs This serves as a bridge for information propagation between features. Due to their representativeness, they allow for better information propagation among features of the same class. This process can be repeated multiple times to fully integrate the information from representative fragments. To avoid gradient vanishing or exploding caused by unfolding the computational graph, we obtain representative fragments that are further refined using the following formula:

[0112]

[0113] Therefore, after propagation through representative segments, we obtain the updated intra-view features X. intra and the updated inter-segment feature X inter .

[0114] (iv) Supervision of False Labels

[0115] Due to the lack of fragment-level annotations, standard MIL-based methods rely solely on video-level supervision provided by video-level action category labels. To further refine the localization of action boundaries, we leverage pseudo-labels provided by behavior proposals and propose a pseudo-label loss that offers more fine-grained supervision than video-level losses.

[0116] Given behavioral suggestions We construct pseudo-labels by assigning action labels to fragments belonging to behavior suggestions and background class labels to all other fragments. in via l1 normalization:

[0117]

[0118] Since behavioral suggestions are generated from model predictions, inaccurate pseudo-labels are inevitable. To address the impact of label noise, we introduce an uncertainty prediction module, using a fully connected (FC) layer to output an uncertainty score. Then, this score is used to reweight the pseudo-label loss at each time step, guiding the model to learn from noisy pseudo-labels. Intuitively, instances with high uncertainty scores don't significantly impact the loss. Adding the uncertainty score, the pseudo-label loss can be written as a time-dependent CAS P-value with respect to the pseudo-label. Average cross-entropy between:

[0119]

[0120] Here, β is a hyperparameter of the weight decay term, which prevents the uncertainty prediction module from predicting infinite uncertainty across all time steps (and thus results in zero loss).

[0121] Therefore, the total loss of the detector in this embodiment is:

[0122] L tol =λ fg L fg +λ bg L bg +λ abg L abg +λ p L p

[0123] Where, λ fg , λ bg , λ abg and λ p It is a trade-off parameter.

[0124] (V) Experimental Verification

[0125] The dataset is Thumos14:

[0126] The THUMOS14 dataset, proposed by the THUMOS challenge, is used for video action recognition and detection tasks. In the video action detection task, this dataset contains 200 validation videos and 213 test videos, covering 20 action categories. The validation set contains 3007 action instances, and the test set contains 3358 action instances. Each video contains an average of 15.5 action instances, belonging to approximately 1-2 action categories. The duration of these action instances varies, typically ranging from a few seconds to several minutes. Furthermore, the duration of each video also varies, unevenly distributed between tens of seconds and 26 minutes, with background areas accounting for 71% of the total video length. All of these factors pose challenges to the model's detection performance.

[0127] The dataset is ActivityNet 1.3:

[0128] The ActivityNet dataset, available in versions 1.2 and 1.3, is currently the largest temporal action detection dataset. Version 1.2 contains 4819 training videos and 2383 validation videos, covering 100 action categories. The training set contains 7151 action instances, and the validation set contains 3582, with each video containing 1.5 action instances. Most videos involve only one action category. Version 1.3 expands upon version 1.2, covering 200 action categories. This dataset contains 10024 training videos, 4926 validation videos, and 5244 test videos. The training and validation sets contain 15410 and 7654 action instances, respectively, while the test set lacks publicly available action instance annotations. Therefore, in this study, the training set is used to train the model, and the validation set is used to test it.

[0129] Example 2

[0130] This embodiment discloses a weakly supervised temporal behavior localization system based on fragment feature propagation.

[0131] A weakly supervised temporal behavior localization system based on fragment feature propagation includes:

[0132] The video segment feature acquisition module is configured to: acquire the video to be processed, divide the video to be processed into multiple video segments, extract the features of each video segment and input them into the detector to be trained;

[0133] The intra-segment video feature and inter-segment video feature acquisition modules are configured to: use expectation-maximization attention to acquire representative segments of each video segment to obtain intra-segment video features; and use a self-attention mechanism to model the relationship between multiple video segments to obtain inter-segment video features.

[0134] The iterative propagation module is configured to iteratively propagate the intra-segment video features and inter-segment video features among multiple video segments to obtain updated intra-segment video features and inter-segment video features.

[0135] The pseudo-label training module is configured to: construct pseudo-labels, obtain uncertainty scores, temporal activation sequences and foreground-background attention based on the updated inter-segment video features, use uncertainty scores to weight pseudo-label loss, guide the detector to be trained to learn from the pseudo-labels, and obtain the finally trained detector;

[0136] The detection module is configured to: obtain attention-weighted video-level action probabilities based on the finally trained detector, and realize the detection of actions in the video to be processed.

[0137] Example 3

[0138] The purpose of this embodiment is to provide a computer-readable storage medium.

[0139] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the weakly supervised temporal behavior localization method based on fragment feature propagation as described in Embodiment 1 of this disclosure.

[0140] Example 4

[0141] The purpose of this embodiment is to provide an electronic device.

[0142] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the weakly supervised temporal behavior localization method based on fragment feature propagation as described in Embodiment 1 of this disclosure.

[0143] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0144] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0145] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A weakly supervised temporal action localization method based on segment feature propagation, characterized in that, The method comprises the following steps: obtaining a to-be-processed video, dividing the to-be-processed video into a plurality of video segments, and respectively extracting video segment features and inputting the video segment features into a to-be-trained detector; obtaining representative segments of each video segment by using expectation maximization attention, and obtaining intra-segment video features; modeling the relationship between the plurality of video segments by using a self-attention mechanism, and obtaining inter-segment video features; iteratively propagating each intra-segment video feature and inter-segment video feature between the plurality of video segments, and obtaining updated intra-segment video features and inter-segment video features; constructing pseudo labels, obtaining an uncertainty score, a temporal class activation sequence and a foreground-background attention based on the updated inter-segment video features, weighting the pseudo label loss by using the uncertainty score, guiding the to-be-trained detector to learn from the pseudo labels, and obtaining a finally trained detector; obtaining an attention weighted video level action probability based on the finally trained detector, and realizing detection of actions in the to-be-processed video; the intra-segment video features and the inter-segment video features are iteratively propagated between the plurality of video segments, and the specific formula is: where t denotes the t-th iteration; is the initial intra-segment video feature; is a parameter that controls the trade-off between feature propagation and original features; is the segment feature of the current video; is the intra-segment video feature of the t-th iteration; is the inter-segment video feature of the t-th iteration; is the inter-segment video feature of the t-1-th iteration; is the new weight of the Gaussian function calculated at the t-th iteration; is the column-wise normalization of is a hyper-parameter that controls the smoothness of the distribution.

2. The weakly supervised temporal action localization method based on segment feature propagation of claim 1, wherein, the extracted video segment features specifically comprise: obtaining embedding features of each video segment; performing background filtering and dynamic sampling on the embedding features to obtain video segment features.

3. The weakly supervised temporal action localization method based on segment feature propagation of claim 2, wherein: The background filtering and dynamic sampling on the embedding features are specifically: outputting foreground weights by passing the embedding features through two convolutional layers and a sigmoid function; multiplying the embedding features by the foreground weights to suppress background frames and filter the background; obtaining an estimated duration of a behavior action in the video segment, updating a sampling weight, calculating a cumulative distribution function of the updated sampling weight, uniformly sampling by using an inverse function of the cumulative distribution function, expanding short-time action segments, and obtaining video segment features.

4. The weakly supervised temporal action localization method based on segment feature propagation of claim 1, wherein, The expectation maximization attention uses an expectation maximization algorithm based on a Gaussian mixture model, and obtains the intra-segment video features of each video segment by using a separate Gaussian mixture model.

5. The weakly supervised temporal action localization method based on segment feature propagation of claim 1, wherein, The pseudo label loss is represented as the average cross entropy between the temporal class activation sequence and the pseudo labels: wherein, is a hyperparameter of the weight decay term; is an uncertainty score; is a pseudo label; is a time class activation sequence; is a duration of the entire segment; is an action class.

6. The weakly supervised temporal action localization method based on segmental feature propagation of claim 5, wherein, The foreground loss, the background loss, the action-aware background loss and the total loss of the detector are as follows: the foreground loss is: wherein, foreground-attention weighted action probability; video-level action label; the background loss is: wherein, , ; is a background class label; is a background attention-weighted action probability; the action-aware background loss is: the total loss is: wherein , , and are trade-off parameters.

7. A weakly supervised temporal action localization system based on segment feature propagation, characterized in that: comprise: a video segment feature acquisition module configured to obtain a to-be-processed video, divide the to-be-processed video into a plurality of video segments, and respectively extract video segment features and input the video segment features into a to-be-trained detector; an intra-segment video feature and inter-segment video feature acquisition module configured to obtain representative segments of each video segment by using expectation maximization attention, and obtain intra-segment video features; modeling the relationship between the plurality of video segments by using a self-attention mechanism, and obtaining inter-segment video features; an iterative propagation module configured to iteratively propagate each intra-segment video feature and inter-segment video feature between the plurality of video segments, and obtain updated intra-segment video features and inter-segment video features; the intra-segment video features and the inter-segment video features are iteratively propagated between the plurality of video segments, and the specific formula is: where t denotes the t-th iteration; is the initial intra-segment video feature; is a parameter that controls the trade-off between feature propagation and original features; is the segment feature of the current video; is the intra-segment video feature of the t-th iteration; is the inter-segment video feature of the t-th iteration; is the inter-segment video feature of the t-1-th iteration; is the new weight of the Gaussian function computed at the t-th iteration; is the column-wise normalization of is a hyper-parameter that controls the smoothness of the distribution; The pseudo-label training module is configured to: construct a pseudo-label, obtain an uncertainty score, a temporal class activation sequence and a foreground-background attention based on the updated inter-segment video feature, guide the detector to be trained to learn from the pseudo-label by weighting the pseudo-label loss with the uncertainty score, and obtain a finally trained detector; The detection module is configured to: obtain an attention weighted video level action probability based on the finally trained detector, and realize detection of actions in the to-be-processed video.

8. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps in the weakly supervised temporal action localization method based on segment feature propagation according to any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and capable of running on the processor, characterized by The processor executes the program to implement the steps in the weakly supervised temporal action localization method based on segment feature propagation according to any one of claims 1-6.

Images