Weakly supervised video temporal action localization method based on adaptive temporal continuity

Abstract

This invention belongs to the field of computer vision and video understanding technology, and discloses a weakly supervised video temporal action localization method based on adaptive temporal continuity. The method includes acquiring an unedited video to be processed, extracting original segment features from the video, and generating an encoded feature sequence using a temporal encoder; performing parallel processing on the encoded feature sequence to generate class-independent action score sequences and class activation sequences, respectively; constructing a temporal continuity label generation module; constructing an instance dependency update module; and supervising the instance dependency update module using temporal continuity pseudo-labels to obtain the action localization result. This invention solves the technical problem in existing technologies where neglecting the temporal continuity of actions leads to severe instance confusion and boundary ambiguity in the model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and video understanding technology, and in particular relates to a weakly supervised video temporal action localization method based on adaptive temporal continuity. Background Technology

[0002] With the rapid development of multimedia technology and the Internet, video data is exploding. Temporal Action Localization (TAL), as a core task in the field of video understanding, aims to simultaneously identify action categories and accurately locate the start and end times of actions in unedited long videos. It has wide application value in fields such as intelligent video surveillance, human-computer interaction analysis, and video content summarization. Although traditional fully supervised TAL methods perform well in terms of accuracy, they heavily rely on frame-level fine-grained manual annotation. This not only requires huge human and material resources but is also extremely time-consuming, making it difficult to scale to massive amounts of video data. To reduce the dependence on high annotation costs, weakly supervised TAL (WTAL) technology, which uses only video-level category labels for training, is gradually replacing fully supervised methods and has become a research hotspot in academia and industry.

[0003] However, most existing mainstream weakly supervised temporal action localization methods are built on standard multiple instance learning frameworks. Their core paradigm is usually based on the "Bag of Frames" assumption, which treats video segments as unordered or independent units through a Top-K aggregation mechanism. While this paradigm simplifies model training, it exposes serious limitations when dealing with video scenes containing dense actions or complex backgrounds. Specifically, because existing methods generally adopt a "stateless" temporal modeling mechanism, the model often independently classifies and scores each video segment, thus ignoring the inherent process and temporal continuity of action instances evolving over time. The class activation sequences (CAS) generated by the model are usually flat and lack internal structure, providing only rough information such as "what action is included" and "approximate location," but failing to effectively distinguish whether a continuous high-response region is a complete long action instance or multiple semantically independent but temporally adjacent short action instances. This defect directly leads to a serious "instance confusion" problem, such as incorrectly merging multiple closely adjacent actions into an excessively long instance, or incorrectly segmenting a complete action instance into multiple fragments due to feature score fluctuations caused by background noise, which seriously affects the integrity and accuracy of the localization boundary.

[0004] Furthermore, existing technologies have significant shortcomings in constructing supervisory signals. Due to the lack of realistic frame-level boundary annotations, most existing pseudo-label generation strategies are based on simple thresholding, providing only binary "foreground / background" distinguishing signals. This coarse supervisory signal cannot convey fine-grained temporal structure information about "when the action starts (New)" and "how it continues (Continue)" to the model, making it difficult for the model to learn the complete lifecycle features of action instances during optimization, further exacerbating the fragmentation and boundary ambiguity of the detection results. Therefore, how to break through the limitations of the traditional "frame set" paradigm and design a method that can explicitly model the temporal state of action instances and adaptively generate high-quality temporal continuity supervisory signals under weak supervision, so as to effectively decouple adjacent action instances and accurately locate action boundaries, has become a key technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a weakly supervised temporal action localization method based on an adaptive temporal continuity network. By introducing an instance dependency update module and a temporal continuity label generation module, action instances are modeled as persistent temporal threads, explicitly decoupling adjacent action instances, thereby improving the accuracy and robustness of temporal action localization. This solves the technical problem in existing technologies where the neglect of the temporal continuity of actions leads to serious instance confusion and boundary ambiguity in the model.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by this invention is a weakly supervised temporal action localization method based on an adaptive temporal continuity network, comprising: S1: Obtain the unedited video to be processed, extract features from the original video segments, and generate an encoded feature sequence using a temporal encoder. ; S2: Encoding feature sequence Parallel processing is performed to generate class-independent action score sequences. and class activation sequences ; S3: Construct a temporal continuity tag generation module to generate fine-grained temporal continuity pseudo-tags containing the states of new instances and continuing instances. ; S4: Build the instance dependency update module to generate the current time. Instance dependency score ; S5: Utilizing Temporally Continuous Pseudo-Labels Supervised training is performed on the instance dependency update module to obtain action localization results. ,in, The start time of the action. The end time of the action. As an action category, This represents the confidence score.

[0007] Furthermore, the specific steps of S1 are as follows: S11: Divide the input video into non-overlapping video segments and use the I3D network pre-trained on the Kinetics dataset as a feature extractor to extract RGB features and optical flow features for each video segment. S12: Concatenate the RGB features and optical flow features to obtain the original fragment feature sequence. Where T is the number of segments. For feature dimension, Represents the set of real numbers; S13: Transform the original fragment feature sequence Input timing encoder , to obtain the encoded feature sequence ,in, This represents the encoded feature dimension.

[0008] Furthermore, the specific steps of S2 are as follows: S2.1: Construct action branches and classifier branches; S2.2: Encode the feature sequence characteristics The inputs are fed in parallel into the action branch and the classifier branch to obtain the action score sequence. and class activation sequences .

[0009] Furthermore, the specific steps of S3 are as follows: S3.1: Activate the class sequence and action score sequence Weighted fusion is performed to obtain the fusion activation sequence. The function is represented as follows: (1) in: To fuse activation sequences, For time steps, In the category dimension The operation of taking the maximum value above. CAS For class activation sequences, For balance coefficient, It is an action fraction sequence. For class activation sequences; S3.2: Utilizing the activation threshold of the fusion sequence and action score threshold For fusion activation sequences Perform binarization to generate a binary mask. As shown in equation (2), the binary mask is extracted. A continuous set of highly activated regions in the middle is used as the initial set of action segments. As shown in equation (3): (2) in: It is a binary mask. For logical AND operation, The activation threshold for the fusion sequence. The threshold for action score; }(3) in: For the initial set of fragments, This is the first initial action segment. This is the second initial action segment; S3.3: Initial set of action segments Perform a segment merging operation based on feature similarity, calculate the average value of features within each segment as a prototype representation, and obtain an optimized set of instances. ; S3.4: Based on the optimized set of instances Generate fine-grained temporal continuity pseudo-tags .

[0010] Furthermore, the specific steps of S4 are as follows: S4.1: Create a dynamic instance library As shown in equation (4), it is used to store the state of currently active action instances; (4) in: For dynamic instance libraries, For the first An action instance at time The cyclic state characteristics, For a moment The total number of active action instances stored in the instance library. The number of action instances; S4.2: At time step From the encoded feature sequence Obtain the frame features at the current time. Map it to a query As shown in equation (5), the dynamic instance library Mapping to key As shown in equation (6), a learnable vector is introduced. It is also mapped to a key representing the new instance. As shown in equation (7): (5) in: For query, For the features of the current frame, This is the first learnable projection matrix; (6) in: Instance library Mapped keys, This is the second learnable projection matrix; (7) in: As the key representing the new instance, The initial state vector for new learnable instances; S4.3: Utilize attention mechanisms for routing decisions, including query... Instance library Mapped key and the key representing the new instance After concatenation, the vector is input into a self-attention encoder to obtain the enhanced current frame query vector. The existing instance key vector enhanced by the self-attention mechanism The new instance key vector enhanced by the self-attention mechanism As shown in Equation (8), the correlation between the current frame and existing and new instances is calculated using the self-attention mechanism to generate the instance allocation probability distribution. As shown in equation (9): (8) in: This is the current frame query vector enhanced by the self-attention mechanism. The existing instance key vectors are enhanced by the self-attention mechanism. The new instance key vector is enhanced by the self-attention mechanism. For self-attention encoders, For connection; (9) in: For similarity calculation function, Assign probability distributions to instances. For the first enhanced by the self-attention mechanism The key vector of an existing action instance; It is a normalized exponential function; Furthermore, the specific steps of S5 are as follows: S5.1: Construct a system based on classification loss and temporal continuity loss The total loss function As shown in equation (10); (10) in: For temporal continuity loss, For hyperparameters, For the total loss function, For classification loss; S5.2: Calculate the temporal continuity loss using Focal Loss The network parameters are iteratively updated by minimizing this loss function to obtain the trained model; S5.3: After completing model training, remove the temporal continuity label generation module and convert the class activation sequence output by the classifier branch into a single class activation sequence. Action score of action branch output Instance dependency score output by the instance dependency update module The data is then fused to obtain the final action score sequence. As shown in equation (11): (11) Where: k is an adjustable parameter. For the final action score sequence, For instance dependency scores; S5.4: Using Soft-NMS to score the final action sequence Post-processing is performed to obtain the final motion localization result.

[0011] Furthermore, in S4.4, probability distributions are assigned based on instances. The specific steps for updating the instance library using a gated loop unit are as follows: S4.4.1 If the current frame is determined to belong to the first... If there is an existing instance, then its features Input the GRU unit corresponding to this instance and update its state characteristics. Its functional representation is shown in equation (12): (12) in: For the updated number An action instance at time The cyclic state characteristics, For the first An action instance at time The cyclic state characteristics, For gated loop unit; S4.4.2 If the current frame is determined to be a new instance, a new state entry is initialized in the instance library and tracking of the new action instance begins.

[0012] Furthermore, the specific steps in S5.4 are as follows: S5.4.1: Using the standard temporal thresholding method, from the final action score sequence Extract consecutive segments exceeding a preset threshold to generate candidate action proposals. ; S5.4.2: Use Soft-NMS to rank all candidate proposals by confidence score Sort in descending order to obtain the final motion localization result.

[0013] Furthermore, the network structure of the temporal continuity label generation module includes: a sequence fusion unit, a binarization mask generation unit, and a segment merging unit based on spatiotemporal features, used to fuse temporal proximity and feature semantic similarity to generate fine-grained temporal continuity pseudo-labels containing the states of new instances and continuing instances. .

[0014] Furthermore, the network structure of the instance dependency update module includes: a dynamic instance library maintenance unit, a self-attention-based routing scheduling unit, and a state update unit based on a gated loop unit. It is used to maintain the dynamic instance library to store the temporal state features of action instances, and to calculate the correlation between the current frame features and the state of each instance in the instance library and the new instance marker through the self-attention mechanism, so as to realize the routing decision of whether the current frame should belong to the continuation of an existing instance or the start of a new instance.

[0015] The beneficial effects of this invention are: 1. This invention breaks through the limitations of the traditional "frame set" paradigm and completely solves the multi-instance confusion problem. Most existing technologies are based on the standard Multiple Instance Learning (MIL) framework, processing video segments independently and ignoring the temporal dependencies of actions. This invention innovatively introduces an Instance Dependency Update Module (IDUM), modeling actions as stateful temporal threads with lifecycles. By maintaining a dynamic instance bank, this invention can, when processing each frame, use an attention mechanism to deeply interact with current features and historical instance states, explicitly deciding whether to "continue" the current action or "start" a new action. This mechanism enables the model to have "memory" capabilities like a human observer, effectively decoupling closely adjacent or semantically similar action instances, greatly reducing the probability of incorrectly merging adjacent actions, and significantly improving instance discrimination capabilities in complex scenes.

[0016] 2. A fine-grained temporal continuity supervision signal was constructed, significantly improving boundary localization accuracy. Addressing the issues of high noise and discontinuity in pseudo-labels under weak supervision, this invention proposes a temporal continuity label generation module (TCLGM). Unlike traditional methods that rely solely on a single score threshold to generate coarse binary labels, this module creatively integrates temporal proximity and feature similarity constraints. This mechanism intelligently identifies and repairs action fragmentation caused by occlusion, camera motion, or confidence fluctuations, reconnecting broken fragments into complete action entities. The resulting fine-grained pseudo-labels, containing "New" and "Continue" states, provide the model with a powerful structured supervision signal, forcing the model to learn the complete evolution of the action. This achieves accurate localization of the action's start and end boundaries, effectively solving the technical pain point of incomplete localization.

[0017] 3. The method exhibits excellent generalization ability and robustness, setting new performance records in multiple benchmark tests. Extensive and rigorous experimental validation was conducted on three mainstream benchmark datasets with different characteristics (THUMOS14, ActivityNet v1.2, and MultiTHUMOS). Experimental results show that the invention demonstrates superior adaptability for both action-intensive videos (such as MultiTHUMOS) and long videos with single actions (such as ActivityNet). Particularly on the highly challenging THUMOS14 dataset, the invention achieves a mean mAP of 51.2% (0.1:0.7), with particularly outstanding performance at high t-IoU thresholds. This performance not only significantly outperforms existing weakly supervised state-of-the-art (SOTA) methods but also rivals fully supervised methods in some metrics, demonstrating the significant advantages and practical application value of this invention in mining temporal structure information from videos.

[0018] 4. A virtuous cycle of online inference and offline optimization is established, improving the stability of model training. This invention achieves complementarity between online instance dependency updates and offline Temporal Continuity Label Generation Module (TCLGM). The refined labels generated by the offline module can correct the prediction bias of the online module, while the state features learned by the online module can, in turn, improve the discriminativeness of feature representation. This collaborative mechanism effectively suppresses noise interference common in weakly supervised learning, making model convergence more stable and enabling efficient extraction of high-quality action features with semantic consistency from unedited videos containing a large amount of background noise. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0020] Figure 1 This is an overall architecture diagram of the method of the present invention; Figure 2 This is a schematic diagram of the instance dependency update module structure in this invention; Figure 3 This is a schematic diagram of the temporal continuity tag generation module in this invention; Figure 4 These are the visualization results of the algorithm proposed in this invention, where (a) is Pole Vault and (b) is Hammer Throw. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, this invention proposes a weakly supervised temporal action localization method based on an adaptive temporal continuity network. The core of this invention lies in transcending the traditional "frame set" assumption and achieving accurate temporal modeling of action instances through online instance dependency updates and offline continuity tag generation. Specific steps include: S1: Obtain the unedited video to be processed, extract the original segment features of the video, and use a temporal encoder to generate a context-aware encoded feature sequence; S11: Divide the input video into non-overlapping video snippets (a video snippet V contains T non-overlapping RGB frames and optical flow frames). Use the I3D (Inflated 3D ConvNet) network pre-trained on the Kinetics dataset as a feature extractor. For each video snippet, extract 1024-dimensional RGB features and 1024-dimensional optical flow features. S12: Concatenate the RGB (red, green, blue) features and optical flow features to obtain the original fragment feature sequence. ,in For the number of segments, For feature dimension, It is the set of real numbers; S13: Transform the original fragment feature sequence The input is fed into a temporal encoder consisting of convolutional layers. In this process, local temporal context is captured through convolutional layers, and the output is an encoded feature sequence containing local temporal context information. ,in, The encoded feature dimension, the encoded feature sequence It serves as the common input for the subsequent three branches (i.e., the action branch, the classifier branch, and the instance dependency update module).

[0023] S2: Encoding feature sequence Parallel processing is performed to generate class-independent action score sequences and class-specific class activation sequences, respectively. S2.1: Construct action branches and classifier branches; The Actionness Head predicts a class-independent sequence of actionness scores using a fully connected layer and a sigmoid activation function. , is used to represent the probability that each frame belongs to a foreground action (not a background action).

[0024] The classifier head, through a fully connected layer, predicts the classifier's behavior. Class activation sequence of each action category (Class Activation Sequence, CAS), ,in This refers to the action category.

[0025] S2.2: Encode feature sequence The inputs are fed in parallel into the action branch and the classifier branch to obtain the action score sequence. and class activation sequences ; S3: Construct the Temporal Continuity Label Generation Module (TCLGM). This module mainly includes a sequence fusion unit, a binarization mask generation unit, and a fragment merging unit based on spatiotemporal features. The role of this temporal continuity label generation module is to overcome the action fragmentation problem caused by single threshold truncation under weak supervision. By fusing temporal proximity and semantic similarity of deep features, it generates fine-grained structured pseudo-labels containing "New" and "Continue" states offline. This provides accurate temporal supervision signals for the subsequent online instance dependency update module, assisting the model in learning the complete evolution process of actions.

[0026] Class-independent action score sequences and class activation sequences By combining temporal proximity and feature similarity, fine-grained temporal continuity pseudo-labels containing New and Continue states are generated. ; like Figure 3 As shown, to address the lack of frame-level annotations under weak supervision, this invention designs an offline module (i.e., a temporal continuity tag generation module) to generate high-quality, fine-grained temporal continuity pseudo-tags that include New and Continue states. This is used to monitor subsequent online modules. The specific steps are as follows: S31: Class activation sequence and action score sequence Weighted fusion is performed to obtain the fusion activation sequence. The function is represented as follows: (1) in: To fuse activation sequences, For time steps, In the category dimension The operation of taking the maximum value above. CAS For class activation sequences, For balance coefficient, It is an action fraction sequence. For class activation sequences; S32: Activate the threshold using the fusion sequence. (Used for initial screening of prospects) and action score threshold (For filtering background noise) fusion activation sequence Perform binarization to generate a binary mask. As shown in equation (2), based on binary mask Extracting contiguous highly activated regions as the initial set of action segments

[0027] (2) in: It is a binary mask. For logical AND operation, The activation threshold for the fusion sequence. The threshold for action score; }(3) in: For the initial set of fragments, This is the first initial action segment. This is the second initial action segment; S33: For the initial set of motion fragments To address the oversegmentation problem (i.e., a complete action is broken due to noise), a segment merging operation based on feature similarity is performed on the initial set of action segments: First, the average value of features within each segment is calculated as a prototype representation. If two adjacent segments... The time interval is less than the preset interval threshold. Furthermore, the cosine similarity of the prototype features is greater than the preset similarity threshold. Then, the two fragments are merged into a complete action instance. This operation uses feature consistency to repair the internal breaks in the action, resulting in an optimized set of instances. ; The specific merging rules are as follows: For any two action segments and If they both meet the following two conditions, then they are merged: 1) Time interval , Set a preset interval threshold (e.g., 5 segment durations); 2) Feature cosine similarity , To set a similarity threshold (e.g., 0.8). Action clips The prototype feature vector, Action clips The prototype feature vector (i.e., the average of the encoded features of all frames contained in the segment) is used to characterize the semantic information of the segment.

[0028] To address the common problems of "fragmented motion" and "incomplete localization" in weakly supervised scenarios, S33 of this invention makes targeted improvements. Since occlusion, deformation, or changes in lighting often cause a sudden drop in confidence during long actions, leading to incorrect segmentation into independent short actions by conventional methods, this step introduces a dual constraint mechanism to determine if the two segments before and after the break are sufficiently close in time ( And they are highly similar in semantic features. This involves merging these into a single action instance, enabling the model to effectively tolerate intermediate noise interference and accurately recover the complete action temporal boundaries. These thresholds were determined through extensive ablation experiments. Set to 5. Setting it to 0.8 provides the best performance.

[0029] S34. Using instance collections Generate fine-grained temporal continuity pseudo-tags , The generation rules are as follows: For each instance The starting frame is marked as "1" (indicating New), subsequent frames within the instance are marked as "0" (indicating Continue), and background frames are marked as ignored.

[0030] This invention breaks through the static limitations of existing WTAL (Weakly-supervised Temporal Action Localization) technology, which relies solely on class activation sequences (CAS) for single-dimensional hard threshold truncation and ignores the semantic connections between segments. It creatively introduces a dual constraint mechanism of "feature similarity" and "temporal proximity," using the cosine similarity of deep features to "stitch" together time segments interrupted by noise. It abandons the traditional "score-oriented" and simple temporal smoothing thinking. On this basis, it further breaks through the limitations of traditional binary labels (0 / 1), generating fine-grained structured state labels containing "New Instance" and "Continue Instance" based on the optimized instance set.

[0031] S4: Build the Instance Dependency Update Module (IDUM); Phase S4 constructs an Instance Dependency Update Module (IDUM). For example... Figure 2 As shown, this module is an online, stateful controller whose core purpose is to solve the instance confusion problem. IDUM utilizes an attention mechanism and a gated loop unit to predict in real time whether the current video frame is a continuation of an existing action instance or the beginning of a completely new instance. The module dynamically updates the state of its instance library based on this prediction result. The Instance Dependency Update Module (IDUM) mainly includes a dynamic instance library maintenance unit, a self-attention-based routing and scheduling unit, and a state update unit based on a gated loop unit (GRU). The role of this instance dependency update module is: as an online, stateful controller, it explicitly stores the evolution state of action instances by maintaining a dynamic instance library, analyzes the correlation between the current frame and historical instances and new instance tokens using a self-attention mechanism, thereby accurately deciding whether the current frame is a continuation of an existing instance or the beginning of a new instance, and uses GRU to update the instance state in real time, effectively solving the instance confusion problem.

[0032] S41: Create a dynamic instance library As shown in equation (4), the state of the currently active action instance is used to store the state of each action instance in the dynamic instance library. It is a feature vector updated via GRU; (4) in: For dynamic instance libraries, For the first An action instance at time The cyclic state characteristics, For time step The total number of active action instances stored in the instance library. The number of action instances; S42: At time step From the encoded feature sequence Obtain the frame features at the current time. (Right now Map it to a query As shown in equation (5), the dynamic instance library Mapping to key As shown in equation (6), a learnable vector is introduced. It is also mapped to a key representing the new instance. As shown in equation (7): (5) in: For query, For the features of the current frame, This is the first learnable projection matrix; (6) in: Instance library Mapped keys, This is the second learnable projection matrix; (7) in: As the key representing the new instance, The initial state vector for new learnable instances.

[0033] S43: Utilize attention mechanisms for routing decisions, including query... Dynamic instance library Mapped key and the key representing the new instance After concatenation, the vector is input into a self-attention encoder to obtain the enhanced current frame query vector. Enhanced existing instance key vector The enhanced new instance key vector As shown in Equation (8), the correlation between the current frame and existing and new instances is calculated using the self-attention mechanism to generate the instance allocation probability distribution. As shown in equation (9): (8) in: This is the current frame query vector enhanced by the self-attention mechanism. The existing instance key vectors are enhanced by the self-attention mechanism. The new instance key vector is enhanced by the self-attention mechanism. For self-attention encoders, For connection; (9) in: For similarity calculation function, Assign probability distributions to instances. For the first enhanced by the self-attention mechanism The key vector of an existing action instance, It is a normalized exponential function.

[0034] The self-attention encoder adopts the standard Transformer encoder structure. Specifically, it takes the concatenated vector sequence as input, generates query (Q), key (K), and value (V) matrices through linear mapping, and calculates the enhanced feature representation. This structure enables the current frame to simultaneously engage in full-to-full information interaction with all historical instances and new instance options.

[0035] S44: Instance assignment probability distribution based on S43 Calculate the confidence score of the current frame belonging to an existing action instance (e.g., take the maximum or sum of the probabilities assigned to existing instances), and use it as the current moment. Instance dependency score And use it as the output of the module, assigning a probability distribution based on the instance. The instance library is updated using a gated loop unit.

[0036] Assign probability distribution based on instance The steps for updating the instance library using a gated loop unit are as follows: if the current frame is determined to belong to the first... If there is an existing instance (i.e., the instance with the highest similarity), the GRU gating update mechanism is triggered, and the current frame features are updated. Enter the first The GRU unit corresponding to each instance uses reset and update gates to fuse the current frame information into the historical state. The updated features are expressed in the form of a function, as shown in equation (10): (10) in: For the updated number An action instance at time The cyclic state characteristics, For the first An action instance at time The cyclic state characteristics.

[0037] If the current frame is determined to be a new instance, then in the dynamic instance library A new state entry is initialized and tracking of the new action instance begins.

[0038] This mechanism ensures that the instance library always maintains the latest accumulated state of action instances, thus effectively solving the problem of insufficient time-series dependency in existing technologies; S5: Utilizing Temporally Continuous Pseudo-Labels Supervised training is performed on the instance dependency update module to obtain action localization results; In the specific implementation of this application, firstly, the instance dependency update module (IDUM) is trained under supervision using the temporal continuity pseudo-labels generated by S3; secondly, the entire network is jointly optimized end-to-end by combining the video-level classification loss function; finally, in the model inference stage, the output scores of the classifier, action branch and instance dependency update module are fused to generate accurate action localization results.

[0039] S51: Construct a system based on classification loss and temporal continuity loss The total loss function As shown in equation (10), the temporal continuity loss As shown in equation (11); (10) in: For hyperparameters, For the total loss function, The classification loss is a standard multi-instance learning classification loss (including Top-K MIL loss), used to optimize the backbone network. The temporal continuity loss is based on FocalLoss and is used to calculate the predicted probability of the IDUM output. Pseudo-tags generated by TCLGM The differences between them; (11) in: For set , For set The cardinality, which is the number of elements in the set, is also the total number of foreground time steps. This is the balancing factor for Focal Loss, used to balance the weights of positive and negative samples. This is the focusing parameter of Focal Loss, used to adjust the degree of attention given to easy and difficult samples.

[0040] Temporal continuity loss The model is forced to learn to distinguish between the initial and sustained states of an action.

[0041] S52: Calculate the temporal continuity loss using Focal Loss The predicted probability of the supervised instance depends on the update module. Approximating the pseudo-labels generated by S3 The network parameters are iteratively updated by minimizing this loss function to obtain the trained model; S53: After model training, remove the temporal continuity label generation module and convert the class activation sequence output by the classifier branch into a single class activation sequence. Action score of action branch output Instance dependency score output by the instance dependency update module The data is then fused to obtain the final action score sequence. As shown in equation (12): (12) Where: k is an adjustable parameter. For the final action score sequence, For instance dependency scores; S54: Using Soft-NMS to score the final motion sequence Post-processing is performed to obtain the final motion localization result. ,in The start time of the action. The end time of the action. As an action category, This represents the confidence score.

[0042] The specific steps are as follows: S5.4.1: Using the standard temporal thresholding method, from the final action score sequence... Extract consecutive segments exceeding a preset threshold to generate candidate action proposals. ; S5.4.2: Use Soft-NMS to rank all candidate proposals by confidence score Sort in descending order to obtain the final motion localization results. .

[0043] In one embodiment of the present invention, the Soft-NMS (Soft Non-Maximum Suppression) used differs from the traditional NMS (Non-Maximum Suppression) which directly deletes low-scoring proposals with excessive overlap (t-IoU, Temporal Intersection over Union) with high-scoring proposals. Soft-NMS attenuates the confidence of low-scoring proposals based on the overlap, that is, the larger the t-IoU between a proposal and a higher-scoring proposal, the more severely its score is attenuated, and the proposals that are ultimately retained constitute the action localization result.

[0044] The inventive point of this invention is: 1. State-based instance modeling mechanism: Unlike existing technologies that treat video as an unordered collection of frames, this invention innovatively proposes an instance dependency update module, introduces a dynamic instance library and [NI] Token mechanism, and treats actions as "threads" with lifecycles, thereby achieving precise decoupling of adjacent similar actions.

[0045] 2. Dual-constraint pseudo-label generation strategy: A temporal continuity label generation module is proposed, which creatively combines two physical dimensions, "temporal proximity" and "feature similarity," to merge fragmented segments and generate fine-grained supervision signals that distinguish between "New" and "Continue," effectively solving the label noise problem under weak supervision.

[0046] To verify the effectiveness of the method of the present invention, extensive comparative experiments were conducted on three internationally recognized standard datasets: THUMOS14 (THUMOS'14 Action Recognition), ActivityNet v1.2, and MultiTHUMOS.

[0047] Quantitative comparison results: As shown in Table 1, on the THUMOS14 dataset, the method of this invention achieved excellent results under different t-IoU thresholds. When t-IoU=0.5, the mAP (mean Average Precision) reached 42.5%, which is not only significantly better than existing weakly supervised methods (such as SDANet's 37.1%), but also comparable to some fully supervised methods (such as TAL-Net's 42.8%). The average mAP (0.1:0.7) reached 51.2%, a significant improvement compared to the baseline method. This demonstrates the significant advantage of this invention in accurately locating action boundaries.

[0048] Ablation experiment analysis: In order to verify the role of each module, the following experiments were conducted, as shown in Table 2: Using only the baseline model: mAP@AVG was 50.3%; adding the IDUM module but without TCLGM (using only simple threshold labels): mAP improved slightly to 50.5%.

[0049] Using the full model (Baseline + IDUM + TCLGM): mAP jumped to 51.2%. This directly proves that the fine-grained continuous labels generated by TCLGM are crucial for training IDUM, and the combination of the two can effectively improve performance.

[0050] Table 1. Comparison with existing methods on the THUMOS14 dataset.

[0051] Table 2 Ablation Experiment Analysis

[0052] Using qualitative results to demonstrate: such as Figure 4 In the visualization results shown, Figure 4 (a) shows “Pole Vault” and Figure 4 (b) In dense action scenarios such as "Hammer Throw", the baseline method tends to merge two adjacent actions or break a long action (i.e. instance confusion and incomplete localization). The prediction results (New / Continue signals) generated by the method of this invention can clearly separate closely adjacent actions, and the prediction boundary is highly consistent with the ground truth (GT).

[0053] In summary, this invention effectively solves the long-standing problems of instance confusion and incomplete localization in weakly supervised temporal action localization by introducing an adaptive temporal continuity network, and has extremely high academic value and application prospects.

[0054] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A weakly supervised video temporal action localization method based on adaptive temporal continuity, implemented according to the following steps: S1: Obtain the unedited video to be processed, extract features from the original video segments, and generate an encoded feature sequence using a temporal encoder. ; S2: Encoding feature sequence Parallel processing is performed to generate class-independent action score sequences. and class activation sequence ; S3: Construct a temporal continuity tag generation module to generate fine-grained temporal continuity pseudo-tags containing the states of new instances and continuing instances. ; S4: Build the instance dependency update module to generate the current time. Instance dependency score ; S5: Utilizing Temporally Continuous Pseudo-Labels Supervised training is performed on the instance dependency update module to obtain action localization results. ,in, The start time of the action. The end time of the action. As an action category, This represents the confidence score.

2. The weakly supervised video temporal action localization method based on adaptive temporal continuity according to claim 1, characterized in that, The specific steps for S1 are as follows: S11: Divide the input video into non-overlapping video segments and use the I3D network pre-trained on the Kinetics dataset as a feature extractor to extract RGB features and optical flow features for each video segment. S12: Concatenate the RGB features and optical flow features to obtain the original fragment feature sequence. Where T is the number of segments, For feature dimension, Represents the set of real numbers; S13: Transform the original fragment feature sequence Input timing encoder , to obtain the encoded feature sequence ,in, This refers to the encoded feature dimensions.

3. The weakly supervised video temporal action localization method based on adaptive temporal continuity according to claim 2, characterized in that, The specific steps for S2 are as follows: S2.1: Construct action branches and classifier branches; S2.2: Encode the feature sequence characteristics The inputs are fed in parallel into the action branch and the classifier branch to obtain the action score sequence. and class activation sequence .

4. The weakly supervised video temporal action localization method based on adaptive temporal continuity according to claim 3, characterized in that, The specific steps for S3 are as follows: S3.1: Activate the class sequence and action score sequence Weighted fusion is performed to obtain the fusion activation sequence. The function is represented as follows: （1） in: To fuse activation sequences, For time steps, In the category dimension The operation of taking the maximum value above. CAS For class activation sequences, For balance coefficient, It is an action fraction sequence. For class activation sequences; S3.2: Utilizing the activation threshold of the fusion sequence and action score threshold For fusion activation sequences Perform binarization to generate a binary mask. As shown in equation (2), the binary mask is extracted. A continuous set of highly activated regions in the middle is used as the initial set of action segments. As shown in equation (3): （2） in: It is a binary mask. For logical AND operation, The activation threshold for the fusion sequence. The threshold for action score; }（3） in: For the initial set of fragments, This is the first initial action segment. This is the second initial action segment; S3.3: Initial set of action segments Perform a segment merging operation based on feature similarity, calculate the average value of features within each segment as a prototype representation, and obtain an optimized set of instances. ; S3.4: Based on the optimized set of instances Generate fine-grained temporal continuity pseudo-tags .

5. The weakly supervised video temporal action localization method based on adaptive temporal continuity according to claim 4, characterized in that, The specific steps for S4 are as follows: S4.1: Create a dynamic instance library As shown in equation (4), it is used to store the state of currently active action instances; （4） in: For dynamic instance libraries, For the first An action instance at time The cyclic state characteristics, For a moment The total number of active action instances stored in the instance library. The number of action instances; S4.2: At time step From the encoded feature sequence Obtain the frame features at the current time. Map it to a query As shown in equation (5), the dynamic instance library Mapping to key As shown in equation (6), a learnable vector is introduced. It is also mapped to a key representing the new instance. As shown in equation (7): （5） in: For query, For the features of the current frame, This is the first learnable projection matrix; （6） in: Instance library Mapped keys, This is the second learnable projection matrix; （7） in: As the key representing the new instance, The initial state vector for new learnable instances; S4.3: Utilize attention mechanisms for routing decisions, including query... Instance library Mapped key and the key representing the new instance After concatenation, the vector is input into a self-attention encoder to obtain the enhanced current frame query vector. The existing instance key vector enhanced by the self-attention mechanism The new instance key vector enhanced by the self-attention mechanism As shown in Equation (8), the correlation between the current frame and existing and new instances is calculated using the self-attention mechanism to generate the instance allocation probability distribution. As shown in equation (9): （8） in: This is the current frame query vector enhanced by the self-attention mechanism. The existing instance key vectors are enhanced by the self-attention mechanism. The new instance key vector is enhanced by the self-attention mechanism. For self-attention encoders, For connection; （9） in: For similarity calculation function, Assign probability distributions to instances. For the first enhanced by the self-attention mechanism The key vector of an existing action instance; It is a normalized exponential function; S4.4: Instance assignment probability distribution generated based on S4.3 Calculate the confidence that the current frame belongs to an existing action instance, and use it as the current moment. Instance dependency score Assign probability distribution based on instance The instance library is updated using a gated loop unit.

6. The weakly supervised video temporal action localization method based on adaptive temporal continuity according to claim 5, characterized in that, The specific steps for S5 are as follows: S5.1: Construct a system based on classification loss and temporal continuity loss The total loss function As shown in equation (10); （10） in: For temporal continuity loss, For hyperparameters, For the total loss function, For classification loss; S5.2: Calculate the temporal continuity loss using Focal Loss The network parameters are iteratively updated by minimizing this loss function to obtain the trained model; S5.3: After completing model training, remove the temporal continuity label generation module and convert the class activation sequence output by the classifier branch into a single class activation sequence. Action score of action branch output Instance dependency score output by the instance dependency update module The data is then fused to obtain the final action score sequence. As shown in equation (11): （11） Where: k is an adjustable parameter. For the final action score sequence, For instance dependency scores; S5.4: Using Soft-NMS to score the final action sequence Post-processing is performed to obtain the final motion localization result.

7. The weakly supervised video temporal action localization method based on adaptive temporal continuity according to claim 5, characterized in that, In S4.4, probability distributions are assigned based on instances. The specific steps for updating the instance library using a gated loop unit are as follows: S4.4.1 If the current frame is determined to belong to the first... If there is an existing instance, then its features Input the GRU cell corresponding to this instance and update its state characteristics. Its functional representation is shown in equation (12): （12） in: For the updated number An action instance at time The cyclic state characteristics, For the first An action instance at time The cyclic state characteristics, For gated loop unit; S4.4.2 If the current frame is determined to be a new instance, a new state entry is initialized in the instance library and tracking of the new action instance begins.

8. The weakly supervised video temporal action localization method based on adaptive temporal continuity according to claim 6, characterized in that, The specific steps in S5.4 are as follows: S5.4.1: Using the standard temporal thresholding method, from the final action score sequence Extract consecutive segments exceeding a preset threshold to generate candidate action proposals. ; S5.4.2: Use Soft-NMS to rank all candidate proposals by confidence score Sort in descending order to obtain the final motion localization result.

9. The weakly supervised video temporal action localization method based on adaptive temporal continuity according to claim 1, characterized in that, The network structure of the temporal continuity label generation module includes: a sequence fusion unit, a binarization mask generation unit, and a segment merging unit based on spatiotemporal features. These units fuse temporal proximity and semantic similarity to generate fine-grained temporal continuity pseudo-labels that include the states of new and continuing instances. .

10. A weakly supervised video temporal action localization method based on adaptive temporal continuity according to claim 1, characterized in that, The network structure of the instance dependency update module includes: a dynamic instance library maintenance unit, a self-attention-based routing scheduling unit, and a state update unit based on a gated loop unit. It is used to maintain the dynamic instance library to store the temporal state features of action instances, and to calculate the correlation between the current frame features and the state of each instance in the instance library and the new instance marker through the self-attention mechanism, so as to realize the routing decision of whether the current frame should belong to the continuation of an existing instance or the start of a new instance.

Images