Point supervision time sequence action positioning method based on global statistical optimization
By employing global statistical optimization and geometric reconstruction methods, the problem of overly complete pseudo-labels in point-supervised temporal action localization was solved, improving pseudo-label quality and localization accuracy, reducing computational costs, and achieving efficient and robust action localization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XI AN JIAOTONG UNIV
- Filing Date
- 2026-02-03
- Publication Date
- 2026-05-12
AI Technical Summary
Existing point-supervised temporal action localization methods suffer from "overcomplete" initial pseudo-labels, and deep learning-based optimization methods carry the risk of "negative optimization" and high costs, lacking effective utilization of the geometric center constraints of point annotations.
A global statistical optimization-based approach is adopted, which utilizes the global category duration distribution pattern and the geometric anchor point characteristics of point annotations to perform differential reconstruction of initial pseudo-labels. By constructing a global statistical optimization strategy that does not require training parameters, combined with a geometric reconstruction module and an exemption strategy, the boundary of pseudo-labels is optimized.
It significantly improves the temporal boundary accuracy and localization performance of pseudo-labels, reduces algorithm complexity and computational overhead, and enhances the robustness and efficiency of detection.
Smart Images
Figure CN122024323A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and video understanding technology, specifically relating to a point-supervised temporal action localization method based on global statistical optimization. Background Technology
[0002] With the rapid development of the internet and multimedia technologies, video data is exploding. Temporal Action Localization (TAL), a core task in the field of video understanding, aims to identify the specific categories of actions occurring in a long, uncropped video and accurately pinpoint their start and end times. This technology has broad application prospects in areas such as intelligent video surveillance, video content retrieval, human-computer interaction, and sports event analysis.
[0003] While traditional fully supervised temporal action localization methods have achieved good detection performance, they rely on frame-accurate boundary annotations, resulting in extremely high manual costs. To reduce annotation costs, point-supervised temporal action localization (P-TAL), which only requires annotation of one time point for each action instance, has gradually become a research hotspot.
[0004] In the field of point-supervised technology, the early mainstream paradigm followed a two-stage framework of "classification-guided localization". SF-Net (ECCV 2020), proposed by Ma et al., serves as the cornerstone of this field, establishing a pseudo-label mining mechanism under single-frame supervision. It uses labeled points as "seeds" to mine adjacent frames to generate pseudo-action labels. To address the problem of missing action "completeness" in point supervision, Lee et al. (ICCV 2021) introduced a greedy algorithm, attempting to search for the best pseudo-sequence among candidate sequences that can cover complete action instances, thereby alleviating over-localization or under-localization.
[0005] In recent years, to further overcome performance bottlenecks, researchers have begun to focus on reliability propagation and the utilization of temporal saliency information. Zhang et al.'s HR-Pro (AAAI 2024) constructed a hierarchical reliability propagation framework, which, for the first time, systematically utilized the inherent reliability of point annotations at different levels by introducing online memory updates at the fragment level and fine-tuning boundaries using regression heads at the instance level. Furthermore, addressing the "confidence-quality misalignment" bottleneck in traditional multi-instance learning frameworks, Xia et al. (CVPR 2024) proposed a method to realign confidence using temporal saliency information, attempting to calibrate the scoring quality of action proposals by introducing saliency priors.
[0006] Despite the continuous improvement in detection performance achieved by the aforementioned methods, the quality bottleneck of pseudo-labels remains significant in practical applications. To overcome this performance ceiling, a new paradigm of "pseudo-label refinement," represented by PseR (IEEE TMM 2025), has recently emerged. PseR proposes a three-stage framework comprising seed proposal generation, proposal propagation, and a refinement network. It attempts to generate proposal bags by perturbing the scale and center of the initial seed and then trains an additional refinement network to filter out better pseudo-labels. Although this paradigm surpasses previous two-stage frameworks in performance, existing point-supervised techniques still suffer from the following significant drawbacks: 1. Initial pseudo-labels suffer from "over-completeness": Taking the seed generation stage of HR-Pro and PseR as examples, the initial proposals generated by relying on multi-threshold truncation often suffer from severe "over-completeness". Due to the lack of explicit boundary constraints, the duration of the proposals generated by the model often far exceeds the actual duration of the action instance, containing too much background fragments.
[0007] 2. Deep learning-based optimization methods carry the risk of "negative optimization" and are costly: Methods such as PseR employ deep learning-based refinement strategies, requiring the construction of complex refinement networks (including selection and ranking modules) and dedicated training. This purely data-driven "self-learning" model lacks deterministic prior constraints. It generates a large number of candidates by randomly perturbing seed proposals and then relies on the network's predicted scores for selection. When the initial pseudo-labels are of high quality, this random perturbation and selection may introduce bias, leading to a decrease in pseudo-label quality. Furthermore, training additional refinement networks significantly increases the algorithm's complexity and computational cost, contradicting the goal of high efficiency and low cost in point-supervised tasks.
[0008] 3. Lack of effective utilization of point annotation geometric center constraints: Existing optimization methods often ignore the strong constraint effect of point annotations as action anchors. Although some methods (such as TSP-Net) attempt to utilize center priors, they still rely on complex deep networks for prediction. While PseR introduces a center perturbation mechanism, it essentially still relies on the network to regress the optimal position, rather than using statistical regularities to determine the optimal position. For most actions, manually annotated points are usually located in the core region of the action, which is a strong statistical regularity. If this regularity can be directly utilized for geometric reconstruction, rather than relying on unstable network regression, boundary drift can be corrected more efficiently and robustly.
[0009] In summary, how to effectively solve the "over-completeness" problem of pseudo-labels by utilizing the geometric characteristics of point annotations and the statistical regularities of datasets without introducing additional complex deep learning modules, and how to differentiate between high-quality and low-quality proposals, are key technical problems that urgently need to be solved in the field of point-supervised temporal action localization. Summary of the Invention
[0010] To overcome the shortcomings of the existing technologies, the present invention aims to provide a point-supervised temporal action localization method based on global statistical optimization. This method introduces a global statistical optimization strategy that requires no training parameters. Utilizing the global category duration distribution and the geometric anchor characteristics of point annotations, it differentially reconstructs the initial pseudo-labels, effectively solving the problem of over-complete pseudo-labels and avoiding the negative optimization impact of deep learning optimization methods on high-quality proposals, thereby significantly improving the accuracy of temporal action localization.
[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A point-supervised temporal action localization method based on global statistical optimization is proposed. Based on a point-supervised learning framework, a global statistical optimization strategy requiring no additional training parameters is constructed. This strategy jointly utilizes the global category duration distribution pattern and the geometric anchor characteristics of point annotations to differentially reconstruct the initial pseudo-labels. Specifically, the method includes the following steps: Step 1: Obtain the uncropped input video and the manually supervised annotation information of the action instances to be detected in the video. Use a pre-trained two-stream 3D convolutional neural network to process the input video and extract the feature sequence of the video. Step 2: Based on point-supervised annotation, generate an initial pseudo-label set using a one-stage point-supervised action detector; the initial pseudo-label set includes the predicted category, start time, end time, and confidence score for each action instance; Step 3: Construct a pseudo-label optimization strategy based on global statistics: Calculate the global reference duration for each action prediction category through the global category statistical modeling module, and establish a global duration standard; Step 4: Use the geometric reconstruction optimization module to differentiate the initial pseudo-label set: implement an exemption strategy for high-confidence pseudo-labels to prevent negative optimization; implement a geometric reconstruction strategy for low-confidence pseudo-labels, using the real annotation points as the center and combining the global reference duration to remove background noise; Step 5: Use the optimized pseudo-labels as supervision signals to train a fully supervised temporal action localization network. Use the trained network to perform action localization on the uncropped video and output the final action category and temporal boundaries.
[0012] Preferably, the pre-trained two-stream 3D convolutional neural network is an I3D network; the one-stage supervised action detector is a segment-level action discrimination network based on reliability-aware attention; the initial pseudo-label set is generated using the one-stage supervised action detector in step two, and the specific operation steps are as follows: Step 1: Input the feature sequence of the video into a one-stage point-supervised action detector, and output class activation sequence and class-independent attention sequence; Step 2: Perform multi-threshold truncation on the class activation sequence to generate candidate action proposals, and calculate the external-internal contrast score of each candidate action proposal as a confidence score; Step 3: Combining the location information of the point supervision annotations, select the proposals that cover the annotation points and have the highest confidence scores from the candidate action proposals as seed proposals to form the initial pseudo-label set.
[0013] Preferably, the steps for step three are as follows: Step 1: Traverse the initial set of pseudo-labels for the entire dataset and aggregate all pseudo-label proposals by action category; Step 2: Sort the pseudo-label proposals for each category by confidence score, and select high-quality pseudo-label proposals to build a reference sample set by setting a confidence threshold or a relative ranking threshold; Step 3: Calculate the statistical average of the duration of all pseudo-label proposals in the reference sample set, and define it as the global reference duration for this action category.
[0014] Preferably, the operation steps of the geometric reconstruction optimization module in step four are as follows: Step 1: For each pseudo-label proposal to be optimized, determine whether it contains real action annotation points within its time range; if it does, lock the annotation point closest to the center of the proposal as the geometric anchor point; Step 2: Determine whether the confidence score of the pseudo-label proposal meets the preset high confidence condition; if it does, the proposal is determined to be a high-confidence pseudo-label, exempted, and its original start and end times remain unchanged. Step 3: If the exemption determination logic is not met, the proposal is determined to be a low-confidence pseudo-label that needs optimization. The proposal is then geometrically reconstructed using the global reference duration and geometric anchor point, and its start and end times are updated.
[0015] Preferably, the specific calculation formula for geometric reconstruction in step 3 is defined as follows: First, the correction duration is calculated using the weighted fusion formula. : in, The original duration of the current pseudo-label proposal. For the current global reference duration of pseudo-label proposals, These are the preset fusion weight coefficients; Secondly, the modified duration is shrunk to obtain the final duration. : in, The preset shrinkage coefficient, and ; Finally, using the aforementioned geometric anchor points Centered on, based on the final duration Recalculate start time and end time : Preferably, if the confidence score of the proposal is higher than the preset absolute high confidence threshold, or higher than the relative high score threshold of the category, it is determined to be a high confidence pseudo-label.
[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Avoids "negative optimization" of high-quality proposals: Unlike existing deep learning-based optimization methods (such as PseR), this invention designs a differentiated reconstruction mechanism (exemption strategy). High-quality proposals with high confidence or high ranking within their class are directly retained, avoiding quality loss caused by random perturbations; only low-quality proposals are reconstructed, thus ensuring the robustness of the overall optimization effect.
[0017] 2. Effectively solves the problem of "over-complete" pseudo-labels: This invention addresses the issue that pseudo-labels generated by existing methods are generally too long. By statistically analyzing the average duration of high-confidence samples in each category as a priori, and introducing a shrinkage coefficient to correct the duration of low-confidence proposals, the invention effectively removes background noise at the proposal boundaries and significantly improves the temporal boundary accuracy (mIoU) of pseudo-labels.
[0018] 3. Fully utilizes the geometric constraints of point annotations: This invention abandons the approach of simply relying on regression prediction boundaries, and instead forces the reconstruction to use manually annotated time points as the geometric center of the action. This strong geometric constraint based on "point as center" not only conforms to the visual distribution patterns of most actions, but also effectively corrects the boundary drift problem caused by model prediction.
[0019] 4. Computationally efficient, requiring no additional training cost: The optimization strategy of this invention is based purely on statistical computation and geometric transformation. Unlike PseR, it does not require training an additional refinement network or increase the number of model parameters. It is a plug-and-play lightweight module that greatly reduces the complexity of the algorithm and the training overhead.
[0020] 5. Significantly improved final localization performance: Experimental results show that training a fully supervised model using the pseudo-labels optimized in this invention achieves detection performance that surpasses the current state-of-the-art methods on benchmark datasets such as THUMOS14, proving the effectiveness of the global statistical optimization strategy. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.
[0022] Figure 1 This is a detailed logic flowchart of the pseudo-label optimization strategy based on global statistics provided in the embodiments of the present invention, which shows the interaction process between global statistics and differentiated reconstruction.
[0023] Figure 2 This is a comparison of the visualization effects before and after different pseudo-label optimizations provided in the embodiments of the present invention, showing the process of reconstructing and shrinking the complete pseudo-label using manually marked points as geometric anchor points. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions adopted by this invention will be further described below in conjunction with the accompanying drawings.
[0025] like Figure 1 As shown, this embodiment proposes a point-supervised temporal action localization method based on global statistical optimization. This method aims to solve the problems of "over-complete" pseudo-labels in existing point-supervised methods and "negative optimization" in deep learning optimization methods. The method mainly includes the following steps: Step 1: Obtain the uncropped input video and the manually supervised annotation information of the action instances to be detected in the video. This information is then processed using a pre-trained 3D convolutional neural network (in this embodiment, the I3D network is used) to extract the video's feature sequence. Specifically, the input video is divided into non-overlapping video segments. The I3D network, pre-trained on the Kinetics-400 dataset, extracts the RGB features and optical flow features of each segment, and concatenates these features to obtain the segment's feature representation. .
[0026] Step 2: Based on point-supervised annotation, an initial pseudo-label set is generated using a one-stage point-supervised action detector (in this embodiment, a one-stage model of the HR-Pro network is selected). In one feasible implementation, this step includes: Step 1: Extract the feature sequence Input a one-stage network to generate a class activation sequence (CAS) and a class-independent attention sequence for each action category.
[0027] Step 2: Use multiple thresholds to truncate CAS to generate candidate action proposals and calculate their external-internal contrast (OIC) scores as confidence levels.
[0028] Step 3: For each labeled point, select the proposal that covers that point and has the highest OIC score as the "seed proposal". At this point, the pseudo-labels have a preliminary positioning capability, but they often have the problem of excessive boundary coverage.
[0029] Step 3: Construct a pseudo-label optimization strategy based on global statistics: Calculate the global statistical features of each action category, and use these global statistical features to perform differentiated reconstruction of the initial pseudo-label set. The core of this step lies in constructing a global category statistical modeling module and a geometric reconstruction optimization module. Step 1: In one feasible implementation, iterate through the initial pseudo-labels of the entire dataset and aggregate them by action category. For each category... : Step 1.1: Set the absolute high score threshold and ranking percentage threshold Select confidence level Or ranked higher The proposals constitute a reference sample set.
[0030] Step 1.2: Calculate the average duration of all proposals in the reference sample set, denoted as the global reference duration ( This parameter reflects the statistical duration distribution of this action category under ideal conditions.
[0031] Step 2: In one feasible implementation, perform an "exemption-restructuring" bidirectional logic for each pseudo-tag proposal: Step 2.1: First, check if the proposal contains real annotation points. If it does, take the nearest annotation point as the geometric anchor point. .
[0032] Step 2.2: If the proposal confidence level Or in the class before It was determined to be a high-quality proposal, and was directly exempted, keeping the original boundaries unchanged.
[0033] Step 2.3: If the exemption conditions are not met, the proposal is judged as low-quality and geometric reconstruction is performed. First, the fusion weight coefficient is used. The original duration of the current pseudo-label proposal is weighted and merged with the global reference duration, and then multiplied by a shrinkage factor. To significantly suppress excessive background noise, the final correction duration is calculated. Then, the motion is forced to expand to both sides based on the correction duration, with the actual annotation point as the geometric center, and the start and end times of the motion are regenerated.
[0034] Step 4: Fully Supervised Model Training and Prediction. The optimized pseudo-labels are used as supervision signals to train a fully supervised temporal action localization network (the ActionFormer network is used in this example). The trained network is then used to perform action localization on the uncropped video, and the final prediction results, namely the action category and time boundary, are output.
[0035] To verify the effectiveness of the method of this invention, a comparative experiment was conducted on the THUMOS14 dataset. Table 1 shows the comparison of pseudo-label quality before and after using the optimization strategy of this invention.
[0036] Table 1 Comparison of pseudo-label quality before and after optimization index Original pseudo-tags Optimized pseudo tags Changes mIoU 0.5783 0.6123 +3.4% High quality ratio 41.45% 49.25% +7.8% Overcompleteness 41.71% 38.21% -3.5% Average duration 6.54s 4.70s Closer to the true value (3.97s) As shown in Table 1, the average duration of the original first-stage proposal was 6.54s, far exceeding the average duration of the actual action (3.97s), and the over-completeness rate was as high as 41.71%, indicating severe background redundancy. After adopting the statistical optimization strategy of this invention, the average duration converged to 4.70s, the over-completeness rate significantly decreased to 38.21%, and the proportion of high-quality pseudo-labels increased significantly by nearly 8 percentage points. This shows that the "geometric reconstruction" strategy of this invention effectively removed redundant background, while the "exemption mechanism" successfully preserved the original high-quality detection results; the combination of the two significantly improved the overall quality of pseudo-labels.
[0037] like Figure 2 As shown, in order to intuitively verify the effectiveness of the global statistical optimization strategy in step three of the present invention, the "basketball dunk" action instance with video ID video_validation_0000906 was selected for visualization.
[0038] The first row of horizontal bars in the figure represents the true time boundary of the action; the second row of horizontal bars represents the initial pseudo-labels generated by the one-stage network of HR-PRO. It can be seen that the initial results have a serious "overcomplete" phenomenon and a large deviation in the starting time (301.44s-305.92s), and the intersection-over-union (IoU) with the true value is only 0.28. The vertical dashed line running through the top and bottom of the figure represents the given manual point supervision annotation position (305.8s), which is used as a geometric anchor point in this invention; The third row of horizontal bars represents the pseudo-labels processed by the global statistical optimization strategy of this invention. As can be seen from the figure, the optimized pseudo-labels are geometrically reconstructed and time-contracted (304.12s-307.48s) centered on the annotation point indicated by the vertical dashed line. Their boundaries are corrected to highly coincide with the true values, and the IoU is improved to 0.97, demonstrating the superiority of the method of this invention in correcting boundary drift and suppressing background noise.
[0039] Furthermore, to verify the advantages of the present invention in final localization performance, the ActionFormer model trained with optimized pseudo-labels was compared with current existing methods, and the results are shown in Table 2.
[0040] Table 2 Comparison of detection performance between the present invention and existing technologies (THUMOS14) Method 0.3 0.4 0.5 0.6 0.7 AVG(0.3:0.7) AVG(0.1:0.7) HR-Pro 74.3 64.3 52.2 39.8 24.8 51.1 60.3 PseR 78.7 70.9 60.1 45.4 29.6 56.9 64.8 Ours 79.9 73.4 63.3 50.9 32.6 60.0 67.2 As shown in Table 2, the method of this invention exhibits significant performance advantages at all IoU thresholds. Compared to HR-Pro, this invention improves performance by 6.9% on the AVG (0.1:0.7) metric, demonstrating that the improvement in pseudo-label quality directly translates into a leap in final detection performance. Compared to the latest PseR method, this invention improves performance by 2.4% on AVG (0.1:0.7) and by 3.1% on AVG (0.3:0.7). Particularly at high IoU thresholds (such as 0.6 and 0.7), this invention achieves improvements of 5.5% and 3.0%, respectively, indicating that the global statistical optimization strategy of this invention is more effective than PseR's deep learning optimization network in improving the accuracy of localization boundaries, while avoiding a complex training process.
Claims
1. A point-supervised temporal action localization method based on global statistical optimization, characterized in that: Based on a point-supervised learning framework, a global statistical optimization strategy requiring no additional training parameters is constructed. This strategy jointly utilizes the global category duration distribution pattern and the geometric anchor characteristics of point annotations to differentially reconstruct the initial pseudo-labels. Specifically, the steps include: Step 1: Obtain the uncropped input video and the manually supervised annotation information of the action instances to be detected in the video. Use a pre-trained two-stream 3D convolutional neural network to process the input video and extract the feature sequence of the video. Step 2: Based on point-supervised annotation, generate an initial pseudo-label set using a one-stage point-supervised action detector; the initial pseudo-label set includes the predicted category, start time, end time, and confidence score for each action instance; Step 3: Construct a pseudo-label optimization strategy based on global statistics: Calculate the global reference duration for each action prediction category through the global category statistical modeling module, and establish a global duration standard; Step 4: Use the geometric reconstruction optimization module to differentiate the initial pseudo-label set: implement an exemption strategy for high-confidence pseudo-labels to prevent negative optimization; implement a geometric reconstruction strategy for low-confidence pseudo-labels, using the real annotation points as the center and combining the global reference duration to remove background noise; Step 5: Use the optimized pseudo-labels as supervision signals to train a fully supervised temporal action localization network. Use the trained network to perform action localization on the uncropped video and output the final action category and temporal boundaries.
2. The point-supervised temporal action localization method based on global statistical optimization according to claim 1, characterized in that: The pre-trained two-stream 3D convolutional neural network is an I3D network; the one-stage supervised action detector is a fragment-level action discrimination network based on reliability-aware attention; in step two, the one-stage supervised action detector is used to generate an initial pseudo-label set, and the specific operation steps are as follows: Step 1: Input the feature sequence of the video into a one-stage point-supervised action detector, and output class activation sequence and class-independent attention sequence; Step 2: Perform multi-threshold truncation on the class activation sequence to generate candidate action proposals, and calculate the external-internal contrast score of each candidate action proposal as a confidence score; Step 3: Combining the location information of the point supervision annotations, select the proposals that cover the annotation points and have the highest confidence scores from the candidate action proposals as seed proposals to form the initial pseudo-label set.
3. The point-supervised temporal action localization method based on global statistical optimization according to claim 1, characterized in that: The steps for step three are as follows: Step 1: Traverse the initial set of pseudo-labels for the entire dataset and aggregate all pseudo-label proposals by action category; Step 2: Sort the pseudo-label proposals for each category by confidence score, and select high-quality pseudo-label proposals to build a reference sample set by setting a confidence threshold or a relative ranking threshold; Step 3: Calculate the statistical average of the duration of all pseudo-label proposals in the reference sample set, and define it as the global reference duration for this action category.
4. The point-supervised temporal action localization method based on global statistical optimization according to claim 1, characterized in that: The operation steps of the geometric reconstruction optimization module in step four are as follows: Step 1: For each pseudo-label proposal to be optimized, determine whether it contains real action annotation points within its time range; if it does, lock the annotation point closest to the center of the proposal as the geometric anchor point; Step 2: Determine whether the confidence score of the pseudo-label proposal meets the preset high confidence condition; if it does, the proposal is determined to be a high-confidence pseudo-label, exempted, and its original start and end times remain unchanged. Step 3: If the exemption determination logic is not met, the proposal is determined to be a low-confidence pseudo-label that needs optimization. The proposal is then geometrically reconstructed using the global reference duration and geometric anchor point, and its start and end times are updated.
5. The point-supervised temporal action localization method based on global statistical optimization according to claim 4, characterized in that, The specific calculation formula for geometric reconstruction in step 3 is defined as follows: First, the correction duration is calculated using the weighted fusion formula. : in, The original duration of the current pseudo-label proposal. For the current global reference duration of pseudo-label proposals, These are the preset fusion weight coefficients; Secondly, the modified duration is shrunk to obtain the final duration. : in, The preset shrinkage coefficient, and ; Finally, using the aforementioned geometric anchor points Centered on, based on the final duration Recalculate start time and end time : 。 6. The point-supervised temporal action localization method based on global statistical optimization according to claim 4, characterized in that, If the confidence score of the proposal is higher than the preset absolute high confidence threshold, or higher than the relative high score threshold of the category, it is determined to be a high confidence pseudo-label.