Weakly supervised temporal action localization method and device based on semantic and saliency knowledge collaborative propagation

By extracting salient foreground features using a two-stream network and a salient object detection network, and employing distillation to propagate pseudo-labels and contrastive learning methods, the problems of false detection and false negative detection in action detection in existing technologies are solved, thereby improving the accuracy of video temporal action localization.

CN119399825BActive Publication Date: 2026-06-23ZHEJIANG UNIV OF TECH
2 Cites 0 Cited by

Patent Information

Application Number
CN202411224619.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2026-06-23
Estimated Expiration
2044-09-03

AI Technical Summary

Technical Problem

Existing weakly supervised temporal action localization methods struggle to distinguish between foreground appearances blending into the background and action segments with drastic temporal changes when dealing with limited representations of appearance and motion features, leading to false positives and false negatives in action detection.

Method used

By extracting salient foreground features through a two-stream network and a salient object detection network, a salient perception branch is established. Pseudo-labels are propagated using a distillation method, and key fragment features are aggregated by combining a contrastive learning method. This achieves complementary learning of semantic and salient information, thereby improving the accuracy of action localization.

Benefits of technology

It improves the accuracy of video temporal action localization, enhances the ability to recognize and locate human behavior and actions, and reduces false detections and missed detections.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The weakly supervised temporal action localization method and device based on semantic and saliency knowledge collaborative propagation comprises the following steps: 1) extracting temporal features and saliency foreground features from an uncropped video; 2) constructing a basic branch and a saliency perception branch to respectively process the temporal features and the saliency target features to obtain a basic class activation sequence, a motion, appearance representation score and a saliency class activation sequence, and weighting and fusing the four sequences to obtain a fused action score sequence; 3) utilizing branch distillation and branch action consistency constraint to interact semantic information and saliency information, and perfecting the fused action score sequence; 4) extracting key segments and ambiguous segments of the basic branch and the saliency perception branch, and utilizing the key segments and the ambiguous segments between branches and within branches for comparative learning to improve feature representation, combining the distillation result to perfect the fused action score sequence and obtaining an action localization result. The present application can perceive subtle human actions and accurate temporal action boundaries in uncropped videos.
Need to check novelty before this filing date? Find Prior Art