Weak supervision video sequential action positioning method and system based on deep learning

Abstract

The invention discloses a weak supervision video sequential action positioning method and system based on deep learning, and the method comprises the following steps: S1, extracting a current frame and a previous frame in a video, extracting an optical flow through an optical flow estimation network, and inputting the optical flow and frames sampled by the video at equal intervals into a double-flow action recognition network to extract video features; S2, performing semantic consistency modeling on the video features to obtain embedded features; S3, mapping the embedded features to a class activation sequence by a training classification module; S4, updating the video features by adopting an attention module; S5, taking the updated video features as the input of the next cycle, and repeating S2 to S4 until stopping; S6, fusing class activation sequences generated by each cycle, and calculating classification loss of estimated action classes and real class labels; S7, fusing the embedded features of each cycle to calculate similarity loss between the action features; and S8, obtaining target loss according to the classification loss and the similarity loss, and updating system model parameters.

Description

technical field [0001] The present invention relates to the field of computer vision based on deep learning, in particular to a method and system for locating time-series actions in weakly supervised videos based on deep learning. Background technique [0002] Weakly supervised video temporal action localization refers to locating the start time and end time of action instances in a video while only relying on video-level action category annotations. Recently, this task has gradually attracted attention due to its wide application in other tasks in the field of computer vision, such as: dense video description, spatio-temporal detection of video actions. [0003] In recent years, temporal action localization technology has made great progress. Deep learning-based methods, especially convolutional neural networks, occupy an important place in it. For example, the research work "UntrimmedNets for Weakly Supervised Action Recognition and Detection" (In Proceedings of the IEEE...

AI Technical Summary

Problems solved by technology

However, the training process of this method is uncontrollable, and the semantic consistency of actions in videos is ignored, and the semantic consistency of actions is not explicitly modeled to guide the action localization process.

Images