Language description guided video timing sequence positioning method

Abstract

The invention discloses a language description guided video timing sequence positioning method, which comprises the following steps of S1, extracting a multi-modal feature coding network of cross-modal features so as to learn cross-modal information of videos and natural languages, and obtaining multi-modal fusion representation of input languages and visual modals; s2, performing hierarchical decomposition on the cross-modal information by adopting a hierarchical tree structure strategy; step S3, employing progressive reinforcement learning mechanisms, correct credit allocation is provided through two task-oriented rewards; mutual promotion of different strategies in the tree structure is encouraged; according to the method, the decision process of human beings from coarse to fine is simulated through the progressive reinforcement learning framework based on the tree structure, complex action strategies can be effectively decomposed, the number of search steps is reduced while the search space is increased, and a more impressive result is obtained in a more reasonable mode.

Description

technical field [0001] The invention relates to the technical field of computer vision, in particular to a video timing positioning method for language description guidance based on tree structure and progressive reinforcement learning. Background technique [0002] The task of natural language video temporal localization is an emerging and challenging task in the field of computer vision and video analysis. Its goal is to determine the temporal boundaries of segments in untrimmed videos that correspond to a given natural language. This task is very related to the action timing localization task. Compared with the action timing localization task, this task is more challenging: 1) Not only does it not have a predefined action list and label, but it may contain complex descriptions. 2) This task requires the model to be able to establish the relationship between language modality and visual modality, to model multimodal features, and to have a deep understanding of natural la...

AI Technical Summary

Problems solved by technology

However, this method relies on the matching and ranking of the external sliding window, resulting in inefficiency, inflexibility of the bounding box and additional space consumption, and the method also deviates from the human perception mechanism

Images