Attention regression-based video time sequence sentence positioning method and device

Abstract

The invention discloses an attention regression-based video time sequence sentence positioning method and device. The method comprises the following steps: encoding a video clip and a sentence by using a bidirectional long short term memory network on the basis of a three-dimensional convolutional neural network and a Glove word vector mechanism to characterize contents of the video clip and the sentence; building a symmetrical association between the video and the sentence through a multimodal attention mechanism according to the contents of the video clip and the sentence, so as to acquire attention weight vectors and attention weighted features of the video and the sentence; and outputting a positioning result of an acquired video time sequence sentence through an attention weight-basedregression mechanism and an attention weighted feature regression mechanism according to the attention weight vectors or attention weighted features of the video and the sentence. The method disclosed by the invention can keep contextual information in the video and the sentence and improve the efficiency of a sentence positioning process, so as to achieve the aim of improving the sentence positioning speed, the positioning accuracy and the positioning robustness.

Description

technical field [0001] The invention relates to the technical field of computer vision, in particular to a method and device for locating video time-series sentences based on attention regression. Background technique [0002] In the prior art, the video sequence sentence positioning method mainly includes: constructing a unified representation space between the video and the sentence, scanning in the video to generate several video segments to be selected for positioning, and projecting the sentence and the video segment to be selected into the unified representation space Compare and locate; scan the video to generate several video segments to be selected for positioning, and fuse the visual features of the video segments to be selected with the text features of sentences to generate multimodal features. Time series regression is performed on the basis of the multimodal features to generate a time deviation value between the video segment to be selected and the video segme...

AI Technical Summary

Problems solved by technology

[0003] The method adopted in the prior art has the following defects: the method of scanning in the video to generate the video segment to be selected has high computational cost and cannot adapt to the processing of long videos, so the scalability of the above video sequence sentence positioning method is not strong ; Separate the video segment to be selected from the global video for independent processing, blocking the interaction between specific video content and video context information, which is crucial to sentence positioning

Therefore, the accuracy rate of the above video temporal sentence positioning methods is not high; the above methods directly use the general long short-term memory network to extract sentence features, ignoring the key information for temporal positioning in the sentence, so they are not enough to mine sentence information.

Images