Video statement positioning method based on multi-stage aggregation Transformer model

Abstract

The invention discloses a video statement positioning method based on a multi-stage aggregation Transformer model, and in the video statement Transformer model, each video slice or word can adaptivelyaggregate and align information from all other video slices or words in two modes according to the semantics of the video slice or word. Through multi-layer superposition, finally obtained video statement joint representation has rich visual language clue capture capability, and finer matching can be realized. In the multi-stage aggregation module, the stage characteristics of the starting stage,the stage characteristics of the intermediate stage and the stage characteristics of the ending stage are connected in series to form the characteristic representation of the candidate segment. Because the obtained characteristics representation captures the specific information of different stages, the method is very suitable for accurately positioning the starting position and the ending position of the video clip. The two modules are integrated together to form an effective and efficient network, and the accuracy of video statement positioning is improved.

Description

technical field [0001] The invention belongs to the technical field of video sentence positioning and retrieval, and more specifically relates to a video sentence positioning method based on a multi-stage aggregation Transformer model. Background technique [0002] Video localization is a fundamental problem in computer vision systems with wide-ranging applications. In the past decade, a lot of research and related application system development have been carried out on video action localization. In recent years, with the rise of multimedia data and the diversification of people's needs, the problem of sentence localization in videos (video sentence localization) has gradually become important. The purpose of video sentence location is to locate a certain video segment corresponding to the sentence to be queried in a very long video. Compared with video action localization, sentence localization has greater challenges and broader application prospects, such as video retrie...

AI Technical Summary

Problems solved by technology

However, the average pooling operation completely discards the stage information, and cannot perform precise matching on different stages to achieve precise positioning.

Although full convolution operations or RoI Pooling operations can characterize different stages to a certain extent, they do not rely on explicit stage-specific features, so they are also deficient in more precise positioning.

Images