Text-to-video cross-modal retrieval method based on multi-face video representation learning

Abstract

The invention discloses a text-to-video cross-modal retrieval method based on multi-face video representation learning. The method comprises the following steps: acquiring preliminary features of a video and a text; after the video initial frames are grouped according to different scenes by using a video splitting tool, the display coding branches are input for explicit coding, and explicit multi-surface representation of the different scenes of the video is obtained; inputting the initial features of the video into an implicit coding branch, and carrying out implicit coding on the initial features of the video through a preamble feature multiple attention network to obtain implicit multi-face representation for expressing different semantic contents of the video; fusing the multi-surface codes of the two branches to obtain multi-surface video feature representation; according to the method, multi-face video feature representation and text features are mapped into a public space, a public space learning algorithm is utilized to learn the relevancy between two modals, a model is trained in an end-to-end mode, and text-to-video cross-modal retrieval is achieved. According to the method, the retrieval performance is improved by utilizing a video multi-surface representation thought.

Description

technical field [0001] The invention relates to the technical field of video cross-modal retrieval, in particular to a text-to-video cross-modal retrieval method based on multi-faceted video representation learning. Background technique [0002] In recent years, due to the popularization of the Internet and mobile smart devices and the rapid development of communication and multimedia technologies, massive amounts of multimedia data are created and uploaded to the Internet every day. The rate of growth is increasing, and these multimedia data have become the most important source of information for modern people. Especially for video data, people will more easily upload and share the videos they create, and how to quickly and accurately retrieve the videos required by users is a daunting challenge. Text-to-video cross-modal retrieval is one of the key techniques to alleviate this challenge. [0003] Existing text-to-video cross-modal retrieval assumes that all videos do no...

AI Technical Summary

Problems solved by technology

However, this traditional model encoding output method has shortcomings: due to the characteristics of video and text, a video may have many different scenes as the photographer moves or the angle of view is switched during the shooting process, and a query text may It does not describe the entire content of the corresponding video, that is, the query text and the video are partially related

If the video is only expressed as a single feature vector, the multi-scene information in the video may be blurred, resulting in an inaccurate representation of the video, which ultimately affects the accuracy of the text-video retrieval results

Images