A Text-to-Video Cross-Modal Retrieval Method Based on Multi-Level Coding

Abstract

The invention discloses a text-to-video cross-modal retrieval method based on multi-level coding. The method includes: firstly obtaining preliminary features of video and text, and then separately analyzing the two modes through two multi-level coding network branches. Global, timing, and local information are encoded; audio features are extracted from the video side, and sentence features are extracted from the text side; finally, multiple encoded features are fused with multi-level features to obtain robust video and text expressions. Map the features of the two modalities into a unified common space through the fully connected layer, use the public space algorithm to learn the relationship between the two modalities, train the model in an end-to-end manner, and automatically learn the matching relationship between text and video, This enables cross-modal retrieval from text to video. The invention is a concept-free method, which can realize cross-modal retrieval without complex concept detection operations, and utilizes deep learning technology to greatly improve the performance and efficiency of retrieval.

Description

technical field [0001] The invention relates to the technical field of video cross-modal retrieval, in particular to a text-to-video cross-modal retrieval method based on multi-level coding. Background technique [0002] In recent years, due to the popularization of the Internet and mobile smart devices and the rapid development of communication and multimedia technologies, massive amounts of multimedia data are created and uploaded to the Internet every day. The rate of growth is increasing, and these multimedia data have become the most important source of information for modern people. With the advent of the 5G era, due to its features such as faster transmission speed, larger bandwidth and lower delay, it will further accelerate the growth of multimedia data, especially for video data, it will be easier for people to It is foreseeable that the amount of video data storage on the Internet will be huge in the future. Faced with such a huge amount of multimedia data, how ...

AI Technical Summary

Problems solved by technology

However, this concept-based retrieval method has the following shortcomings: First, text and video have very rich content, and it is generally difficult to fully describe its content through several concepts, resulting in loss of information; second, the performance of the retrieval model depends on the text and video content. A video concept extractor, but how to build an effective concept extractor is not easy; Third, because this type of retrieval method relies on complex concept modeling and concept matching, its retrieval efficiency is relatively low

Images