Video description method based on space-time super-resolution and electronic equipment

Abstract

The invention relates to a video description method based on space-time super-resolution and electronic equipment, and the method is realized based on a video description model, and comprises the following steps: obtaining an input video, and sampling the input video to obtain a video frame sequence containing a plurality of compression size frames; performing multi-modal feature extraction and feature coding on the video frame sequence through the video description model, dynamically fusing the coded multi-modal features, and gradually decoding to generate a video description statement; wherein when the video description model is trained, a frame with an original resolution and a middle missing frame between adjacent sampling frames are reconstructed from two dimensions of space and time, a loss function is constructed by a reconstruction error and a decoding prediction error, and model training is realized. Compared with the prior art, the method has the advantages of rich and accurate description, strong generalization ability, low calculation overhead and the like.

Description

technical field [0001] The invention relates to the fields of computer vision and natural language, and in particular to a video description method and electronic device based on spatiotemporal super-resolution. Background technique [0002] In recent years, with the popularization of 5G network, video as a medium of information interaction has been widely spread in people's daily life, and it has also brought various new challenges, such as automatic classification and retrieval of large-scale videos. , motion and event detection and other video understanding tasks. As one of the key tasks of video understanding tasks, video description aims to automatically generate a natural language description for a given video clip, which has a very broad application prospect in the fields of human-computer interaction, infant teaching and visual impairment assistance. Due to the richness and complex timing of video scenes, it is difficult to model video information. Compared with sta...

AI Technical Summary

Problems solved by technology

However, this approach ignores the information loss caused by frame sampling and image compression. At the same time, if frame sampling is not performed and the high resolution of the original image is maintained for feature extraction, a large amount of computing costs will be introduced, and the running speed of the model will drop significantly.

Images