Video content description method and system based on frame selection

Abstract

The invention relates to a video content description method based on frame selection, comprising the following steps: a feed-forward neural network is used for constructing a screening model, and thescreening model screens the video frame according to the visual richness and semantic consistency of the video frame; constructing a description model for describing the content of the video to be described; training the screening model and the description model with training data; Selecting a description frame in the video to be described through the filtering model; the visual features of the description frame are extracted and the description model is input to obtain the description sentence of the video to be described.

Description

technical field [0001] The invention relates to technologies in the fields of digital image processing and natural language processing, in particular to a technology for generating natural language descriptions for video content. Background technique [0002] Video content description (video captioning) is the task of converting video content into natural language. As early as 2002, Kojima et al. proposed the first video content description system to describe human behavior. Since then, a series of studies on image and video description have been born. Early methods used a bottom-up approach to deal with this problem, that is, first generated descriptors through attribute learning or object detection, and then concatenated these descriptors into a complete sentence through a language model. With the development of neural networks and deep learning, most modern description systems are based on convolutional neural networks and recurrent neural networks, and adopt an encoder...

AI Technical Summary

Problems solved by technology

Such methods waste a lot of computing power on the one hand, and are also susceptible to noise or unimportant visual content on the other;

[0010] 2. The video content description method based on the attention mechanism, especially the method based on the spatial attention mechanism, needs to extract the features of the video frames first, and then perform weighted fusion on them. This can only be done when the video is of limited length and can be completely It is used in the case of observation, which is not suitable for real-time video or real-world scenarios, such as navigation for the blind;

[0011] 3. The video content description method based on key frame extraction also needs to first summarize the global content of the video and then describe it in text according to the summary, which is very difficult to apply to real scenes, because in real scenes, there is no cutting A good video can be used for global content summarization, and the annotation data of video summarization is very subjective and expensive. It is difficult to train a key frame extractor that can be well used for video content description.

Images