Video description generation system based on graph convolution network

Abstract

The invention belongs to the technical field of cross-media generation, and particularly relates to a video description generation system based on a graph convolution network. The video description generation system comprises a video feature extraction network, a graph convolution network, a visual attention network and a sentence description generation network. The video feature extraction network performs sampling processing on videos to obtain video features and outputs the video features to the graph convolution network. The graph convolution network recreates the video features accordingto semantic relations and inputs the video features into sentence descriptions to generate a recurrent neural network; and the sentence description generation network generates sentences according tofeatures of video reconstruction. The features of a frame-level sequence and a target-level sequence in the videos are reconstructed by adopting graph convolution, and the time sequence information and the semantic information in the videos are fully utilized when description statements are generated, so that the generation is more accurate. The invention is of great significance to video analysisand multi-modal information research, can improve the understanding ability of a model to video visual information, and has a wide application value.

Description

technical field [0001] The invention belongs to the technical field of cross-media generation, and in particular relates to a video description generation system based on a graph convolutional network. Background technique [0002] Among various multimodal information processing tasks, video description automatic generation (Video Captioning) is a very important task, and it is a basic research task in the field of video analysis. Its task is to give a video, the computer automatically analyzes the content of the video, and generates sentences describing the main content of the video. This task is developed on the basis of the image description automatic generation task (Image Captioning), but there are significant differences between video and images. Video can be regarded as a collection of multiple images, accompanied by audio information, and its features are diverse. The complexity of nature and content scenes far exceeds that of images, so the technical difficulty and...

AI Technical Summary

Problems solved by technology

[0005] Although the existing work has made great progress in the automatic video description task, the characteristics of the video itself have not been fully exploited, and the timing of the video and the object information in different frames have not been fully utilized. For the generative model, The input is generally the word at the previous moment, and the information of other words at the previous moment is also lost

Images