Video caption generation method based on semantic segmentation and multilayer attention frame

Abstract

The invention relates to a video caption generation method based on semantic segmentation and a multilayer attention frame, and the method comprises the steps: 1, extracting multiple image frames froma video, wherein the caption of the video needs to be generated; 2, sensing a semantic segmentation model through a full convolution instance, and extracting the feature information of one deconvolution layer from the video; 3, extraction motion features and audio features of the video; 4, sensing the semantic segmentation model through the full convolution instance, and extracting an attribute label from an image extracted at step 1, wherein the attribute label comprises the object information in each image frame; 5, generating context matrixes in different modes according to the informationextracted at former steps, carrying out the layered fusion of the context matrixes in different modes, and generating a fused context matrix; 6, obtaining words, serving as the components of the caption, through employing LSTM and employing a multilayer sensing mechanism for processing; 7, carrying out the series combination of all obtained words, and generating a final caption.

Description

technical field [0001] The present invention relates to the technical field of computer vision and natural language processing, in particular to three-dimensional feature extraction technology and semantic segmentation technology based on computer vision, and time series model technology based on natural language processing, more particularly, to full convolution-based semantic segmentation technology and a multi-layer attention framework for video caption generation. Background technique [0002] Video subtitle generation refers to the automatic generation of natural language descriptions for a video. Such research has received increasing attention in the fields of artificial intelligence and computer vision. In today's society, it has a very wide range of applications, such as helping blind people in their daily lives, improving the quality of online video retrieval, and so on. In addition to related applications, video subtitle generation technology has played a huge ro...

AI Technical Summary

Problems solved by technology

For example, due to the characteristics of video, video needs to extract a large number of picture features, and resizing each picture will lose the structural information of the picture, and extracting 3D convolution and 2D convolution seems to extract different features, but due to the weight sharing factor of the convolution, a large amount of information is repeatedly extracted

At present, although the attention mechanism is used to improve the fusion effect between modalities, the use of the same attention operation for different modalities does not consider the differences between modalities, which will lead to cross-interference of information between modalities

Images