A method for generating natural language descriptions of open-domain videos based on multimodal feature fusion

Abstract

An open-domain video natural language description method based on multi-modal feature fusion, using a deep convolutional neural network model to extract RGB image features and grayscale optical flow image features, adding video spatiotemporal information and audio information to form a multi-modal feature system , when extracting C3D features, dynamically adjust the coverage of continuous frame blocks input into the 3D convolutional neural network, solve the limitation of training data size, and have robustness to the length of the video that can be processed, and the audio information makes up for the visual Insufficient, and finally for multi-modal feature fusion. The invention uses the data standardization method to standardize the eigenvalues ​​of each mode within a certain range, which solves the problem of eigenvalue differences; uses the PCA method to reduce the dimension of individual modal features, and effectively retains 99% of important information, which solves the problem of excessive dimensionality. It effectively improves the accuracy of the generated open-domain video description sentences, and has high robustness to scenes, characters, and events.

Description

technical field [0001] The invention belongs to video analysis technology, in particular to an open-domain video natural language description generation method based on multimodal feature fusion. Background technique [0002] With the popularity of smart mobile devices in recent years, a large amount of video data on network platforms needs to be analyzed and managed urgently. Therefore, it is of great practical value to study the natural language description technology of videos. Illegal videos on social platforms such as Weibo and WeChat emerge in endlessly, but now we mainly rely on manual methods such as reporting by the masses to report and control the spread of such videos, which is not effective. In addition to controlling the dissemination of pornographic, violent and reactionary and other illegal videos and maintaining network security, the language description of videos can also provide intelligent technology for the blind and other people with visual impairments t...

AI Technical Summary

Problems solved by technology

The S2VT model achieved a METEOR value of 29.8% on a standard video description dataset, which is higher than all previous model results, but S2VT only considers the image features and optical flow features of the video, and other information of the video has not been passed. more research

[0004] Later, some models were proposed, such as the bidirectional LSTM model (Yi B, Yang Y, Shen F, et al. Bidirectional Long-Short Term Memory for Video Description[C] / / ACM onMultimedia Conference.ACM,2016:436-440.) , multi-scale multi-instance model (Xu H, Venugopalan S, Ramanishka V, et al. A Multi-scale Multiple Instance Video Description Network [J]. Computer Science, 2015, 6738: 272-279.) but did not consider the image and other features besides optical flow

In 2017, Pasunuru et al. proposed a multi-task model (Pasunuru R, Bansal M. Multi-Task Video Captioning with Video and Entailment Generation [J]. 2017.), between unsupervised video prediction tasks (encoding) and language generation tasks (decoding) shared parameters among them, achieved the best result so far, with a METEOR value of 36%, but the model only used RGB image features

Images