Video description text generation method based on multi-modal fusion

Abstract

The invention provides a video description text generation method based on multi-modal fusion, and the method comprises the steps: obtaining a to-be-described video which comprises a video frame, wherein the to-be-described video is provided with a corresponding video description statement; obtaining text theme information of the video description statement, and setting a text theme information code for each piece of text theme information; respectively obtaining dynamic time domain information codes, static information codes and audio feature vector codes of the to-be-described video; performing fusion processing on the dynamic time domain information code, the static information code and the audio feature vector code of the to-be-described video to obtain a fusion result; inputting the fusion result and the text theme information code into a first recurrent neural network for iterative processing, and determining a video content description text of the to-be-described video. The beneficial effects of the invention are that the method achieves the generation of the natural language description of the video on the basis of the fusion of various modes of the video, audio and text, and improves the generation accuracy and robustness.

Description

technical field [0001] The invention relates to the technical field of image processing, in particular to a method for generating video description text based on multimodal fusion. Background technique [0002] Video resources have become the most popular and favorite way for people to obtain information, especially after the emergence of some video APPs, watching videos every day has become an indispensable way of leisure and entertainment for many people. In order to better serve users, it is necessary to express the core information in the video in text form for recommendation display. Therefore, there must be a method capable of outputting the core content information of a given video. [0003] At present, video content description (video captioning) is usually performed on a video. The video content description is to generate a piece of text describing the video content by giving a piece of video. The video content description needs to describe the video content in a ...

AI Technical Summary

Problems solved by technology

[0004] Then the above-mentioned existing technology will frame the video, and use the image after the frame extraction as an independent feature to output the description text, but the independent image after the frame extraction cannot reflect the dynamic content and time domain information of the video; and naturally The output of language description text requires the support of text-level information. However, the above-mentioned existing technologies do not integrate the characteristics of text-level information, which leads to large divergence of the output description text content and unstable semantic direction.

Images