Video content description method based on text auto-encoder

Abstract

The invention discloses a video content description method based on a text auto-encoder. The method comprises the following steps: firstly, constructing a convolutional neural network to extract two-dimensional and three-dimensional features of a video; secondly, constructing a text auto-encoder, namely extracting text hidden space features and a decoder-multi-head attention residual error networkreconstruction text by using an encoder-text convolution network; thirdly, obtaining estimated text hidden space features through a self-attention mechanism and full-connection mapping; and finally,alternately optimizing the model through an adaptive moment estimation algorithm, and obtaining corresponding video content description for the new video by using the constructed text auto-encoder andthe convolutional neural network. According to the method, a potential relationship between video content semantics and video text description can be fully mined through the training of the text auto-encoder, and the action time sequence information of the video in a long time span is captured through the self-attention mechanism, so that the calculation efficiency of the model is improved, and the text description more conforming to the real content of the video is generated.

Description

technical field [0001] The invention belongs to the technical field of computers, in particular to the technical field of video content description, and relates to a video content description method based on a text autoencoder. Background technique [0002] In recent years, with the continuous development of information technology and the iterative upgrade of smart devices, people are more inclined to use video to convey information, which makes the scale of various types of video data larger and larger, but also brings great challenges. For example, hundreds of thousands of video data are uploaded to the server every minute on the video content sharing website. It is time-consuming and labor-intensive to manually review whether these videos comply with the rules, but the method of video description can significantly improve the review work. Efficiency, saving a lot of time and labor costs. Video content description technology can be widely used in practical scenarios such ...

AI Technical Summary

Problems solved by technology

[0005] The shortcomings of the above methods are mainly manifested in the following aspects: First, the mainstream video description method mainly uses cross-entropy to calculate the loss, which has the disadvantage of error accumulation. Although reinforcement learning can be used to avoid this disadvantage, the calculation amount is large and it is difficult to converge; Second, the above method only considers the features of the video, and does not make full use of the rich features contained in the video text, ignoring the guiding role of the text as prior information on the update of the description model parameters; third, the recurrent neural network belongs to the sequential structure, and the current moment The unit depends on the output of all previous units and cannot be processed in parallel, resulting in a waste of computing resources. Sometimes the gradient disappears and the weights cannot be updated accurately, making it difficult to accurately generate coherent sentences that match the video content.

Images