Extraction method of semantic information of video images

Abstract

The invention provides an extraction method of semantic information of video images, and relates to the technical field of video description and annotation. Firstly, frame sequences of a video are extracted according to a certain inter-frame space, a feature vector of each frame image is extracted according to a convolutional neural network, the feature vectors are regarded as input of a LSTM network encoder, output of each time step of the LSTM network encoder and output of the previous time step of a LSTM network decoder are regarded as input of an external storage EMM, and contents of a stored matrix in the external storage EMM are updated; the external storage EMM outputs two reading vectors which are regarded as input vectors of decoding and encoding of the subsequent time step respectively. Through two LSTM network dynamics, reading and writing of the external storage EMM are controlled, and storing of the feature vector of each frame image of the video at the encoding phase is achieved; at the decoding phase, through forecasting of feedback of words, the output of the subsequent time step of the external storage is adjusted, so that when an annotation of the video is generated, the feature vectors of a context are adjusted according to a generated word sequence.

Description

technical field [0001] The present invention relates to the technical field of video description and video labeling, in particular to a method and device for extracting semantic information of video images. Background technique [0002] Currently, with the development of the Internet, digital devices, and multimedia technologies, video has attracted more attention from multimedia users because video is more vivid than text and pictures. At present, the rapid development of short video applications such as WeChat and Kuaishou and various online live broadcast platforms makes video play an increasingly important role in people's lives. In order to facilitate people to better understand the content of video images, it is extremely important to describe and label video images to obtain semantic information of video images. Video description is to use natural language to describe the characteristics of video content according to the specific content of a video. Using an applicat...

AI Technical Summary

Problems solved by technology

[0005] Embodiments of the present invention provide a method and device for extracting semantic information of video images to solve the problem that in the prior art, when the input video takes a long time and the number of extracted frames is large, the context vector generated by the prior art model through encoding cannot Contains all the global information, especially the information in the picture of the input frame at an earlier time may be forgotten, leading to the problem of model performance degradation

Images