Video description method based on target space semantic alignment

Abstract

The invention discloses a video description method based on target space semantic alignment. According to the method, firstly, appearance features and action features are extracted from sampled video frames containing text description, and the appearance features and the action features are spliced and then input into a time sequence Gaussian mixture cavity convolution encoder to obtain time sequence Gaussian features; constructing a decoder by using two layers of long and short-term memory neural networks to obtain probability distribution and hidden vectors of generated statements; establishing a semantic reconstruction network and calculating semantic reconstruction loss; and optimizing the model by using a stochastic gradient descent algorithm, sequentially carrying out the steps on a new video to obtain statement generation probability distribution, and obtaining a video description statement by using a greedy search algorithm. According to the method, modeling is carried out on the long-term time sequence relation of the video by using the time sequence Gaussian mixture cavity convolution, and the statement-level probability distribution difference is obtained through the semantic reconstruction network, so that the semantic gap between the generated statement and the video content can be reduced, and the natural statement which more accurately describes the video content is generated.

Description

technical field [0001] The invention belongs to the technical field of computer vision, in particular to the field of visual content understanding and analysis, and relates to a video description method based on target space semantic alignment. Background technique [0002] In the era of Internet + 2.0, people, machines, and things are closely connected by the network and edge devices, and information transmission is very important. As a multimedia data form with an increasing share, video contains richer visual features than text images. Accurate understanding of video content has become an urgent need for various practical applications such as video surveillance, autonomous driving, and navigation for the visually impaired. Describing video content in natural language that is easy for humans to understand is an important research direction for visual understanding, which is called video description. [0003] The video description task is to describe the video content with ...

AI Technical Summary

Problems solved by technology

[0005] The above video description methods mainly have the following deficiencies: (1) When extracting features, only two-dimensional static features and three-dimensional dynamic features of the video are considered, and the relationship between target objects in the video is not fully considered, which often leads to the description of the relationship between different targets Semantic confusion occurs in sentences, such as combining two unrelated target words; (2) When using the attention mechanism, the correspondence between words and video frames is often considered, while the relationship between words and target objects in video frames is ignored. The corresponding relationship leads to the description of the target object irrelevant in the generated sentence; (3) the time and space complexity of the traditional attention mechanism is too high, which is quadratically proportional to the number of video frames, and it is difficult to be used in actual tasks with high real-time performance

Images