Video description method based on dual-path fractal network and LSTM

Abstract

The invention discloses a video description method based on a dual-path fractal network and an LSTM. According to the method, key frame sampling is carried out on a to-be-describe video and an optical flow characteristic between two adjacent frames of an original video is extracted; learning is carried out respectively by using two fractal networks and high-level feature expressions of the video frame and the optical flow characteristic are obtained; the high-level feature expressions are inputted into two LSTM-unit-based recurrent neural network models; and then weighted averaging is carried out on output values of two independent modules at all times to obtain a description statement corresponding to the video. According to the method provided by the invention, the original video frames and the optical flow information are used respectively for the to-be-describe video; the added optical flow characteristic compensates dynamic information that is lost by the sampling frame inevitably; and the changes of the video in the spatial dimension and the time dimension are considered. Besides, the abstract visual feature expression is carried out on the bottom feature by using the novel fractal networks, so that the person, object and behavior and space position relation that are involved in the video can be analyzed and dug out precisely.

Description

technical field [0001] The invention belongs to the technical field of video description and deep learning, and in particular relates to a video description method based on a two-way fractal network and LSTM. Background technique [0002] With the advancement of technology and the development of society, all kinds of video camera terminals, especially smart phones, have become very popular, and the price of hardware storage has become increasingly low, which makes the flow of multimedia information grow exponentially. In the face of a large number of video information streams, how to efficiently and automatically analyze, recognize and understand massive video information with minimal human intervention, and thus describe it semantically, has become a hot topic in the current image processing and computer vision research fields. topic. For most people, it may be a simple matter to watch a short video and then describe the video in words. However, for a machine, it is a cha...

AI Technical Summary

Problems solved by technology

However, for a machine, it is a challenging task to generate a natural language description by extracting the pixel information of each frame image in the video, analyzing and processing it.

Images