A Convolutional Neural Network-Based Generation Method for Spatiotemporal Consistent Depth Map Sequences

Abstract

The invention discloses a method for generating a spatial-temporal consistency depth map sequence based on a convolution neural network, which can be used in a film and television work 2D-to-3D technology. The method comprises the following steps: (1) collecting a training set, wherein each training sample in the training set is composed of a continuous RGB image sequence and a corresponding depth map sequence; (2) carrying out spatial-temporal consistency super pixel segmentation on each image sequence in the training set, and constructing a spatial similarity matrix and a temporal similarity matrix; (3) constructing a convolution neural network composed of a single super pixel depth regression network and a spatial-temporal consistency conditional random field loss layer; (4) training the convolution neural network; and (5) for an RGB image sequence of unknown depth, using the trained neural network to recover a corresponding depth map sequence through forward propagation. The problem that a depth recovery method based on clues relies too much on the scene hypothesis and the problem that the frames of a depth map generated by the existing depth recovery method based on a convolution neural network are discontinuous are avoided.

Description

technical field [0001] The invention relates to the field of computer vision stereoscopic video, in particular to a method for generating a spatiotemporal consistency depth map sequence based on a convolutional neural network. Background technique [0002] The basic principle of stereoscopic video is to superimpose and play two images with horizontal parallax, and viewers can see the pictures of left and right eyes respectively through stereoscopic glasses, thereby generating stereoscopic perception. Stereoscopic video can provide people with an immersive three-dimensional perception, and is very popular among consumers. However, as the popularity of 3D film and television hardware continues to rise, the shortage of 3D film and television content will follow. It is expensive to directly shoot with a 3D camera, and the post-production is difficult. Usually, it can only be used in high-cost movies. Therefore, the 2D / 3D conversion technology of film and television works is an...

AI Technical Summary

Problems solved by technology

The jump of the depth map in adjacent frames will cause flickering of the synthesized virtual view, which seriously affects the user's perception

In addition, the continuity between frames also provides important clues for depth recovery, while in existing methods, this information is simply ignored

Images