Video depth map estimation method and device with space-time consistency

Abstract

The invention provides a video depth map estimation method and device with space-time consistency. The method comprises the following steps: generating a training set: taking a central frame as a target view, taking a front frame and a rear frame as source views, and generating a plurality of sequences; for a static object in a scene, constructing a framework for jointly training monocular depth and camera pose estimation from an unmarked video sequence, which comprises constructing a depth map estimation network structure, constructing a camera pose estimation network structure, and constructing a loss function of the part; for a moving object in the scene, cascading a previous optical flow network after the created framework to simulate the motion in the scene, which comprises establishing an optical flow estimation network structure and establishing a loss function of the part; proposing a loss function of the deep neural network for the space-time consistency test of the depth map;continuously optimizing the model, carrying out joint training on monocular depth and camera attitude estimation, and then training an optical flow network; and realizing depth map estimation of continuous video frames by utilizing the optimized model.

Description

technical field [0001] The invention belongs to the field of geometric information understanding of video scenes, and relates to a technique for estimating a depth map of a video frame, in particular to a technical scheme for estimating a depth map of continuous video frames with temporal and spatial consistency. Background technique [0002] Understanding 3D scene geometry in video is a fundamental problem in visual perception, which includes many basic computer vision tasks, such as depth estimation, camera pose estimation, optical flow estimation, etc. A depth map refers to an image that contains information about the distance from the surface of objects in the scene to the viewpoint. Estimating depth is an important part of understanding the geometric relationship in the scene, and a general image-based method for extracting depth maps is very necessary. The distance relationship helps to provide a richer object and environment representation, and can further realize th...

AI Technical Summary

Problems solved by technology

There are some defects in these types of methods: the device scanning method mainly uses physical equipment for manual scanning and acquisition, but the existing three-dimensional scanners (such as Kinect) are not only expensive, but also not suitable for general application scenarios; traditional mathematical methods The depth estimation accuracy is too low, and for some complex scenes, this kind of method usually cannot perform an effective processing; supervised deep learning methods mainly rely on deep learning and network architecture and mathematical models to obtain results, such methods are usually for The data set has a strong dependence, and the acquisition of the data set usually requires a lot of manpower and material resources, and such methods usually have poor generalization; unsupervised deep learning methods, existing video depth estimation methods, The spatiotemporal discontinuity of the depth map is usually ignored, and large errors often occur in some occluded areas or non-Lambertian surface areas during actual processing

Images