Single-depth camera depth map real-time enhancement method and device based on neural network

Abstract

The invention discloses a single-depth camera depth map real-time enhancement method and device based on a neural network. The single-depth camera depth map real-time enhancement method comprises thefollowing steps: acquiring a depth image and an RGB image of a sample through a depth camera, aligning the depth image with the RGB image according to a calibrated internal and external parameter matrix of the depth camera, and converting the RGB image into a gray scale space and carrying out radiation transformation to obtain a gray scale image aligned with the characteristics of the depth image;constructing a forward neural network and a loss function, inputting the depth image and the grayscale image into the forward neural network for training, and updating the weight of the forward neural network according to the back propagation of the loss function; and inputting the grayscale image into a forward neural network after training and weight updating to output an enhanced depth image.According to the single-depth camera depth map real-time enhancement method, the depth camera is used for shooting the sample and collecting the depth image, and high-precision scanning equipment is not needed for collecting the real depth image as supervision information, so that the manual calibration process is omitted, and good interactive three-dimensional reconstruction experience is provided for a user.

Description

technical field [0001] The invention relates to the technical fields of computer vision and computer graphics, in particular to a neural network-based method and device for real-time enhancement of a depth map of a single-depth camera. Background technique [0002] In recent years, consumer-level depth cameras have gradually become popular, especially the latest Iphone X has a built-in depth camera based on structured light. This enables many new mobile applications in mixed reality, from 3D scanning to virtual reality. Although the resolution and quality of the raw data collected by the sensor has improved, the depth map obtained from the consumer depth camera at this stage still has a lot of noise and lacks sufficient details. For example, volumetric 3D reconstruction is a key issue in the fields of computer graphics and computer vision. High-quality 3D human body models have broad application prospects and important application values ​​in the fields of film and televis...

AI Technical Summary

Problems solved by technology

The traditional bilateral filtering method cannot enhance the details while ensuring the authenticity of the depth map. The method of restoring the three-dimensional shape of the object from light and dark usually requires complex optimization algorithms and is invalid for specific inputs. The fusion method based on temporal smoothing cannot be used alone. Frame real-time augmentation of depth maps, whereas data-driven machine learning algorithms cannot do unsupervised without real depth map data

Images