Flight time depth image iterative optimization method based on convolutional neural network

Abstract

The invention discloses an iterative optimization method of a time of flight (Time-of-Flight: ToF) depth image based on a convolutional neural network. The method comprises the following steps: performing basic triangular transformation and a multi-frequency phase deblurring algorithm on a correlation coefficient graph obtained by imaging of a multi-frequency amplitude modulation continuous wave ToF camera to obtain an initial depth graph and a reflection intensity graph of a scene; constructing a convolutional neural network based on iterative optimization, and constructing a data set by using computer graphics and a three-dimensional reconstruction technology to train the neural network and search for optimal parameters for the neural network; inputting ToF original correlation measurement, an initial depth map and a reflection intensity map into the convolutional neural network, and gradually reducing error influences of various sources and different characteristics through iterative optimization of a multi-stage isomorphic network, so the depth map quality is improved from coarse to fine.

Description

technical field [0001] The invention relates to the field of three-dimensional vision, in particular to an iterative optimization method of a Time-of-Flight depth image based on a convolutional neural network. Background technique [0002] Depth acquisition is not only the key to most 3D vision tasks, but also plays an increasingly important role in traditional RGB-based vision tasks, such as semantic segmentation and gesture recognition. Previously popular structured light and stereo vision either had a too small ranging range or required scene textures. ToF technology overcomes these shortcomings and becomes the most promising way to acquire depth. [0003] ToF cameras measure distance by measuring the time it takes for a beam or pulse to travel from the transmitter to the object and back to the receiver. For certain amplitude-modulated CW ToF cameras, depth is obtained indirectly by measuring the phase difference between the outgoing and incoming waves. Modern ToF camer...

AI Technical Summary

Problems solved by technology

One is that the end-to-end CNN architecture cannot realize the mapping from the multi-frequency correlation coefficient map to the real depth map in principle

The second is that ToF depth map optimization needs to run on millimeter precision. The results completely output by CNN have low reliability and accuracy in the prediction of pixel-level millimeter precision, and there are prone to failure cases and serious deviations.

The third is that the error sources of the ToF depth map are different, so different principles need to be used to remove them. It is difficult for an end-to-end network to encapsulate these principles and predict the nonlinear coupling error accurately at one time.

Images