A data-driven tof depth imaging method that can remove multipath errors

Abstract

The application discloses a data-driven TOF depth imaging method capable of removing multipath error. The steps are as follows: three-dimensional modeling is carried out for a scene with multipath effect, depth rendering is carried out on the scene to obtain a depth map label and corresponding scene parameters; the scene is described in the form of a triangular mesh, transient rendering based on pbrt is carried out to obtain a pulse response transient map; a learnable modulation function is convolved with the pulse response transient to obtain echo after the modulation function illuminates the scene; a learnable TOF demodulation function is used, multiplied with the output of the previous step, integrated and added with ambient light and noise, and input into a reconstruction subnetwork; the network is trained, a modulation function is obtained, laser is modulated, the scene is illuminated, echo is measured by a photodetector, multiplied with the obtained demodulation function and filtered; and a voltage value is input into the trained reconstruction subnetwork to obtain a depth map. The method enhances the robustness of TOF depth imaging to multipath error and improves imaging accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of graphics rendering, computer photography and deep learning, and in particular to a data-driven TOF depth imaging technique capable of removing multipath error. BACKGROUND

[0002] In recent years, amplitude-modulated continuous-wave (AMCW) time-of-flight cameras (such as Microsoft's Kinect One) have not only been widely adopted in interactive commercial applications, but have also become an exciting imaging approach in computer vision. Research on depth imaging has important significance for fields such as autonomous driving, geographic remote sensing, and medical imaging. Depth maps have a wide range of applications, and relative to two-dimensional images, they can obtain more rich positional relationships between objects by distance information, i.e., distinguishing foreground from background. After further deep processing, three-dimensional modeling and other applications can be completed, and target recognition and tracking can be quickly completed. At the same time, depth information can still complete traditional applications such as target image segmentation, labeling, recognition, and tracking.

[0003] TOF cameras measure depth by periodically amplitude-modulating floodlight illumination of a scene, which reflects back to the camera along direct and indirect light paths. The camera then measures the phase shift of the incident signal relative to the illumination signal. In order to extract depth from these raw phase measurements, many challenging reconstruction problems must be solved. In the presence of global illumination, multiple light paths interfere with each other along direct and indirect paths, resulting in severe multipath interference (MPI) distortion of the depth map.

[0004] With the rise of the artificial intelligence wave in recent years, applying artificial intelligence to the field of computer photography has become a frontier research hotspot in the fields of computer vision, digital signal processing, and optics. Processing of depth maps as an application of artificial intelligence is attracting widespread attention, but using deep learning for depth map processing requires a large number of suitable and reliable data sets to train the deep learning model. Advances in rendering technology make it possible to obtain a large number of data sets that can be customized according to the needs of deep learning tasks through simulation. Using the latest ray tracing algorithm (Physically Based Rendering [J]. Physically Based Rendering (Third Edition)) can obtain data with strong multipath effects based on different object materials and viewing angles. SUMMARY

[0005] To solve the defects in the above existing TOF imaging methods, the purpose of the present application is to propose a data-driven TOF depth imaging method that can remove multipath error.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] A data-driven TOF depth imaging method capable of removing multipath error, comprising the following steps:

[0008] Step 1, three-dimensional modeling is performed on a scene with multipath effect, and then depth rendering is performed on the scene to obtain a depth map label and corresponding scene parameters;

[0009] Step 2, the scene of step 1 is described in the form of a triangular mesh, and a scene description file is generated in combination with the scene parameters to perform pbrt-based transient rendering to obtain a plurality of frame impulse response transient maps;

[0010] Step 3, inputting the plurality of frame impulse response transient maps into a TOF imaging network, the TOF imaging network comprising a simulation network and a GAN-based reconstruction subnetwork, wherein the simulation network obtains simulated original output data by simulating a modulation and demodulation process, and the reconstruction subnetwork is responsible for reconstructing the original output data into a depth map;

[0011] Step 4, describing a modulation function in the form of a sum of multiple harmonics in the simulation network, and setting the amplitude and phase of the harmonics as learnable parameters, so that the modulation parameters are trainable;

[0012] Step 5, convolving the learnable modulation function of step 4 with the plurality of frame impulse response transient maps of step 2 to obtain a reflected echo after the scene is illuminated by the light wave corresponding to the modulation function, and the echo can fully reflect the multipath effect of the scene;

[0013] Step 6, setting four trainable demodulation functions in the manner of step 4, and performing cross-correlation operation on the reflected echo of step 5;

[0014] Step 7, adding ambient light and Gaussian white noise to the result of step 6 to obtain a noisy measurement map;

[0015] Step 8, inputting the noisy measurement map into the GAN-based reconstruction subnetwork, inputting the measurement map into a generator G to obtain a reconstruction result, and then judging the authenticity of the generated result by a discriminator, and updating the parameters of the generator G based on this in the training;

[0016] Step 9, the simulation network and the GAN-based reconstruction subnetwork are trained simultaneously by a deep learning method to obtain trained modulation functions, demodulation functions and a reconstruction subnetwork;

[0017] Step 10, modulating a laser diode using the modulation function trained in step 9 to drive the laser diode to emit laser light illuminating the scene with the same waveform as the modulation function;

[0018] Step 11, the reflected light reflected by the surface of the scene object is received by a photodetector;

[0019] Step 12, multiply the reflection signal collected by the photodetector in step 11 with the demodulation function trained in step 9 and perform low-pass filtering, and use an analog-to-digital converter to collect the voltage output after low-pass filtering;

[0020] Step 13, repeat steps 10 to 12, scan each point in the scene, measure all points in the scene, and obtain the measurement result;

[0021] Step 14, input the measurement result obtained in step 13 into the reconstruction subnetwork trained in step 9 to obtain the depth map of the scene.

[0022] Further, the mathematical forms of the modulation function and the demodulation function in steps 4 and 6 are: F(t) = Σ sin(kωt+φ k ), wherein k represents the number of harmonics, k = 1, 2, 3, 4, 5, 6; ω represents the frequency of the fundamental wave; φ represents the phase corresponding to each harmonic; and t is time.

[0023] Further, in step 5, the continuous form of the reflection echo is represented as: E(t) = ∫α(τ)F(t-τ)dτ, and the discrete form of the reflection echo is represented as: E(t) = Σα(τ)*F(t-τ) / f, wherein F(t-τ) is the form of the intensity modulated light source, α(τ) is the impulse response transient function; f is the frame number of the impulse response transient function used, here τ = kT0, k = 0, 1, 2, 3…, and T0 is the interval between two frames of the impulse response transient function.

[0024] The present application solves the problem of lack of accurate description of multipath effect in existing data sets by using the customized data set based on pbrt rendering, and uses the learnable modulation and demodulation function and the GAN-based reconstruction subnetwork to better extract information from the above-mentioned data set. This solves the problem that the TOF imaging method is difficult to obtain an accurate depth map in the case of multipath effect in the scene. The method of the present application still has high depth imaging accuracy in the case of strong multipath effect in the scene. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The method flowchart of the embodiment of the present application is provided. DETAILED DESCRIPTION

[0026] The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0027] The present embodiment provides a data-driven TOF depth imaging method, and the specific process is shown in Figure 1 , comprising the following steps:

[0028] Step 1, select a 3D modeling with strong multipath effect such as corner scene for three-dimensional modeling, by setting the camera to select several frames of different view angles, using depth rendering to get depth map label, and exporting the view matrix corresponding to these view angles and the material parameters of the object surface.

[0029] Step 2, re-describe the above three-dimensional modeling in the form of triangular mesh, use python script to combine the triangular mesh modeling with view matrix and object surface material parameters to generate scene description file.

[0030] Step 3, set the transient rendering renderer based on pbrt, set the camera to "perspective" mode, give a view angle of 60°, render based on Monte Carlo sampling, set the sampler to "random" mode, set the sampling number of each pixel to 100, give the setting and render, get several frames of impulse response transient diagram.

[0031] Step 4, select Pytorch as the framework of deep learning, set the training batch size to 8, select the learning rate as 1e-4, and set the learning rate exponential decay, decay by 10% every 10 epochs, input several frames of impulse response transient diagram in the data set into the TOF imaging network, simulate the network and the reconstruction sub-network based on GAN, where the simulation network obtains the simulated hardware system original output data by simulating the modulation and demodulation process, and the reconstruction sub-network is responsible for reconstructing the original output data into depth map.

[0032] Step 5, select the cumulative signal of 6 harmonics to simulate the modulation function F(t) = Σ sin(kωt+φk), k = 1, 2, 3, 4, 5, 6 in the simulation network, where k represents the number of harmonics, ω represents the frequency of the fundamental wave, and φ represents the phase corresponding to each harmonic.

[0033] Step 6, convolve the learnable TOF modulation function in step 5 with several frames of impulse response transient diagram to get the echo after the light wave illuminates the scene, which is expressed in discrete form as E(t) = Σ α(τ) * F(t-τ) / f.

[0034] Step 7, the simulation network needs 4 demodulation functions D(t), because the reflected signal can be approximated as αF(t-t0)+β; where α is the reflectivity, t0 is the time delay in space transmission, and β is the radiation component caused by external light source. In order to have a definite solution for 3 unknowns, at least 3 or more receiving quantities are needed. The specific function form is also simulated in the manner of step 5, and the received echo E(t) and demodulation function D(t) are cross-correlated: Get the cross-correlation sequence of modulation and demodulation functions.

[0035] Step 8, the result obtained in step 7 is added to Gaussian white noise and input to the GAN-based reconstruction subnetwork. The specific steps are as follows: first, input the noisy measurement map to the generator G to obtain the reconstruction result, then use the discriminator to judge the authenticity of the generated result, and update the parameters of the generator G based on this in the training. The reconstructed depth map is calculated with the depth map label obtained in step 1 using the loss function. The loss function selects the mean square error function and the adversarial loss. At the same time, the modulation function, the demodulation function and the reconstruction subnetwork are trained. This step finds the best modulation and demodulation functions for imaging through deep learning, and applies them to the modulation and demodulation circuit of the light source.

[0036] Step 9, use the modulation function F(t) trained in step 8 to modulate the laser diode, and drive the laser diode to emit laser light with the same waveform as the modulation function to illuminate the scene.

[0037] Step 10, the light source encounters an object in the scene and reflects. The reflected light from each path passes through the beam splitter and is focused on the APD photodetector through the lens. The APD photodetector acquires the reflected signal.

[0038] Step 11, multiply the reflected signal by the demodulation function D(t) obtained by network training through the multiplier and perform low-pass filtering to obtain a voltage value representing scene information. Use a digital-to-analog converter to collect the above voltage value.

[0039] Step 12, repeat steps 9-11 to complete the depth measurement of all points in the scene and obtain a noisy depth measurement map.

[0040] Step 13, input the noisy depth measurement map into the trained reconstruction subnetwork to obtain the final depth map.

Claims

1. A data-driven TOF depth imaging method that can remove multipath errors, characterized in that, The method comprises the following steps: Step 1, three-dimensional modeling is performed for a scene with multipath effect, and then depth rendering is performed on the scene to obtain a depth map label and corresponding scene parameters; Step 2, the scene in step 1 is described in the form of a triangular mesh, and a scene description file is generated in combination with the scene parameters to perform pbrt-based transient rendering to obtain a plurality of frame impulse response transient maps; Step 3, the plurality of frame impulse response transient maps are input into a TOF imaging network, the TOF imaging network comprising a simulation network and a GAN-based reconstruction subnetwork, wherein the simulation network requires four demodulation functions, and simulated original output data is obtained through simulation of a modulation and demodulation process, and the reconstruction subnetwork is responsible for reconstructing the original output data into a depth map; Step 4, a modulation function is described in the form of a sum of multiple harmonics in the simulation network, and the amplitudes and phases of the harmonics are set as learnable parameters, so that the modulation parameters are trainable; Step 5, the learnable modulation function in step 4 is convolved with the plurality of frame impulse response transient maps in step 2 to obtain a reflection echo after the light wave of the modulation function illuminates the scene, and the echo can fully reflect the multipath effect of the scene; Step 6, four trainable demodulation functions are set in the manner of step 4, and cross-correlation operation is performed on the reflection echo in step 5; Step 7, the result in step 6 is added to ambient light and Gaussian white noise to obtain a noisy measurement map; Step 8, the noisy measurement map is input into the GAN-based reconstruction subnetwork, the measurement map is input into a generator G to obtain a reconstruction result, and the authenticity of the reconstruction result is judged by a discriminator, and the parameters of the generator G are updated based on this in the training; Step 9, the simulation network and the GAN-based reconstruction subnetwork are trained simultaneously by using a deep learning method to obtain trained modulation functions, demodulation functions and a reconstruction subnetwork; Step 10, the laser diode is modulated using the modulation function trained in step 9, and the laser diode is driven to emit laser with the same waveform as the modulation function to illuminate the scene; Step 11, the reflected light reflected by the surface of the scene object is received by a photodetector; Step 12, the reflected signal collected by the photodetector in step 11 is multiplied by the demodulation function trained in step 9 and low-pass filtered, and the voltage output after low-pass filtering is collected using an analog-to-digital converter; Step 13, steps 10 to 12 are repeated to measure each point in the scene, and measurement results are obtained; Step 14, the measurement results obtained in step 13 are input into the trained reconstruction subnetwork in step 9 to obtain a depth map of the scene.

2. The data-driven TOF depth imaging method of claim 1, wherein, The mathematical forms of the modulation function and demodulation function in step 4 and step 6 are: F(t) =∑sin(kωt+φ k ) Wherein k represents the order of the harmonic, k = 1, 2, 3, 4, 5, 6; ω represents the frequency of the fundamental wave; φ represents the phase corresponding to each harmonic; and t is time.

3. The data-driven TOF depth imaging method of claim 1, wherein, In step 5, the continuous form of the reflection echo is represented as: E(t) = ∫α(τ)F(t-τ)dτ, The discrete form of the reflection echo is represented as: E(t) = Σα(τ)*F(t-τ) / f Wherein, F(t-τ) is the intensity modulation light source form, α(τ) is the impulse response transient function; f is the frame number of the impulse response transient function used, here τ=kT0, k=0, 1, 2, 3…, T0 is the interval between two frames of the impulse response transient function.

Images