Wide-range noise robust depth imaging method and system based on rgb guidance

Abstract

The application discloses a wide-range noise robust depth imaging method and system based on RGB guidance. The method steps are as follows: constructing a calibrated RGB-D database; establishing a differentiable physical imaging model, and adding physical constraints of an actual scene to obtain multiple noise-free measurement values; constructing a visual TOF double-branch multi-scale fusion depth reconstruction network, taking depth feature extraction and visual feature extraction fusion as an end-to-end network for iterative optimization, and using color pseudo-edge and Fisher information loss to supervise the network reconstruction effect, and jointly training the encoding function and the depth reconstruction network; inputting test data into the trained encoding function and reconstruction network to reconstruct the depth map of the test scene; and building an RGB-guided wide-range noise robust depth imaging hardware prototype system, and using the optimized encoding function and reconstruction network to reconstruct a high-quality depth map in an actual scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computational imaging and deep learning, and particularly relates to a wide-range noise-robust depth imaging method and system based on RGB guidance. BACKGROUND

[0002] Three-dimensional imaging technology has a wide range of applications in autonomous driving, augmented reality, medical surgery, industrial automation, and other fields. These computer vision tasks all rely on high-resolution and high-precision depth perception. Depth imaging can be divided into passive and active light imaging according to whether there is active light: passive imaging, also known as visual imaging, includes monocular and binocular imaging, and active light imaging includes direct and indirect time-of-flight (TOF).

[0003] Since the release of Microsoft Kinect in 2010, consumer-grade depth sensors have been widely used. Kinect is based on a monocular structured light approach, and the iPhone X released in 2017 also uses this approach. However, it cannot obtain depth measurements of distant objects and performs poorly in outdoor scenes with strong light. Binocular stereo vision systems have a larger measurement range than structured light systems and can work in outdoor environments with strong sunlight, but they are easily affected by the texture of object surfaces. TOF technology is a technique for measuring the distance of an object and has a wide range of applications in machine vision, augmented reality, and three-dimensional imaging. However, due to environmental conditions, TOF images are often disturbed by noise and have poor adaptability to low reflectivity objects and distant objects.

[0004] Therefore, a single type of sensor has its own limitations in three-dimensional information perception, but the error sources of each camera ranging principle can be considered independent. Therefore, by using appropriate passive and active imaging fusion technology, we can take advantage of each other's strengths and weaknesses to obtain more accurate scene depth information than any single detection technology. SUMMARY

[0005] In view of the problems and considerations in the above existing three-dimensional imaging methods, the purpose of the present application is to propose a wide-range noise-robust TOF imaging method and system based on RGB guidance.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] The wide-range noise-robust depth imaging method based on RGB guidance includes the following steps:

[0008] Step 1, construct a database, including a training set and a test set, and containing color images and depth images of the same scene;

[0009] Step 2, a differentiable physical imaging model is established to simulate the time-of-flight imaging process, and physical constraints of the actual scene are added, and the encoding function including the light source modulation function and the sensor demodulation function is set as a learnable parameter; the data in the training set is input into the differentiable physical imaging model to obtain noise-free multiple measurements;

[0010] Step 3, a visual TOF double-branch multi-scale fusion depth reconstruction network is constructed, including a depth feature extraction network for extracting pixel-by-pixel multi-scale TOF depth information, a monocular depth generation network for extracting monocular depth information, and an edge feature extraction network for extracting edge structure information.

[0011] Step 4, the pixel-by-pixel TOF depth information is obtained through the depth feature extraction network, the monocular depth is extracted through the monocular depth generation network, the monocular depth is used to supervise and correct the TOF depth through multi-scale fusion, and the corrected depth map is obtained; the high-resolution color picture edge is extracted through the edge feature extraction network, and optimized using edge structure loss;

[0012] Step 5, the corrected depth map and the high-resolution color picture edge are fused, and an edge-assisted depth fusion network is constructed as an end-to-end network for iterative optimization;

[0013] Step 6, color pseudo-edge guided loss and Fisher information quantity guided loss are used as loss functions, and combined with multiple measurement difference loss function and depth fidelity loss function, the encoding function and the fusion depth reconstruction network are jointly trained;

[0014] Step 7, the test data is input into the trained encoding function and fusion depth reconstruction network to reconstruct the test scene depth map; the encoding function and the fusion depth reconstruction network trained in step 6 are used to reconstruct the actual scene high-quality depth map.

[0015] Further, in step 1, the database includes virtual synthesis and real-world scene RGBD images and fine edge images based on color pictures, and each sample in the training set and the test set includes scene depth, surface reflectivity and ambient light intensity; when constructing the database, the intrinsic image decomposition is performed on the RGB image of the scene image to obtain the reflectivity image of the scene, the surface reflectivity of the scene is obtained by extracting the R channel of the reflectivity image, and the ambient light intensity is obtained by averaging the RGB three channels of the reflectivity image; the gradient and divergence operation is performed on the RGB image of the scene to obtain the edge map containing rich texture information.

[0016] Further, in step 2, the differentiable physical imaging model is specifically:

[0017] Assuming that the emitted light is modulated by the function Mi (t) modulation, the reflected signal of the scene point p is:

[0018]

[0019] where a(p) is the amplitude coefficient due to the scene reflection, β is the ambient component due to other light sources, is the distance-dependent time delay of light propagation, i is the number of modulation functions, t is the time delay;

[0020] The reflected signal is specifically:

[0021]

[0022] where z(p) is the depth of light propagation, c is the speed of light, F falloff is the light attenuation coefficient;

[0023] The demodulation function D i (t) is used to demodulate the reflected signal Ri(p, t) to derive the measurement value

[0024] s i (p) = ∫0 T R i (p, t) D i (t) dt,

[0025] where T is the integration time.

[0026] Further, in step 4, the pixel-by-pixel TOF depth information is obtained by a depth feature extraction network, specifically: first, a feature extraction block of an encoding module is used to extract multi-scale features to obtain features of different scales; a multi-scale feature fusion module is used to fuse the features of different scales, and the fused features are input to a decoding module; the decoding module uses a residual depth reconstruction module to perform convolution integration on the depth map obtained by upsampling and the fused features output by the multi-scale feature fusion module, in a coarse-to-fine manner, to generate a large-size depth map with more details.

[0027] Further, in step 4, monocular depth is extracted by a monocular depth generation network, specifically: a RefineNet network is used for monocular extraction to learn the relative depth sequence contained in the RGB image.

[0028] Further, in step 4, edge features are extracted by an edge feature extraction network, specifically: an MSRN network is used to learn the mapping between the RGB image and the image edge to extract the soft edge information of the RGB image.

[0029] Further, in step 5, the modified depth map and the high-resolution color image edge are fused by attention mechanism channel by channel, and then the fused result is input into a CNN convolutional layer to obtain a reconstructed fine depth map.

[0030] The application further provides a system for implementing the RGB-guided wide-range noise-robust depth imaging method, which comprises a laser diode, a function signal generator, a beam splitter, a scanning device, a light collecting lens, an avalanche photodiode, a multiplier, an amplifier, a low-pass filter and a color camera, specifically: the laser diode is used as a light source; the learned encoding function is generated by the function signal generator, and a modulation function in the encoding function modulates the laser emitted by the laser diode; the laser emitted by the laser diode passes through the beam splitter and reaches the scanning device, and the scanning device scans the scene; the laser irradiates a point of the scene and is reflected according to the original light path; the reflected light is focused on the avalanche photodiode through the light collecting lens; the avalanche photodiode converts the reflected light signal into an electric signal, and then the converted signal is multiplied by a demodulation function in the encoding function through the multiplier, and the output voltage value is further amplified and low-pass filtered through the amplifier and the low-pass filter; finally, the amplified and low-pass filtered voltage value is sampled, quantized and converted into a measurement value through an analog-to-digital converter.

[0031] The method of the application considers that a depth image and a color image of the same scene have high correlation at a texture edge, features are extracted from the depth measurement value and the color image respectively, and the features are fused for final depth prediction, the prior information provided by clear edge information in the color image can further improve the performance of the depth image.

[0032] (1) A learnable physical imaging model is established to simulate the iToF imaging process, and actual physical constraints, such as light path attenuation, are added, so that the modulation and demodulation functions that can be physically realized can be learned.

[0033] (2) The noise model of the iToF imaging system is derived, and Fisher information is used for guidance, and the color pseudo-edge guidance loss, the multi-measurement difference loss and the depth fidelity loss are combined to supervise the entire system framework.

[0034] (3) A visual TOF double-branch multi-scale fusion depth reconstruction network is designed, which uses RGB monocular depth information and edge structure features for fusion guidance to realize depth imaging under a large noise level. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The flowchart of the method of the application;

[0036] Figure 2A network structure diagram of the method of the present invention;

[0037] Figure 3 A diagram of the hardware prototype system for time-of-flight imaging built for an example of the present invention;

[0038] Figure 4 This is a simulation diagram of depth scenes with different light source photon quantities and ambient photon quantities in an embodiment of the present invention;

[0039] Figure 5 This is a calibrated optical path attenuation diagram in an embodiment of the present invention;

[0040] Figure 6 This is a comparison result diagram of the method of the present invention on the dataset NYU-V2;

[0041] Figure 7 This is the depth reconstruction result of the method of the present invention on the dataset NYU-V2;

[0042] Figure 8 Comparison of the reconstruction results of the proposed method on (a) SUN RGB-D dataset and (b) 4D Light Field dataset scenes using different encoding functions and networks. DETAILED DESCRIPTION

[0043] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Reference Figure 1 The present embodiment provides a method for robust depth imaging of a large range of noise based on RGB guidance, and the specific steps are as follows:

[0045] Step 1: Construct a database, including a training set and a test set, which contains color images and depth images of the same scene.

[0046] The database includes RGBD images of virtual synthesis and real-world scenes, as well as fine edge images based on color pictures. Each sample in the training set and test set includes the depth, surface reflectivity and ambient light intensity of the scene. When constructing the database, the RGB image of the scene image is subjected to intrinsic image decomposition to obtain the reflectivity map of the scene. The surface reflectivity of the scene is obtained by extracting the R channel of the reflectivity map, and the ambient light intensity is obtained by averaging the three RGB channels of the reflectivity map. The RGB image of the scene is subjected to gradient and divergence operations to obtain an edge map containing rich texture information.

[0047] This example uses the NYU-V2 dataset for training and performs generalization verification on 17 scenes of the SUN RGB-D dataset and 300 scenes of the 4D Light Field dataset.

[0048] For different noise level (i.e. signal to noise ratio level) scenarios, the system simulates different light source photon amount and ambient photon amount depth scenarios, such as Figure 4 As shown in the figure, the first column is the GT image and the corresponding RGB image, the second and third columns are the simulated measurement value images under higher signal to noise ratio, and the fourth and fifth columns are the simulated measurement value images under lower signal to noise ratio:

[0049] Step 2, a differentiable physical imaging model is established to simulate the time-of-flight imaging process. Assuming that the emitted light is modulated by a modulation function

[0050] Mi(t), then the reflection signal of the scene point p is:

[0051]

[0052] wherein α(p) is the amplitude coefficient caused by scene reflection, β is the ambient component caused by other light sources, is the distance-dependent time delay of light propagation, i is the number of modulation functions, and t is the time delay;

[0053] The specific form of the reflection signal is:

[0054]

[0055] wherein z(p) is the depth of light propagation, c is the speed of light, F falloff is the light attenuation coefficient;

[0056] The demodulation function D i (t) is used to demodulate the reflection signal Ri(p, t) to derive the measurement value

[0057] s i (p) = ∫0 T R i (p, t) D i (t) dt,

[0058] wherein T is the integration time, and s i (p) contains three unknowns: α(p), β, and z(p), so N≥3 measurements are usually needed to reconstruct the depth z(p). When α(p), β, and z(p) are set as differentiable parameters, the above iToF imaging process can be regarded as a differentiable iToF physical imaging model. By setting the amplitude and phase of the encoding function (i.e. modulation and demodulation functions) as learnable parameters, the entire encoding function becomes a learnable function, which is applied to the forward model of the entire iToF imaging system, and the imaging system becomes a differentiable form.

[0059] But in the actual imaging system, the theoretical value simulated in the forward model and the real physical measurement value will have obvious errors due to the limitations of devices. Therefore, when the physical imaging model is applied to the actual system, physical constraints need to be made according to the actual system.

[0060] When a point light source illuminates a scene, it emits a spherical wave. Since the total energy is constant, the light intensity received per unit area is attenuated with the square of the propagation distance of the light source. The light path attenuation coefficient is very important. When designing the reconstruction algorithm, considering the attenuation of the light path can make the theoretical and measured values correspond. The iToF imaging system in the embodiment considers the attenuation of the light path and adds a differentiable iToF imaging model. The light path attenuation calibrated by the present application is shown in Figure 5

[0061] After the above physical constraint conditions, the data in the training set are input into the differentiable physical imaging model, and then the noise-free multiple measurement values corresponding to the actual system are obtained.

[0062] Step 3, constructing a visual TOF double-branch multi-scale fusion depth reconstruction network, including a depth feature extraction network for extracting pixel-by-pixel multi-scale TOF depth information, a monocular depth generation network for extracting monocular depth information, and an edge feature extraction network for extracting edge structure information;

[0063] Step 4, obtaining the pixel-by-pixel TOF depth information through the depth feature extraction network, extracting the monocular depth through the monocular depth generation network, performing supervision correction on the TOF pixel-level depth through multi-scale fusion, extracting the high-resolution color picture edge through the edge feature extraction network, and optimizing using the edge structure loss.

[0064] ​The deep feature extraction network obtains the pixel-by-pixel TOF depth information, specifically: first, using the multi-scale features extracted by the feature extraction block of the encoding module, features of different scales are obtained; using the multi-scale feature fusion module to fuse the features of different scales, input to the decoding module; the decoding module uses the residual depth reconstruction module to convolve and integrate the depth map obtained by upsampling and the fusion features output by the multi-scale feature fusion module, in a coarse-to-fine manner, to generate a large-size depth map with more details. The monocular depth generation network uses the RefineNet network for monocular extraction, learns the relative depth sequence contained in the RGB image, and its architecture can generate a depth map from a single RGB image. The process of monocular depth feature extraction is completed through a series of convolution layers and pooling layers. The convolution layer is responsible for sliding the filter on the image to extract local features of the image, and the pooling layer is responsible for down-sampling the features to reduce the amount of calculation and avoid overfitting. This feature extraction process can effectively capture the depth information in the image and correct the depth map by incorporating the pixel-by-pixel TOF depth information.

[0065] The edge feature extraction network is used to directly reconstruct the super-resolution image soft edge from the RGB image. The embodiment is improved and optimized on a basic network structure called multi-scale residual network (MSRN), and uses the edge GT picture in the data set for supervision. MSRN is an efficient network that can adaptively detect image features of different scales, which is beneficial to image soft edge extraction. However, the original MSRN is too complex and is designed for super-resolution tasks, and is not suitable for image soft edge feature extraction. In order to make MSRN adapt to the soft edge reconstruction task, the following modifications are made: (1) reduce the number of multi-scale residual blocks. (2) learn the mapping between RGB images and image edges, rather than the mapping between low-resolution RGB images and high-resolution images. The original multi-scale residual block is designed for super-resolution tasks, and the input low-resolution RGB picture is used to obtain the output super-resolution RGB image, that is, the network learns the mapping between low-resolution RGB images and high-resolution images. In order to adapt to the task of soft edge extraction in this embodiment, the output is changed to high-resolution RGB edge during training, so that the network can learn the mapping between RGB images and image edges. The multi-scale residual block can adaptively detect features of images of different sizes. Since the original task is to obtain a super-resolution RGB image from a low-resolution RGB image, 8 to 10 multi-scale residual blocks are used to achieve high-quality reconstruction. This example only learns the mapping from RGB images to edges, and focuses more on the extraction of high-frequency features, so it does not need to retain all high-frequency and low-frequency information. Therefore, reducing the number of multi-scale residual blocks to 3 can still obtain relatively high reconstruction quality, and reducing the number of multi-scale residual blocks can also reduce the parameter amount of the network, which is beneficial to reducing the resources and energy consumption occupied by the network.

[0066] Step 5, the modified depth map in step 4 above is fused with the color picture edge extracted by the edge feature extraction network to construct an edge-assisted depth fusion network, which is iteratively optimized as an end-to-end network.

[0067] The modified depth map and the color picture edge are fused in a progressive manner, and the fused information and the depth image have a nonlinear feature mapping relationship. The rough depth image features obtained by the depth feature extraction network contain a large number of low-frequency features, and the soft edge obtained by the edge feature extraction network contains rich high-frequency details. In order to fully play the image features and realize the soft edge guidance, a progressive fusion layer is applied to fuse the low-frequency and high-frequency features, and then the fused feature map is sent to the edge-assisted depth fusion network to reconstruct a clear depth image. The edge-assisted depth fusion network realizes effective improvement of the details and levels of the reconstructed image. At the same time, the network also adopts a cross-layer connection method, so that the feature information can be transmitted more smoothly, improving the expression ability and training effect of the fusion network.

[0068] Step 6, construct the loss function of the coding function and the visual TOF double-branch multi-scale fusion depth reconstruction network, including color pseudo-edge guidance loss, Fisher information guidance loss function, multi-measurement difference loss function and depth fidelity loss function, which can be used for joint training optimization of the end-to-end framework of the coding function and the visual TOF double-branch multi-scale fusion depth reconstruction network. The overall loss function formula of the system is:

[0069] L = L Fidclity + λ1L fishcr + λ2L cdgc + λ3L dis

[0070] Wherein, L Fidclity is the depth fidelity loss, L fisher is the Fisher information guidance loss, L Edge is the edge loss, L dis is the multi-measurement difference loss, and λ1, λ2, λ3 are hyperparameters for network training.

[0071] First, analyze the system noise to obtain the Fisher information guidance loss. The measurement value X i in the iToF system can be divided as follows:

[0072] X i (p) = S i (p) + N dark +N readout

[0073] where the photon noise is the statistical noise due to the process of collecting photons by the imaging sensor and exposing the process. Since the process of photons reaching the imaging sensor is random and the measurement result is subject to Poisson statistics, the measurement result is the photon signal related to the scene depth following Poisson distribution. Therefore, in iToF imaging, the size of the photon noise only depends on the measured signal. The dark current noise has Poisson distribution and exists as long as the iToF imaging system is working. It is caused by the heat generated by the silicon lattice of the imaging sensor. Therefore, the size of the dark current noise is not related to light, only when the temperature of the sensor is higher, the dark current noise is larger. readout ~ N(0, σ r 2 is the readout noise following Gaussian distribution, which is the inherent noise of the readout circuit of iToF imaging, σ r is the standard deviation of the readout noise following Gaussian distribution, which is generated when the op-amp amplified electrical signal is converted to digital quantity readout by the analog-to-digital converter. dark ~ P(λd) is the dark current noise following Poisson distribution, and λd is the expectation of the dark current noise.

[0074] The noise distribution model of iToF imaging measurement value is obtained by approximating Poisson noise with variable parameter additive Gaussian noise:

[0075]

[0076]

[0077] s i (p) is the measurement value of the theoretical time-of-flight imaging, X(p) is the multiple measurement values, μ i (p) and σ i (p) are the mean and standard deviation of the readout noise following Gaussian distribution.

[0078] For each pixel point in iToF imaging, multiple measurement values are obtained by using different combinations of emitted and demodulated functions, N is the measurement number of each pixel point, and the probability of detecting these measurement values is:

[0079]

[0080] The Fisher information of the observation variable X(p) with respect to the unknown parameter depth z(p) is:

[0081]

[0082] It can be deduced that the Fisher information of the measurement value X(p) of iToF imaging with respect to the depth z(p) is:

[0083]

[0084] Thus, the Fisher information guided loss function is obtained:

[0085]

[0086] In order to further improve the efficiency of multiple modulation and demodulation functions and strengthen the difference between different modulation and demodulation functions, Manhattan distance is used as the difference loss of multiple measurements:

[0087]

[0088] Where i, j are the serial numbers of the measurement times, s i (p) Measuring the system value as described in step 2 above.

[0089] Points with sharp changes in image brightness are usually organized into a group of curve segments called image edges. Various methods have been proposed to extract image edges, including Sobel, Prewitt, Roberts and Canny. However, these methods have some limitations and are difficult to apply to any image. At the same time, these ready-made edge detectors use binary measurement to convert all edge values to 0 and 1, which leads to the loss of a large number of image features and the emergence of false edges. In order to avoid these problems, the present application uses image soft edges instead of image edges. Soft edges are obtained by eliminating the binary strategy, which can retain more accurate image edge information, so the following curvature formula is used to obtain the corresponding soft edge I HR from I Edge Because it can accurately describe the change in the gradient domain:

[0090] I Edge = div(u x ,u y )

[0091]

[0092] x and y represent the horizontal and vertical directions respectively, i represents selecting one direction from the x and y directions, u i represents the gradient component in the i direction, u x , u y respectively represent the gradient components in the horizontal or vertical direction when i takes the value of x or y. At the same time, and div(·) represent gradient and divergence operations respectively. Thus, the edge loss is defined as:

[0093]

[0094] where E(·) represents the Edge-Net edge feature extraction network, E(I LR) represents the reconstructed soft edge, I Edge is the soft edge detected from the corresponding super-resolution RGB image.

[0095] Step 7, in the test stage, the imaging process simulation of the samples in the test set is carried out using the optimized encoding function, the measurement value with noise is obtained, and is transmitted to the trained deep reconstruction neural network, and finally the depth map of the test scene is obtained;

[0096] A hardware prototype system of RGB edge guided large range noise robust depth imaging is built, and the optimized encoding function is used as the light source modulation and sensor demodulation function of the hardware prototype system. The measurement value and the RGB image of the real scene are collected, input into the optimized visual TOF double-branch multi-scale depth reconstruction neural network, and finally the depth map of the real scene is reconstructed to complete indirect time-of-flight imaging and optimization.

[0097] The structure of the visual TOF double-branch multi-scale fusion depth reconstruction network involved in the embodiment is as shown in Figure 2 The network mainly consists of two branches. The upper branch is a depth extraction network that obtains TOF depth information. It mainly consists of a feature pyramid structure multi-scale neural network and a pixel-level depth extraction neural network, can simultaneously utilize spatial structure and measurement value to depth mapping to guide depth reconstruction, can ensure fast convergence of encoding function learning, and the obtained depth map can better preserve the feature details of the scene. In order to further improve the efficiency of multiple modulation and demodulation functions and strengthen the difference between different modulation and demodulation functions, Manhattan distance is used as the multiple measurement difference loss.

[0098] The lower branch mainly consists of a monocular depth generation network and an edge feature extraction network. The monocular depth network extracts monocular depth, and the TOF pixel-level depth is supervised and corrected through multi-scale fusion. On the other hand, the high-resolution edge structure information of the RGB image extracted by the edge feature extraction network and the corrected TOF depth map are added through the attention mechanism according to the channel to perform feature fusion, and then pass through a CNN convolution layer to obtain the reconstructed fine depth map. The high-quality edge reconstruction guide is provided for the depth image, so that the reconstruction quality of the depth image is improved.

[0099] The entire network of the present application uses color pseudo-edge guided loss, Fisher information guided loss, multiple measurement difference loss and depth fidelity loss for supervision. According to the Fisher information of the iToF imaging measurement value, the Fisher information guided loss L fisher .

[0100] As Figure 3As shown, the embodiment also builds a hardware prototype system, which can realize visual radar fusion imaging of any coding function, and the system includes a laser diode, a function signal generator, a beam splitter, a scanning device, a light collecting lens, an avalanche photodiode, a multiplier, an amplifier, a low-pass filter, a single-chip microcomputer and a color camera. A laser diode with a wavelength of 638 nm and a maximum power of 200 mw is used as the light source of the system; the learned coding function is generated by the function signal generator, and the modulation function in the coding function is coupled to the laser diode by a Bias-T coupling circuit to modulate the laser emitted by the laser diode; the laser passes through the beam splitter and reaches the scanning device; the scanning device scans the scene by using a two-dimensional scanning galvanometer; the laser irradiates a point of the scene, and the reflected light is focused on an avalanche photodiode through a light collecting lens; the photodiode converts the reflected light signal into an electrical signal, then the converted signal is multiplied with the learned demodulation function by the multiplier, and the output voltage value is further amplified and low-pass filtered by the amplifier and the low-pass filter; finally, the voltage value amplified and low-pass filtered is sampled, quantized and converted into a measurement value by an analog-to-digital converter, and the scanning and collection of the system are controlled by the single-chip microcomputer. The same view color picture is taken by the color camera. A color picture and TOF measurement values are transmitted to the optimized depth reconstruction network, and finally the fusion depth map of the real scene is reconstructed.

[0101] Please refer to Figure 6 The method of the present application is compared with the existing time-of-flight imaging method, coding function and reconstruction algorithm respectively on the data NYU-V2. The method achieves the best result in the MAE index of three noise levels.

[0102] As Figure 7 shown, the reconstruction results of the scenes in the NYUv2 dataset on different coding functions are compared. The top images in the first column are GroundTruth (GT), and the bottom images are the corresponding RGB images. The top images in the second to fifth columns are the reconstructed depth maps (where ours represents the method of the present application, sine represents the method of the modulation and demodulation function being a sine function, squre represents the method of the modulation and demodulation function being a square wave function, and double frequency sine represents the method of the modulation and demodulation function being a double frequency sine function), and the bottom images are error maps (showing the difference between the images reconstructed by the network and the high-resolution images). It can be seen that the method of the present application can effectively eliminate the influence of noise and reconstruct a depth map close to the real depth value.

[0103] As Figure 8The reconstruction results of scenes of the SUN RGB-D dataset and the 4D Light Field dataset on different encoding functions and networks are shown for comparison. The images in the first column on the top are Ground Truth (gt), and the images on the bottom are corresponding RGB images. The images in the second to fifth columns on the top are reconstructed depth maps (wherein ours represents the method of the present application, hami+network represents that the reconstruction method is a network with a hamilton function as a modulation and demodulation function, squre+PS represents that the reconstruction method is a phase shift method with a square function as a modulation and demodulation function, and sine+PS represents that the reconstruction method is a phase shift method with a sine function as a modulation and demodulation function), and the images on the bottom are error maps. It can be seen that the method of the present application can effectively eliminate the influence of noise and reconstruct a depth map close to the true depth value.

Claims

1. A large-range noise-robust depth imaging method based on RGB-guided, characterized in that, The method comprises the following steps: Step 1, constructing a database, including a training set and a test set, and containing color images and depth images of the same scene; Step 2, establishing a differentiable physical imaging model to simulate the time-of-flight imaging process, adding physical constraints of the actual scene, setting the encoding function including the light source modulation function and the sensor demodulation function as a learnable parameter, inputting the data in the training set into the differentiable physical imaging model to obtain noise-free multiple measurements; Step 3, constructing a visual TOF double-branch multi-scale fusion depth reconstruction network, including a depth feature extraction network for extracting pixel-by-pixel multi-scale TOF depth information, a monocular depth generation network for extracting monocular depth information, and an edge feature extraction network for extracting edge structure information; Step 4, obtaining pixel-by-pixel TOF depth information through the depth feature extraction network, extracting monocular depth through the monocular depth generation network, and supervising and correcting the TOF depth through multi-scale fusion of the monocular depth to obtain a corrected depth map; extracting high-resolution color picture edges through the edge feature extraction network and optimizing using edge structure loss; Step 5, fusing the corrected depth map and the high-resolution color picture edges, and constructing an edge-assisted depth fusion network as an end-to-end network for iterative optimization; Step 6, using color pseudo-edge guided loss and Fisher information guided loss as loss functions, and combining multiple measurement difference loss function and depth fidelity loss function to jointly train the encoding function and the fusion depth reconstruction network; Step 7, inputting test data into the trained encoding function and fusion depth reconstruction network to reconstruct the test scene depth map; using the encoding function and the fusion depth reconstruction network trained in step 6 to reconstruct the actual scene high-quality depth map.

2. The RGB-guided large-range noise-robust depth imaging method of claim 1, wherein, In step 1, the database includes virtual synthesis and real-world scene RGBD images and fine edge images based on color pictures, and each sample in the training set and the test set includes scene depth, surface reflectivity and ambient light intensity; when constructing the database, the RGB image of the scene image is subjected to intrinsic image decomposition to obtain the reflectivity map of the scene, the surface reflectivity of the scene is obtained by extracting the R channel of the reflectivity map, and the ambient light intensity is obtained by averaging the RGB three channels of the reflectivity map; The gradient and divergence of the scene RGB image are calculated to obtain an edge map containing rich texture information.

3. The RGB-guided large-range noise-robust depth imaging method of claim 1, wherein, In step 2, the differentiable physical imaging model is specifically: Assume that the emitted light is modulated by a function M i (t) modulated, then the reflected signal of the scene point p is: where a(p) is the amplitude coefficient due to the scene reflection, β is the ambient component due to other light sources, is the distance-dependent time delay of light propagation, i is the number of modulation functions, t is the time delay; The specific form of the reflected signal is: where z(p) is the depth of light propagation, c is the speed of light, F falloff is the optical attenuation coefficient; Using a demodulation function D i (t) demodulating the reflected signal Ri(p, t) to derive a measurement value Where T is the integration time.

4. The RGB-guided large-scale noise-robust depth imaging method according to claim 1, characterized in that: In the step 4, the pixel-by-pixel TOF depth information is obtained through a deep feature extraction network, specifically: first, a feature extraction block of an encoding module is used to extract multi-scale features to obtain features of different scales; a multi-scale feature fusion module is used to fuse the features of different scales and input the fused features to a decoding module; the decoding module uses a residual depth reconstruction module to perform convolution integration on the depth map obtained by upsampling and the fused features output by the multi-scale feature fusion module, and generate a large-size depth map with more details in a coarse-to-fine manner.

5. The RGB-guided large-range noise-robust depth imaging method of claim 1, wherein, In the step 4, monocular depth is extracted through a monocular depth generation network, specifically: a RefineNet network is used for monocular extraction to learn the relative depth sequence contained in the RGB image.

6. The RGB-guided, large-range noise-robust depth imaging method of claim 1, wherein, In the step 4, high-resolution color picture edges are extracted through an edge feature extraction network, specifically: an MSRN network is used to learn the mapping between the RGB image and the image edges to extract the soft edge information of the RGB picture.

7. The RGB-guided, large-range noise-robust depth imaging method of claim 6, wherein, In step 4, the following curvature formula is used to obtain the corresponding soft edge I HR from the super-resolved RGB image I Edge ​ I Edge = div(u x , u y ) x and y denote horizontal and vertical directions, i denotes a direction chosen from x and y, u i denotes a gradient component in the i direction, u x , u y denote gradient components in the horizontal or vertical direction when i takes the value x or y, respectively, and div(·) denote gradient and divergence operations, respectively.

8. The RGB-guided, large-range noise-robust depth imaging method of claim 1, wherein, In the step 5, the corrected depth map and the high-resolution color picture edges are added by channel through an attention mechanism for feature fusion, and a CNN convolution layer is used to obtain a reconstructed fine depth map.

9. The RGB-guided, large-range noise-robust depth imaging method of claim 1, wherein, In the step 6, the Fisher information I is: where z(p) is the depth of light propagation, s i (p) is the measured value of the theoretical time-of-flight imaging, X(p) is the multiple measurements, σ i (p) is the standard deviation of the readout noise of the Gaussian distribution.

10. A system for implementing the method of wide-range noise-robust depth imaging based on RGB guidance as claimed in claim 1, characterized in that, The system comprises a laser diode, a function signal generator, a beam splitter, a scanning device, a light collecting lens, an avalanche photodiode, a multiplier, an amplifier, a low-pass filter and a color camera, specifically: a laser diode is used as a light source; a learned encoding function is generated by a function signal generator, and a modulation function in the encoding function modulates the laser emitted by the laser diode; the laser emitted by the laser diode passes through the beam splitter and reaches the scanning device, and the scanning device scans the scene; the laser irradiates the points of the scene and is reflected according to the original light path; the reflected light is focused on the avalanche photodiode through the light collecting lens; the avalanche photodiode converts the reflected light signal into an electric signal, then multiplies the converted signal with a demodulation function in the encoding function through the multiplier, and further amplifies and low-pass filters the output voltage value through the amplifier and the low-pass filter; finally, the voltage value amplified and low-pass filtered is sampled, quantized and converted into a measurement value through an analog-to-digital converter.

Images