Large-range noise robust depth imaging method and system based on RGB guidance

Abstract

The invention discloses a large-range noise robust depth imaging method and system based on RGB guidance. The method comprises the following steps: constructing a calibration RGB-D database; establishing a differentiable physical imaging model, and adding physical constraints of an actual scene to obtain a plurality of noise-free measurement values; constructing a visual TOF double-branch multi-scale fusion depth reconstruction network, extracting and fusing depth features and visual features to serve as an end-to-end network for iterative optimization, supervising a network reconstruction effect by using color pseudo edges and Fischer information loss, and jointly training a coding function and the depth reconstruction network; inputting the test data into the trained coding function and the reconstruction network, and reconstructing a depth map of the test scene; and building a hardware prototype system for RGB-guided large-range noise robust depth imaging, and reconstructing a high-quality depth map in an actual scene by using an optimized coding function and a reconstruction network.

Description

Technical Field

[0001] The present invention belongs to the field of computational imaging and deep learning technology, and in particular to a method and system for large-scale noise robust depth imaging based on RGB guidance. Background Art

[0002] Three-dimensional imaging technology has been widely used in autonomous driving, augmented reality, medical surgery, industrial automation and many other fields. These computer vision tasks all rely on high-resolution and high-precision depth perception. Depth imaging can be divided into passive and active light imaging according to the presence or absence of active light: passive imaging, that is, visual imaging, includes monocular and binocular imaging, and active light imaging includes direct and indirect time of flight (TOF).

[0003] Since the release of Microsoft Kinect in 2010, consumer depth sensors have been widely used. Kinect is based on a monocular structured light method, which was also adopted by the iPhone X released in 2017. However, it cannot obtain depth measurements of distant objects and performs poorly in outdoor scenes under strong light. Binocular stereo vision systems have a larger measurement range than structured light systems and can also work in outdoor environments with strong sunlight, but are easily affected by the surface texture of objects. TOF technology is a technology for measuring the distance of objects and has a wide range of applications in machine vision, augmented reality, 3D imaging and other fields. However, due to the influence of environmental conditions, TOF images are often interfered by noise and have poor adaptability to low-reflectivity objects and distant objects.

[0004] Therefore, a single type of sensor has its own limitations in terms of 3D information perception, but from the principle of ranging of each camera, its error sources can be considered independent of each other. Therefore, through appropriate active and passive imaging fusion technology, taking advantage of each other's strengths and weaknesses, more accurate scene depth information can be obtained than any of the above detection technologies alone. Summary of the invention

[0005] In view of the problems and considerations existing in the above existing three-dimensional imaging methods, the purpose of the present invention is to propose a large-scale noise robust TOF imaging method and system based on RGB guidance.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A large-scale noise robust depth imaging method based on RGB guidance, the method comprises the following steps:

[0008] Step 1: construct a database, including a training set and a test set, and contain color images and depth images of the same scene;

[0009] Step 2, establishing a differentiable physical imaging model to simulate the time-of-flight imaging process, adding physical constraints of the actual scene, setting the encoding function to a learnable parameter, the encoding function including a light source modulation function and a sensor demodulation function; inputting the data in the training set into the differentiable physical imaging model to obtain multiple noise-free measurements;

[0010] Step 3, constructing a visual TOF dual-branch multi-scale fusion depth reconstruction network, including a deep feature extraction network for extracting pixel-by-pixel multi-scale TOF depth information, a monocular depth generation network for extracting monocular depth information, and an edge feature extraction network for extracting edge structure information;

[0011] Step 4, obtaining pixel-by-pixel TOF depth information through a depth feature extraction network, extracting monocular depth through a monocular depth generation network, and supervising and correcting the TOF depth through multi-scale fusion of the monocular depth to obtain a corrected depth map; extracting the edge of the high-resolution color image through an edge feature extraction network, and optimizing it using edge structure loss;

[0012] Step 5, fusing the corrected depth map with the edge of the high-resolution color image, and constructing an edge-assisted deep fusion network as an end-to-end network for iterative optimization;

[0013] Step 6, using the color pseudo edge guided loss and the Fisher information guided loss as the loss function, and combining the multiple measurement difference loss function and the depth fidelity loss function, jointly training the encoding function and the fusion deep reconstruction network;

[0014] In step 7, the test data is input into the trained encoding function and fused depth reconstruction network to reconstruct the depth map of the test scene; the encoding function and fused depth reconstruction network trained in step 6 are used to reconstruct a high-quality depth map of the actual scene.

[0015] Furthermore, in step 1, the database includes RGBD images of virtual synthesis and real-world scenes and fine edge images based on color pictures, and each sample in the training set and the test set includes the depth, surface reflectivity and ambient light intensity of the scene; when constructing the database, the RGB image of the scene image is subjected to intrinsic image decomposition to obtain a reflectivity map of the scene, the surface reflectivity of the scene is obtained by extracting the R channel of the reflectivity map, and the ambient light intensity is obtained by averaging the three RGB channels of the reflectivity map; the RGB image of the scene is subjected to gradient and divergence operations to obtain an edge map containing rich texture information.

[0016] Furthermore, in step 2, establishing a differentiable physical imaging model is specifically as follows:

[0017] Assume that the emitted light is modulated by the function Mi (t) modulation, the reflected signal of scene point p is:

[0018]

[0019] where α(p) is the amplitude coefficient due to scene reflection, β is the ambient component due to other light sources, is the distance-dependent time delay of light propagation, i is the number of modulation functions, and t is the time delay;

[0020] The specific form of the reflected signal is:

[0021]

[0022] where z(p) is the depth of light propagation, c is the speed of light, and F falloff is the light attenuation coefficient;

[0023] Using the demodulation function D i (t) Demodulate the reflected signal Ri(p,t) to derive the measured value

[0024] s i (p) = ∫0 T R i (p,t)D i (t)dt,

[0025] Where T is the integration time.

[0026] Furthermore, in step 4, the pixel-by-pixel TOF depth information is obtained through a deep feature extraction network, specifically: first, the feature extraction block of the encoding module is used to extract multi-scale features to obtain features of different scales; the features of different scales are fused using a multi-scale feature fusion module and input into a decoding module; the decoding module uses a residual depth reconstruction module to convolute and integrate the upsampled depth map with the fused features output by the multi-scale feature fusion module, and generates a large-size depth map with more details in a coarse-to-fine manner.

[0027] Furthermore, in step 4, the monocular depth is extracted through a monocular depth generation network, specifically: a RefineNet network is used for monocular extraction to learn a relative depth sequence contained in the RGB image.

[0028] Furthermore, in step 4, the edge of the high-resolution color image is extracted through an edge feature extraction network, specifically: the MSRN network is used to learn the mapping between the RGB image and the image edge, and the soft edge information of the RGB image is extracted.

[0029] Furthermore, in step 5, the corrected depth map and the edge of the high-resolution color image are feature fused by channel addition through an attention mechanism, and a reconstructed fine depth map is obtained after passing through a CNN convolution layer.

[0030] The present invention also provides a system for realizing the above-mentioned RGB-guided large-scale noise-robust depth imaging method, which includes a laser diode, a function signal generator, a beam splitter, a scanning device, a light-collecting lens, an avalanche photodiode, a multiplier, an amplifier, a low-pass filter and a color camera, and specifically: a laser diode is used as an emitting light source; a learned encoding function is generated by a function signal generator, and a modulation function in the encoding function modulates the laser emitted by the laser diode; the laser emitted by the laser diode passes through a beam splitter and reaches a scanning device, and the scanning device scans the scene; the laser irradiates a point on the scene and is reflected according to the original optical path; the reflected light is focused onto an avalanche photodiode through a light-collecting lens; the avalanche photodiode converts the reflected light signal into an electrical signal, and then multiplies the converted signal with the demodulation function in the encoding function through a multiplier, and the output voltage value is further amplified and low-pass filtered through an amplifier and a low-pass filter; finally, the amplified and low-pass filtered voltage value is sampled, quantized and converted into a measurement value through an analog-to-digital converter.

[0031] The method of the present invention takes into account that the depth image and the color image of the same scene have a high correlation at the texture edge, extracts features from the depth measurement value and the color image respectively, and fuses them for the final depth prediction. The prior information provided by the clear edge information in the color image can further improve the performance of the depth image. Compared with the prior art, the present invention has the following advantages:

[0032] (1) A learnable physical imaging model is established to simulate the iToF imaging process, and actual physical constraints are added, including optical path attenuation, so that physically realizable modulation and demodulation functions can be learned.

[0033] (2) The noise model of the iToF imaging system is derived and guided by the Fisher information, combined with the color pseudo-edge guidance loss, multiple measurement difference loss and depth fidelity loss to supervise the entire system framework.

[0034] (3) A visual TOF dual-branch multi-scale fusion depth reconstruction network was designed, which used RGB monocular depth information and edge structure features for fusion guidance to achieve depth imaging under high noise levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the method of the present invention;

[0036] Figure 2A network structure diagram of the method of the present invention;

[0037] Figure 3 A diagram of a hardware prototype system for time-of-flight imaging built for an example of the present invention;

[0038] Figure 4 It is a simulation diagram of depth scenes with different light source photon quantities and ambient photon quantities in an embodiment of the present invention;

[0039] Figure 5 is a light path attenuation diagram calibrated in an embodiment of the present invention;

[0040] Figure 6 This is a comparison result diagram of the method of the present invention on the data set NYU-V2;

[0041] Figure 7 This is the depth reconstruction result of the method of the present invention in the data set NYU-V2;

[0042] Figure 8 Comparison of reconstruction results of the method of the present invention on (a) SUN RGB-D dataset and (b) 4D Light Field dataset scenes on different encoding functions and networks. DETAILED DESCRIPTION

[0043] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Reference Figure 1 , a large-scale noise robust depth imaging method based on RGB guidance in this embodiment, the specific steps are as follows:

[0045] Step 1: construct a database, including a training set and a test set, which contains color images and depth images of the same scene.

[0046] The database includes RGBD images of virtual synthesis and real-world scenes, as well as fine edge images based on color pictures. Each sample in the training set and test set includes the depth, surface reflectivity and ambient light intensity of the scene. When constructing the database, the RGB map of the scene image is subjected to intrinsic image decomposition to obtain the reflectivity map of the scene. The surface reflectivity of the scene is obtained by extracting the R channel of the reflectivity map, and the ambient light intensity is obtained by averaging the three RGB channels of the reflectivity map. The RGB image of the scene is subjected to gradient and divergence operations to obtain an edge map containing rich texture information.

[0047] In this embodiment, the NYU-V2 dataset is selected for training, and generalization verification is performed on 17 scenes of the SUN RGB-D dataset and 300 scenes of the 4DLight Field dataset.

[0048] For scenes with different noise levels (i.e. signal-to-noise ratio levels), the system simulates depth scenes with different light source photon quantities and ambient photon quantities, such as Figure 4 As shown, the first column is the GT image and the corresponding RGB image, the second and third columns are the simulated measurement value images under higher signal-to-noise ratio, and the fourth and fifth columns are the simulated measurement value images under lower signal-to-noise ratio:

[0049] Step 2: Establish a differentiable physical imaging model to simulate the time-of-flight imaging process. Assume that the emitted light is modulated by the function

[0050] Mi(t) modulation, the reflected signal of scene point p is:

[0051]

[0052] where α(p) is the amplitude coefficient due to scene reflection, β is the ambient component due to other light sources, is the distance-dependent time delay of light propagation, i is the number of modulation functions, and t is the time delay;

[0053] The specific form of the reflected signal is:

[0054]

[0055] where z(p) is the depth of light propagation, c is the speed of light, and F falloff is the light attenuation coefficient;

[0056] Using the demodulation function D i (t) Demodulate the reflected signal Ri(p,t) to derive the measured value

[0057] s i (p) = ∫0 T R i (p,t)D i (t)dt,

[0058] Where T is the integration time, s i (p) contains three unknowns: α(p), β, z(p), so N ≥ 3 measurements are usually required to reconstruct the depth z(p). When α(p), β, z(p) are set as differentiable parameters, the above iToF imaging process can be regarded as a differentiable iToF physical imaging model. By setting the amplitude and phase of the encoding function (i.e., modulation and demodulation functions) as learnable parameters, the entire encoding function becomes a learnable function, which is applied to the forward model of the entire iToF imaging system, and the imaging system becomes a differentiable form.

[0059] However, in actual imaging systems, there will be obvious errors between the theoretical values ​​simulated in the forward model and the actual physical measurement values ​​due to device limitations. Therefore, when applying the physical imaging model to the actual system, physical constraints need to be imposed according to the actual system.

[0060] When a point light source illuminates a scene, it emits a spherical wave. Since the total energy is constant, the light intensity received per unit area decays with the square of the propagation distance of the light source. The optical path attenuation coefficient is very important. When designing a reconstruction algorithm, the attenuation of the optical path must be considered to make the theoretical and measured values ​​correspond. The attenuation of the optical path is considered in the iToF imaging system of this embodiment and is added to the differentiable iToF imaging model. The optical path attenuation calibrated by the present invention is as follows: Figure 5 As shown:

[0061] After the above physical constraints are met, the data in the training set is input into the differentiable physical imaging model, and noise-free multiple measurement values ​​corresponding to the actual system are obtained.

[0062] Step 3, constructing a visual TOF dual-branch multi-scale fusion depth reconstruction network, including a deep feature extraction network for extracting pixel-by-pixel multi-scale TOF depth information, a monocular depth generation network for extracting monocular depth information, and an edge feature extraction network for extracting edge structure information;

[0063] Step 4: Obtain pixel-by-pixel TOF depth information through the deep feature extraction network, extract monocular depth through the monocular depth generation network, supervise and correct the TOF pixel-level depth through multi-scale fusion, extract the edge of the high-resolution color image through the edge feature extraction network, and optimize it using the edge structure loss.

[0064] The deep feature extraction network obtains pixel-by-pixel TOF depth information, specifically: first, the multi-scale features extracted by the feature extraction block of the encoding module are used to obtain features of different scales; the features of different scales are fused using the multi-scale feature fusion module and input to the decoding module; the decoding module uses the residual depth reconstruction module to convolve the upsampled depth map with the fused features output by the multi-scale feature fusion module, and generates a large-size depth map with more details in a coarse-to-fine manner. The monocular depth generation network uses the RefineNet network for monocular extraction and learns the relative depth sequence contained in the RGB image. Its architecture can generate a depth map using a single RGB image. The process of monocular depth feature extraction is completed through a series of convolutional layers and pooling layers. The convolutional layer is responsible for sliding the filter on the image to extract local features of the image, while the pooling layer is responsible for downsampling the features to reduce the amount of calculation and avoid overfitting. This feature extraction process can effectively capture the depth information in the image and integrate it into the pixel-by-pixel TOF depth information to correct the depth map.

[0065] The edge feature extraction network is used to directly reconstruct the soft edge of the super-resolution image from the RGB image. This embodiment improves and optimizes a basic network structure called a multi-scale residual network (MSRN), and uses the edge GT image in the data set for supervision. MSRN is an efficient network that can adaptively detect image features of different scales. This feature is conducive to the extraction of soft edges of images. However, the original MSRN is too complex and is designed for super-resolution tasks. It is not suitable for the extraction of soft edge features of images. In order to make MSRN suitable for the soft edge reconstruction task, the present invention makes the following modifications: (1) reduce the number of multi-scale residual blocks. (2) learn the mapping between RGB images and image edges, rather than the mapping between low-resolution RGB images and high-resolution images. The original multi-scale residual block is designed for super-resolution tasks. The input is a low-resolution RGB image to obtain the output super-resolution RGB image, that is, the network learns the mapping between low-resolution RGB images and high-resolution images. In order to adapt to the task of soft edge extraction in this embodiment, the output is changed to a high-resolution RGB edge during training, so that the network can learn the mapping between RGB images and image edges. The multi-scale residual block can adaptively detect the features of images of different sizes. Since the original task is to obtain a super-resolution RGB image from a low-resolution RGB image, 8 to 10 multi-scale residual blocks are used to achieve high-quality reconstruction. This example only learns the mapping from RGB images to edges, focusing more on the extraction of high-frequency features. It does not need to retain all high-frequency and low-frequency information, so reducing the number of multi-scale residual blocks to 3 can still achieve high reconstruction quality. Reducing the number of multi-scale residual blocks can also reduce the number of network parameters, which is conducive to reducing the resources and energy consumption occupied by the network.

[0066] In step 5, the depth map corrected in step 4 above is fused with the edge of the color image extracted by the edge feature extraction network, and an edge-assisted deep fusion network is constructed as an end-to-end network for iterative optimization.

[0067] The corrected depth map and the edge of the color image are progressively fused, and there is a nonlinear feature mapping relationship between the fused information and the depth image. The rough depth image features obtained by the deep feature extraction network contain a large number of low-frequency features, and the soft edges obtained by the edge feature extraction network contain rich high-frequency details. In order to give full play to the image characteristics and realize soft edge guidance, a progressive fusion layer is applied to fuse low-frequency and high-frequency features, and then the fused feature map is sent to the edge-assisted deep fusion network to reconstruct a clear depth image. The edge-assisted deep fusion network effectively improves the details and layering of the reconstructed image. At the same time, the network also adopts a cross-layer connection method, which enables the feature information to be transmitted more smoothly, improving the expression ability and training effect of the fusion network.

[0068] Step 6, construct the loss function of the encoding function and the visual TOF dual-branch multi-scale fusion deep reconstruction network, including the color pseudo-edge guided loss, the Fisher information guided loss function, the multiple measurement difference loss function and the depth fidelity loss function, and the encoding function and the visual TOF dual-branch multi-scale fusion deep reconstruction network can be jointly trained and optimized in an end-to-end framework. The overall loss function formula of the system is:

[0069] L=L Fidclity +λ1L fishcr +λ2L cdgc +λ3L dis

[0070] Among them, L Fidclity is the depth fidelity loss, L fisher is the Fisher information guided loss, L Edge is the edge loss, L dis To measure the difference loss multiple times, λ1, λ2, and λ3 are the hyperparameters of network training.

[0071] First, the system noise is analyzed to obtain the Fisher information guidance loss. The measured value X in the iToF system i At pixel point p, it can be divided into:

[0072] X i (p) = S i (p)+N dark +N readout

[0073] Among them, photon noise is the statistical noise caused by the imaging sensor collecting photons and performing the exposure process. Since the process of photons reaching the imaging sensor is random and the measurement result conforms to the Poisson statistical law, the measurement result is a photon signal related to the scene depth that follows the Poisson distribution. Therefore, in iToF imaging, the size of the photon noise depends only on the measured value signal. Dark current noise has a Poisson distribution and exists as long as the iToF imaging system is working. It is caused by electrons generated by the heat of the silicon lattice of the imaging sensor. Therefore, the size of the dark current noise has nothing to do with light. Only when the temperature of the sensor is higher, the dark current noise will be greater. N readout ~N(0,σ r 2 ) is the Gaussian distributed readout noise, which is the inherent noise of the readout circuit of iToF imaging, σ r is the standard deviation of the Gaussian distribution of read noise. When the electrical signal amplified by the operational amplifier is converted into a digital quantity for reading through an analog-to-digital converter, read noise will be generated. N dark ~P(λd) is the dark current noise with Poisson distribution, and λd is the expectation of the dark current noise.

[0074] Using additive Gaussian noise with variable parameters to approximate Poisson noise, we get the noise distribution model of iToF imaging measurement values:

[0075]

[0076]

[0077] s i (p) is the theoretical time-of-flight imaging measurement value, X(p) is the multiple measurement value, μ i (p) and σ i (p) is the mean and standard deviation of the Gaussian distributed read noise.

[0078] For each pixel in iToF imaging, multiple measurements are obtained by emitting different combinations of modulation functions and demodulation functions. N is the number of measurements for each pixel, and the probability of detecting these measurements is:

[0079]

[0080] The Fisher information of the observed variable X(p) relative to the unknown parameter depth z(p) is:

[0081]

[0082] It can be deduced that the Fisher information of the measured value X(p) of iToF imaging with respect to the depth z(p) is:

[0083]

[0084] This gives us the Fisher information guided loss function:

[0085]

[0086] In order to further improve the efficiency of multiple groups of modulation and demodulation functions and enhance the differences between different modulation and demodulation functions, Manhattan distance is used as the difference loss of multiple measurements:

[0087]

[0088] Among them, i, j are the serial numbers of the measurement times, s i (p) is the system measurement value as described in step 2 above.

[0089] Points where the brightness of an image changes sharply are usually organized into a group of curve segments called image edges. Many methods have been proposed to extract image edges, including Sobel, Prewitt, Roberts, and Canny. However, these methods have some limitations and are difficult to apply to arbitrary images. At the same time, these off-the-shelf edge detectors use binarization measurements to convert all edge values ​​to 0 and 1, which leads to the loss of a large number of image features and the appearance of false edges. In order to avoid these problems, the present invention uses image soft edges instead of image edges. Soft edges are obtained by eliminating the binarization strategy, which can retain more accurate image edge information, so the following curvature formula is used from I HR (Super-resolution RGB image) to obtain the corresponding soft edge I Edge , because it can accurately describe the changes in the gradient domain:

[0090] I Edge =div(u x ,u y )

[0091]

[0092] x and y represent the horizontal and vertical directions respectively, i represents selecting a direction from the x and y directions, u i represents the gradient component in the i direction, u x 、u y Respectively represent the gradient components in the horizontal or vertical direction when i is x or y. and div(·) represent the gradient and divergence operations respectively. Thus, the marginal loss is defined as:

[0093]

[0094] Where E(·) represents the Edge-Net edge feature extraction network, E(I LR) represents the reconstructed soft edge, I Edge are the soft edges detected from the corresponding super-resolved RGB image.

[0095] Step 7: In the test phase, the optimized encoding function is used to simulate the imaging process of the samples in the test set to obtain a noisy measurement map, which is then transmitted to the trained deep reconstruction neural network to finally obtain a depth map of the test scene.

[0096] A hardware prototype system for RGB edge-guided large-range noise-robust depth imaging is built, and the optimized encoding function is used as the light source modulation and sensor demodulation function of the hardware prototype system. The measurement values ​​and RGB images of the real scene are collected and input into the optimized visual TOF dual-branch multi-scale depth reconstruction neural network. Finally, the depth map of the real scene is reconstructed to complete indirect time-of-flight imaging and optimization.

[0097] The structure of the visual TOF dual-branch multi-scale fusion deep reconstruction network involved in this embodiment is as follows: Figure 2 As shown in the figure, the network mainly consists of two branches. The upper branch is a depth extraction network to obtain TOF depth information. It is mainly composed of a feature pyramid structure multi-scale neural network and a pixel-level depth extraction neural network. It can simultaneously use the spatial structure and the mapping of the measurement value to the depth to guide the depth reconstruction, which can ensure the rapid convergence of the coding function learning, and the obtained depth map better retains the characteristic details of the scene. In order to further improve the efficiency of multiple groups of modulation and demodulation functions and strengthen the differences between different modulation and demodulation functions, the Manhattan distance is used as the difference loss of multiple measurements.

[0098] The lower branch is mainly composed of a monocular depth generation network and an edge feature extraction network. The monocular depth is extracted through the monocular depth network and the TOF pixel-level depth is supervised and corrected through multi-scale fusion. On the other hand, the high-resolution edge structure information of the RGB image extracted by the edge feature extraction network and the corrected TOF depth map are added by channel through the attention mechanism for feature fusion. After the CNN convolution layer, the reconstructed fine depth map is obtained, which provides high-quality edge reconstruction guidance for the depth image, thereby improving the reconstruction quality of the depth image.

[0099] The entire network of the present invention uses color pseudo-edge guided loss, Fisher information guided loss, multiple measurement difference loss, and depth fidelity loss for supervision. According to the derived Fisher information of the iToF imaging measurement value, the Fisher information guided loss L can be obtained. fisher .

[0100] like Figure 3As shown, this embodiment also builds a hardware prototype system that can realize visual radar fusion imaging of any coding function. The system includes a laser diode, a function signal generator, a beam splitter, a scanning device, a light-collecting lens, an avalanche photodiode, a multiplier, an amplifier, a low-pass filter, a single-chip microcomputer and a color camera. A laser diode with a wavelength of 638nm and a maximum power of 200mw is used as the light source emitted by the system; the learned encoding function is generated by the function signal generator, and the modulation function in the encoding function modulates the laser emitted by the laser diode through the Bias-T coupling circuit; the laser passes through the beam splitter and reaches the scanning device; the scanning device uses a two-dimensional scanning galvanometer to scan the scene; the laser irradiates the point of the scene and reflects according to the original light path; the reflected light is focused on an avalanche photodiode through a light-collecting lens; the photodiode converts the reflected light signal into an electrical signal, and then multiplies the converted signal with the learned demodulation function through a multiplier, and the output voltage value is further amplified and low-pass filtered through an amplifier and a low-pass filter; finally, the amplified and low-pass filtered voltage value is sampled, quantized and converted into a measurement value through an analog-to-digital converter, and the scanning and acquisition of the system are controlled by a single-chip microcomputer. The same-view color picture is taken by a color camera. A color picture and the TOF measurement value are transmitted to the optimized depth reconstruction network, and finally a fused depth map of the real scene is reconstructed.

[0101] See also Figure 6 The method of the present invention is compared with the existing time-of-flight imaging method, encoding function and reconstruction algorithm on the data NYU-V2. The method achieves the best results in the MAE index in three noise levels.

[0102] like Figure 7 As shown, it is a comparison of the reconstruction results of the scenes in the NYUv2 dataset on different encoding functions. The upper image in the first column is GroundTruth (GT), the lower image is the corresponding RGB image, the upper images in the second to fifth columns are reconstructed depth maps (wherein, ours represents the method of the present invention, sine represents the modulation and demodulation function as a sine function method, squre represents the modulation and demodulation function as a square wave function method, and dual-frequency sine represents the modulation and demodulation function as a dual-frequency sine function method), and the lower image is an error map (showing the difference between the image reconstructed by the network and the high-resolution image). It can be seen that the method of the present invention can effectively eliminate the influence of noise and reconstruct a depth map close to the true depth value.

[0103] like Figure 8As shown, the reconstruction results of the scenes of the SUN RGB-D dataset and the 4D Light Field dataset on different encoding functions and networks are compared. The upper image in the first column is GroundTruth (gt), the lower image is the corresponding RGB image, the upper images in the second to fifth columns are reconstructed depth maps (wherein, ours represents the method of the present invention, hami+network represents the modulation and demodulation function is the Hamilton function and the reconstruction method is reconstructed by the network, squre+PS represents the modulation and demodulation function is the square wave function and the reconstruction method is reconstructed by the phase shift method, sine+PS represents the modulation and demodulation function is the sine function and the reconstruction method is reconstructed by the phase shift method), and the lower image is the error map. It can be seen that the method of the present invention can effectively eliminate the influence of noise and reconstruct a depth map close to the true depth value.

Claims

1. A large-scale noise-robust depth imaging method based on RGB guidance, characterized in that: The method comprises the following steps: Step 1: construct a database, including a training set and a test set, and contain color images and depth images of the same scene; Step 2, establishing a differentiable physical imaging model to simulate the time-of-flight imaging process, adding physical constraints of the actual scene, setting the encoding function to a learnable parameter, the encoding function including a light source modulation function and a sensor demodulation function; inputting the data in the training set into the differentiable physical imaging model to obtain multiple noise-free measurements; Step 3, constructing a visual TOF dual-branch multi-scale fusion depth reconstruction network, including a deep feature extraction network for extracting pixel-by-pixel multi-scale TOF depth information, a monocular depth generation network for extracting monocular depth information, and an edge feature extraction network for extracting edge structure information; Step 4, obtaining pixel-by-pixel TOF depth information through a depth feature extraction network, extracting monocular depth through a monocular depth generation network, and supervising and correcting the TOF depth through multi-scale fusion of the monocular depth to obtain a corrected depth map; The edge of high-resolution color images is extracted through the edge feature extraction network and optimized using edge structure loss; Step 5, fusing the corrected depth map with the edge of the high-resolution color image, and constructing an edge-assisted deep fusion network as an end-to-end network for iterative optimization; Step 6, using the color pseudo edge guided loss and the Fisher information guided loss as the loss function, and combining the multiple measurement difference loss function and the depth fidelity loss function, jointly training the encoding function and the fusion deep reconstruction network; In step 7, the test data is input into the trained encoding function and fused depth reconstruction network to reconstruct the depth map of the test scene; the encoding function and fused depth reconstruction network trained in step 6 are used to reconstruct a high-quality depth map of the actual scene.

2. The RGB-guided large-scale noise-robust depth imaging method according to claim 1, characterized in that: In the step 1, the database includes RGBD images of virtual synthesis and real-world scenes and fine edge images based on color pictures, and each sample in the training set and the test set includes the depth, surface reflectivity and ambient light intensity of the scene; when constructing the database, the RGB image of the scene image is subjected to intrinsic image decomposition to obtain a reflectivity map of the scene, the surface reflectivity of the scene is obtained by extracting the R channel of the reflectivity map, and the ambient light intensity is obtained by averaging the three RGB channels of the reflectivity map; The gradient and divergence operations are performed on the RGB image of the scene to obtain an edge map containing rich texture information.

3. The RGB-guided large-scale noise-robust depth imaging method according to claim 1, characterized in that: In step 2, the differentiable physical imaging model is established as follows: Assume that the emitted light is modulated by the function M i (t) modulation, the reflected signal of scene point p is: where α(p) is the amplitude coefficient due to scene reflection, β is the ambient component due to other light sources, is the distance-dependent time delay of light propagation, i is the number of modulation functions, and t is the time delay; The specific form of the reflected signal is: where z(p) is the depth of light propagation, c is the speed of light, and F falloff is the light attenuation coefficient; Using the demodulation function D i (t) Demodulate the reflected signal Ri(p,t) to derive the measured value Where T is the integration time.

4. The RGB-guided large-scale noise-robust depth imaging method according to claim 1, characterized in that: In the step 4, the pixel-by-pixel TOF depth information is obtained through the deep feature extraction network, specifically: first, the feature extraction block of the encoding module is used to extract multi-scale features to obtain features of different scales; the features of different scales are fused using the multi-scale feature fusion module and input to the decoding module; the decoding module uses the residual depth reconstruction module to convolute the upsampled depth map with the fused features output by the multi-scale feature fusion module, and generates a large-size depth map with more details in a coarse-to-fine manner.

5. The RGB-guided large-scale noise-robust depth imaging method according to claim 1, characterized in that: In step 4, the monocular depth is extracted through a monocular depth generation network, specifically: a RefineNet network is used to perform monocular extraction and learn a relative depth sequence contained in the RGB image.

6. The RGB-guided large-scale noise-robust depth imaging method according to claim 1, characterized in that: In the step 4, the edge of the high-resolution color image is extracted through the edge feature extraction network, specifically: the MSRN network is used to learn the mapping between the RGB image and the image edge, and the soft edge information of the RGB image is extracted.

7. The RGB-guided large-scale noise-robust depth imaging method according to claim 6, characterized in that: In step 4, the following curvature formula is used to calculate the curvature of the super-resolution RGB image I HR Get the corresponding soft edge I Edge : AND Edge =div(in x ,in y ) x and y represent the horizontal and vertical directions respectively, i represents selecting a direction from the x and y directions, u i represents the gradient component in the i direction, u x 、u y Respectively represent the gradient components in the horizontal or vertical direction when i is x or y, and div(·) denote the gradient and divergence operations, respectively.

8. The RGB-guided large-scale noise-robust depth imaging method according to claim 1, characterized in that: In step 5, the corrected depth map and the edge of the high-resolution color image are subjected to feature fusion by channel addition through an attention mechanism, and a reconstructed fine depth map is obtained through a CNN convolutional layer.

9. The RGB-guided large-scale noise-robust depth imaging method according to claim 1, characterized in that: In step 6, the Fisher information amount I is: Where z(p) is the depth of light propagation, s i (p) is the theoretical measurement value of time-of-flight imaging, X(p) is the multiple measurement value, σ i (p) is the standard deviation of the Gaussian distributed read noise.

10. A system for implementing the RGB-guided large-scale noise robust depth imaging method as claimed in claim 1, characterized in that: The system includes a laser diode, a function signal generator, a beam splitter, a scanning device, a light-collecting lens, an avalanche photodiode, a multiplier, an amplifier, a low-pass filter and a color camera. Specifically, a laser diode is used as an emitting light source; a learned encoding function is generated by a function signal generator, and a modulation function in the encoding function modulates the laser emitted by the laser diode; the laser emitted by the laser diode passes through a beam splitter and reaches a scanning device, and the scanning device scans the scene; the laser irradiates a point on the scene and is reflected along the original optical path; the reflected light is focused onto an avalanche photodiode through a light-collecting lens; the avalanche photodiode converts the reflected light signal into an electrical signal, and then multiplies the converted signal with the demodulation function in the encoding function through a multiplier, and the output voltage value is further amplified and low-pass filtered through an amplifier and a low-pass filter; finally, the amplified and low-pass filtered voltage value is sampled, quantized and converted into a measurement value through an analog-to-digital converter.

Images