ToF Depth Image Denoising Method Based on Expansion Modeling and Multimodal Fusion

Abstract

The present invention discloses a ToF depth image denoising method based on unfolded modeling and multimodal fusion. The steps include: 1. Obtain image data; 2. Construct two convolutional neural networks with non-shared parameters, Φ<subgt;A< / subgt;(·) and Φ<subgt;D< / subgt;(·); 3. Construct an initial reconstruction unit including a global feature fusion module and an initial residual regression module; 4. Construct a local update unit including a hidden layer and an output layer of a local feature fusion module. Under the loop iteration framework of ToF depth data denoising, the present invention can achieve a robust and accurate ToF depth image denoising effect by introducing multimodal data fusion technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of 3D image processing, and specifically relates to a ToF depth image denoising method based on unfolded modeling and multimodal fusion. Background Art

[0002] With the development of digital imaging technology, people have higher and higher requirements for images. Not only are high resolution, frame rate, and wide dynamic range required, but also distance information of the scene needs to be obtained. However, traditional two-dimensional imaging technology cannot obtain the distance information of the scene. Therefore, depth imaging technology has emerged. Depth imaging technology can capture the three-dimensional coordinate information of the measured scene in real time using a specific distance sensor and save it in the form of 3D point clouds and depth images.

[0003] Currently, the more widely used depth acquisition technologies mainly include binocular vision, structured light, time-of-flight method and other technical means. Binocular vision technology mimics the parallax imaging principle of human eyes. Two RGB cameras are used to obtain the left and right views of the scene at the same moment from different positions. Then, the parallax is obtained through feature matching, and the depth information is calculated according to the geometric relationship between the binocular cameras and the measured scene. Binocular vision technology has high requirements for hardware, consumes a large amount of computing resources, and is easily affected by factors such as changes in ambient light intensity and the complexity of the object surface texture in terms of measurement accuracy.

[0004] Structured light is a depth image acquisition system composed of a projector and a two-dimensional camera. The projector emits light encoded by a certain method and projects it onto the measured object, and then it is collected by the two-dimensional camera. The position and depth information of the object are calculated by calculating the distortion of the returned encoded pattern through a specific algorithm. It can be divided into fringe structured light, coded structured light, and speckle structured light according to different encoded patterns. This method has high measurement accuracy, but it is easily affected by ambient light, resulting in poor outdoor experience, short measurement distance, and the accuracy will deteriorate as the detection distance increases.

[0005] In recent years, with the rapid development of integrated circuit and semiconductor manufacturing processes, the ToF (Time-of-Flight) depth image acquisition technology has witnessed significant progress. This method obtains the depth information of the measured scene by recording the time of flight of light. The time-of-flight depth acquisition technology can be roughly divided into two categories: the scanning method and the non-scanning method. The representative product of the scanning method is lidar, whose working principle is to scan point by point, from point to line and then to surface, so as to obtain the depth information of objects within a spatial range. The advantage of this ToF depth acquisition technology based on the scanning method is that it can measure at a relatively long distance with high measurement accuracy, while the disadvantages are that it is too complex, has a large volume, and high production cost, and it is commonly found in various large equipment such as vehicles, engineering machinery, and factory assembly lines. The non-scanning method is used in various consumer-grade ToF depth sensors that can be frequently encountered in daily life, which are commonly found in small devices such as mobile phones, computers, and small household robots. The biggest difference from the scanning method is that it does not require point-by-point scanning and can calculate the scene depth by obtaining the reflected light signals of all pixel points in the scene at one time. Compared with other depth acquisition technologies such as monocular / stereo vision and structured light, the ToF depth acquisition technology based on the non-scanning method has the advantages of long detection distance, high imaging accuracy, small volume, and easy integration, and has been widely applied to fields such as face recognition, human-computer interaction, 3D scanning, and autonomous driving. This is also the research focus of this article. Unless otherwise specified, the ToF depth acquisition technology in the following text refers to the non-scanning method.

[0006] The ToF depth image acquisition technology will be affected by various noises during the imaging process, such as Flying Pixel (FP), Multi-Path Interference (MPI), and shot noise, resulting in serious deviations in the obtained depth. The flying pixel noise originates from the aliasing of the modulated light signals reflected by foreground and background objects in the receiver, and its visual manifestation is the blurring and thickening of the object edge region in the depth image. Shot noise is a random noise during the ToF depth image imaging process, and its visual manifestation is generally salt-and-pepper noise distributed in a scattered dot pattern. Multi-path interference means that the reflected modulated light signal corresponding to any pixel point in the ToF depth image not only comes from the direct light path representing the true time of flight but also includes multiple non-direct reflection light paths. In the ToF depth image, multi-path interference is manifested as regionally distributed depth deviations and depth discontinuities.

[0007] It should be emphasized that active depth sensors, including structured light and time-of-flight ranging sensors, will have flying pixel noise and shot noise, and their distribution characteristics are relatively regular, and the removal of noise is relatively simple. In comparison, the generation process of multipath interference noise involves the modeling problem of the light transmission process in 3D space, and the removal difficulty is higher. And due to the existence of multipath interference, the current single-image ToF depth image denoising algorithms all have a high computational complexity and a slow processing speed. A typical representative is the residual pyramid network of the classic method SHARP-Net in ToF denoising. SHARP-Net believes that the spatial distribution of multipath interference noise is related to the geometric structure. This phenomenon inspired the author to use a residual pyramid to model the light transmission process of multipath interference noise at the spatial level, and fuse the prediction residuals of multipath interference noise at multiple scales, so as to achieve the purpose of removing multipath interference noise. However, this ToF depth image denoising algorithm based on spatial hierarchical decomposition still has certain problems.

[0008] First, in the early scales of the residual pyramid, if the denoising result of the ToF depth image has a large deviation compared with the real depth image, its prediction error will be gradually transmitted along the upsampling direction to the finer-grained and larger-scale multipath interference prediction residuals, which can be called the early error accumulation problem of the residual pyramid.

[0009] Second, existing ToF depth image denoising algorithms, including SHARP-Net, still use convolutional operations to model the light transmission process of multipath interference noise, and represent the generation process of multipath interference noise by aggregating adjacent depth pixels. However, according to the analysis of the mathematical model of multipath interference, this light transmission process modeling method tends to directly predict the multipath interference noise here by "memorizing" the scene geometric structure, with poor performance and unreliability. Summary of the Invention

[0010] The present invention is to solve the above-mentioned deficiencies of the existing technologies, and proposes a ToF depth image denoising method based on expansion modeling and multi-modal fusion, in order to introduce multi-modal data fusion technology into the cyclic iterative framework of ToF depth data denoising, so as to achieve a robust and accurate ToF depth image denoising effect.

[0011] In order to achieve the above invention purpose, the present invention adopts the following technical solutions:

[0012] The characteristics of a ToF depth image denoising method based on expansion modeling and multi-modal fusion of the present invention are as follows: it is carried out according to the following steps:

[0013] Step 1: Obtain a ToF depth image and its corresponding true depth image, as well as an amplitude image corresponding to the true depth image, and preprocess them to obtain a preprocessed ToF depth image D and a preprocessed true depth image D. GT and the preprocessed amplitude image A;

[0014] Step 2: Construct two convolutional neural networks Φ with no shared parameters D (·) and Φ A (·); and used to obtain the global features of the ToF depth image D and the global features of the amplitude image A

[0015] Step 3: Construct a global feature fusion module The initial reconstruction unit of the initial residual regression module is used to obtain the initial depth residual R;

[0016] Step 4: Construct a local feature fusion module The local update unit of the hidden layer and the output layer is used to obtain the denoised ToF depth image D * ;

[0017] Step 5: Convolutional neural network Φ D (·) and Φ A (·), the initial reconstruction unit and the local update unit constitute the ToF depth image denoising network and perform parameter updates;

[0018] Step 5.1: Use formula (12) to construct the loss function L of the ToF depth image denoising network seq :

[0019]

[0020] In formula (1), L k represents the loss function of the k-th cycle update, and is obtained from formula (2):

[0021]

[0022] In formula (2), Indicates the use of the Sobel operator for gradient detection, ||·|| indicates the calculation of the norm modulo 1; D k (m,n) represents the predicted depth map D updated in the kth cycle k The depth value at the pixel coordinate (m, n); D GT (m,n) represents the real depth image D after preprocessing GT The depth value at the pixel coordinate (m, n); M represents the length of the image, and N represents the width of the image;

[0023] Step 5.2: Based on the ToF depth image, amplitude image and true depth map, the Adam gradient descent optimization algorithm is used to train the ToF depth image denoising network and calculate the loss function L seq Used to update network parameters in reverse until the loss function L seq Until convergence, the trained ToF depth image denoising model is obtained.

[0024] The ToF depth image denoising method based on expansion modeling and multimodal fusion of the present invention is also characterized in that: the convolutional neural network Φ in step 2 D (·) By local feature extraction module and global feature extraction module Composition, convolutional neural network Φ A (·) By local feature extraction module and global feature extraction module constitute;

[0025] Step 2.1: Local feature extraction module Extract local features of the ToF depth image D to obtain local features of the ToF depth image D

[0026] The local feature extraction module Extract local features of the amplitude image A to obtain local features of the amplitude image A

[0027] Step 2.2: Global feature extraction module Extract the global features of the ToF depth image D to obtain the global features of the ToF depth image D

[0028] The global feature extraction module Extract the global features of the amplitude image A to obtain the global features of the amplitude image A

[0029] The step 3 comprises:

[0030] Step 3.1: The global feature fusion module It consists of multiple scale residual channel attention layers, which take the global features of the ToF depth image D into account. and the global features of the amplitude image A After concatenation in the channel dimension, it is input into the global feature fusion module Fusion at different scales is performed to obtain the multi-scale global fusion feature F G ;

[0031] Step 3.2. The initial residual regression module consists of two convolutional layers with a kernel size of 1×1, and is used to reduce the channel dimension of the global fusion feature F G . After that, a Sigmoid function σ is used to activate the global fusion feature with reduced dimension, so as to obtain the initial depth residual R

[0032] The local feature fusion module in the above Step 4 consists of 1 residual channel attention layer and 1 convolutional layer with a kernel size of 3×3, and obtains the ToF depth image D according to the following process * :

[0033] Step 4.1. Define the current loop update count as k, and initialize k = 1;

[0034] Step 4.2. Calculate the predicted depth map D k = D k-1 + R k ; where, R k represents the depth residual of the k-th loop update. When k = 1, let R k = R; let D k-1 = D;

[0035] Step 4.3. Input the predicted depth map D k of the k-th loop update into the local feature extraction module for processing to obtain the local feature of the predicted depth map D k of the k-th loop update

[0036] Step 4.4. Input the local feature of the predicted depth image D k of the k-th loop update and the local feature of the amplitude image A into the local feature fusion module . After passing through the local feature fusion processing of the residual channel attention layer and the local feature aggregation processing in the spatial dimension by the 3×3 convolutional layer in sequence, the local fusion feature of the k-th loop update is obtained

[0037] Step 4.5. Use Equation (3) to construct the input variable x in the k-th loop update[[ID=5_3]] k :

[0038]

[0039] In Equation (3), Concat represents the concatenation operation in the channel dimension;

[0040] Step 4.6. The hidden layer uses Equations (4) and (5) to obtain two coefficients \(z\) and \(r\) for updating the neuron states in the hidden layer at the \(k\)-th iteration: k and \(r\) k :

[0041] \(z\) k =\(\sigma(\text{Conv}\) z \((\text{Concat}(h\) k-1 , x\) k ))) (4)

[0042] \(r\) k =\(\sigma(\text{Conv}\) r \((\text{Concat}(h\) k-1 , x\) k ))) (5)

[0043] In Equations (4) and (5), \(\sigma(\cdot)\) is the Sigmoid activation function, \(h_{k - 1}\) represents the neuron states in the hidden layer at the \((k - 1)\)-th iteration; \(\text{Conv}(\cdot)\) and \(\text{Conv}(\cdot)\) respectively represent two convolutional layers with a \(1\times1\) convolutional kernel in the hidden layer. 1 Step 4.7. The hidden layer uses Equation (6) to map the input variable \(x\) into the variable space of the hidden layer, thereby obtaining the updated variable of the input variable \(x\) in the hidden layer at the \(k\)-th iteration. z \((\cdot)\) and \(\text{Conv}\) r \((\cdot)\) respectively represent two convolutional layers with a \(1\times1\) convolutional kernel in the hidden layer.

[0044] In Equation (6), \(\tanh(\cdot)\) is the hyperbolic tangent activation function, \(\text{Conv}(\cdot)\) is a convolutional layer with a \(1\times1\) convolutional kernel in the hidden layer, and \(\odot\) represents the Hadamard product. k Step 4.8. The hidden layer uses Equation (7) to obtain the neuron states \(h\) in the hidden layer corresponding to the input variable \(x\) at the \(k\)-th iteration: k In the \(k\)-th iteration update of the hidden layer.

[0045]

[0046] In Equation (6), \(\tanh(\cdot)\) is the hyperbolic tangent activation function, \(\text{Conv}(\cdot)\) is a convolutional layer with a \(1\times1\) convolutional kernel in the hidden layer, and \(\odot\) represents the Hadamard product. h \((\cdot)\) is a convolutional layer with a \(1\times1\) convolutional kernel in the hidden layer, and \(\odot\) represents the Hadamard product.

[0047] Step 4.9. The output layer uses Equation (8) to calculate the predicted depth residual \(R\) output by the output layer for the input variable \(x\) at the \(k\)-th iteration: k corresponding to the input variable \(x\) in the hidden layer at the \(k\)-th iteration. k :

[0048]

[0049] Step 4.10. The output layer uses Equation (9) to calculate the predicted depth residual \(R\) output by the output layer for the input variable \(x\) at the \(k\)-th iteration: k in the output layer for the input variable \(x\) at the \(k\)-th iteration. k :

[0050] \(R\) k =\(\sigma(\text{Conv}\) L1(Conv L2 (h k ))) (8)

[0051] In Equation (8), Conv L1 and Conv L2 are respectively convolutional layers with a 1×1 convolutional kernel in the output layer;

[0052] Step 4.10: After assigning k + 1 to k, return to Step 4.2 and execute sequentially until k > K, thereby obtaining the denoised ToF depth image D * ; where K represents the maximum number of cyclic updates.

[0053] An electronic device according to the present invention includes a memory and a processor, characterized in that the memory is used to store a program that supports the processor to execute the ToF depth image denoising method, and the processor is configured to execute the program stored in the memory.

[0054] A computer-readable storage medium according to the present invention, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, it executes the steps of any of the ToF depth image denoising methods.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0056] 1. The present invention proposes a cyclic update architecture that iteratively optimizes the ToF depth residual at a single scale. Different from the existing residual pyramid architecture based on multi-scale residual prediction, the cyclic update architecture does not have the problem of early error accumulation, ensuring the robustness of the method.

[0057] 2. The present invention proposes a multi-modal data fusion technology for local-global joint of ToF amplitude image and depth image, which fuses the ToF amplitude image and depth image at local and global scales respectively. The global scale ensures the reliability of the initialization of the ToF depth residual, and the local scale ensures the introduction of new information in the cyclic iteration. This technology strengthens the feature expression ability of the network model for ToF noise in the feature extraction stage and improves the denoising performance of the method.

[0058] 3. The present invention proposes an explicit solution technology for multi-path interference based on a cyclic gating unit. By using the superposition property of the ToF echo signal along the indirect reflection optical path, a cyclic gating unit is introduced to define the accumulation generation process of multi-path interference as the iterative update of hidden variables, thereby improving the accuracy and generalization of the ToF depth image denoising method. Description of the Drawings

[0059] Figure 1This is the structural diagram of the recurrent neural network based on expansion modeling and multimodal fusion in the present invention. Among them, there are two sub-modules: a global reconstruction unit and a local update unit;

[0060] Figure 2 This is the structural diagram of the local encoder of the present invention;

[0061] Figure 3 This is the structural diagram of the global encoder of the present invention;

[0062] Figure 4 This is the structural diagram of the residual channel attention block (RCAL) of the present invention

[0063] Figure 5 This is the structural diagram of the global reconstruction unit of the present invention;

[0064] Figure 6 This is the structural diagram of the local update unit of the present invention;

[0065] Figure 7 This is the visual comparison effect diagram of the present invention in the test set. Detailed implementation manners

[0066] In this embodiment, a ToF depth image denoising method based on expansion modeling and multimodal fusion is as Figure 1 shown and is carried out according to the following steps:

[0067] Step 1: Obtain the ToF amplitude image and its corresponding ToF depth image, as well as the true depth image corresponding to the ToF depth image, and perform preprocessing to obtain the preprocessed amplitude image A, the preprocessed ToF depth image D, and the preprocessed true depth image D GT ;

[0068] The preprocessing is mainly divided into three parts, including normalization, filtering, and cropping.

[0069] First is normalization. Set the maximum depth value D max , and divide the depth map and the true depth map by D max so as to normalize the depth map and the true depth map between 0 and 1. In this example, D max is set to 4.096 m. Set the maximum amplitude value A max , and use the same operation to normalize the amplitude map between 0 and 1. In this example, A max is set to 255.

[0070] Secondly is filtering. To better mine the information of the amplitude map and the depth map, in this embodiment, bilateral filtering is applied to the amplitude map to remove noise, and minimum-maximum filtering is applied to the depth map to remove the pixel points below and beyond the depth range.

[0071] Finally, there is cropping. In this embodiment, the input ToF depth image, amplitude image, and real depth image are all cropped to a size of 384×512, which is convenient for applying the residual pyramid to solve the ToF denoising problem.

[0072] Step 2: Construct two convolutional neural networks Φ D (·) and Φ A (·); among them, the convolutional neural network Φ D (·) is composed of a local feature extraction module and a global feature extraction module , and the convolutional neural network Φ A (·) is composed of a local feature extraction module and a global feature extraction module ;

[0073] Step 2.1: The local feature extraction module is sequentially connected by three convolutional layers, three ReLU activation functions, three residual channel attention blocks, and two pooling layers, as Figure 2 shown, to extract local features of the ToF depth image D and obtain the local features

[0074] of the ToF depth image D. The local feature extraction module has the same structure as the local feature extraction module Figure 2 shown, the difference is that it extracts local features of the amplitude image A to obtain the local features of the amplitude image A and has different learnable parameters from the local feature extraction module ;

[0075] Step 2.2: The global feature extraction module is sequentially connected by three convolutional layers, three ReLU activation functions, three residual channel attention blocks, and three pooling layers, as Figure 3 shown. Its input is the output F of the local feature extraction module D3 , so as to complete the extraction of global features of the ToF depth image D and obtain the global features

[0076] of the ToF depth image D. The structure of the global feature extraction module is the same as that of the global feature extraction module Figure 3 shown. Its input is the output F of the local feature extraction module A3, thus completing the extraction of the global features of the amplitude image A and obtaining the global features of the amplitude image A

[0077] The structures of the residual channel attention blocks (RCAL) in Steps 2.1 and 2.2 are as Figure 4 shown. First, two 3×3 convolutional layers and a ReLU activation function are used to aggregate the input feature F. Then, a global pooling operation is used to reduce the spatial scale of the aggregated feature, reducing the scale from 384×512 to 1×1. After that, two 1×1 convolutional layers, a ReLU activation function, and a Sigmoid activation function are used to calculate the channel attention. Finally, the channel attention is used to weight the aggregated feature, and the weighted result is added to the input feature as a residual to obtain the final result F^' of this module.

[0078] Step 3. Construct an initial reconstruction unit that includes a global feature fusion module and an initial residual regression module;

[0079] Step 3.1. The global feature fusion module consists of residual channel attention layers (RCAL) of multiple scales, as Figure 5 shown. After connecting the global features of the ToF depth image D and the global features of the amplitude image A in the channel dimension and inputting them into the global feature fusion module , fusion at different scales is performed to obtain the multi-scale global fusion feature F G ;

[0080] Step 3.2. The initial residual regression module (IRRM) consists of two convolutional layers with a kernel size of 1×1 and is used to reduce the dimensionality of the channel dimension of the global fusion feature F G , and then a Sigmoid function σ is used to activate the dimension-reduced global fusion feature to obtain the initial depth residual R;

[0081] Step 4. Construct a local update unit that includes a local feature fusion module hidden layer and an output layer; among them, the local feature fusion module consists of 1 residual channel attention layer and 1 convolutional layer with a kernel size of 3×3, as Figure 6 shown;

[0082] Step 4.1. Define the current loop update count as k and initialize k = 1;

[0083] Step 4.2. Calculate the predicted depth map D k = D k-1 + Rk ; wherein, R k represents the depth residual updated in the k-th loop. When k = 1, let R k = R; let D k-1 = D;

[0084] Step 4.3: Input the predicted depth map D k updated in the k-th loop into the local feature extraction module for processing to obtain the local features of the predicted depth map D k updated in the k-th loop

[0085] Step 4.4: Input the local features k of the predicted depth image D updated in the k-th loop and the local features of the amplitude image A into the local feature fusion module . After sequentially undergoing local feature fusion processing through the residual channel attention layer and local feature aggregation processing in the spatial dimension by a 3×3 convolutional layer, the locally fused features

[0086] updated in the k-th loop are obtained k :

[0087]

[0088] In Equation (1), Concat represents the concatenation operation in the channel dimension;

[0089] Step 4.6: The hidden layer uses Equations (2) and (3) to respectively obtain two coefficients z k and r k of the neuron state in the hidden layer updated in the k-th loop:

[0090] z k = σ(Conv z (Concat(h k-1 , x k ))) (2)

[0091] r k = σ(Conv r (Concat(h k-1 , x k ))) (3)

[0092] In Equations (2) and (3), σ(·) is the Sigmoid activation function, h k-1 represents the neuron state of the hidden layer updated in the (k - 1)-th loop; Conv z(·) and Conv r (·) respectively represent two convolutional layers with 1×1 convolutional kernels in the hidden layer;

[0093] Step 4.7: The hidden layer uses Equation (4) to map the input variable x k into the variable space of the hidden layer, thereby obtaining the input variable x k in the updated variable of the hidden layer in the k-th loop update

[0094]

[0095] In Equation (4), tanh(·) is the hyperbolic tangent activation function, and Conv h (·) is the convolutional layer with a 1×1 convolutional kernel in the hidden layer, and ⊙ represents the Hadamard product;

[0096] Step 4.8: The hidden layer uses Equation (5) to obtain the state h of the hidden layer neuron corresponding to the input variable x k in the k-th loop update: k :

[0097]

[0098] Step 4.9: The output layer uses Equation (6) to calculate the predicted depth residual R k output by the output layer for the input variable x k :

[0099] R k = σ(Conv L1 (Conv L2 (h k ))) (6)

[0100] In Equation (6), Conv L1 and Conv L2 are respectively convolutional layers with 1×1 convolutional kernels in the output layer;

[0101] Step 4.10: After assigning k + 1 to k, return to Step 4.2 and execute sequentially until k > K, thereby obtaining the denoised ToF depth image D * ; where K represents the maximum number of loop updates;

[0102] Step 5: The ToF depth image denoising network is composed of the convolutional neural network Φ D (·) and Φ A (·), the initial reconstruction unit, and the local update unit, and the parameters are updated;

[0103] Step 5.1: Use Equation (7) to construct the loss function L of the ToF depth image denoising networkseq :

[0104]

[0105] In Equation (7), L k represents the loss function updated in the k-th iteration, and is obtained by Equation (8):

[0106]

[0107] In Equation (8), represents gradient detection using the Sobel operator, and ||·|| represents calculating the norm with a modulus of 1; D k (m,n) represents the predicted depth map D updated in the k-th iteration k at the depth value of the pixel coordinates (m, n); D GT (m,n) represents the true depth image D after preprocessing GT at the depth value of the pixel coordinates (m, n); M represents the length of the image, and N represents the width of the image. In this embodiment, M = 384 and N = 512.

[0108] Step 5.2: Based on the ToF depth image, amplitude image, and true depth map, use the Adam gradient descent optimization algorithm to train the ToF depth image denoising network, and calculate the loss function L seq for backpropagating to update the network parameters until the loss function L seq converges, thereby obtaining the trained ToF depth image denoising model.

[0109] To verify the effectiveness of the present invention, visual metrics and numerical metrics were tested on two publicly available synthetic training datasets, ToF-FlyingThings3D (TFT3D) and Cornell-Box dataset, and one publicly available real dataset, True Box dataset, as Figure 7 shown in Table 1.

[0110] Table 1 Comparative performance results of the ToF depth data denoising method of the present invention on multiple datasets

[0111]

[0112] ]>

[0113]

[0114] In Table 1 and Figure 7In this embodiment, comparisons with existing methods are mainly made on three datasets. Among them, both TFT3D and Cornell-Box datasets are large synthetic datasets, and True Box is a small real dataset. The comparison methods are mostly ToF denoising methods newly proposed in conference papers and journal papers in the past two years, namely the time-of-flight imaging algorithm based on deep learning (DeepToF), the time-of-flight denoising algorithm based on a coarse-to-fine prediction strategy (CFN), the time-of-flight denoising algorithm based on a kernel prediction network (ToF-KPN), the time-of-flight denoising algorithm based on spatial hierarchical deconstruction (SHARP-Net), and the time-of-flight denoising algorithm based on ray alignment update (RADU). Among them, Ours represents the method of the present invention based on unfolded modeling, and the metric used is the mean absolute error (MAE). It can be seen from the data in Table 1 that the method of the present invention obtains the lowest MAE, further reducing the noise in the ToF depth image. Combining Figure 7 From the visualization of the error map after denoising in

Claims

1. A ToF depth image denoising method based on expansion modeling and multi-modal fusion, characterized in that, It is carried out according to the following steps: Step 1: Obtain the ToF depth image, its corresponding ground truth depth image, and the amplitude image corresponding to the ground truth depth image, and perform preprocessing to obtain the preprocessed ToF depth image , the preprocessed ground truth depth image and the preprocessed amplitude image ; Step 2: Construct two convolutional neural networks with non-shared parameters and ; and are used to obtain the global features of the ToF depth image and the global features of the amplitude image ; ;​ Step 3: Construct a global feature fusion module and an initial reconstruction unit of the initial residual regression module, and use it to obtain the initial depth residual ; Step 3.1, the global feature fusion module is composed of residual channel attention layers of multiple scales, and the global features of the ToF depth image and the global features of the amplitude image are concatenated in the channel dimension and then input into the global feature fusion module for fusion at different scales to obtain multi-scale global fusion features ; Step 3.

2. The initial residual regression module consists of two convolutional layers with a convolutional kernel of and is used to reduce the channel dimension of the global fusion feature . After that, a Sigmoid function is used to activate the globally fused feature after dimensionality reduction, thereby obtaining the initial depth residual ; Step 4: Construct a local update unit that includes a local feature fusion module , a hidden layer, and an output layer, and is used to obtain a denoised ToF depth image ; Step 5: A ToF depth image denoising network is constructed by a convolutional neural network and , an initial reconstruction unit, and a local update unit, and the parameters are updated; Step 5.

1. Construct the loss function of the ToF depth image denoising network using Equation (1). : (1) In formula (1), K represents the maximum number of cyclic updates, represents the loss function of the th cyclic update and is obtained from formula (2); (2) In formula (2), represents gradient detection using the Sobel operator, represents calculating the norm with a modulus of 1; represents the predicted depth map updated in the th cycle, at the depth value at the pixel coordinates (m, n); represents the true depth image after preprocessing at the depth value at the pixel coordinates (m, n); M represents the length of the image, and N represents the width of the image; Step 5.2: Based on the ToF depth image, amplitude image, and real depth map, use the Adam gradient descent optimization algorithm to train the ToF depth image denoising network, and calculate the loss function for backpropagating to update the network parameters until the loss function converges, thereby obtaining the trained ToF depth image denoising model.

2. The ToF depth image denoising method based on expansion modeling and multimodal fusion according to claim 1, characterized in that: The convolutional neural network in step 2 consists of a local feature extraction module and a global feature extraction module and is composed of a local feature extraction module and a global feature extraction module and is formed; Step 2.1, the local feature extraction module extracts local features from the ToF depth image to obtain the local features of the ToF depth image ; ; The local feature extraction module extracts local features from the amplitude image to obtain the local features of the amplitude image ; ; Step 2.2, the global feature extraction module extracts the global features from the ToF depth image to obtain the global features of the ToF depth image ; ; The global feature extraction module extracts global features from the amplitude image to obtain the global features of the amplitude image . .

3. The ToF depth image denoising method based on expansion modeling and multimodal fusion according to claim 2, wherein, The local feature fusion module in step 4 consists of 1 residual channel attention layer and 1 convolutional layer with a convolutional kernel of and obtains the ToF depth image according to the following process : Step 4.1: Define the current loop update count as k and initialize k = 1; Step 4.2: Calculate the predicted depth map updated in the k-th iteration ; where represents the depth residual updated in the k-th iteration. When k = 1, let ; Let ; Step 4.3: Input the predicted depth map updated in the k-th loop into the local feature extraction module for processing to obtain the local features of the predicted depth map updated in the k-th loop ; ; Step 4.4: The local features of the predicted depth image updated in the k-th loop are input, together with the local features of the amplitude image , into the local feature fusion module . In the module, through the local feature fusion processing of the residual channel attention layer and the local feature aggregation processing in the spatial dimension by the convolutional layer , the locally fused features updated in the k-th loop are obtained; Step 4.5, construct the input variable in the th loop update using Equation (3) : (3) In formula (3), represents the concatenation operation in the channel dimension; Step 4.6: The hidden layer respectively obtains two coefficients for updating the neuron states in the hidden layer in the k-th cycle by using Equation (4) and Equation (5). and : (4) (5) In Equations (4) and (5), is the Sigmoid activation function, represents the state of the neurons in the hidden layer updated in the (k - 1)-th iteration; and respectively represent two convolutional layers in the hidden layer with convolutional kernels of ; Step 4.

7. The hidden layer uses Equation (6) to map the input variable into the variable space of the hidden layer, thereby obtaining the input variable and the updated variable of the hidden layer in the k-th loop update : (6) In Equation (6), is the hyperbolic tangent activation function, is the convolutional layer with a convolutional kernel of in the hidden layer, represents the Hadamard product; Step 4.

8. The hidden layer uses Equation (7) to obtain the input variable corresponding to the hidden layer neuron state in the (7) Step 4.

9. The output layer calculates the input variable in the predicted depth residual output by the output layer in the th loop update using Equation (8): (8) In formula (8), and are respectively convolutional layers with convolutional kernels of in the output layer; Step 4.10: After assigning k + 1 to k, return to Step 4.2 and execute sequentially until k > to obtain the denoised ToF depth image. Here, K represents the maximum number of loop updates.

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor to execute any one of the ToF depth image denoising methods recited in claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it executes the steps of any one of the ToF depth image denoising methods recited in claims 1-3.

Images