Transparent object depth completion method based on background guidance and frequency domain enhancement

Abstract

The invention discloses a transparent object depth completion method based on background guidance and frequency domain enhancement, and belongs to the field of computer vision. According to the method, an end-to-end background guided deep completion network BGD-Net is constructed; according to the network, an original RGB image, a restored background image, an original depth image and a transparent object mask are used as input, depth noise is purified through a gating depth feature extractor, background information is fused through a multi-scale frequency domain feature enhancer, and high-frequency edge details are enhanced. And finally, outputting a high-precision depth map through a multi-scale feature fusion and refinement module. According to the method, the repaired background is obviously utilized as guidance, visual confusion of a foreground transparent object and the background is effectively decoupled, and the boundary definition is obviously improved in combination with frequency domain processing. Experiments on a public data set show that the method reaches an advanced level in the aspects of depth completion precision and robustness of the transparent object, can be realized only by a common RGB-D camera, and is suitable for scenes such as robot grabbing, AR / VR and the like.

Description

A method for depth completion of transparent objects based on background guidance and frequency domain enhancement Technical Field

[0001] This invention belongs to the field of computer vision and deep learning, specifically relating to a method for depth completion of transparent objects, and more particularly to a depth completion network that utilizes a restored background image as a guiding signal, combined with frequency domain feature enhancement and multi-scale fusion. This method is applicable to scenarios requiring precise perception of the three-dimensional geometry of transparent objects, such as robot grasping, augmented reality (AR), virtual reality (VR), and scene reconstruction. Background Technology

[0002] Transparent objects, such as glassware and plastic packaging, are ubiquitous in daily life and industrial applications. In robotics, accurately perceiving the three-dimensional position and geometry of transparent objects is a crucial foundation for reliable grasping, manipulation, and interaction. Simultaneously, in augmented reality (AR) and virtual reality (VR) applications, achieving natural visual fusion, occlusion, and interaction between virtual content and glass surfaces in the real environment (such as desktops and shop windows) also relies on accurate perception of the geometry of transparent object surfaces. However, this critical requirement faces fundamental bottlenecks in current mainstream 3D perception technologies.

[0003] Widely used RGB-D depth sensors operate based on the Lambertian surface assumption, which requires light to be uniformly scattered back to the sensor. Transparent objects, due to refraction, completely violate this assumption: incident light penetrates the object and is deflected, causing the signal received by the sensor to originate from the background or a cluttered path. This results in the calculated "depth" value not being the actual surface location of the object, but often being background depth or meaningless noise. Consequently, transparent object regions in the original depth map often appear as large holes, incorrectly filled with background depth, or filled with outliers, severely compromising the reliability of 3D geometric information and directly causing the failure of downstream tasks such as robot grasping and scene reconstruction. Developing depth completion algorithms capable of recovering accurate geometry from such incomplete and noisy data has become a key issue in promoting the implementation of related applications.

[0004] Depth completion methods based on single-view RGB-D have become the most practical technical approach to solving this problem due to their strong hardware universality. However, this task faces three intertwined core challenges: First, weak RGB information in transparent areas. The object surface lacks inherent texture, and RGB pixels are actually background blending under complex refraction. This "visual camouflage" weakens key shape and contour cues. Second, severe misalignment between depth and geometric boundaries. Depth errors or omissions caused by refraction result in a complete spatial misalignment between the depth map and the clear object contours in the RGB image, making simple early feature fusion strategies prone to failure. Third, blurred edges in depth prediction. Existing methods are insufficient in reconstructing fine object boundaries, resulting in blurred and distorted prediction results at edges, directly affecting the accuracy of downstream tasks.

[0005] To address these challenges, researchers have primarily followed two technical approaches, both of which have limitations. One approach is learning-based methods (such as ClearGrasp and HDCNet). While these methods continuously improve their network architecture, they generally suffer from over-reliance on geometric priors, lack of explicit modeling of background interference in RGB features, and insufficient boundary refinement. The other approach is based on multimodal hardware (such as polarization and infrared cameras). These methods approach the problem from a physical perception level, providing additional, material-related physical information, and can more directly address the challenges of transparent objects under certain controlled conditions. However, they suffer from inherent drawbacks such as high hardware costs, system complexity, and poor environmental adaptability, hindering widespread adoption.

[0006] Therefore, exploring a depth completion algorithm that relies solely on a general-purpose RGB-D sensor but can explicitly eliminate transparent interference and achieve accurate boundary restoration is of significant research value. Summary of the Invention

[0007] This invention aims to overcome the shortcomings of existing technologies and provide a method for depth completion of transparent objects based on background guidance and frequency domain enhancement. This method introduces the repaired background image as key guiding information and combines innovative gated depth feature extraction, multi-scale frequency domain feature enhancement, and multi-scale feature fusion techniques to systematically solve the core difficulties in depth completion of transparent objects.

[0008] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0009] A method for depth completion of transparent objects based on background guidance and frequency domain enhancement, characterized by the following steps:

[0010] Step 1: Obtain Input Data: Obtain a scene dataset containing transparent objects, and edit the original RGB image using a large image editing model to obtain the repaired background image. The input data includes the original RGB image. The restored background image generated by image editing of a large model. Original depth map containing noise And the binary mask of transparent objects ;

[0011] Step 2: Construct the background-guided deep completion network BGD-Net, which includes a gated deep feature extractor GDFE, a multi-scale frequency domain feature enhancer MSFFE, and a multi-scale feature fusion and refinement module connected in sequence.

[0012] (2.1) The gated depth feature extractor receives the... and The concatenated data, combined with intermediate features obtained from the previous layer of the encoder. Feature purification is performed through a dual-path gating attention mechanism, resulting in clean deep features. ;

[0013] (2.2) The multi-scale frequency domain feature enhancer receives the output from the gated depth feature extractor. The fused features of the current layer are obtained through convolutional layers. Input at the same time and Through collaborative processing of spatial context and frequency domain detail paths, the final output is enhanced features. ;

[0014] (2.3) The multi-scale feature fusion and refinement module aggregates feature maps from S different levels of the decoder backbone network. After upsampling and concatenating the feature maps from the S different levels, a multilayer perceptron (MLP) is used as the final refinement head to learn to nonlinearly integrate all multi-scale feature information and introduce residual connections from coarse-grained depth prediction to finally output a refined depth map. .

[0015] In step 2 (2.1), the dual-path gating attention mechanism specifically includes:

[0016] (2.1.1) Feature stitching and geometric perception extraction: The stitched original depth map is combined with the binary mask of the transparent object [ , Input a 3×3 convolutional layer to extract geometrically perceptual features. Then, combine these features with intermediate features. By splicing the images again, enhanced features are obtained. ;

[0017] (2.1.2) Spatial attention filtering: Selecting features... Spatial attention weights are generated by sequentially passing the pooling layer, convolutional layer, and sigmoid activation function. These spatial attention weights are then combined with... Element-wise multiplication is used to filter the feature space locations, and then a lightweight multilayer perceptron (MLP) is used to obtain the features. ;

[0018] (2.1.3) Channel attention recalibration: intermediate features After 1×1 convolution dimensionality reduction, and spliced ​​[ , A lightweight MLP is used as the common input to generate channel attention weights. These channel attention weights are then combined with the features. Element-wise multiplication is used to recalibrate the feature channels. Finally, the features filtered by spatial and channel attention are input into a multi-scale dual-attention module to obtain the output features. ;

[0019] In step 2 (2.2), the processing procedures for the spatial context path and the frequency domain detail path specifically include:

[0020] (2.2.1) Spatial context feature extraction: and After concatenation, the data is input into a lightweight MLP to obtain image features. At the same time Preliminary image feature weights are generated through a 3×3 convolutional layer, and... After multiplication, the data is input into a multi-scale dual-attention module to obtain spatial context features. ;

[0021] (2.2.2) Frequency domain detail enhancement: enhancement of spatial context features Features fused with the current layer Two-dimensional discrete wavelet transforms are performed separately to decompose the data into low-frequency sub-bands and multiple high-frequency sub-bands. After concatenating all sub-bands, frequency domain feature transformation and enhancement are performed on each sub-band through independent convolutional layers. Finally, the processed sub-bands are reconstructed into the spatial domain using inverse discrete wavelet transforms to obtain the frequency domain enhanced features. ;

[0022] (2.2.3) Feature fusion and secondary frequency domain enhancement: Integrating spatial context features With and from the restored background image The extracted background features are added together. The added features are then summed. The enhanced features are then processed again through the frequency domain detail enhancement step described in (2.2.2) above, and the final output is the enhanced feature. ;

[0023] The specific process of the multi-scale dual attention module is as follows:

[0024] (1) Let the features of the input multi-scale dual attention module be: , First, the features are fed into four parallel convolutional layers with kernel sizes of 1×1, 3×3, 5×5, and 7×5 to capture multi-scale contextual information. The output features of the four parallel convolutional layers are concatenated, and then multi-scale contextual space attention weights are generated using an MLP and a sigmoid activation function. These multi-scale contextual space attention weights are then combined with… Multiply to obtain the characteristics ;

[0025] (2) At the same time, The average pooling layer, convolutional layer, and sigmoid activation function are input sequentially to generate multi-scale contextual channel attention weights. These multi-scale contextual channel attention weights are then combined with... Multiply, and then multiply the result with... The summation ultimately yields the output features. ;

[0026] Step 3: Input the input data into BGD-Net and train the network;

[0027] The background-guided depth completion network is trained using a composite loss function, which is applied to binary masks of transparent objects. Internal calculation, its expression is: ;

[0028] in, To predict depth maps With true depth Between loss; For the reason The calculated normal map and the result of Calculate the cosine distance loss between normal maps; To balance the weights.

[0029] Step 4: Depth Completion. Input the scene data to be processed into the trained BGD-Net network. After forward propagation, the network directly outputs the completed depth map.

[0030] The beneficial effects of this invention are as follows:

[0031] 1. The BGD-Net network proposed in this invention creatively introduces the restoration of the background image as an explicit guiding signal into the depth completion process, providing the network with background context information that "should be visible", effectively decoupling the visual confusion between the transparent foreground object and the background, and significantly alleviating the problem of weak RGB cues.

[0032] 2. This invention designs a gated depth feature extractor, which achieves adaptive filtering and feature enhancement of noise and invalid information in the original depth data by integrating the attention mechanism of depth, mask and context features, providing a more reliable geometric starting point for subsequent processing.

[0033] 3. This invention innovatively integrates frequency domain analysis (discrete wavelet transform) into the feature enhancement module, which can explicitly separate and enhance high-frequency edge information. Combined with traditional spatial domain convolution, it jointly overcomes the problem of blurred depth boundary reconstruction of transparent objects.

[0034] 4. This invention utilizes a systematic multi-scale feature fusion and refinement strategy to fully leverage the complementary information of features at different levels, ensuring high fidelity in both the overall structure and local details of the final depth map.

[0035] 5. Experimental results show that the method of this invention achieves state-of-the-art performance on authoritative public datasets such as TransCG and ClearPose, and demonstrates excellent robustness and generalization ability in a variety of complex and challenging scenarios such as occlusion, new backgrounds, non-planar backgrounds, and liquid filling. Attached Figure Description

[0036] Figure 1 is an overall architecture diagram of the background-guided deep completion network of the present invention, which also shows the structural details of its core components—gated deep feature extractor and multi-scale frequency domain feature enhancer.

[0037] Figure 2 shows a qualitative comparison of the method of this invention and the TDCNet method on the ClearPose (top two rows) and TransCG (bottom two rows) datasets (from left to right: the input RGB image, the absolute error map of TDCNet, the absolute error map of this invention, and the depth ground truth map).

[0038] Figure 3 shows the qualitative ablation results of the multi-scale frequency domain feature enhancer (first two rows) and the gated depth feature extractor (bottom two rows) (from left to right: the input RGB image, the absolute error map with the corresponding module added, the absolute error map without the corresponding module added, and the depth ground truth map). Detailed Implementation

[0039] The present invention will now be described in further detail with reference to the accompanying drawings and technical solutions.

[0040] This invention provides a method for depth completion of transparent objects based on background guidance and frequency domain enhancement, the overall network architecture (BGD-Net) of which is shown in Figure 1. The specific implementation steps are as follows: 1. Data Preparation

[0041] Use publicly available, large-scale real-world transparent object datasets, such as TransCG or ClearPose. For each frame of data, obtain:

[0042] (1) Original RGB image .

[0043] (2) Incomplete original depth map .

[0044] (2) Corresponding ground depth map (For evaluation purposes only).

[0045] (3) Transparent object instance segmentation mask .

[0046] (4) Repair the background image : Utilizing a large-scale image inpainting model (based on the Stable Diffusion Inpaint Anything model) to mask To guide, from Remove transparent objects and generate a visually plausible background image.

[0047] 2. Network Structure and Parameter Configuration

[0048] BGD-Net receives four inputs: , , , Output the completed depth map .

[0049] (1) Encoder-decoder backbone: Common U-Net-like structures can be used as the basic skeleton for feature extraction and fusion.

[0050] (2) Gated Deep Feature Extractor: As shown in Figure 1, a GDFE module is embedded in the encoder part of the backbone network. Specifically, the intermediate features... The number of channels is set to The lightweight MLP in dual-path attention can be designed as a two-layer module. In multi-scale dual-attention modules, the number of output channels for convolutional kernels of different sizes can be set to... Finally, the data is stitched together. The gated depth feature extractor is designed to clean up noise in the original depth map and enhance effective geometric features. It employs a dual-path gated attention mechanism, combining the original depth map, object mask, and encoder intermediate features to generate spatial and channel attention weights. This adaptively suppresses noise in invalid regions, preserves and enhances effective background geometric cues, and outputs clean depth features.

[0051] (3) Multi-scale frequency domain feature enhancer: designed to enhance the perception of edges and details of transparent objects. This module contains two pathways: the spatial context pathway utilizes the original RGB and restored background images, extracting and calibrating global context features through channel attention and multi-scale convolution; the frequency domain detail pathway performs discrete wavelet transform on the features, explicitly separating and enhancing the high-frequency components representing object boundaries in the frequency domain, then reconstructing them back to the spatial domain through inverse transform, and fusing them with the spatial pathway information, significantly improving the reconstruction capability of sharp boundaries. As shown in Figure 1, this module operates on the feature fusion layer of the backbone network. In the spatial context pathway, the MLP used to generate channel weights can be designed as a simple two-layer network. The configuration of the multi-scale convolutional blocks is similar to that in GDFE. In the frequency domain detail pathway, Haar wavelets are used for discrete wavelet transform.

[0052] (4) Multi-scale feature fusion and refinement module: This module is responsible for aggregating features from different levels of the decoder backbone network. It upsamples feature maps from multiple scales to the same resolution and concatenates them to form a comprehensive representation that integrates low-level details and high-level semantics. This representation undergoes non-linear fusion and decoding through a lightweight multilayer perceptron (MLP), and residual connections are introduced to ultimately output a high-quality depth map with clear boundaries and geometric accuracy. As shown in Figure 1, features are extracted from the last S=3 levels of the backbone network. The final refinement head's MLP can contain two fully connected layers and a ReLU activation function.

[0053] 3. Network Training

[0054] (1) Loss function: adopting ,in Set to 0.001. All losses are only in the mask. Calculated within the region.

[0055] (2) Optimization settings: The Adam optimizer was used with an initial learning rate of 0.001 and a batch size of 4. Training was performed on an NVIDIA GeForce RTX 3090 GPU for 30 epochs.

[0056] 4. Experimental Verification

[0057] To verify the effectiveness of the method of the present invention, a large number of experiments were conducted on the TransCG and ClearPose datasets.

[0058] (1) Comparative Experiment: TDCNet, the current state-of-the-art method for depth completion of transparent objects, was selected as the main baseline for comparison. The comparative results of this invention on authoritative datasets fully demonstrate its superiority. On the TransCG and ClearPoseSet1 test sets, the depth completion accuracy of this invention is significantly better than the comparative method. In a series of more challenging sub-scenes in ClearPose, such as semi-transparent overlays, non-planar backgrounds, opaque interference objects, new backgrounds, severe occlusion, and liquid filling, this invention also shows comprehensive and significant advantages, and can more clearly and completely recover the three-dimensional structure of objects, and exhibits better generalization ability and robustness. The qualitative error map comparison (as shown in Figure 2) also intuitively demonstrates the significant improvement of this invention in the sharpness of transparent object boundaries and internal geometric consistency.

[0059] (2) Ablation Experiments: Ablation studies were conducted on each core module on the TransCG dataset. The experiments show that the Gated Depth Feature Extractor (GDFE) plays a key role in suppressing depth noise and purifying input features; the Multi-Scale Frequency Domain Feature Enhancer (MSFFE) is crucial for introducing background context information and enhancing high-frequency edge details; and the Multi-Scale Feature Fusion and Refinement module further optimizes the overall smoothness and geometric consistency of the depth map. Removing any module will lead to a considerable decrease in model performance. Qualitative error map comparisons (as shown in Figure 3) also intuitively verify the important contributions of each module to improving the quality of depth prediction, especially in enhancing the clarity of transparent object boundaries and the continuity of internal structures.

[0060] 5. Application

[0061] The trained BGD-Net model can be deployed in robotics or AR / VR systems. When the system captures a scene containing transparent objects using a regular RGB-D camera (such as the Intel RealSense D435), it can calculate object masks in real-time or offline (e.g., using a segmentation model) and generate images using image inpainting techniques. Subsequently, , , , By inputting BGD-Net, a high-quality completed depth map can be obtained, which can then be used for downstream tasks such as robot grasping path planning and precise interaction between virtual objects and real glass surfaces.

[0062] In summary, this invention provides an efficient and robust depth completion scheme for transparent objects. It only requires a general-purpose RGB-D sensor and can significantly improve the three-dimensional perception accuracy of transparent objects, which has important practical value.

Claims

1. A method for depth completion of transparent objects based on background guidance and frequency domain enhancement, characterized in that, Includes the following steps: Step 1: Obtain Input Data: Obtain a scene dataset containing transparent objects, and edit the original RGB image using a large image editing model to obtain the repaired background image; the input data includes the original RGB image. The restored background image generated by image editing of a large model. Original depth map containing noise And the binary mask of transparent objects Step 2: Construct a background-guided deep completion network BGD-Net, which includes a gated deep feature extractor GDFE, a multi-scale frequency domain feature enhancer MSFFE, and a multi-scale feature fusion and refinement module connected in sequence; (2.1) The gated deep feature extractor receives the... and The concatenated data, combined with intermediate features obtained from the previous layer of the encoder. Feature purification is performed through a dual-path gating attention mechanism, resulting in clean deep features. ； (2.2) The multi-scale frequency domain feature enhancer receives the output from the gated depth feature extractor. The fused features of the current layer are obtained through convolutional layers. Input at the same time and Through collaborative processing of spatial context and frequency domain detail paths, the final output is enhanced features. ； (2.3) The multi-scale feature fusion and refinement module aggregates feature maps from S different levels of the decoder backbone network. After upsampling and concatenating the feature maps from the S different levels, a multilayer perceptron (MLP) is used as the final refinement head to learn to nonlinearly integrate all multi-scale feature information and introduce residual connections from coarse-grained depth prediction to finally output a refined depth map. Step 3: Input the input data into BGD-Net and train the network; Step 4: Depth completion; Input the scene data to be processed into the trained BGD-Net network, and after the network forward propagates, it directly outputs the completed depth map.

2. The method for depth completion of transparent objects based on background guidance and frequency domain enhancement according to claim 1, characterized in that, In step 2 (2.1), the dual-path gating attention mechanism specifically includes: (2.1.1) Feature stitching and geometric perception extraction: The stitched original depth map and the binary mask of the transparent object are combined... ， Input a 3×3 convolutional layer and extract geometrically perceptual features; then combine these features with intermediate features. By splicing the images again, we obtain enhanced features. (2.1.2) Spatial attention filtering: Selecting features... Spatial attention weights are generated by sequentially passing through pooling layers, convolutional layers, and a sigmoid activation function; these spatial attention weights are then combined with... Element-wise multiplication is used to filter the feature space locations, and then a lightweight multilayer perceptron (MLP) is used to obtain the features. (2.1.3) Channel attention recalibration: intermediate features After 1×1 convolution dimensionality reduction, and spliced ​​[ ， A lightweight MLP is used as the common input to generate channel attention weights; these channel attention weights are then combined with features. Element-wise multiplication is performed to recalibrate the feature channels; finally, the features filtered by spatial and channel attention are input into a multi-scale dual attention module to obtain the output features. 。 3. The method for depth completion of transparent objects based on background guidance and frequency domain enhancement according to claim 1, characterized in that, In step 2 (2.2), the processing of the spatial context path and the frequency domain detail path specifically includes: (2.2.1) Spatial context feature extraction: ... and After concatenation, the data is input into a lightweight MLP to obtain image features. At the same time Preliminary image feature weights are generated through a 3×3 convolutional layer, and... After multiplication, the data is input into a multi-scale dual-attention module to obtain spatial context features. (2.2.2) Frequency domain detail enhancement: enhancement of spatial context features Features fused with the current layer Two-dimensional discrete wavelet transforms are performed separately to decompose the data into low-frequency sub-bands and multiple high-frequency sub-bands. After concatenating all sub-bands, frequency domain feature transformation and enhancement are performed on each sub-band through independent convolutional layers. Finally, the processed sub-bands are reconstructed into the spatial domain through inverse discrete wavelet transforms to obtain the frequency domain enhanced features. (2.2.3) Feature fusion and secondary frequency domain enhancement: spatial context features With and from the restored background image The extracted background features are added together; the added features are then summed. The enhanced features are then processed again through the frequency domain detail enhancement step described in (2.2.2) above, and the final output is the enhanced feature. 。 4. A method for depth completion of transparent objects based on background guidance and frequency domain enhancement according to claim 2 or 3, characterized in that, The specific process of the multi-scale dual attention module is as follows: (1) Let the feature of the input multi-scale dual attention module be denoted as ， First, the data is fed into four parallel convolutional layers with kernel sizes of 1×1, 3×3, 5×5, and 7×5 to capture multi-scale contextual information. The output features of the four parallel convolutional layers are concatenated, and then multi-scale contextual space attention weights are generated using an MLP and a sigmoid activation function. These multi-scale contextual space attention weights are then combined with… Multiply to obtain the characteristics (2) At the same time, The average pooling layer, convolutional layer, and sigmoid activation function are input sequentially to generate multi-scale contextual channel attention weights. These multi-scale contextual channel attention weights are then combined with... Multiply, and then multiply the result with... The summation ultimately yields the output features. 。 5. The method for depth completion of transparent objects based on background guidance and frequency domain enhancement according to claim 1, characterized in that, The background-guided depth completion network is trained using a composite loss function, which is applied to binary masks of transparent objects. Internal calculation, its expression is: ;in, To predict depth maps With true depth Between loss; For the reason The calculated normal map and the result of Calculate the cosine distance loss between normal maps; To balance the weights.

Images