Transparent object depth completion method based on input robustness

Abstract

The invention discloses a transparent object depth completion method based on input robustness, and belongs to the field of computer vision and deep learning. According to the method, an RIDCNet network is constructed, and the RIDCNet network comprises an RGB-to-Normal branch and a Depth-to-Depth branch. The RGB-to-Normal branch is used for predicting a surface normal from the RGB image; the double-branch features are fused through a feature fusion module FFModule, the module adopts a double-branch attention mechanism, and RGB geometric information and depth measurement information are dynamically integrated through self-enhancement and collaborative enhancement; and finally, the Depth-to-Depth decoder is used for outputting a complemented depth map. The core purpose of the method is to solve the problem that the RGB-D camera fails to sense the depth of transparent objects such as glass, and the method can be applied to tasks requiring robust depth sensing, such as robot grabbing of the transparent objects, augmented reality (AR) scene fusion, automatic driving obstacle recognition and 3D scene reconstruction. According to the method, the dependence on defect depth input is remarkably reduced, and the robustness and precision of the model in a cross-sensor scene and under the condition that input disturbance exists are improved.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning, and specifically relates to a method for depth completion of transparent objects based on input robustness. Background Technology

[0002] Transparent objects are ubiquitous in applications such as robotics and augmented reality, making accurate depth perception crucial. Typically, depth information for objects and the entire scene can be obtained using depth estimation algorithms or RGB-D cameras. However, standard RGB-D cameras, which rely on active infrared patterns or time-of-flight measurements, systematically fail on transparent surfaces due to light transmission and refraction, resulting in missing or severely compromised depth values.

[0003] Existing solutions are mainly divided into optimization-based methods and learning-based methods. Optimization-based methods utilize geometric priors (such as surface normals and smoothness) to iteratively optimize depth, but they suffer from error accumulation and scalability issues. Learning-based methods, especially deep neural networks, have become mainstream by learning the mapping from RGB-D to the complete depth map.

[0004] However, existing learning-based methods have significant drawbacks: 1) Over-reliance on the original depth map; the network mainly learns interpolation from the damaged depth map rather than extracting complementary geometric information from the RGB image, resulting in limited performance improvement. 2) Sensor domain offset; networks trained on different sensor data have poor generalization ability. 3) Perturbation sensitivity; when there are large-scale local faults in the original depth map, the network performance drops sharply.

[0005] Therefore, there is an urgent need for a transparent object depth completion method that can reduce dependence on the original depth map, improve cross-sensor generalization ability and perturbation robustness. Summary of the Invention

[0006] To address the aforementioned technical problems in existing technologies, this invention proposes a transparent object depth completion method based on input robustness. The core of this method lies in constructing a depth completion network. This network guides RGB feature extraction by introducing a surface normal estimation task and integrates RGB and depth features using a feature fusion module. This solves the problems of modal dependency imbalance, weak domain generalization ability, and poor input robustness in existing transparent object depth completion methods.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A method for depth completion of transparent objects based on input robustness is proposed. The method is implemented using a depth completion network RIDCNet, which includes an RGB-to-Normal branch, a Depth-to-Depth branch, and a feature fusion module FFModule connecting the two. The method includes: S1. Input the RGB image to the RGB-to-Normal branch, extract multi-scale features of the RGB image through the branch and predict and generate surface normal maps, and obtain intermediate features of the RGB branch for subsequent fusion. S2. The input original depth map is fed to the Depth-to-Depth branch, through which multi-scale depth features of the original depth map are extracted as intermediate features of the depth branch for subsequent fusion. S3. Using the feature fusion module FFModule, multi-scale fusion is performed on the intermediate features of the RGB branch obtained in step S1 and the intermediate features of the depth branch obtained in step S2 to obtain multi-scale fused features. S4. Based on the multi-scale fusion features obtained in step S3, generate and output the completed depth map.

[0008] Preferably, the RGB-to-Normal branch includes an RGB encoder and an RGB-to-Normal decoder connected in sequence; step S1 specifically includes: S11, the RGB encoder processes the input RGB image. Encode the data and extract the deepest RGB encoding features. and at least one scale of intermediate RGB encoding features S12, the RGB-to-Normal decoder addresses the deepest RGB encoded features. and at least one scale of intermediate RGB encoding features Decode the surface and output the predicted surface normal map. and at least one scale of intermediate RGB decoder features The intermediate RGB decoder features serve as the intermediate features of the RGB branch in step S1.

[0009] Preferably, the depth-to-depth branch includes a depth encoder; step S2 specifically includes: S21, the depth encoder processes the input raw depth map. Encode the data and extract the deepest encoded features. and at least one intermediate depth coding feature at a scale The deepest depth coding features and intermediate depth coding features are used as intermediate features of the depth branch in step S2.

[0010] Preferably, the feature fusion module FFModule includes at least one feature fusion block FFBlock, which employs a dual-branch attention mechanism; in step S3, at each fusion scale, FFBlock performs the following operations: S31, for the input RGB feature map and depth feature map Calculate the channel attention weights respectively. , S32, Utilizing Channel Attention Weights , Calculate the self-enhancing features of RGB features and depth features respectively. , and enhanced features based on collaborative attention , S33. Add the self-enhanced features and the enhanced features based on collaborative attention respectively to obtain the enhanced RGB features. and depth features S34. Enhanced RGB features and depth features Perform fusion and output the fusion features at that scale. .

[0011] Preferably, in step S31, the channel attention weights The calculation method is as follows: global average pooling is performed on the input feature map, and then it is obtained by sequentially passing it through a one-dimensional convolutional layer with ReLU activation function and Sigmoid activation function.

[0012] Preferably, step S32 includes: S321, calculating self-enhancing features: , ;That middle This indicates multiplication by position; S322. Calculate the collaborative attention weights: ; S323. Calculate the enhanced features based on collaborative attention: ; .

[0013] Preferably, in step S34, the fusion operation specifically involves: combining the enhanced RGB features... and depth features To splice them together, and then through a Convolutional layer Processing is performed to obtain fusion features. .

[0014] Preferably, the deep completion network RIDCNet adopts an encoder-decoder architecture; Both the encoder and decoder consist of multiple basic modules. The basic module is a convolution-dense block-convolution module, i.e., a CDC module. The decoder contains multiple upsampling units, each of which contains a CDC module and an upsampling layer. The upsampling layer is implemented using dense upsampling convolution.

[0015] Preferably, the RGB-to-Normal decoder performs vector normalization on the output result before outputting the surface normal map, so that the output surface normal vector is a unit vector.

[0016] Preferably, when training the deep completion network RIDCNet, the loss function used is a combined loss function. The expression is: ; in, To fully compensate for the loss, To estimate the loss for the surface normal, These are the weighting coefficients; Surface normal estimation loss Defined as: ; in, It is an estimated surface normal diagram. It is a true surface normal diagram. It is the total number of pixels with valid true values. For cosine similarity calculation, The total number of pixels with valid true values. This represents the number of pixels in the transparent object region. These are the weighting coefficients; Deep Completion Loss Defined as: ; in, These are the weighting coefficients; for Regression loss is defined as: ; in, It is an estimated depth map. It is a true depth map. for Norm, These are the weighting coefficients; To smooth the loss, it is defined as: ; in, This represents the number of pixels with valid true values.

[0017] The beneficial technical effects of this invention are as follows: 1. Reduce depth dependence and enhance RGB utilization: By introducing surface normal estimation as an auxiliary task, the network is forced to delve deeper into the geometric cues in the RGB image, reducing the dependence on the defective original depth map and making the network pay more attention to the complementary information of the RGB modes.

[0018] 2. Excellent cross-sensor generalization ability; the surface normal characterizes the local surface orientation and is independent of the absolute depth scale, exhibiting inherent robustness to different sensor noise modes. Therefore, this invention performs exceptionally well on unseen sensor data.

[0019] 3. Strong resistance to input disturbances: Since the network does not overly rely on the continuity of the original depth map, when there are large local missing blocks or noise in the depth map, this invention can maintain stable depth completion performance through RGB information and learned priors.

[0020] 4. Improved accuracy: Through an effective dual-branch attention fusion mechanism, this invention can integrate multimodal information more precisely, outperforming existing methods on multiple standard evaluation metrics. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall architecture of the RIDCNet network provided in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the internal structure of the feature fusion module FFModule provided in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0024] This embodiment provides a complete method for depth completion of transparent objects based on input robustness, implemented using the PyTorch framework, and trained and tested on an RGB-D dataset containing transparent objects.

[0025] The specific steps are as follows: 1. Network architecture implementation (e.g.) Figure 1 (As shown). Construct the RIDCNet network, which includes RGB-to-Normal branch, Depth-to-Depth branch, and special... The fusion module FFModule; the RIDCNet network adopts a U-Net-based structure.

[0026] The core components are as follows: Base Block: Employs Conv-Dense-Conv (CDC) blocks. Each CDC block contains: Convolutional layer → DenseBlock (facilitating feature reuse) → Convolutional layer. At the end of the decoder in the RGB-to-Normal branch, the final ReLU activation is removed, and a normalization layer is introduced to produce unit vectors.

[0027] Encoder: The input image is halved in resolution after passing through an initial convolutional layer, followed by three CDC blocks, each of which is then followed by a downsampling layer, for a total of four downsampling operations, resulting in the deepest encoded features. and features at three intermediate scales .

[0028] The decoder consists of four CDCU blocks (Conv-Dense-Conv-Upsample). Each CDCU block contains one CDC block and one dense upsampling convolutional layer (learning upsampling weights using convolution). The RGB-to-Normal decoder outputs a normal map. and intermediate features , , The Depth-to-Depth decoder outputs the final depth map. .

[0029] Key design features: 1. In the CDC and CDCU blocks, the original depth map is not used as additional input, forcing the network to learn from RGB and extracted features, reducing the dependency path on the original depth input. 2. Dual-branch parallel processing.

[0030] RGB-to-Normal branch: The RGB image first goes through an RGB encoder ( Feature extraction is performed to generate deep coding features. and multiple intermediate scale features These features are then input into the RGB-to-Normal decoder ( The final output is the predicted surface normal map. And simultaneously output multiple intermediate decoder features For subsequent integration.

[0031] Depth-to-Depth branch: The original depth map is processed by the depth encoder ( Processing yields deep coding features. and multiple intermediate scale features .

[0032] The specific process for generating three intermediate scale features is as follows: RGB-to-Normal branch: Input: RGB image .

[0033] process: ; .

[0034] Output: Predicted surface normal map and features used for fusion , , .

[0035] Depth-to-Depth branch: Input: Original depth map .

[0036] process: .

[0037] Output: Deep features used for fusion , , , .

[0038] 3. Feature fusion (e.g.) Figure 2 (As shown).

[0039] The feature fusion module FFModule receives corresponding scale features from the two branches (e.g., and First, channel attention vectors for RGB and depth features are generated through global average pooling (GAP) and fully connected layers (implemented in practice using 1×1 convolutions), respectively. and These two vectors are used to perform self-enhancement on their respective features. Simultaneously, the two attention vectors are summed and then normalized using Softmax to obtain the collaborative attention weights. and will Simultaneously acting on features from two modalities, generating collaboratively enhanced features ( Finally, the self-enhanced and collaboratively enhanced features are added together, concatenated, and then fused across modalities using a 1×1 convolutional layer to output the fused features at that scale. This process occurs at multiple scales (e.g., ) can be executed in parallel.

[0040] The feature fusion module FFModule consists of multiple FFBlocks, each of which performs the following two-branch attention fusion operation; Input: RGB feature map Deep feature map .

[0041] Calculate attention weights: , ;in, For global average pooling, and It is a 1×1 convolutional layer with ReLU activation. It is the sigmoid function; and It is the channel attention vector.

[0042] Self-enhancing path: , ;in, This indicates the broadcast multiplication along the channel direction. This path allows each modal feature to reinforce itself.

[0043] Collaborative attention path: . Make The sum of 1 represents the co-importance distribution of the two modes. Applicable to two modes: , This path allows the features of the two modalities to be cross-adjusted based on the other's attention.

[0044] Combination and Integration: , ; . for Convolutional layers are used to fuse and adjust the number of channels.

[0045] Multi-scale fusion: Fusion is performed at four scales; , , , .

[0046] 4. Depth map generation.

[0047] Features after fusion of various scales ( Input Depth-to-Depth decoder ( Through a series of upsampling and convolution operations, the final output is a completed depth map. .

[0048] .

[0049] 5. Network Training: A combined loss function is used. Surface normal estimation loss is employed. Cosine similarity loss is used, with additional weights assigned to transparent object regions. Depth completion loss is also employed. include The network employs two losses: a loss based on surface normal consistency and a smoothing loss. By jointly optimizing these two losses, the network can simultaneously learn accurate surface geometry and depth values.

[0050] Combination loss function The expression is: ,in To fully compensate for the loss, To estimate the loss for the surface normal, These are the weighting coefficients. Surface normals estimate the loss. Defined as; ; Among them, It is an estimated surface normal diagram. It is a true surface normal diagram. It is the total number of pixels with valid true values. For cosine similarity calculation, The total number of pixels with valid true values. This represents the number of pixels in the transparent object region. These are the weighting coefficients.

[0051] Deep Completion Loss Defined as: ;in, These are the weighting coefficients.

[0052] for Regression loss is defined as: ; in, It is an estimated depth map. It is a true depth map. for Norm, These are the weights for smoothing the loss.

[0053] To smooth the loss, it is defined as: ; in, This represents the number of pixels with valid true values.

[0054] 6. Training and testing.

[0055] In the specific experiments, the PyTorch framework was used for training on an NVIDIA A40 GPU. The AdamW optimizer was chosen, with an initial learning rate of 0.001 and a step-wise decay. The batch size was set to 16. Comparative experiments with the baseline model (which only concatenates RGB-D as input) show that the RIDCNet network, which incorporates the RGB-to-Normal branch and the FFModule module, exhibits superior robustness and higher accuracy in both cross-sensor tests and depth map perturbation tests.

[0056] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for depth completion of transparent objects based on input robustness, characterized in that, The method is based on a deep completion network RIDCNet, which includes an RGB-to-Normal branch, a Depth-to-Depth branch, and a feature fusion module FFModule connecting the two; the method includes: S1. Input the RGB image to the RGB-to-Normal branch, extract multi-scale features of the RGB image through the branch and predict and generate surface normal maps, and obtain intermediate features of the RGB branch for subsequent fusion. S2. The input original depth map is fed to the Depth-to-Depth branch, through which multi-scale depth features of the original depth map are extracted as intermediate features of the depth branch for subsequent fusion. S3. Using the feature fusion module FFModule, multi-scale fusion is performed on the intermediate features of the RGB branch obtained in step S1 and the intermediate features of the depth branch obtained in step S2 to obtain multi-scale fused features. S4. Based on the multi-scale fusion features obtained in step S3, generate and output the completed depth map.

2. The method for depth completion of transparent objects based on input robustness according to claim 1, characterized in that, The RGB-to-Normal branch includes an RGB encoder and an RGB-to-Normal decoder connected in sequence; step S1 specifically includes: S11, the RGB encoder processes the input RGB image. Encode the data and extract the deepest RGB encoding features. and at least one scale of intermediate RGB encoding features S12, the RGB-to-Normal decoder addresses the deepest RGB encoded features. and at least one scale of intermediate RGB encoding features Decode the surface and output the predicted surface normal map. and at least one scale of intermediate RGB decoder features The intermediate RGB decoder features serve as the intermediate features of the RGB branch in step S1.

3. The method for depth completion of transparent objects based on input robustness according to claim 1, characterized in that, The Depth-to-Depth branch includes a depth encoder; step S2 specifically includes: S21, the depth encoder processes the input raw depth map. Encode the data and extract the deepest encoded features. and at least one intermediate depth coding feature at a scale The deepest depth coding features and intermediate depth coding features are used as intermediate features of the depth branch in step S2.

4. The method for depth completion of transparent objects based on input robustness according to claim 1, characterized in that, The feature fusion module FFModule includes at least one feature fusion block FFBlock, which employs a dual-branch attention mechanism. In step S3, at each fusion scale, FFBlock performs the following operations: S31, for the input RGB feature map... and depth feature map Calculate the channel attention weights respectively. , S32, Utilizing channel attention weights , Calculate the self-enhancing features of RGB features and depth features respectively. , and enhanced features based on collaborative attention , S33. Add the self-enhanced features and the enhanced features based on collaborative attention respectively to obtain the enhanced RGB features. and depth features S34. Enhanced RGB features and depth features Perform fusion and output the fusion features at that scale. .

5. The method for depth completion of transparent objects based on input robustness according to claim 4, characterized in that, In step S31, channel attention weights The calculation method is as follows: global average pooling is performed on the input feature map, and then it is obtained by passing it through a one-dimensional convolutional layer with ReLU activation function and Sigmoid activation function in sequence.

6. The method for depth completion of transparent objects based on input robustness according to claim 4, characterized in that, Step S32 includes: S321. Calculate self-enhancing features: , ,in This indicates multiplication by position; S322. Calculate the collaborative attention weights: ; S323. Calculate the enhanced features based on collaborative attention: ; .

7. The method for depth completion of transparent objects based on input robustness according to claim 4, characterized in that, In step S34, the fusion operation specifically involves: combining the enhanced RGB features... and depth features To splice them together, and then through a Convolutional layer Processing is performed to obtain fusion features. .

8. The method for depth completion of transparent objects based on input robustness according to claim 1, characterized in that, The deep completion network RIDCNet adopts an encoder-decoder architecture; Both the encoder and decoder consist of multiple basic modules. The basic module is a convolution-dense block-convolution module, i.e., a CDC module. The decoder contains multiple upsampling units, each of which contains a CDC module and an upsampling layer. The upsampling layer is implemented using dense upsampling convolution.

9. The method for depth completion of transparent objects based on input robustness according to claim 2, characterized in that, Before outputting the surface normal map, the RGB-to-Normal decoder performs vector normalization on the output result, so that the output surface normal vector is a unit vector.

10. The method for depth completion of transparent objects based on input robustness according to claim 1, characterized in that, When training the deep completion network RIDCNet, the loss function used is the combined loss function. The expression is: ； in, To fully compensate for the loss, To estimate the loss for the surface normal, These are the weighting coefficients; Surface normal estimation loss Defined as: ； in, It is an estimated surface normal diagram. It is a true surface normal diagram. It is the total number of pixels with valid true values. For cosine similarity calculation, The total number of pixels with valid true values. This represents the number of pixels in the transparent object region. These are the weighting coefficients; Deep Completion Loss Defined as: ； in, These are the weighting coefficients; for Regression loss is defined as: ； in, It is an estimated depth map. It is a true depth map. for Norm, These are the weighting coefficients; To smooth the loss, it is defined as: ； in, This represents the number of pixels with valid true values.

Images