A method and system for edge-preserving depth inpainting of transparent objects

Abstract

The application discloses a transparent object depth repairing method and system with edge preservation. The method comprises the following steps: S1, establishing a transparent object image dataset; the transparent object image dataset comprises RGB images with a transparent object in several scenes and real depth images obtained by a depth camera; S2, respectively performing image preprocessing on the RGB images and the corresponding depth images, and splicing the preprocessed RGB images and the depth images to obtain spliced images; S3, establishing a transparent object depth repairing model, and training the model by using the preprocessed transparent object image dataset to obtain a trained model; and S4, constructing a loss function with edge preservation to optimize the transparent object depth repairing model. The application can solve the problems of depth error and depth loss generated when the depth camera captures the depth of the transparent object, and can solve the problems of poor effect and blurred boundary of the current depth repairing algorithm.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for depth restoration of transparent objects while preserving their edges. Background Technology

[0002] Depth images are acquired by depth cameras, and their pixel values ​​represent the distance between the camera and the object. High-quality depth images play a crucial role in fields such as robotics, autonomous driving, and 3D reconstruction. Due to light reflection and refraction, current consumer-grade depth cameras suffer from depth loss and errors when dealing with transparent objects, significantly interfering with the analysis and processing of depth images in downstream tasks. Therefore, depth image restoration for transparent objects is particularly important.

[0003] Traditional methods mainly rely on special equipment such as polarization cameras and special light field sensors to capture the properties of transparent objects. These methods are too dependent on external equipment and require fixed backgrounds and fixed devices, which limits their generalization ability in different scene environments and makes them unsuitable for large-scale application.

[0004] Deep learning methods mainly obtain depth images of transparent objects by matching corresponding points in multiple views. These methods are computationally expensive and have low accuracy. Some methods using neural networks are not effective in depth restoration of transparent objects, resulting in blurred object boundaries and difficulty in obtaining satisfactory depth images. Summary of the Invention

[0005] The purpose of this invention is to address the defects and shortcomings of existing technologies by proposing an edge-preserving method and system for depth restoration of transparent objects, solving the problems of depth errors and depth loss when depth cameras capture the depth of transparent objects, as well as the problems of poor performance and blurred boundaries in current depth restoration algorithms.

[0006] The technical solution of the present invention is as follows.

[0007] On the one hand, a method for depth restoration of transparent objects while preserving their edges includes:

[0008] S1, Establish a transparent object image dataset; the transparent object image dataset includes RGB images of transparent objects in several scenes, original depth images obtained by a depth camera, and real depth images;

[0009] S2, perform image preprocessing on the RGB image and the corresponding original depth image respectively, and stitch the preprocessed RGB image and the original depth image to obtain a stitched image;

[0010] S3. Establish a transparent object depth restoration model and train it using a preprocessed transparent object image dataset to obtain the trained model. The transparent object depth restoration model includes a feature encoding module, a cross-layer feature fusion module, and a depth image restoration module. The feature encoding module is used to extract features from the stitched image to generate the first image features. The cross-layer feature fusion module is used to fuse the first image features from the upper and lower layers to obtain the second image features. The depth image restoration module is used to combine the first image features and the second image features to obtain the depth-restored image.

[0011] S4. Construct a loss function with edge preservation to optimize the depth restoration model for transparent objects.

[0012] Preferably, S2 specifically includes:

[0013] A geometric transformation operation is performed on the RGB image, and the original depth image is initially repaired using the nearest neighbor interpolation method. The initially repaired depth image is then stitched together with the geometrically transformed RGB image to obtain the first data matrix.

[0014] Preferably, the feature encoding module includes a first convolutional layer, a first backbone feature extraction layer, and a first downsampling layer connected in sequence;

[0015] The first convolutional layer consists of a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation layer connected in sequence, used to obtain the first preliminary features I of the stitched image. 1(conv) ;

[0016] The first backbone feature extraction layer adopts an improved ELAN network structure, which extracts the first preliminary feature I. 1(conv) The input is the backbone feature extraction layer. The output of the first backbone feature extraction layer is fused with the CBAM attention module and combined with skip connections to obtain the first feature-encoded image I with the region of interest. backbon ;

[0017] The first downsampling layer consists of a max pooling layer, a batch normalization layer, and a ReLU activation layer connected in sequence, encoding the first feature image I containing the region of interest. backbone Input the first downsampling module and output the first image feature.

[0018] Preferably, the first preliminary feature I 1(conv) After inputting into the backbone feature extraction layer, the first feature-encoded image I with the region of interest is obtained. backbone , means as follows:

[0019] I backbone =I 1(conv) +CBAM(ELAN(I 1(conv) ))

[0020] Here, ELAN represents ELAN network; CBAM represents attention network.

[0021] Preferably, the cross-layer feature fusion module includes a max pooling layer and a CBAM attention module; the first image feature I output from layer M is... 1(M) After passing through a max pooling layer, the input to the CBAM attention module yields a feature encoding containing information from the upper layers. This feature encoding is then combined with the first image feature I from layer M+1. 1(M+1) By combining these features, we obtain the second image feature I of the (M+1)th layer. 2(M) , means as follows:

[0022] I 2(M) =CBAM(Maxpool(I 1(M) ))*I 1(M+1)

[0023] Wherein, CBAM represents the attention network; Maxpool represents the max pooling layer network.

[0024] Preferably, the depth image restoration module includes a second convolutional layer, a second backbone feature extraction layer, and a pixel shuffling operation layer;

[0025] The second convolutional layer includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation layer connected in sequence, used to obtain the second preliminary features of the second image features;

[0026] The second backbone feature extraction layer adopts an improved ELAN network structure. The second preliminary feature is input into the second backbone feature extraction layer. The output of the second backbone feature extraction layer is fused with the CBAM attention module and combined with skip connections to obtain a second feature-encoded image with region of interest.

[0027] The pixel shuffling operation layer obtains a high-resolution feature map from the low-resolution second feature encoded image through convolution and multi-channel recombination.

[0028] Preferably, the loss function is expressed as follows:

[0029]

[0030] Where L represents the loss value; D p Represents the predicted depth image; D gt Represents the true depth image; α and β both represent hyperparameters; Lap represents the Laplacian operation; This represents the gradient of the predicted depth image in the horizontal direction; This represents the gradient of the true depth image in the horizontal direction; This represents the gradient of the predicted depth image in the vertical direction; This represents the gradient of the true depth image in the vertical direction.

[0031] On the other hand, a depth restoration system for transparent objects that preserves edges includes:

[0032] The image dataset construction module establishes a transparent object image dataset; the transparent object image dataset includes RGB images of transparent objects in several scenes and real depth images obtained by a depth camera;

[0033] The stitching module performs image preprocessing on the RGB image and the corresponding original depth image, and then stitches the preprocessed RGB image and the original depth image to obtain a stitched image.

[0034] The model training module establishes a transparent object depth restoration model and trains it using a preprocessed transparent object image dataset to obtain the trained model. The transparent object depth restoration model includes a feature encoding module, a cross-layer feature fusion module, and a depth image restoration module. The feature encoding module extracts features from the stitched image to generate the first image features; the cross-layer feature fusion module fuses the first image features from upper and lower layers to obtain the second image features; and the depth image restoration module combines the first and second image features to obtain the depth-restored image.

[0035] The model optimization module constructs a loss function with edge preservation to optimize the depth restoration model for transparent objects.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) The present invention uses a cross-layer feature fusion module to extract multi-layer context information. By combining upper-layer features, it effectively alleviates the problem that the convolutional network pays more and more attention to local areas as the convolutional depth increases and that transparent objects are easily confused with the background, thus effectively improving the model's prediction accuracy for input data.

[0038] (2) The present invention adopts a loss function that combines Laplacian loss and MSE loss, which can effectively extract high-frequency features of the image, preserve the edge details of the object, and help downstream tasks to accurately understand the object contour, boundary and geometric features, laying the foundation for subsequent segmentation, 3D perception, reconstruction and other tasks. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1This is a flowchart of a method for edge-preserving depth restoration of transparent objects according to an embodiment of the present invention;

[0041] Figure 2 This is a structural diagram of the backbone feature extraction layer in an embodiment of the invention;

[0042] Figure 3 This is a structural diagram of the cross-layer feature fusion module according to an embodiment of the present invention;

[0043] Figure 4 This is a structural diagram showing the combination of the modules in an embodiment of the present invention;

[0044] Figure 5 This is a comparison image of the original RGB image, the original depth image, the true depth image, and the predicted depth-restoring image in an embodiment of the present invention; wherein, (a) represents the original RGB image; (b) represents the original depth image; (c) represents the true depth image; and (d) represents the predicted depth-restoring image.

[0045] Figure 6 This is a structural block diagram of the edge-preserving transparent object depth restoration system according to an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0047] Preserving edge details provides more refined and clear contour information, helping to capture detailed features of an object's surface, thus enabling more accurate depth perception and 3D reconstruction. By preserving object edges, we can better distinguish the hierarchical relationships between different parts of a transparent object, identify the shape and structure of the object's boundaries, and improve our understanding and analysis of transparent objects, providing a more reliable data foundation for visual perception and computer vision tasks.

[0048] See Figure 1 As shown in the figure, this embodiment provides a method for depth restoration of a transparent object while preserving its edges, which includes the following steps.

[0049] S1, Establish a transparent object image dataset; the transparent object image dataset includes RGB images of transparent objects in several scenes, original depth images obtained by a depth camera, and real depth images.

[0050] When photographing transparent objects, the special optical properties of transparent objects can cause image defects such as missing depth and depth distortion. Therefore, after photographing transparent objects, the dataset will cover the transparent objects with an opaque mask to mitigate the depth errors caused by the transparent objects and form a true depth image.

[0051] S2, perform image preprocessing on the RGB image and the corresponding original depth image respectively, and stitch the preprocessed RGB image and the original depth image together to obtain a stitched image.

[0052] Specifically, in S2, bilinear interpolation is used to scale the RGB image and the original depth image according to a set threshold, resulting in a scaled image size of (320, 240). Furthermore, in S2, a geometric transformation operation is performed on the RGB image to highlight or adjust specific features in the image of objects surrounding the transparent object, making the image more prominent or adaptable to specific needs. The original depth image acquired by the depth camera contains numerous erroneous depth values ​​and missing depth values. Nearest neighbor interpolation is used to perform preliminary repair using the depth values ​​surrounding the missing depth pixels. The repaired depth image is then stitched together with the RGB image to obtain a first data matrix, which is then imported into the transparent object depth repair model.

[0053] S3. Establish a transparent object depth restoration model and train it using a preprocessed transparent object image dataset to obtain a trained model. The transparent object depth restoration model includes a feature encoding module, a cross-layer feature fusion module, and a depth image restoration module. The feature encoding module is used to extract features from the stitched image to generate the first image features. The cross-layer feature fusion module is used to fuse the first image features from the upper and lower layers to obtain the second image features. The depth image restoration module is used to combine the first image features and the second image features to obtain the depth-restored image.

[0054] Specifically, the feature encoding module includes a first convolutional layer, a first backbone feature extraction layer, and a first downsampling layer, which import the first data matrix obtained by splicing into the feature encoding module.

[0055] The first convolutional layer contains a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation layer to obtain the first preliminary input feature I. 1(conv) The 1×1 convolutional layer extracts features and preserves spatial structure by performing convolution operations on the input data, helping to capture the features of the input data. The batch normalization layer is used to normalize each feature during neural network training, which helps to accelerate network convergence, reduce the gradient vanishing problem, and enhance the network's robustness to input data. The ReLU activation function layer introduces non-linearity by setting negative input values ​​to zero, which helps the network learn non-linear relationships.

[0056] The first backbone feature extraction layer adopts an improved ELAN network structure, see network structure [link to network structure]. Figure 2 As shown, ELAN is an efficient network architecture that enables the network to learn more features and has stronger robustness by controlling the shortest and longest gradient paths.

[0057] The ELAN network output is fused with the CBAM attention module, and skip connections are added to obtain the first feature-encoded image I with the region of interest. backbone CBAM is an attention module for computer vision tasks designed to enhance the ability of convolutional neural networks to model image features. CBAM combines channel attention and spatial attention modules to improve the model's ability to represent features. By using skip connections to add the original data to the output of subsequent layers, it accelerates network convergence and enhances the network's expressive power.

[0058] The first preliminary feature I 1(conv) After inputting into the backbone feature extraction layer, we obtain a feature-encoded image I containing the region of interest. backbone The formula is as follows:

[0059] I backbone =I 1(conv) +CBAM(ELAN(I 1(conv) ))

[0060] The first downsampling layer contains a max pooling layer, a batch normalization layer, and a ReLU activation layer, which encodes the features of the region of interest into the image I. backbone The first downsampling module outputs the first image features. The max-pooling layer selects the maximum value within each region to retain the most significant features, achieving feature dimensionality reduction. The batch normalization layer normalizes each feature during neural network training, helping to accelerate network convergence, reduce the vanishing gradient problem, and enhance the network's robustness to input data. The ReLU activation function layer introduces non-linearity by setting negative input values ​​to zero, helping the network learn non-linear relationships.

[0061] Specifically, the cross-layer feature fusion module includes a max pooling layer and a CBAM attention module, see [link to relevant documentation]. Figure 3 As shown, the first image features output from layer M and the first image features input from layer M+1 are used to obtain the second image features through a cross-layer fusion extraction module.

[0062] In this embodiment, the first image feature I output by layer M is... 1(M) After passing through a max pooling layer, the image is fed into the CBAM attention module to obtain a feature encoding containing information from the upper layers. This encoding is then combined with the first image feature I from layer M+1. 1(M+1) By combining these features, we obtain the second image feature I of the (M+1)th layer. 2(M)The formula is as follows:

[0063] I 2(M) =CBAM(Maxpool(I 1(M) ))*I 1(M+1)

[0064] Using a cross-layer feature fusion module can extract multi-layer contextual information. By combining features from upper layers, it can effectively alleviate the problem that convolutional networks become increasingly focused on local areas as the convolutional depth increases, as well as the problem that transparent objects are easily confused with the background. This effectively improves the model's prediction accuracy for the input data.

[0065] Specifically, the depth image restoration module includes a second convolutional layer, a second backbone feature extraction layer, and a pixel shuffle operation layer.

[0066] The first image features are imported into the depth image restoration module. Through N layers of upsampling combined with the second image features, the depth restoration process for transparent objects is completed. (See [link to module]). Figure 4 As shown.

[0067] Specifically, the second convolutional layer includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation layer. The 1×1 convolutional layer extracts features and preserves spatial structure by performing convolution operations on the input data, helping to capture the features of the input data. The batch normalization layer normalizes each feature during neural network training, helping to accelerate network convergence, reduce the vanishing gradient problem, and enhance the network's robustness to input data. The ReLU activation function layer introduces non-linearity by setting negative input values ​​to zero, helping the network learn non-linear relationships.

[0068] The second backbone feature extraction layer employs an improved ELAN network structure. ELAN is an efficient network structure that, by controlling the shortest and longest gradient paths, enables the network to learn more features and exhibits stronger robustness. The output of the ELAN network is fused with the CBAM attention mechanism and combined with skip connections to obtain a feature-encoded image with the region of interest.

[0069] The pixel shuffling operation layer transforms low-resolution feature images into high-resolution feature maps through convolution and multi-channel recombination, effectively improving the quality of low-resolution features.

[0070] S4. Construct a loss function with edge preservation to optimize the depth restoration model for transparent objects.

[0071] The image containing a transparent object was processed using both the edge-preserving depth restoration method described above and existing depth restoration methods without edge preservation. The processing results are shown in [link to results]. Figure 5As shown in the figure, (a) represents the original RGB image; (b) represents the original depth image; (c) represents the true depth image; and (d) represents the depth-restored image predicted by the method of this invention.

[0072] See Figure 6 As shown, this embodiment also discloses an edge-preserving depth restoration system for transparent objects, including:

[0073] Image dataset construction module 601 establishes a transparent object image dataset; the transparent object image dataset includes RGB images of transparent objects in several scenes and real depth images obtained by a depth camera;

[0074] The stitching module 602 performs image preprocessing on the RGB image and the corresponding depth image respectively, and stitches the preprocessed RGB image and depth image to obtain a stitched image.

[0075] Model training module 603 establishes a transparent object depth restoration model and trains it using a preprocessed transparent object image dataset to obtain the trained model. The transparent object depth restoration model includes a feature encoding module, a cross-layer feature fusion module, and a depth image restoration module. The feature encoding module extracts features from the stitched image to generate first image features; the cross-layer feature fusion module fuses the first image features from upper and lower layers to obtain second image features; and the depth image restoration module combines the first and second image features to obtain the depth-restored image.

[0076] Model optimization module 604 constructs a loss function with edge preservation to optimize the depth restoration model for transparent objects.

[0077] A specific implementation of an edge-preserving transparent object depth restoration system is described in this embodiment, which is the same as the edge-preserving transparent object depth restoration method.

[0078] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for depth restoration of transparent objects while preserving their edges, characterized in that, include: S1, Create a dataset of transparent object images; The transparent object image dataset includes RGB images of transparent objects in several scenes, raw depth images obtained by a depth camera, and real depth images. S2, perform image preprocessing on the RGB image and the corresponding original depth image respectively, and stitch the preprocessed RGB image and the original depth image to obtain a stitched image; S3. Establish a transparent object depth restoration model and train it using a preprocessed transparent object image dataset to obtain the trained model. The transparent object depth restoration model includes a feature encoding module, a cross-layer feature fusion module, and a depth image restoration module. The feature encoding module is used to extract features from the stitched image to generate the first image features. The cross-layer feature fusion module is used to fuse the first image features from the upper and lower layers to obtain the second image features. The depth image restoration module is used to combine the first image features and the second image features to obtain the depth-restored image. S4, construct a loss function with edge preservation to optimize the depth restoration model for transparent objects; The feature encoding module includes a first convolutional layer, a first backbone feature extraction layer, and a first downsampling layer connected in sequence. The first convolutional layer consists of a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation layer connected in sequence, used to obtain the first preliminary features I of the stitched image. 1(conv) ; The first backbone feature extraction layer adopts an improved ELAN network structure, which extracts the first preliminary feature I. 1(conv) The input is the backbone feature extraction layer. The output of the first backbone feature extraction layer is fused with the CBAM attention module and combined with skip connections to obtain the first feature-encoded image I with the region of interest. backbon ; The first downsampling layer consists of a max pooling layer, a batch normalization layer, and a ReLU activation layer connected in sequence, encoding the first feature image I containing the region of interest. backbone Input the first downsampling module and output the first image feature; The cross-layer feature fusion module includes a max pooling layer and a CBAM attention module; it integrates the first image features I output from layer M. 1(M) After passing through a max pooling layer, the input to the CBAM attention module yields a feature encoding containing information from the upper layers. This feature encoding is then combined with the first image feature I from layer M+1. 1(M+1) By combining these features, we obtain the second image feature I of the (M+1)th layer. 2(M) , means as follows: ； in, Represents an attention network; This represents a max-pooling layer network; The depth image restoration module includes a second convolutional layer, a second backbone feature extraction layer, and a pixel shuffling operation layer; The second convolutional layer includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation layer connected in sequence, used to obtain the second preliminary features of the second image features; The second backbone feature extraction layer adopts an improved ELAN network structure. The second preliminary feature is input into the second backbone feature extraction layer. The output of the second backbone feature extraction layer is fused with the CBAM attention module and combined with skip connections to obtain a second feature-encoded image with region of interest. The pixel shuffling operation layer obtains a high-resolution feature map from the low-resolution second feature encoded image through convolution and multi-channel recombination.

2. The method for edge-preserving depth restoration of transparent objects according to claim 1, characterized in that, S2 specifically includes: A geometric transformation operation is performed on the RGB image, and the original depth image is initially repaired using the nearest neighbor interpolation method. The initially repaired depth image is then stitched together with the geometrically transformed RGB image to obtain the first data matrix.

3. The method for edge-preserving depth restoration of transparent objects according to claim 2, characterized in that, The first preliminary feature I 1(conv) After inputting into the backbone feature extraction layer, the first feature-encoded image I with the region of interest is obtained. backbone , means as follows: ； in, Indicates an ELAN network; This represents an attention network.

4. The method for edge-preserving depth restoration of transparent objects according to claim 1, characterized in that, The loss function is expressed as follows: ； Where L represents the loss value; D p Represents the predicted depth image; D gt Represents a true depth image; and Both represent hyperparameters; Lap represents the Laplace operation; This represents the gradient of the predicted depth image in the horizontal direction; This represents the gradient of the true depth image in the horizontal direction; This represents the gradient of the predicted depth image in the vertical direction; This represents the gradient of the true depth image in the vertical direction.

5. A depth restoration system for transparent objects with edge preservation, characterized in that, For implementing the method of any one of claims 1 to 4, comprising: The image dataset construction module establishes a transparent object image dataset; the transparent object image dataset includes RGB images of transparent objects in several scenes and real depth images obtained by a depth camera; The stitching module performs image preprocessing on the RGB image and the corresponding original depth image, and then stitches the preprocessed RGB image and the original depth image to obtain a stitched image. The model training module establishes a transparent object depth restoration model and trains it using a preprocessed transparent object image dataset to obtain the trained model. The transparent object depth restoration model includes a feature encoding module, a cross-layer feature fusion module, and a depth image restoration module. The feature encoding module extracts features from the stitched image to generate the first image features; the cross-layer feature fusion module fuses the first image features from upper and lower layers to obtain the second image features; and the depth image restoration module combines the first and second image features to obtain the depth-restored image. The model optimization module constructs a loss function with edge preservation to optimize the depth restoration model for transparent objects.

Images