A method for transparent object depth completion based on dcdu-mamba

Abstract

The application discloses a transparent object depth completion method based on DCDU-Mamba, which comprises the following steps: 1) constructing a DCDU-Mamba network framework; 2) acquiring an RGB image and a depth image of a transparent object; 3) obtaining RGB features and depth features; 4) a skip connection module fuses the RGB features and the depth features through a double-head cross-attention mechanism to obtain a shallow feature map; 5) generating a transparent object depth completion image; and 6) recovering the size of the transparent object depth completion image through a linear embedding layer to make the size of the transparent object depth completion image align with that of an original depth image. The application improves the complex details and edge features of the transparent object, further integrates and optimizes the effective areas of multiple modes in the RGB feature map and the depth feature map, and significantly improves the capturing capacity of the network on the edge region of the transparent object.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, and in particular to a transparent object depth completion method based on DCDU-Mamba. BACKGROUND

[0002] In contemporary medical treatment and industrial production, the wide use of transparent objects has become an inevitable trend. Robots play an important role in modern industry, and in the fields of manufacturing, medical treatment and home service, it is inevitable to grasp high-reflective or transparent objects. However, the current robot grasping method seriously depends on the depth information collected by the RGB-D camera. For non-transparent objects, this information can be obtained by a depth camera. However, for transparent objects such as chemical test tubes, beakers and glass bottles, due to their refractive and high light properties, they do not meet the geometric light path assumption in the classic vision algorithm, which makes it complex to capture the edges and details related to transparent objects. In addition, the lack of texture features makes it challenging to determine the actual shape and position of the transparent object. Therefore, it is difficult for the depth camera to accurately measure the depth of the transparent object and to obtain its coordinates in the three-dimensional space, thereby leading to the failure of grasping the transparent object.

[0003] For the problem of transparent object depth completion, although the neural network based on convolution or Transformer has made certain progress in transparent object depth completion, there are still significant limitations. The local receptive field of the convolution-based model is difficult to capture the global correlation of long-range light propagation, while the Transformer can model global dependence, but it is easy to lose high-frequency details and has high computational cost. Its linear complexity is not suitable for processing high-resolution images, which is particularly important for the task of extracting accurate edge features of transparent objects. The depth map predicted by these methods still has inaccurate depth values and blurred contours. SUMMARY

[0004] The purpose of the present application is to provide a transparent object depth completion method based on DCDU-Mamba, comprising the following steps:

[0005] 1) Constructing a DCDU-Mamba network framework, including an encoder, a skip connection module and a decoder;

[0006] 2) Obtaining the RGB image and depth image of the transparent object;

[0007] 3) The encoder inputs the RGB image and depth image of the transparent object into two independent VMamba feature extraction networks respectively for feature extraction, obtaining RGB features and depth features;

[0008] 4) The skip connection module fuses the RGB features and depth features through a double-headed cross-attention mechanism to obtain a shallow feature map;

[0009] The RGB feature and the depth feature are fused by a decoder to obtain a deep feature map;

[0010] 5) The deep feature map is spliced with a shallow feature map, and up-sampling is realized by a DOSA network in the decoder to generate a transparent object depth completion image;

[0011] 6) The size of the transparent object depth completion image is recovered by a linear embedding layer, so that the transparent object depth completion image is aligned with the size of the original depth image.

[0012] Further, in step 2), the RGB image and the depth image of the transparent object are obtained by a depth camera, and are denoted as I c and I s , respectively.

[0013] Further, in step 3), the two independent VMamba feature extraction networks are mirror images of each other.

[0014] Further, in step 3), the VMamba feature extraction network comprises a Patch Embedding module and a four-stage structure.

[0015] The four-stage structure comprises a VSS module, a VSS and Patch Merging integrated module I, a Patch Merging II, and a Patch Merging III in sequence.

[0016] The Patch Embedding module divides the RGB image or the depth image into non-overlapping image blocks, and projects the image blocks onto a dimension C through a linear embedding layer; the size of each image block is SxSx3; H and W are the height and width of the RGB image.

[0017] The VSS and Patch Merging integrated module I, the Patch Merging II, and the Patch Merging III each comprise two VSS modules and a Patch Merging module.

[0018] The VSS module is used to extract local feature information of the image block, thereby generating an RGB feature or a depth feature.

[0019] The Patch Merging module is used to perform down-sampling operation, reduce the size of the image block, and increase the number of channels.

[0020] Further, in the four-stage structure of the VMamba feature extraction network, the output of each stage is as follows:

[0021]

[0022] where F c is the feature map of RGB image, F s is the feature map of sparse depth image, i∈{1,2,3,4} represents each module in the VMamba four-stage structure.

[0023] Further, in the four-stage structure, the image block size of each stage is C is the dimension.

[0024] Further, in step 4), the step of fusing RGB features and depth features includes:

[0025] 4.1) Given the RGB feature F c and the depth feature F s , multiply them by the weight matrix W q , W k and W v respectively to generate the query matrix (Query, Q), the key matrix (Key, K) and the value matrix (Value, V);

[0026] 4.2) Take the depth feature F s as the query source and the RGB feature F c as the key-value source, that is:

[0027]

[0028] 4.3) Construct the fusion block across RGB features to depth features as shown in the following formula:

[0029]

[0030] 4.4) Take the RGB feature F c as the query source and the depth feature F s as the key-value source, that is:

[0031]

[0032] 4.5) Construct the fusion block across depth features to RGB features as shown in the following formula:

[0033]

[0034] where W q c , W k c and W k c are the RGB feature mapping learnable matrix, W q s , Wq s and W v s is a learnable matrix of depth feature mapping, c represents the RGB feature map, s represents the depth feature map, and i represents the i-th stage of the four stages in the VMamba feature extraction network. and respectively represent cross-attention mechanism operations in different directions, and d is a scaling factor.

[0035] 4.6) The cross-RGB-to-depth fusion block and the cross-RGB-to-depth fusion block are fused together by a concatenation operation and a convolution operation with a 3x3 convolution kernel to obtain a fused shallow feature map That is:

[0036]

[0037] Further, in step 5), the step of generating a transparent object depth completion image comprises:

[0038] 5.1) The channel half-split operation is used to divide the channel of the fused feature into two equal and independent branches, which are respectively denoted as the first branch and the second branch.

[0039] 5.2) The first branch mixes the fusion result from the lower layer decoder and the shallow feature map of the current stage by a convolution operation to obtain the fused feature h;

[0040] wherein the input H i of the first branch of the i-th layer and the output h i are respectively as follows:

[0041] H i = Concat(h0, h1,..., h i-1 )(8)

[0042] h i = ReLU(BN(Conv(Conv(H i ,1×1,k×4),3×3,k)))(9)

[0043] wherein k represents the output channel number of the 1x1 convolution operation.

[0044] 5.3) The second branch mixes the fusion result from the lower layer decoder and the shallow feature map of the current stage by a convolution operation to obtain the fused feature y cat , that is:

[0045] y i = Conv3×3(xi-1 Z, i = 1, 2, …, L (10)

[0046] y cat = Concat (y1, y2, …, y L ) (11)

[0047] where x0 is the input, L is the number of layers, Z is the number of output channels of the i-th convolutional layer; y i is the fusion feature corresponding to the i-th convolutional layer;

[0048] 5.4) Merge the fusion feature y cat and the fusion feature h through the channel splicing operation, and shuffle the merged feature map in the channel dimension to generate a feature fusion image;

[0049] 5.5) Up-sample the feature fusion image to twice the original size through the dynamic sampling layer, thereby generating a transparent object depth completion image.

[0050] Further, the first branch adopts an improved DenseNet network structure, including a plurality of dense connection blocks and a size alignment module;

[0051] Each dense connection block contains a plurality of cascaded convolutional layers;

[0052] The second branch adopts an improved OSA network, including a plurality of convolutional layers.

[0053] Further, step 5.5) of up-sampling the feature fusion image to the current scale through the dynamic sampling layer includes:

[0054] 5.5.1) Initialize to generate a normalized two-dimensional grid, and copy the second dimension of the grid g times, so that the features of each group of the two-dimensional grid share the same sampling set, adapt to the grouping mechanism, and obtain the original sampling grid ε, that is:

[0055]

[0056] 5.5.2) Given an up-sampling scale factor s' and a feature map λ of size C×H×W; map the input to an offset λ1 through a linear layer, and the size is 2g(s') 2 ×H×W; g is the number of offsets;

[0057] 5.5.3) Adjust the offset λ1 to obtain the offset λ2, that is:

[0058] θ2 = 0.25λ1 = 0.25Linear(λ) (13)

[0059] 5.5.4) Reshaping the offset λ2 through the pixel_shuffling function to obtain an offset grid λ3 of size 2g x s'H x s', that is:

[0060] λ3 = Pixel_shuffling(λ2) (14)

[0061] 5.5.5) Constructing the sampling set η, that is:

[0062] η = λ3 + ε (15)

[0063] 5.5.6) Re-sampling the feature map to the output λ' of size C x s'H x s'W using the bilinear interpolation method in the grid_sample function, as follows:

[0064] λ' = Grid_sample(η, λ) (16).

[0065] The technical effects of the present application are self-evident. The present application integrates VMamba with U-Net and introduces the U-Mamba framework. This framework enhances multi-scale characteristics and can effectively capture context information related to transparent objects while preserving local details. In addition, through the selective state space mechanism adopted by VMamba, U-Mamba dynamically focuses on key areas related to transparent objects in the image, which helps to fully understand and estimate transparent objects.

[0066] The present application integrates VMamba with U-Net and introduces the double-headed U-Mamba structure, which enhances the use of multi-scale features, enabling the model to effectively capture context information related to transparent objects while preserving local details. In addition, through the selective state space mechanism of VMamba, the double-headed U-Mamba structure can dynamically focus on key areas related to transparent objects in the image, such as edge refraction and background distortion, while ignoring irrelevant noise.

[0067] The present application uses a double-headed cross-attention mechanism to fuse RGB features and depth features, improving the complex details and edge features of transparent objects, and further integrating and optimizing the effective areas of multi-modal RGB feature maps and depth feature maps, significantly improving the network's ability to capture edge regions of transparent objects.

[0068] The present application uses DOSA network in the decoder, combining low-level and multi-scale fusion features, enhancing multi-level feature representation, and effectively restoring the resolution of transparent object images.

[0069] The present application evaluates the proposed algorithm on the TransCG dataset and compares the test results with other advanced methods, confirming the effectiveness and superiority of the proposed method. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 is a whole network framework based on DCDU-Mamba;

[0071] Figure 2 is a feature extraction network based on VMamba;

[0072] Figure 3 is a feature fusion network based on a double-head cross-attention mechanism;

[0073] Figure 4 is an up-sampling network based on DOSA.

[0074] Figure 5 is a dynamic up-sampling module based on DySample.

[0075] Figure 6 is the depth prediction effect of three depth completion methods in real scenes. DETAILED DESCRIPTION

[0076] The application will be further described below in conjunction with the embodiments, but should not be understood as limiting the above-mentioned subject matter of the application only to the following embodiments. Various substitutions and changes can be made according to ordinary technical knowledge and conventional means in the art without departing from the above-mentioned technical idea of the application, and all should be included in the protection scope of the application.

[0077] Example 1:

[0078] A transparent object depth completion method based on DCDU-Mamba includes the following steps:

[0079] 1) Construct a DCDU-Mamba network framework, including an encoder, a skip connection module and a decoder;

[0080] 2) Obtain the RGB image and depth image of the transparent object;

[0081] 3) The encoder inputs the RGB image and depth image of the transparent object into two independent VMamba feature extraction networks respectively for feature extraction, to obtain RGB features and depth features;

[0082] 4) The skip connection module fuses the RGB features and depth features through a double-head cross-attention mechanism to obtain a shallow feature map;

[0083] The decoder is used to fuse the RGB features and depth features to obtain a deep feature map;

[0084] 5) The deep feature map and the shallow feature map are spliced, and the DOSA network in the decoder is used to realize up-sampling to generate a transparent object depth completion image;

[0085] Specifically, the deep feature map, i.e., the deep high-level fusion feature map, is Figure 1 the feature map at the bottom of the decoder; the shallow feature map is the feature map fused by the double-headed cross attention mechanism in step 4).

[0086] The splicing process here is Figure 1 the process of the dashed box of the decoder part, and the dashed box with "+" is the deep high-level fusion feature map corresponding to the fusion result of the lower decoder;

[0087] The shallow feature map corresponds to the fusion feature of the current stage;

[0088] 6) The size of the transparent object depth completion image is restored through a linear embedding layer, so that the transparent object depth completion image is aligned with the size of the original depth image.

[0089] In step 2), the RGB image and the depth image of the transparent object are obtained by a depth camera, and are denoted as I c and I s , respectively.

[0090] In step 3), the two independent VMamba feature extraction networks are mirror images of each other.

[0091] In step 3), the VMamba feature extraction network includes a Patch Embedding module and a four-stage structure.

[0092] The four-stage structure includes a VSS module, a VSS and Patch Merging integrated module I, a Patch Merging II, and a Patch Merging III in turn.

[0093] The Patch Embedding module divides the RGB image or the depth image into non-overlapping image blocks, and projects these image blocks onto a dimension C through a linear embedding layer; the size of each image block is SxSx3; H and W are the height and width of the RGB image.

[0094] The VSS and Patch Merging integrated module I, the Patch Merging II, and the Patch Merging III each include two VSS modules and a Patch Merging module.

[0095] The VSS module is used to extract local feature information of the image block, thereby generating an RGB feature or a depth feature.

[0096] The Patch Merging module is used to perform a downsampling operation, reduce the size of the image block, and increase the number of channels.

[0097] The output of each stage in the four-stage structure of the VMamba feature extraction network is as follows:

[0098]

[0099] where F c is the feature map of the RGB image, F s is the feature map of the sparse depth image, and i∈{1,2,3,4} represents each module in the four-stage structure of VMamba.

[0100] In the four-stage structure, the image block size of each stage is

[0101] C is the dimension.

[0102] In step 4), the step of fusing RGB features and depth features includes:

[0103] 4.1) Given the RGB feature F c and the depth feature F s , multiply them by the weight matrices W q , W k , and W v , respectively, to generate the query matrix (Query, Q), the key matrix (Key, K), and the value matrix (Value, V);

[0104] 4.2) Take the depth feature F s as the query source and the RGB feature F c as the key-value source, that is:

[0105]

[0106] 4.3) Construct the fusion block across RGB features to depth features, as shown in the following formula:

[0107]

[0108] where d is the scaling factor; the parameter

[0109] 4.4) Take the RGB feature F c as the query source and the depth feature F s as the key-value source, that is:

[0110]

[0111] 4.5) Construct the fusion block across depth features to RGB features, as shown in the following formula:

[0112]

[0113] where W q c , W k c and W k c are the learnable matrices of RGB feature mapping, W q s , W q s and W v s are the learnable matrices of depth feature mapping, c represents the RGB feature map, s represents the depth feature map, and i represents the i-th stage of the four stages in the VMamba feature extraction network. and represent cross-attention mechanism operations in different directions, respectively, and d is a scaling factor.

[0114] 4.6) The cross-RGB-to-depth feature fusion block and the cross-RGB-to-depth feature fusion block are fused together by a concatenation operation and a convolution operation with a 3x3 convolution kernel to obtain the fused shallow feature map i.e.

[0115]

[0116] In step 5), the step of generating a transparent object depth completion image includes:

[0117] 5.1) The channel half-split operation is used to divide the channel of the fused feature into two equal and independent branches, which are denoted as the first branch and the second branch, respectively.

[0118] 5.2) The first branch mixes the fusion result from the lower layer decoder and the shallow feature map of the current stage by a convolution operation to obtain the fused feature h;

[0119] where the input H i of the first branch i-th layer and the output h i are as follows:

[0120] H i = Concat(h0, h1,..., h i-1 )(8)

[0121] h i = ReLU(BN(Conv(Conv(H i ,1x1,kx4),3x3,k)))(9)

[0122] where k represents the output channel number of the 1x1 convolution operation.

[0123] 5.3) The second branch fuses the results from the lower decoder and the shallow feature map of the current stage by convolution operation to obtain the fused feature y cat , i.e.

[0124] y i = Conv3x3(x i-1 , Z), i = 1, 2, …, L (10)

[0125] y cat = Concat(y1, y2, …, y L ) (11)

[0126] wherein x0 is the input, L is the number of layers, Z is the output channel number of the i-th convolution layer; y i is the fused feature corresponding to the i-th convolution layer;

[0127] 5.4) The fused feature y cat and the fused feature h are merged by channel concatenation operation, and the merged feature map is shuffled in the channel dimension to generate a feature fusion image;

[0128] 5.5) The feature fusion image is upsampled to twice the original size by a dynamic sampling layer, thereby generating a transparent object depth completion image.

[0129] The first branch adopts an improved DenseNet network structure, including multiple densely connected blocks and size alignment modules.

[0130] Each densely connected block contains multiple cascaded convolution layers.

[0131] The second branch adopts an improved OSA network, including multiple convolution layers.

[0132] Step 5.5) includes:

[0133] 5.5.1) Initialize a normalized two-dimensional grid network, duplicate the second dimension of the grid g times, so that each group of features of the two-dimensional grid shares the same sampling set, adapt to the grouping mechanism, and obtain the original sampling grid ε, i.e.

[0134]

[0135] 5.5.2) Given an upsampling scale factor s' and a feature map λ of size CxHxW; map the input to an offset λ1 of size 2g(s') 2 xHxW through a linear layer; g is the number of offsets;

[0136] 5.5.3) Adjust the offset λ1 to obtain the offset λ2, that is:

[0137] λ2 = 0.25λ1 = 0.25Linear(λ) (13)

[0138] 5.5.4) Reshape the offset λ2 through the pixel_shuffling function to obtain the offset grid λ3 with a size of 2g×s'H×s', that is:

[0139] λ3 = Pixel_shuffling(λ2) (14)

[0140] 5.5.5) Construct the sampling set η, that is:

[0141] η = λ3 + ε (15)

[0142] 5.5.6) Use the bilinear interpolation method in the grid_sample function to resample the feature map to the output λ' with a size of C×s'H×s', as follows:

[0143] λ' = Grid_sample(η, λ) (16).

[0144] Embodiment 2:

[0145] A transparent object depth completion method based on DCDU-Mamba includes the following steps:

[0146] 1) Construct a DCDU-Mamba network framework, including an encoder, a skip connection module, and a decoder;

[0147] 2) Obtain the RGB image and depth image of the transparent object;

[0148] 3) The encoder inputs the RGB image and depth image of the transparent object into two independent VMamba feature extraction networks for feature extraction, respectively, to obtain RGB features and depth features;

[0149] 4) The skip connection module fuses the RGB features and depth features through a double-headed cross-attention mechanism to obtain a shallow feature map;

[0150] The decoder is used to fuse the RGB features and depth features to obtain a deep feature map;

[0151] 5) The deep feature map and the shallow feature map are spliced, and the DOSA network in the decoder is used for upsampling to generate a transparent object depth completion image;

[0152] 6) Recover the size of the transparent object depth completion image by linear embedding layer, so that the size of the transparent object depth completion image is aligned with the original depth image.

[0153] Embodiment 3:

[0154] A transparent object depth completion method based on DCDU-Mamba, the technical content is the same as embodiment 2, further, in step 2), the RGB image and the depth image of the transparent object are obtained by the depth camera, and are respectively denoted as I c and I s .

[0155] Embodiment 4:

[0156] A transparent object depth completion method based on DCDU-Mamba, the technical content is the same as any one of embodiments 2-3, further, in step 3), the two independent VMamba feature extraction networks are mirror images of each other.

[0157] Embodiment 5:

[0158] A transparent object depth completion method based on DCDU-Mamba, the technical content is the same as any one of embodiments 2-4, further, in step 3), the VMamba feature extraction network comprises a Patch Embedding module and a four-stage structure;

[0159] The four-stage structure comprises a VSS module, a VSS and Patch Merging integrated module I, a Patch Merging II, and a Patch Merging III in turn;

[0160] The Patch Embedding module divides the RGB image or the depth image into non-overlapping image blocks, and projects these image blocks onto a dimension C through a linear embedding layer; the size of each image block is S×S×3;

[0161] The VSS and Patch Merging integrated module I, the Patch Merging II, and the Patch Merging III each comprise two VSS modules and a Patch Merging module;

[0162] The VSS module is used to extract local feature information of the image block, thereby generating RGB features or depth features;

[0163] The Patch Merging module is used to perform downsampling operation, reduce the size of the image block, and increase the number of channels.

[0164] Embodiment 6:

[0165] A method for transparent object depth completion based on DCDU-Mamba, the technical content is the same as any one of embodiments 2-5, further, in the four-stage structure of the VMamba feature extraction network, the output of each stage is as follows:

[0166]

[0167] wherein F c is the feature map of the RGB image, F s is the feature map of the sparse depth map, and i∈{1,2,3,4} respectively represents each module in the four-stage structure of the VMamba.

[0168] Embodiment 7:

[0169] A method for transparent object depth completion based on DCDU-Mamba, the technical content is the same as any one of embodiments 2-6, further, in the four-stage structure, the image block size of each stage is C is the dimension.

[0170] Further, in step 4), the step of fusing the RGB feature and the depth feature includes:

[0171] 4.1) Given the RGB feature F c and the depth feature F s , multiply them by the weight matrices W q , W k and W v respectively to generate the query matrix (Query, Q), the key matrix (Key, K) and the value matrix (Value, V);

[0172] 4.2) Take the depth feature F s as the query source and the RGB feature F c as the key-value source, that is:

[0173]

[0174] 4.3) Construct the fusion block across the RGB feature to the depth feature, as shown in the following formula:

[0175]

[0176] 4.4) Take the RGB feature F c as the query source and the depth feature F s as the key-value source, that is:

[0177]

[0178] 4.5) Construct the fusion block across the depth feature to the RGB feature, as shown in the following formula:

[0179]

[0180] wherein W q c , W k c and W k c are the learnable matrices of RGB feature mapping, W q s , W q s and W v s are the learnable matrices of depth feature mapping, c represents the RGB feature map, s represents the depth feature map, and i represents the i-th stage of the four stages in the VMamba feature extraction network. and represent cross-attention mechanism operations in different directions, respectively, and d is a scaling factor.

[0181] 4.6) The cross-RGB-to-depth feature fusion block and the cross-RGB-to-depth feature fusion block are fused together by a concatenation operation and a convolution operation with a 3x3 convolution kernel to obtain the fused shallow feature map i.e.

[0182]

[0183] Example 8:

[0184] A transparent object depth completion method based on DCDU-Mamba, the technical content of which is the same as any one of embodiments 2-7, further, in step 5), the step of generating a transparent object depth completion image comprises:

[0185] 5.1) Adopting a channel half-split operation, the channel of the fusion feature is divided into two equal and independent branches, which are respectively denoted as the first branch and the second branch.

[0186] 5.2) The first branch mixes the fusion result from the lower layer decoder and the shallow feature map of the current stage by a convolution operation to obtain the fusion feature h;

[0187] wherein the input H i of the first branch i-th layer and the output h i are respectively as follows:

[0188] H i = Concat(h0, h1,..., h i-1 )(8)

[0189] h i= ReLU(BN(Conv(Conv(H i , 1x1, kx4), 3x3, k))(9)

[0190] wherein k represents the output channel number of the 1x1 convolution operation;

[0191] 5.3) The second branch mixes the fusion result from the lower decoder and the shallow feature map of the current stage through convolution operation to obtain the fusion feature y cat i.e.:

[0192] y i = Conv3x3(x i-1 , Z), i = 1, 2, …, L (10)

[0193] y cat = Concat(y1, y2, …, y L )(11) wherein x0 is the input, L is the number of layers, and Z is the output channel number of the i-th convolution layer;

[0194] 5.4) The fusion feature y cat and the fusion feature h are merged through channel concatenation operation, and the merged feature map is shuffled in the channel dimension to generate a feature fusion image;

[0195] 5.5) The feature fusion image is upsampled to twice the original size through a dynamic sampling layer to generate a transparent object depth completion image.

[0196] Embodiment 9:

[0197] A transparent object depth completion method based on DCDU-Mamba, the technical content of which is the same as any one of embodiments 2-8, further, the first branch adopts an improved DenseNet network structure, including multiple dense connection blocks and size alignment modules;

[0198] Each dense connection block contains multiple cascaded convolution layers;

[0199] The second branch adopts an improved OSA network, including multiple convolution layers.

[0200] Embodiment 10:

[0201] A transparent object depth completion method based on DCDU-Mamba, the technical content of which is the same as any one of embodiments 2-9, further, the step of upsampled the feature fusion image to the current scale through a dynamic sampling layer in step 5.5) includes:

[0202] ​5.5.1) Initialize to generate a normalized two-dimensional grid, duplicate the second dimension of the grid g times, so that each group of the two-dimensional grid shares the same sampling set of the features, adapt to the grouping mechanism, and obtain the original sampling grid ε, that is:

[0203]

[0204] 5.5.2) Given the up-sampling scale factor s and the feature map λ with size CxHxW; map the input to the offset λ1 through the linear layer, and the size is 2gs 2 xHxW; g is the number of offsets;

[0205] 5.5.3) Adjust the offset λ1 to obtain the offset λ2, that is:

[0206] λ2=0.25λ1=0.25Linear(λ) (13)

[0207] 5.5.4) Remodel the offset λ2 through the pixel_shuffling function to obtain the offset grid λ3 with size 2gxsHxS, that is:

[0208] λ3=Pixel_shuffling(λ2) (14)

[0209] 5.5.5) Construct the sampling set η, that is:

[0210] η=λ3+ε (15)

[0211] 5.5.6) Use the bilinear interpolation method in the grid_sample function to resample the feature map to the output λ' with size CxsHxS, as follows:

[0212] λ'=Grid_sample(η,λ) (16)。

[0213] Embodiment 11:

[0214] A transparent object depth completion method based on DCDU-Mamba, the content is as follows:

[0215] Due to the reflection and refraction effects of light, it is difficult for a depth camera to accurately obtain the depth information of a transparent object; at the same time, a transparent object usually has no rich surface texture, resulting in unclear image contour based on visual feature matching. In view of these problems, the present application proposes a transparent object depth completion algorithm based on DCDU-Mamba, and the overall network framework of DCDU-Mamba is as Figure 1 shown. The network framework mainly consists of an encoder, a jump connection and a decoder, including the following steps:

[0216] Step 101: The encoder inputs the RGB image and the depth image into two separate VMamba networks for feature extraction.

[0217] Step 102: The skip connection fuses multi-scale features of the RGB image and the depth image through a dual-head cross-attention mechanism to capture multi-level features from coarse to fine.

[0218] Step 103: The deep high-level fusion feature map is concatenated with the shallow feature map obtained in this stage, and upsampling is achieved through the DOSA network in the decoder.

[0219] Step 104: The feature size is recovered through a linear embedding layer to ensure that it maintains a consistent size alignment with the original depth map, thereby forming a depth map that integrates color, texture and depth information.

[0220] like Figure 2 The diagram shown is a flowchart of step 101, which includes the following steps:

[0221] Step 201, the input to the VMamba-based feature extraction network includes RGB image I c and sparse depth map D s Both of these were acquired by a depth camera, among which and To balance the difference in the number of channels between the RGB image and the depth image, the depth image was expanded to a size of H×W×3.

[0222] Step 202 involves simultaneously inputting the RGB image and the expanded depth map into the mirrored VMamba encoder, and extracting RGB and depth features at multiple scales. In one of the VMamba encoders, the Patch Embedding module divides the input RGB image or depth map into non-overlapping segments. Patches, where S×S×3 represents the size of the patches. This represents the number of patches. These patches are then projected onto dimension C through a linear embedding layer.

[0223] Step 203 involves constructing a four-stage structure using multiple VSS modules and a Patch Merging module. The first stage uses only two VSS modules, while the latter three stages employ two VSS modules and a Patch Merging module, with the number of output channels for each stage corresponding to [C, 2C, 4C, 8C]. The VSS modules are responsible for extracting local feature information and focusing on transparent objects. The Patch Merging module performs downsampling, halving the size and doubling the number of channels to capture more features or information. The output of each stage can be represented as:

[0224]

[0225] where F c is the feature map of RGB image, F s is the feature map of sparse depth image, i∈{1,2,3,4} represents each stage of VMamba respectively, and the size of each stage is

[0226] As shown in FIG. 1, it is a flow chart of step 102, comprising the following steps: Figure 3

[0227] Step 301, given RGB feature F c and depth feature F s , multiply them by weight W q , W k and W v matrix respectively to generate query matrix (Query, Q), key matrix (Key, K) and value matrix (Value, V).

[0228] Step 302, take depth feature F s as query source and RGB feature F c as key value source, as shown in the following formula:

[0229]

[0230] Step 303, construct fusion block across RGB feature to depth feature, as shown in the following formula:

[0231]

[0232] Step 304, take RGB feature F c as query source and depth feature F s as key value source, as shown in the following formula:

[0233]

[0234] Step 305, construct fusion block across depth feature to RGB feature, as shown in the following formula:

[0235]

[0236] where W q c , W k c and W k c are the learnable matrix of RGB feature mapping, W q s , W​q s and W v s is a learnable matrix of depth feature mapping, c represents the RGB feature map, s represents the depth feature map, and i represents the i-th stage of the four stages in the VMamba feature extraction network. and respectively represent cross-attention mechanism operations in different directions, and d is a scaling factor.

[0237] Step 306, in order to facilitate the direct transmission of RGB information and depth information and solve the gradient disappearance problem, a skip connection is introduced to enhance the interaction between the input information and the cross-attention mechanism fusion information. Through splicing operation and convolution operation with 3x3 convolution kernel, the two kinds of multi-modal information are further fused together, focusing on the depth discontinuous region of the transparent object edge, and the final fused multi-modal feature is obtained as shown in the following formula:

[0238]

[0239] As shown in Figure 4 , it is a flow chart of step 103, including the following steps:

[0240] Step 401, through 3x3 or 1x1 convolution operation, the fusion results from the lower decoder, the original depth map of the same size and the fusion features of the current stage are mixed.

[0241] Step 402, using channel half-split operation, the channels of the fusion features are divided into two equal and independent branches.

[0242] Step 403, the first branch uses an improved DenseNet network structure, the core of which is composed of multiple dense connection blocks (Dense Block). At the same time, in order to keep the feature map size consistent with the other branch, a size alignment module of 3x3 pointwise convolution operation is designed at the end of the branch. Each dense connection block contains 7 cascaded convolution layers, and the input of each convolution layer is the splicing of the output of all previous layers and the input (i.e. dense connection), which realizes feature reuse, explicitly retains the shallow details such as transparent object edge texture and deep semantic information such as transparent object contour. Therefore, the input H i of the i-th layer is:

[0243] H i = Concat(h0, h1,..., h i-1 )(8)

[0244] where h i is the output of the i-th layer, and Concat(·) is the splicing in the channel dimension.

[0245] For each layer, 1×1 convolution and 3×3 convolution operations are first applied sequentially, followed by batch normalization (BN) and the ReLU activation function to obtain the output h of the i-th layer. i for:

[0246] h i =ReLU(BN(Conv(Conv(H) i ,1×1,k×4),3×3,k))) (9)

[0247] Where k represents the number of output channels of a 1×1 convolution operation, and is a constant, set to 20.

[0248] Step 405: The second path uses an improved OSA (One-Shot Aggregation Network), whose core improves feature reuse efficiency through layer-by-layer connections and feature concatenation.

[0249] Each layer of the network consists of a 3×3 convolutional layer, and the outputs of all layers are finally merged through a concatenation operation. Therefore, the output y of the i-th layer... i for:

[0250] y i =Conv3×3(x i-1 ,Z),i=1,2,...,L (10)

[0251] Where x0 is the input, L is the number of layers, and Z is the number of output channels of the i-th convolutional layer.

[0252] The final output y of the splicing operation cat for:

[0253] y cat =Concat(y1,y2,…,y L (11)

[0254] Step 406: The output feature maps of the two branches in steps 404 and 405 are merged by channel concat operation, and the merged feature maps are shuffled in the channel dimension to improve the network generalization performance.

[0255] Step 407: In order to better adapt to features at different scales, the features are upsampled to the current scale through the DySample (dynamic sampling) layer, which can effectively restore the resolution of the image.

[0256] like Figure 5 The diagram shown is a flowchart of step 407, which includes the following steps:

[0257] Step 501, initialize a normalized two-dimensional grid, duplicate the second dimension of the grid g times, so that each group of features shares the same sampling set, adapt to the grouping mechanism, and obtain the original sampling grid ε as:

[0258]

[0259] Step 502, given an up-sampling scale factor s and a feature map λ of size C×H×W. Through a linear layer, the input is mapped to an offset λ1, which is 2gs 2 ×H×W in size, where g is the number of groups into which the feature map is divided along the channel dimension, and g groups of offsets are generated. Multiply the offset λ1 by 0.25 to satisfy the theoretical margin condition between overlapping and non-overlapping, then the output offset λ2 is:

[0260] λ2=0.25λ1=0.25Linear(λ) (13)

[0261] Step 503, the offset λ2 is reshaped into the final output offset grid λ3 through the pixel_shuffling function, which is 2g×sH×sW in size, then the final output offset grid λ3 is:

[0262] λ3=Pixel_shuffling(λ2) (14)

[0263] Step 504, the sampling set η is the sum of the offset grid λ3 and the original sampling grid ε, that is:

[0264] η=λ3+ε (15)

[0265] Step 505, use the bilinear interpolation method in the grid_sample function to resample the feature map to an output λ' of size C×sH×sW, as follows:

[0266] λ'=Grid_sample(η,λ) (16)。

[0267] (1) Dataset

[0268] To evaluate the performance of DCDU-Mamba compared with other methods, experiments were conducted on the TransCG dataset. TransCG is the first large-scale, real transparent object dataset, which was taken from different angles of 130 scenes by a camera connected to a mechanical arm, totaling 57715 RGB-D images.

[0269] (2) Loss function

[0270] Depth completion of transparent objects is considered as a dense regression task, which aims to predict a complete depth map from an RGB image and a sparse depth map. In view of the optical characteristics of refraction and reflection of transparent objects and the depth camera noise problem, a joint loss function framework is proposed, which divides the loss into accuracy loss and smoothness loss, respectively optimizing the geometric accuracy and spatial continuity of the predicted transparent object.

[0271] The accuracy loss aims to minimize the difference between the predicted depth and the true value, so that the model converges to a high-precision depth map, and is defined as follows:

[0272]

[0273] Where L c is the joint loss of L1 and L2, which penalizes depth inaccuracy; α is the balance coefficient of L1 loss and L2 loss, D pre represents the predicted depth map, D gt represents the true depth map, and V represents the pixel set of the depth map under the mask area, and |V| represents the size of set V.

[0274] The smoothness loss L s is composed of the cosine distance between the surface normal of the predicted depth map D pre and the true depth map, which penalizes the smoothness, and is defined as follows:

[0275]

[0276] Where and are the gradient vectors along the height and width axes of the predicted depth map, respectively, and are the gradient vectors along the height and width axes of the true depth map, respectively. The smoothness loss L s is also calculated on the mask of the transparent area.

[0277] The total loss function is defined as the weighted sum of the accuracy loss and the smoothness loss:

[0278]

[0279] Where β is the weight coefficient, balancing the accuracy loss and the smoothness loss.

[0280] (3) Evaluation index

[0281] To comprehensively evaluate the performance of the DCDU-Mamba model, the following four key evaluation metrics are adopted: Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Relative Error (REL), and Threshold δ (where δ is set to 1.05, 1.10, and 1.25). Correspondingly, these metrics are calculated on the transparent region mask.

[0282] The Mean Absolute Error (MAE) is the average of the absolute errors between the predicted depth map and the actual depth map, and its calculation formula is:

[0283]

[0284] The Root Mean Square Error (RMSE) is the square root of the ratio of the square of the deviation between the predicted depth map and the actual depth map to the size of the effective depth image pixel set. Its calculation formula is:

[0285]

[0286] The Mean Absolute Relative Error (REL) is the average of the ratio of the absolute error to the actual depth map, and its calculation formula is:

[0287]

[0288] where represents the predicted depth map, represents the true depth map, V represents the pixel set of the depth map under the mask region, and |V| represents the size of the set V.

[0289] The threshold δ is defined as the predicted depth satisfying the following conditions:

[0290]

[0291] where, and represent the pixel values of the predicted depth map and the corresponding pixel values of the actual depth map, respectively, and δ is set to 1.05, 1.10, and 1.25, respectively.

[0292] (4) Parameter Settings

[0293] To update the network parameters and accelerate the convergence speed of the model, the Adam optimizer is used to train the DCDU-Mamba model. The basic model is built using the DFNet network framework. The initial learning rate is set to 0.001, and is reduced to 0.2 times of the original value after every 10 training cycles (Epoch). The weighting coefficients α and β in the loss function are set to 0.01 and 0.001, respectively. For all experiments, the training cycle (Epoch) and batch size are set to 40 and 8, respectively, and the resolution of all input RGB images and depth images is unified to 320x240, which is used for training and testing. In order to prevent overfitting, random flipping, rotation, noise addition and color transformation in the HLS color space are used for data enhancement during the training process. For the VMamba-based feature extraction network, the encoder weights are initialized using the VMamba-T weights pre-trained on ImageNet-1k.

[0294] (5) Comparative experiment and analysis

[0295] The comparison results of the proposed method with other advanced methods on the TransCG dataset are shown in Table 1, including the ClearGrasp algorithm based on three Deeplabv3+ models, the LIDF-Refine algorithm based on local implicit neural representation, the DFNet algorithm based on dense block network, the FDCT algorithm based on improved DFNet, and the TODE-Trans algorithm based on Swin Transformer feature extraction.

[0296] The results in Table 1 demonstrate that the proposed method not only significantly outperforms the DFNet-based model, but also outperforms many state-of-the-art techniques. In particular, the DCDU-Mamba model achieves 91.25 at the threshold δ 1.05 of 0.2, which is 0.82 higher than the best-performing TODE-Trans model, with an improvement of 0.82%. It is worth noting that, as shown in Table 1, compared with TODE-Trans, the DCDU-Mamba model has similar error indicators such as RMSE, REL and MAE.

[0297] Table 1 Comparison results of the proposed method with other advanced methods on the TransCG dataset

[0298]

[0299] Different depth completion methods are applied to estimate the depth of transparent objects, from left to right using DFNet, TODE-Trans and the proposed DCDU-Mamba for depth estimation, as shown in Figure 6The depth-completed depth map can better reflect the contour of the original object, so as to prove the quality of the depth completion algorithm. The accuracy of the DFNet method is lower than that of the other two methods, the predicted depth map of the TODE-Trans method has some wrong pixel points, and the predicted depth map of the DCDU-Mamba is smoother and can better reflect the contour of the transparent object.

Claims

1. A method for transparent object depth completion based on DCDU-Mamba, characterized in that, The method comprises the following steps: 1) constructing a DCDU-Mamba network framework, including an encoder, a skip connection module and a decoder; the encoder adopts a VMamba-based feature extraction network, the skip connection module adopts a multi-modal fusion network based on a double-head cross attention mechanism, and the decoder adopts a DOSA-based up-sampling network, including a convolutional layer, a fully connected layer, an OSA network and an up-sampler; 2) obtaining an RGB image and a depth image of a transparent object; 3) the encoder inputs the RGB image and the depth image of the transparent object into two independent VMamba feature extraction networks respectively for feature extraction, to obtain RGB features and depth features; 4) the skip connection module fuses the RGB features and the depth features through a double-head cross attention mechanism, to obtain a shallow feature map; the decoder fuses the RGB features and the depth features, to obtain a deep feature map; 5) the deep feature map is spliced with the shallow feature map, and the DOSA network in the decoder is used for up-sampling, to generate a transparent object depth completion image; 6) a linear embedding layer is used to restore the size of the transparent object depth completion image, so that the transparent object depth completion image is aligned with the original depth image in size; In step 3), the VMamba feature extraction network comprises a Patch Embedding module and a four-stage structure; the four-stage structure comprises a VSS module, a VSS and Patch Merging integrated module I, a Patch Merging II and a Patch Merging III in sequence; The Patch Embedding module divides the RGB image or the depth image into non-overlapping image patches, projects the image patches onto a dimension C through a linear embedding layer; the size of each image patch is SxSx3; H and W are the height and width of the RGB image; the VSS and Patch Merging integrated module I, the Patch Merging II and the Patch Merging III each comprise two VSS modules and a Patch Merging module; the VSS module is used for extracting local feature information of an image block, thereby generating RGB features or depth features; the Patch Merging module is used for performing a down-sampling operation, reducing the size of the image block and increasing the number of channels.

2. The DCDU-Mamba based transparent object depth completion method of claim 1, wherein, In step 2), the RGB image and the depth image of the transparent object are acquired by the depth camera, denoted as I c and I s , respectively.

3. The DCDU-Mamba based transparent object depth completion method of claim 1, wherein, In step 3), the two independent VMamba feature extraction networks are mirror images of each other.

4. The DCDU-Mamba based transparent object depth completion method of claim 1, wherein, In the four-stage structure of the VMamba feature extraction network, the outputs of each stage are as follows: where F c is the feature map of RGB image, F s is the feature map of sparse depth image, i∈{1,2,3,4} represents each module in the VMamba four-stage structure; I c and I s are RGB image and depth image.

5. The DCDU-Mamba based transparent object depth completion method of claim 4, wherein, In the four-stage structure, the image block size of each stage is 6. The DCDU-Mamba based transparent object depth completion method of claim 1, wherein, In step 4), the step of fusing the RGB features and the depth features comprises: 4.1) Given RGB features F c and depth features F s , multiply them by weight matrices W q , W k and W v , respectively, to generate query matrix (Query, Q), key matrix (Key, K) and value matrix (Value, V); 4.2) The deep features F s As a query source, the RGB features F c As a key-value source, i.e.: In the formula, is a weight; 4.3) constructing a fusion block from RGB features to depth features, as shown in the following formula: wherein d is a scaling factor; 4.4) RGB features F c As a source of queries, depth features F s As a source of key-values, i.e.: 4.5) constructing a fusion block from depth features to RGB features, as shown in the following formula: wherein W q c , W k c and W k c are learnable matrices for RGB feature mapping, W q s , W q s and W v s are learnable matrices for depth feature mapping, c represents the RGB feature map, s represents the depth feature map, and i represents the i-th stage of the four stages in the VMamba feature extraction network. and represent cross-attention mechanism operations in different directions, respectively, and d is a scaling factor. 4.6) Fuse the cross-RGB-to-depth feature fusion block and the cross-RGB-to-depth feature fusion block together by concatenation operation and convolution operation with 3x3 convolution kernel to get the fused shallow feature map That is:

7. The DCDU-Mamba based transparent object depth completion method of claim 1, wherein, In step 5), the step of generating the transparent object depth completion image comprises: 5.1) using a channel half-split operation to divide the channels of the fused features into two equal and independent branches, which are denoted as a first branch and a second branch respectively; 5.2) the first branch fuses the results from the lower decoder and the shallow feature maps of the current stage by a convolution operation mixing is performed to obtain the fused features h; wherein the input H i , output h i of the first branch i-th layer are respectively as follows: H i = Concat(h0, h1, …, h i-1 )(8)h i = ReLU(BN(Conv(Conv(H i , 1x1, kx4), 3x3, k))(9) wherein k represents the output channel number of a 1×1 convolution operation; 5.3) The second branch fuses the results from the lower decoder and the shallow feature maps of the current stage by a convolution operation Mixing is performed to obtain the fused features y cat That is: y i = Conv3x3(x i-1 , z), i = 1, 2,..., L (10)y cat = Concat(y1, y2,..., y L ) (11) In the formula, x0 is an input, L is a number of layers, Z is an output channel number of an i-th convolutional layer; y i is a fusion feature corresponding to the i-th convolutional layer. 5.4) Merge the fusion features y cat and h by channel concatenation operation, and shuffle the merged feature map in the channel dimension to generate a feature fusion image; 5.5) using a dynamic sampling layer to up-sample the feature fusion image to twice the original size, thereby generating the transparent object depth completion image.

8. The method of claim 7, wherein the DCDF-Mamba based transparent object depth completion method is characterized by, The first branch adopts an improved DenseNet network structure, including multiple dense connection blocks and a size alignment module; Each dense connection block contains multiple cascaded convolution layers; The second branch adopts an improved OSA network, including multiple convolution layers.

9. The DCDU-Mamba based transparent object depth completion method of claim 7, wherein, Step 5.5) includes: 5.5.1) initialize a normalized two-dimensional grid network, duplicate the second dimension of the grid g times, so that the features of each group of the two-dimensional grid share the same sampling set, adapt to the grouping mechanism, and obtain an original sampling grid ε, that is: 5.5.2) Given an upsampling scale factor s' and a feature map λ of size C x H x W; map the input to offsets λ1 of size 2g(s') x H x W through a linear layer; g is the number of offsets; 2 ×H×W; g is the number of offsets; 5.5.3) adjust the offset λ1 to obtain the offset λ2, that is: λ2=0.25λ1=0.25Linear(λ) (13) 5.5.4) reshape the offset λ2 through the pixel_shuffling function to obtain an offset grid λ3 with a size of 2g×s'H×s', that is: λ3=Pixel_shuffling(λ2) (14) 5.5.5) construct a sampling set η, that is: η=λ3+ε (15) 5.5.6) use the bilinear interpolation method in the grid_sample function to resample the feature map to an output λ' with a size of C×s'H×s'W, as follows: λ'=Grid_sample(η,λ) (16)。

Images