Depth completion method and apparatus, electronic device, and storage medium

Abstract

The application provides a depth completion method and device, electronic equipment and storage medium, which are applied to the technical field of data processing. The method comprises the following steps: acquiring an RGB image, a sparse depth map and a sparse normal map of the sparse depth map; inputting the RGB image into a Transformer model to obtain a first dense depth map and a first confidence map, inputting the sparse depth map into the Transformer model to obtain a second dense depth map and a second confidence map, and inputting the sparse normal map into the Transformer model to obtain a third dense depth map and a third confidence map; and performing fusion processing on the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map and the third confidence map to obtain a target dense depth map.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a depth completion method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of technologies such as autonomous driving, 3D reconstruction, augmented reality, and pose estimation, the requirements for the completeness and accuracy of depth information have also increased accordingly. However, depth maps directly obtained from depth sensors are too sparse and contain a lot of noise to be directly applied to subsequent tasks. Recovering dense depth maps from sparse depth maps is the main task of depth completion. The generated high-precision depth maps can be better applied to subsequent tasks such as autonomous driving, scene understanding, 3D reconstruction, pose estimation, and augmented reality, thus demonstrating practical value.

[0003] In indoor environments, without the influence of strong light, depth maps obtained by structured light sensors typically contain approximately 20% missing values. Depth loss can be categorized into two types: the first is randomly distributed holes on the object's surface, where missing values ​​can be predicted and estimated based on valid depth values ​​in their neighborhood; the second is larger, continuous patches of missing depth, usually appearing at the boundary between the foreground and background of an image. Therefore, early depth completion tasks generally employed traditional image processing methods to process relatively dense depth data, such as hole-filling methods based on a joint bilateral filtering framework, simultaneously utilizing spatial and temporal information to process pixels with missing depth values; or using probabilistic models to capture various uncertainties in the depth measurement process of the structured light system, effectively distinguishing foreground and background content.

[0004] However, traditional image processing methods can only handle relatively dense depth data. For outdoor tasks, current depth data is mostly acquired by LiDAR, which has too low spatial resolution for the depth information it collects, making it difficult to obtain satisfactory results using traditional image processing methods. Summary of the Invention

[0005] This application provides a depth completion method, apparatus, electronic device, and storage medium to solve the problem that traditional image processing methods cannot process depth images with too low resolution.

[0006] This application provides a depth completion method, comprising: acquiring an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map; obtaining a first dense depth map and a first confidence map by inputting the RGB image into a Transformer model, obtaining a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model, and obtaining a third dense depth map and a third confidence map by inputting the sparse normal map into the Transformer model; obtaining a target dense depth map by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map; wherein the image density of the target dense depth map is higher than the image density of the sparse depth map, and the Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map.

[0007] According to the depth completion method provided in this application, after acquiring an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map, the method further includes: extracting shallow features of the RGB image, shallow features of the sparse depth map, and shallow features of the sparse normal map, respectively; performing a linear transformation on the shallow features of the RGB image to obtain a first input vector matching the Transformer model; performing a linear transformation on the shallow features of the sparse depth map to obtain a second input vector matching the Transformer model; and performing a linear transformation on the shallow features of the sparse normal map to obtain a third input vector matching the Transformer model.

[0008] According to the deep completion method provided in this application, the Transformer model includes three prediction branches, each of which includes an encoder network and a decoder network. The encoder network includes three self-attention modules and an image patch merging layer. The decoder network includes sub-pixel convolutional layers and two self-attention modules. The self-attention modules are used to extract bottleneck features. The image patch merging layer is used to adjust the number of channels and downsample the feature map. The sub-pixel convolutional layers are used to upsample and reconstruct features.

[0009] According to the deep completion method provided in this application, the combination of the guided attention module and the self-attention module includes: parallel pre-guided guidance, parallel post-guided guidance, serial pre-guided guidance, and serial post-guided guidance.

[0010] According to the deep completion method provided in this application, the guidance structure of the guidance attention module includes: a parallel guidance structure between encoders, a serial guidance structure between encoders, a parallel guidance structure across encoders and decoders, and a serial guidance structure across encoders and decoders.

[0011] According to the depth completion method provided in this application, the method involves inputting the RGB image into a Transformer model to obtain a first dense depth map and a first confidence map, inputting the sparse depth map into the Transformer model to obtain a second dense depth map and a second confidence map, and inputting the sparse normal map into the Transformer model to obtain a third dense depth map and a third confidence map. The method includes: inputting the RGB image into the Transformer model to obtain a first feature map; performing feature reconstruction and size restoration on the first feature map to obtain the first dense depth map and the first confidence map; inputting the sparse depth map into the Transformer model to obtain a second feature map; performing feature reconstruction and size restoration on the second feature map to obtain the second dense depth map and the second confidence map; and inputting the sparse normal map into the Transformer model to obtain a third feature map; performing feature reconstruction and size restoration on the third feature map to obtain the third dense depth map and the third confidence map.

[0012] According to the depth completion method provided in this application, the step of obtaining a target dense depth map by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map includes:

[0013] According to the formula Determine the target dense depth map;

[0014] in, This represents the depth value of the first dense depth map. This represents the depth value of the second dense depth map. C represents the depth value of the third dense depth map. rgb This represents the first confidence level plot, C. d This represents the second confidence plot, C. n This represents the third confidence level plot.

[0015] This application also provides a depth completion device, including: an acquisition module and a processing module; the acquisition module is used to acquire an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map; the processing module is used to obtain a first dense depth map and a first confidence map by inputting the RGB image into a Transformer model, to obtain a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model, and to obtain a third dense depth map and a third confidence map by inputting the sparse normal map into the Transformer model; and to obtain a target dense depth map by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map; wherein the image density of the target dense depth map is higher than the image density of the sparse depth map, and the Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map.

[0016] According to the depth completion device provided in this application, the processing module is further configured to: extract shallow features of the RGB image, shallow features of the sparse depth map, and shallow features of the sparse normal map, respectively; perform a linear transformation on the shallow features of the RGB image to obtain a first input vector matching the Transformer model; perform a linear transformation on the shallow features of the sparse depth map to obtain a second input vector matching the Transformer model; and perform a linear transformation on the shallow features of the sparse normal map to obtain a third input vector matching the Transformer model.

[0017] According to the present application, a deep completion device is provided, wherein the Transformer model includes three prediction branches, each prediction branch including an encoder network and a decoder network. The encoder network includes three layers of self-attention modules and an image patch merging layer. The decoder network includes sub-pixel convolutional layers and two layers of self-attention modules. The self-attention modules are used to extract bottleneck features. The image patch merging layer is used to adjust the number of channels and downsample the feature map. The sub-pixel convolutional layers are used to upsample and reconstruct features.

[0018] According to the depth completion device provided in this application, the combination of the guided attention module and the self-attention module includes: parallel pre-guided guidance, parallel post-guided guidance, serial pre-guided guidance, and serial post-guided guidance.

[0019] According to the depth completion device provided in this application, the guidance structure of the guidance attention module includes: an inter-encoder parallel guidance structure, an inter-encoder serial guidance structure, a cross-encoder parallel guidance structure, and a cross-encoder serial guidance structure.

[0020] According to the depth completion device provided in this application, the processing module is configured to: input the RGB image into the Transformer model to obtain a first feature map; perform feature reconstruction and size restoration on the first feature map to obtain a first dense depth map and a first confidence map; input the sparse depth map into the Transformer model to obtain a second feature map; perform feature reconstruction and size restoration on the second feature map to obtain a second dense depth map and a second confidence map; input the sparse normal map into the Transformer model to obtain a third feature map; perform feature reconstruction and size restoration on the third feature map to obtain a third dense depth map and a third confidence map.

[0021] According to the depth completion device provided in this application, the processing module is used for:

[0022] According to the formula Determine the target dense depth map;

[0023] in, This represents the depth value of the first dense depth map. This represents the depth value of the second dense depth map. C represents the depth value of the third dense depth map. rgb This represents the first confidence level plot, C. d This represents the second confidence plot, C. n This represents the third confidence level plot.

[0024] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described deep completion methods.

[0025] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the depth completion method as described above.

[0026] The depth completion method, apparatus, electronic device, and storage medium provided in this application can acquire an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map; obtain a first dense depth map and a first confidence map by inputting the RGB image into a Transformer model; obtain a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model; obtain a third dense depth map and a third confidence map by inputting the sparse normal map into the Transformer model; obtain a target dense depth map by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map; wherein the image density of the target dense depth map is higher than the image density of the sparse depth map; the Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map. This scheme can introduce the normal information of the sparse normal map into the depth completion task, and provide the model with more three-dimensional spatial information through the normal. Therefore, it can eliminate depth noise and error to a certain extent, thereby improving the accuracy and quality of the output dense depth. Since RGB image features can be used as a guide term to achieve deep fusion of three-modal features by guiding the attention module, the prediction results of sparse depth and normal branches can be optimized. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the deep completion method provided in this application;

[0029] Figure 2 This is a schematic diagram of the structure of the deep completion network provided in this application;

[0030] Figure 3 This is a schematic diagram of the image block embedding layer provided in this application;

[0031] Figure 4 This is a schematic diagram of the structure of the self-attention module provided in this application;

[0032] Figure 5 This is a schematic diagram of the image block merging layer processing flow provided in this application;

[0033] Figure 6 This is a schematic diagram of the subpixel convolutional layer processing flow provided in this application;

[0034] Figure 7 This is a schematic diagram of the structure of the attention guidance module provided in this application;

[0035] Figure 8 This is one of the schematic diagrams illustrating the combination of the guided attention module and the self-attention module provided in this application;

[0036] Figure 9 This is the second schematic diagram of the combination of the guided attention module and the self-attention module provided in this application;

[0037] Figure 10 This is the third schematic diagram of the combination of the guided attention module and the self-attention module provided in this application;

[0038] Figure 11 This is the fourth schematic diagram of the combination of the guided attention module and the self-attention module provided in this application;

[0039] Figure 12 This is one of the schematic diagrams of the guidance structure of the guidance attention module provided in this application;

[0040] Figure 13 This is the second schematic diagram of the guidance structure of the guidance attention module provided in this application;

[0041] Figure 14 This is the third schematic diagram of the guidance structure of the guidance attention module provided in this application;

[0042] Figure 15 This is the fourth schematic diagram of the guidance structure of the guidance attention module provided in this application;

[0043] Figure 16 This is a schematic diagram of the depth completion device provided in this application;

[0044] Figure 17 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0047] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0048] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0049] This application describes some exemplary embodiments for illustrative purposes. It should be understood that this application may be implemented in other ways not specifically shown in the accompanying drawings.

[0050] The above implementation method will be described in detail below with reference to specific embodiments and accompanying drawings.

[0051] like Figure 1 As shown, this application provides a depth completion method that can be applied to a depth completion device. The depth completion method may include steps S101-S103:

[0052] S101, The depth completion device acquires an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map.

[0053] Optionally, such as Figure 2As shown, this application provides a depth completion network, which may include an image patch embedding layer, a Transformer model, and an image reconstruction layer. The depth completion device can use this network to perform depth completion of sparse depth maps.

[0054] Optionally, after acquiring the RGB image, the sparse depth map, and the sparse normal map of the sparse depth map, the depth completion device can extract the shallow features of the RGB image, the shallow features of the sparse depth map, and the shallow features of the sparse normal map, respectively; perform a linear transformation on the shallow features of the RGB image to obtain a first input vector matching the Transformer model; perform a linear transformation on the shallow features of the sparse depth map to obtain a second input vector matching the Transformer model; and perform a linear transformation on the shallow features of the sparse normal map to obtain a third input vector matching the Transformer model.

[0055] Specifically, such as Figure 2 As shown, in the RGB image, sparse depth map, and sparse normal vector... Figure 3 Before modal information is input into the Transformer model, the depth completion device can convert the original input image into a form suitable for processing by the Transformer model through an image patch embedding layer. For example... Figure 3 As shown, the image patch embedding layer includes convolutional layers, multiple residual blocks, and a linear transformation layer. The depth completion device extracts shallow features from the RGB image, sparse depth map, and sparse normal map through the convolutional layers and multiple residual blocks to obtain a feature map. The size of this feature map is reduced to one-quarter of the original input image size. Then, the extracted shallow features are flattened by the linear transformation layer to obtain a one-dimensional input vector suitable for input into the Transformer model.

[0056] It should be noted that most current visual Transformer methods convert the original input image into image patches using large strides and convolution kernels, and then obtain one-dimensional image patch embeddings through linear transformations. However, while the method of dividing the image into patches using only convolution operations is simple, it makes model convergence difficult. Therefore, this application adds multiple residual blocks, which can accelerate model convergence and improve training efficiency.

[0057] For example, such as Figure 3 As shown, taking an RGB image as an example, the depth completion device can extract shallow features of the RGB image through convolutional layers and multiple residual blocks to obtain the feature map of the RGB image. Then, the feature map is flattened through a linear transformation layer to generate a first input vector that matches the Transformer model.

[0058] S102. The depth completion device obtains a first dense depth map and a first confidence map by inputting the RGB image into the Transformer model, obtains a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model, and obtains a third dense depth map and a third confidence map by inputting the sparse normal map into the Transformer model.

[0059] The Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map.

[0060] Optionally, such as Figure 2 As shown, the Transformer model described above can also include three prediction branches, each of which includes an encoder network and a decoder network. The encoder network can include three layers of self-attention modules and an image patch merging layer. The decoder network has a symmetrical structure with the encoder network. The decoder network can include sub-pixel convolutional layers and two layers of self-attention modules. The self-attention modules are used to extract bottleneck features. The image patch merging layer is used to adjust the number of channels and downsample the feature map. The sub-pixel convolutional layers are used to upsample and reconstruct features.

[0061] The following sections provide a detailed description of the self-attention module, image patch merging layer, subpixel convolutional layer, and guided attention module in the Transformer model.

[0062] I. Self-Attention Module

[0063] like Figure 4 As shown, this embodiment employs the multi-head self-attention mechanism based on shifted windows proposed in Swin Transformer, performing self-attention within non-overlapping shifted local windows. In the self-attention module, the input feature map (i.e., one of the first, second, and third input vectors) first undergoes Layer Normalization (LN), and is then divided into layers by a specified window size. The self-attention for each window is then calculated. The generated attention is passed through an LN layer and a feedforward layer to obtain the output feature map. The feedforward operation consists of two linear layers, with a depth-wise convolution (DWC) layer added between them. The purpose of DWC is to better model the local context within the self-attention module, which helps to better preserve edge and image detail features.

[0064] Optionally, the formula for calculating self-attention is as follows:

[0065]

[0066] Where Q, K, and V represent the query, key, and value matrices, respectively, all of which are obtained by projecting the input window through different matrices. B represents the relative position offset, and d represents the dimension of the input window.

[0067] II. Image Patch Merging Layer

[0068] like Figure 5 As shown, the image patch merging layer functions similarly to pooling. Elements are selected at intervals of 2 in both the row and column directions to form new image patches. These patches are then connected by channels and stretched and normalized. At this point, the channel dimension becomes four times the original. Finally, a fully connected layer adjusts the channel dimension back to twice its original size. The purpose of the image patch merging layer is to downsample the feature map, reducing its resolution, while simultaneously adjusting the number of channels to achieve a hierarchical design.

[0069] III. Subpixel Convolutional Layer

[0070] like Figure 6 As shown, for a low-resolution input image of size H×W, the sub-pixel convolutional layer can first extract its feature map through multiple convolutions, with the feature map size after each convolution layer being consistent with the original input image size. The output size of the last convolutional operation is r. 2 The feature map is defined as ×H×W, where r is the upsampling factor, i.e., the desired magnification of the image. After the convolution operation, the r values ​​of each pixel in the feature map are... 2 The channels are rearranged and combined into an r×r region, corresponding to each r×r sub-region in the high-resolution image, thus obtaining a high-resolution output image magnified r times.

[0071] It should be noted that, compared with the conventional upsampling method through interpolation, in the process of upsampling through subpixel convolution in this application, the interpolation function is implicitly contained in multiple convolutional layers and can be automatically learned. Furthermore, since the image size transformation is only performed in the last layer, and the preceding convolution operations are all performed on the low-resolution image, the running efficiency is improved and the upsampling results are optimized.

[0072] IV. Attention Guiding Module

[0073] like Figure 7 The diagram shown is a structural schematic of the attention guidance module. The calculation formula for this attention guidance module is as follows:

[0074]

[0075] Among them, K G V GQ is the key matrix and value matrix obtained by performing layer normalization, windowing, and linear projection operations on the features of the guiding mode. N The query matrix is ​​obtained by performing a series of preprocessing operations on the features of the guided modality.

[0076] The guided attention mechanism focuses on V through the cross-modal similarity between Q and K. RGB This enables cross-modal guidance, explicitly transferring information from the RGB image branches to the depth and normal branches.

[0077] It should be noted that, apart from the attention calculation formula, the guided attention module operates the same as the self-attention module.

[0078] Optionally, the combination of the guided attention module and the self-attention module can include: parallel pre-guided guidance, parallel post-guided guidance, serial pre-guided guidance, and serial post-guided guidance.

[0079] Specifically, in all four combination methods, RGB features serve as guiding terms. First, a self-attention module captures the correlations within the data. Then, for parallel pre-leading methods, such as... Figure 8 As shown, two guided attention modules can be placed before the self-attention modules of the depth and normal modes to guide the depth and normal modes respectively; for parallel post-guided attention, such as Figure 9 As shown, the depth and normal features are consistent with the RGB features. First, a self-attention module is used, followed by a guided attention module to guide the depth and normal features; for sequential pre-attention, such as... Figure 10 As shown, a guided attention module between RGB and normal modes is first set up before the self-attention calculation of the normal features. Then, the normal features capture the correlation within the data through the self-attention module, and then the depth is guided by the guided attention module between normal and depth. For serial post-guided attention, such as Figure 11 As shown, the trimodal approach first uses a self-attention module to guide the normal direction through RGB features, and then uses the output of the attention module to guide the depth features through RGB-normal direction.

[0080] Optionally, the guidance structure of the above-mentioned guidance attention module includes: a parallel guidance structure between encoders, a serial guidance structure between encoders, a parallel guidance structure across encoders and decoders, and a serial guidance structure across encoders and decoders.

[0081] Specifically, such as Figure 12 As shown, the guidance structure of the attention module is a parallel guidance structure between encoders, such as... Figure 13 As shown, the guidance structure of the attention module is a serial guidance structure between encoders, such as... Figure 14As shown, the guidance structure for the attention module is a parallel guidance structure across the codec, such as... Figure 15 As shown, the guidance structure of the guidance attention module is a cross-codec serial guidance structure. Both the inter-encoder parallel guidance structure and the cross-encoder parallel guidance structure are parallel guidance structures. In the parallel guidance structure, the guidance attention modules between RGB-depth and RGB-normal simultaneously act on multiple branches, with the RGB branch simultaneously providing guidance for both the depth and normal branches. Both the inter-encoder serial guidance structure and the cross-encoder serial guidance structure are serial guidance structures. Serial guidance first acts on the RGB and normal branches; RGB features first guide the normal branch, and then the normal branch features guide the depth branch. The parallel guidance structure applies to parallel pre-guided and parallel post-guided guidance attention modules; the serial guidance structure applies to serial pre-guided and serial post-guided guidance attention modules. Both the inter-encoder parallel guidance structure and the inter-encoder serial guidance structure are inter-encoder guidance structures, meaning that features generated by the encoder provide guidance for other branches; the cross-encoder parallel guidance structure and the cross-encoder serial guidance structure are cross-encoder guidance structures, meaning that features generated by the decoder provide guidance for other branches.

[0082] Let the continuous depth features in the depth branch encoder structure be... The continuous depth features in the decoder are The continuous normal features in the normal branch encoder structure are: Continuous features in the decoder The continuous RGB features in the RGB branch encoder structure are Continuous features in the decoder L represents the number of layers in both the encoder and decoder, and both have the same number of layers.

[0083] In the parallel bootstrapping structure across codecs, the bootstrapping attention module between the RGB and depth branches can be represented as:

[0084]

[0085] The guided attention module between the RGB and normal branches can be represented as:

[0086]

[0087] Where t∈{1,2,…L}.

[0088] In the serial bootstrapping structure across codecs, the bootstrapping attention module between the RGB and depth branches can be represented as:

[0089]

[0090] The guided attention module between the RGB and normal branches can be represented as:

[0091]

[0092] Similarly, the formula for the guided attention module between encoders only requires replacing the first input item of the guided attention module with the corresponding encoder feature, which will not be elaborated here.

[0093] It should be noted that ablation experiments show that, under the same conditions, the prediction results of the combination of the guiding attention module and the self-attention module in a serial structure are better than those of the combination in a parallel structure; the prediction results of the guiding attention module with a cross-codec guiding structure are better than those with a encoder-inter-encoder guiding structure.

[0094] Optionally, the depth completion device can input the RGB image into the Transformer model to obtain a first feature map, and then perform feature reconstruction and size restoration on the first feature map through an image reconstruction layer to obtain a first dense depth map and a first confidence map; input the sparse depth map into the Transformer model to obtain a second feature map, and then perform feature reconstruction and size restoration on the second feature map through an image reconstruction layer to obtain a second dense depth map and a second confidence map; input the sparse normal map into the Transformer model to obtain a third feature map, and then perform feature reconstruction and size restoration on the third feature map through an image reconstruction layer to obtain a third dense depth map and a third confidence map.

[0095] It should be noted that the three prediction branches, after passing through the Transformer model, can generate feature maps with a size one-quarter that of the original image. The image reconstruction layer described above consists of two transposed convolutional layers, which can be used to restore the image size and reconstruct a dense depth map with corresponding confidence maps.

[0096] S103, the depth completion device obtains the target dense depth map by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map.

[0097] The image density of the target dense depth map is higher than that of the sparse depth map.

[0098] Alternatively, the depth completion device can be based on the formula Determine the target dense depth map; wherein, This represents the depth value of the first dense depth map. This represents the depth value of the second dense depth map. C represents the depth value of the third dense depth map. rgb This represents the first confidence level plot, C. d This represents the second confidence plot, C. n This represents the third confidence level plot.

[0099] Specifically, this application can use the Softmax function to calculate weight values ​​from the confidence map and use the weight values ​​calculated from the confidence map to guide the fusion of the dense depth map output from the three prediction branches, thereby improving the output quality of the target dense depth map.

[0100] Optionally, each predicted branch in this application outputs a dense depth map, and the dense depth map output by each predicted branch has its own emphasis. Therefore, it is necessary to constrain the dense depth map output of each predicted branch. In addition, the image reconstruction layer in S102 above can also be used to reconstruct a dense normal map based on the third feature map. Therefore, the depth completion network in this application also introduces normal constraints.

[0101] Optionally, the loss function of the deep completion network is:

[0102]

[0103] in, To address the depth consistency loss in dense depth maps of the target, To address the depth consistency loss in the first dense depth map, To address the depth consistency loss in the second dense depth map, To address the depth consistency loss in the third dense depth map, For normal-depth consistency loss, ω out ω rgb ω d ω n ω nd The weights of each loss term.

[0104] Optionally, the above The depth consistency loss can be defined as:

[0105]

[0106] Where Ω represents the domain of the truth value of dense depth.

[0107] Optionally, the above The normal-depth consistency loss can be defined as:

[0108]

[0109] Where N represents the neighborhood of the current pixel in three-dimensional space.

[0110] It should be noted that, for the normal-depth consistency loss, this application can use the inner product between the current pixel normal n(x) and the tangent vector v(x,y) between its neighboring pixels and the pixel at that point as the loss, thereby constraining the dense normal map.

[0111] In this embodiment, since the normal information of the sparse normal map can be introduced into the depth completion task, and more three-dimensional spatial information can be provided to the model through the normal, depth noise and error can be eliminated to a certain extent, thereby improving the accuracy and quality of the output dense depth. Since RGB image features can be used as guiding terms to achieve deep fusion of three-modal features through the guiding attention module, the prediction results of sparse depth and normal branches can be optimized.

[0112] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0113] The depth completion method provided in this application can be executed by a depth completion device or a control module for depth completion within that device. This application uses the execution of the depth completion method by a depth completion device as an example to illustrate the depth completion device provided in this application.

[0114] It should be noted that the embodiments of this application can divide the deep completion device into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. Optionally, the module division in the embodiments of this application is illustrative and is only a logical functional division; other division methods may be used in actual implementation.

[0115] like Figure 16As shown in the figure, this application embodiment provides a depth completion device 1600. The depth completion device 1600 includes: an acquisition module 1601 and a processing module 1602. The acquisition module 1601 is used to acquire an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map; the processing module 1602 is used to obtain a first dense depth map and a first confidence map by inputting the RGB image into a Transformer model, to obtain a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model, and to obtain a third dense depth map and a third confidence map by inputting the sparse normal map into the Transformer model; and to obtain a target dense depth map by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map; wherein the image density of the target dense depth map is higher than that of the sparse depth map, and the Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map.

[0116] Optionally, the processing module 1602 is further configured to: extract shallow features of the RGB image, shallow features of the sparse depth map, and shallow features of the sparse normal map, respectively; perform a linear transformation on the shallow features of the RGB image to obtain a first input vector that matches the Transformer model; perform a linear transformation on the shallow features of the sparse depth map to obtain a second input vector that matches the Transformer model; and perform a linear transformation on the shallow features of the sparse normal map to obtain a third input vector that matches the Transformer model.

[0117] Optionally, the Transformer model includes three prediction branches, each of which includes an encoder network and a decoder network. The encoder network includes three layers of self-attention modules and an image patch merging layer. The decoder network includes sub-pixel convolutional layers and two layers of self-attention modules. The self-attention modules are used to extract bottleneck features. The image patch merging layer is used to adjust the number of channels and downsample the feature map. The sub-pixel convolutional layers are used to upsample and reconstruct features.

[0118] Optionally, the combination of the guided attention module and the self-attention module includes: parallel pre-guided guidance, parallel post-guided guidance, serial pre-guided guidance, and serial post-guided guidance.

[0119] Optionally, the guidance structure of the guidance attention module includes: a parallel guidance structure between encoders, a serial guidance structure between encoders, a parallel guidance structure across encoders and decoders, and a serial guidance structure across encoders and decoders.

[0120] Optionally, the processing module 1602 is configured to: input the RGB image into the Transformer model to obtain a first feature map; perform feature reconstruction and size restoration on the first feature map to obtain a first dense depth map and a first confidence map; input the sparse depth map into the Transformer model to obtain a second feature map; perform feature reconstruction and size restoration on the second feature map to obtain a second dense depth map and a second confidence map; input the sparse normal map into the Transformer model to obtain a third feature map; perform feature reconstruction and size restoration on the third feature map to obtain a third dense depth map and a third confidence map.

[0121] Optionally, the processing module 1602 is used for:

[0122] According to the formula Determine the target dense depth map;

[0123] in, This represents the depth value of the first dense depth map. This represents the depth value of the second dense depth map. C represents the depth value of the third dense depth map. rgb This represents the first confidence level plot, C. d This represents the second confidence plot, C. n This represents the third confidence level plot.

[0124] In this embodiment, since the normal information of the sparse normal map can be introduced into the depth completion task, and more three-dimensional spatial information can be provided to the model through the normal, depth noise and error can be eliminated to a certain extent, thereby improving the accuracy and quality of the output dense depth. Since RGB image features can be used as guiding terms to achieve deep fusion of three-modal features through the guiding attention module, the prediction results of sparse depth and normal branches can be optimized.

[0125] Figure 17 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 17As shown, the electronic device may include: a processor 1710, a communications interface 1720, a memory 1730, and a communications bus 1740, wherein the processor 1710, the communications interface 1720, and the memory 1730 communicate with each other through the communications bus 1740. Processor 1710 can call logic instructions in memory 1730 to execute a depth completion method, which includes: acquiring an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map; obtaining a first dense depth map and a first confidence map by inputting the RGB image into a Transformer model, obtaining a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model, obtaining a third dense depth map and a third confidence map by inputting the sparse normal map into the Transformer model; obtaining a target dense depth map by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map; wherein the image density of the target dense depth map is higher than the image density of the sparse depth map, and the Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map.

[0126] Furthermore, the logical instructions in the aforementioned memory 1730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the depth completion method provided by the above methods. The method includes: acquiring an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map; obtaining a first dense depth map and a first confidence map by inputting the RGB image into a Transformer model; obtaining a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model; and obtaining a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model. The sparse normal map is input into the Transformer model to obtain a third dense depth map and a third confidence map. A target dense depth map is obtained by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map. The image density of the target dense depth map is higher than that of the sparse depth map. The Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map.

[0128] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the depth completion methods provided above. The method includes: acquiring an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map; obtaining a first dense depth map and a first confidence map by inputting the RGB image into a Transformer model; obtaining a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model; and obtaining a second dense depth map and a second confidence map by inputting the sparse normal map into the Transformer model. The Transformer model obtains a third dense depth map and a third confidence map; by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map, a target dense depth map is obtained; wherein, the image density of the target dense depth map is higher than the image density of the sparse depth map, and the Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A depth completion method, characterized in that, include: Obtain the RGB image, the sparse depth map, and the sparse normal map of the sparse depth map; By inputting the RGB image into the Transformer model, a first dense depth map and a first confidence map are obtained. By inputting the sparse depth map into the Transformer model, a second dense depth map and a second confidence map are obtained. By inputting the sparse normal map into the Transformer model, a third dense depth map and a third confidence map are obtained. The Transformer model includes three prediction branches, each including an encoder network and a decoder network. The encoder network includes three layers of self-attention modules and an image patch merging layer. The decoder network includes sub-pixel convolutional layers and two layers of self-attention modules. The self-attention modules are used to extract bottleneck features. The image patch merging layer is used to adjust the number of channels and downsample the feature map. The sub-pixel convolutional layers are used to upsample and reconstruct features. The target dense depth map is obtained by fusing the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map using the following formula: ； in, This represents the depth value of the first dense depth map. This represents the depth value of the second dense depth map. This represents the depth value of the third dense depth map. This represents the first confidence level plot. This represents the second confidence level plot. This represents the third confidence level plot; Wherein, the image density of the target dense depth map is higher than the image density of the sparse depth map, and the Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map.

2. The depth completion method according to claim 1, characterized in that, After acquiring the RGB image, the sparse depth map, and the sparse normal map of the sparse depth map, the method further includes: The shallow features of the RGB image, the shallow features of the sparse depth map, and the shallow features of the sparse normal map are extracted respectively. A linear transformation is performed on the shallow features of the RGB image to obtain a first input vector that matches the Transformer model; A linear transformation is performed on the shallow features of the sparse depth map to obtain a second input vector that matches the Transformer model; A linear transformation is performed on the shallow features of the sparse normal graph to obtain a third input vector that matches the Transformer model.

3. The depth completion method according to claim 1, characterized in that, The combination of the guided attention module and the self-attention module includes: parallel pre-guided guidance, parallel post-guided guidance, serial pre-guided guidance, and serial post-guided guidance.

4. The depth completion method according to claim 1, characterized in that, The guidance structure of the guidance attention module includes: a parallel guidance structure between encoders, a serial guidance structure between encoders, a parallel guidance structure across encoders and decoders, and a serial guidance structure across encoders and decoders.

5. The depth completion method according to claim 1, characterized in that, The process of obtaining a first dense depth map and a first confidence map by inputting the RGB image into a Transformer model, obtaining a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model, and obtaining a third dense depth map and a third confidence map by inputting the sparse normal map into the Transformer model includes: The RGB image is input into the Transformer model to obtain a first feature map. The first feature map is then reconstructed and its size restored to obtain a first dense depth map and a first confidence map. The sparse depth map is input into the Transformer model to obtain the second feature map. The second feature map is then reconstructed and its size restored to obtain the second dense depth map and the second confidence map. The sparse normal map is input into the Transformer model to obtain the third feature map. The third feature map is then reconstructed and its size restored to obtain the third dense depth map and the third confidence map.

6. A depth completion device, characterized in that, include: Acquisition module and processing module; The acquisition module is used to acquire an RGB image, a sparse depth map, and a sparse normal map of the sparse depth map; The processing module is used to obtain a first dense depth map and a first confidence map by inputting the RGB image into a Transformer model, to obtain a second dense depth map and a second confidence map by inputting the sparse depth map into the Transformer model, and to obtain a third dense depth map and a third confidence map by inputting the sparse normal map into the Transformer model. The Transformer model includes three prediction branches, each including an encoder network and a decoder network. The encoder network includes three layers of self-attention modules and an image patch merging layer. The decoder network includes sub-pixel convolutional layers and two layers of self-attention modules. The self-attention modules are used to extract bottleneck features. The image patch merging layer is used to adjust the number of channels and downsample the feature map. The sub-pixel convolutional layers are used to upsample and reconstruct features. The module then fuses the first dense depth map, the first confidence map, the second dense depth map, the second confidence map, the third dense depth map, and the third confidence map according to the following formula to obtain a target dense depth map: ； in, This represents the depth value of the first dense depth map. This represents the depth value of the second dense depth map. This represents the depth value of the third dense depth map. This represents the first confidence level plot. This represents the second confidence level plot. This represents the third confidence level plot; Wherein, the image density of the target dense depth map is higher than the image density of the sparse depth map, and the Transformer model includes a guided attention module, which is used to pass the branch information of the RGB image to the depth branch of the sparse depth map and the normal branch of the sparse normal map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the depth completion method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the depth completion method as described in any one of claims 1 to 5.

Images