Depth map completion method based on two-branch progressive fusion

Abstract

The present application relates to the technical field of automatic driving, in particular to a depth map completion method based on double-branch gradual fusion, comprising: processing color images and laser radar sparse depth images of the same target scene through a double-branch depth completion network based on mixed attention fusion and gradual refinement; training the double-branch depth completion network based on mixed attention fusion and gradual refinement using a training set in a depth completion dataset, obtaining network weight parameters after each round of training, and verifying the network weight parameters through a verification set, selecting the network weight parameters with the smallest error as the optimal network weight parameters; and performing depth completion on the double-branch depth completion network using a test set and the optimal network weight parameters to obtain a target dense depth image. The present application optimizes feature processing by extracting, fusing and refining the features of sparse depth images and color images, and finally obtains a dense depth image with rich details and accurate depth values.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically to a depth graph completion method based on bi-branch progressive fusion. Background Technology

[0002] In autonomous driving technology, depth perception is crucial for machines to understand the 3D information of a scene. LiDAR, as a common depth sensor, can directly acquire high-precision sparse depth images. Depth completion aims to complete the sparse depth images acquired by LiDAR, thereby obtaining denser depth images of the scene.

[0003] Among related solutions, methods that combine color images with sparse depth images from LiDAR based on deep learning technology have achieved significant results in depth completion tasks. These methods generally focus on improving the effectiveness of depth completion by addressing either the network model architecture or the depth refinement methods.

[0004] The network model architecture primarily employs CNNs or Transformers. CNNs excel at handling local features, but extracting global features requires a deep network structure, making the model overly complex. Transformers are adept at obtaining global features, but their ability to capture detailed features is still inferior to CNNs, and their computational efficiency needs improvement.

[0005] Depth thinning methods are typically based on spatial propagation networks, refining depth images by calculating affinity matrices. These methods improve coarse depth images iteratively, but often require a large number of iterations, resulting in low algorithmic efficiency.

[0006] Chinese patent CN118154654A discloses a depth completion method based on a distance-aware mask converter and a sparse attention masking mechanism. This invention is based on an innovative macro-to-micro network, utilizing prior knowledge within a macro dataset while enhancing specific details in the sparse depth at a micro level. However, the method suffers from poor integration of macro and micro features (i.e., global and local features), resulting in an overly complex network model. While this patent optimizes the computational complexity of the depth refinement process, it still requires a significant number of iterations to achieve good depth completion results. Chinese patent CN118864556A discloses a guided depth completion method based on a dual encoder. This method uses a dual-encoder backbone network to extract information from RGB images and sparse depth images to obtain a preliminary completed depth image; a spatial propagation network is then used to propagate and refine the preliminary completed depth image, outputting a dense depth image. However, this method does not fully exploit global information when processing sparse depth features, and the depth refinement effect is less than ideal due to a lack of sufficient prior knowledge. Therefore, how to achieve the coordinated processing of global depth and detail depth information of depth images while reducing algorithm complexity has become a difficult problem in current research. Summary of the Invention

[0007] This invention provides a depth map completion method based on dual-branch progressive fusion, which overcomes the problems of existing algorithms having high algorithm complexity when constructing depth images and difficulty in taking into account both global depth and detail depth of the depth map, resulting in insufficient depth map accuracy.

[0008] This invention provides a depth map completion method based on dual-branch progressive fusion, comprising:

[0009] Image acquisition, including the acquisition of color images and sparse depth images;

[0010] The construction of a dual-branch deep completion network is based on hybrid attention fusion and progressive refinement to construct a dual-branch deep completion network including a depth coarse estimation branch and a depth refinement branch;

[0011] The training of the dual-branch deep completion network is carried out by training the network on the training set in the public deep completion dataset and retaining the network weight parameters at the end of each training round. The network weight parameters are validated on the validation set in the public deep completion dataset, and the network weight parameters with the lowest error are selected as the optimal network weight parameters.

[0012] The dense depth image of the target is constructed by performing depth completion calculation on the dual-branch depth completion network based on the test set in the publicly available depth completion dataset and the optimal network weight parameters, thereby obtaining the dense depth image of the target.

[0013] Furthermore, the coarse depth estimation branch first merges the color image with the sparse depth image, then extracts multi-scale features of the merged image based on the Swin-Unet network, and predicts the initial depth image based on the depth prediction module DP.

[0014] Furthermore, the depth prediction module DP first upscales and merges the image resolution based on interpolation upsampling, then extracts local features of the upscaled image through 3×3 convolution, and simultaneously downsamples while extracting features through 3×3 strided convolution. Finally, the final depth prediction value is normalized to between 0 and 1 through the Sigmoid function, where nonlinearity is introduced based on the ReLU activation function.

[0015] Furthermore, the method for merging the color image and the sparse depth image in the coarse depth estimation branch is as follows: first, the color image and the sparse depth image are merged based on the Concate function; then, the number of channels in the merged image is adjusted to a 1×1 convolution; and downsampling is performed while extracting features based on a 3×3 strided convolution, wherein the ReLU activation function is used as the activation function.

[0016] Furthermore, the deep refinement branch includes an image classifier network ResNet50, a hybrid attention fusion module HAF, a recursive modulation module RM, and a depth prediction module DP.

[0017] Furthermore, in the deep refinement branch, the number of ResNet50 image classifier networks is selected from 1;

[0018] The number of Hybrid Attention Fusion (HAF) modules is selected from 3;

[0019] The number of recursive modulation modules (RMs) is selected from 3;

[0020] The number of depth prediction modules (DP) is selected from 1;

[0021] The initial parameters of the image classifier network ResNet50 are determined based on the pre-training results of the ImageNet dataset.

[0022] Furthermore, the Hybrid Attention Fusion Module (HAF) first performs local residual fusion and global self-attention fusion on the features of the image classifier network and the features extracted by the Swin-Unet network decoder, and finally performs selective fusion to generate two spatial attention maps. The spatial attention maps are then multiplied pixel-by-pixel with the output features of the local residual fusion and global self-attention fusion, and the multiplication results are summed to obtain the output features of the Hybrid Attention Fusion Module (HAF).

[0023] The Swin-Unet network decoder selects features from the first three stages, and the image classifier network selects features of the same scale.

[0024] The local residual fusion is used to fuse local features. The local residual fusion process includes: first, stitching the images output by the coarse depth estimation branch and the fine depth branch by using the Concate function; then, adjusting the number of channels by 1×1 convolution; and finally, processing by residual blocks, wherein the ReLU activation function is used as the activation function.

[0025] The global self-attention fusion is used to capture global features, and the global self-attention fusion is calculated based on the BiFormer cross self-attention calculation method;

[0026] The selective fusion first concatenates the results of the local residual fusion and the global self-attention fusion through the Concate function. Then, the number of channels is adjusted through a 1×1 convolution and the ReLU activation function is used as the activation function. Local features are extracted again through a 3×3 convolution and the ReLU activation function is used as the activation function. Then, the local features are refined again through a 3×3 convolution and two spatial attention maps are generated through the Sigmoid function.

[0027] Furthermore, the recursive modulation module (RM), based on the output of the hybrid attention fusion module (HAF), refines the fourth-stage features of the Swin-Unet network decoder downsampled to 1 / 8 of the original scale, including:

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] in, Represents the Adama product. This represents the Sigmoid function. ReLU represents a 3×3 convolution operation, and ReLU represents the ReLU activation function. This indicates a double interpolation upsampling operation. tanh Represents the hyperbolic tangent function. Indicates the first i During the secondary refinement process, the output features of the Hybrid Attention Fusion (HAF) module... Indicates the first i During the secondary refinement process, intermediate features are recursively modulated and refined. Indicates the first iDuring the secondary refinement process, recursive modulation refinement features are obtained. Indicates the first i During the secondary refinement process, the spatial domain two-dimensional attention map, Indicates the first i The adaptive factor in the secondary refinement process is the initial recursive modulation feature, which is the fourth-stage feature of the Swin-Unet network decoder downsampled to 1 / 8 of the original scale.

[0033] Furthermore, during the training of the dual-branch deep completion network, the loss of the coarse depth estimation branch and the loss of the refined depth branch are summed to obtain the loss of the dual-branch deep completion network based on hybrid attention fusion and feature refinement, wherein the loss function adopts the scale-invariant logarithmic scaling loss function.

[0034] Furthermore, the Adam gradient optimization algorithm is selected during the training of the dual-branch deep completion network;

[0035] The base learning rate was set to 0.0001 when training the dual-branch deep completion network, and decreased by 10% every 5 epochs. The parameters... β 1 = 0.9 β 2 = 0.999, and use an attenuation rate of 0.0001;

[0036] When training the dual-branch deep completion network, the size of the network training parameter Batch is 16;

[0037] When validating the network weight parameters using the validation set of the publicly available deep completion dataset, the RMSE evaluation metric is selected to determine the magnitude of the error.

[0038] Compared with existing technologies, the advantages of this invention lie in its proposed dual-branch depth completion method based on hybrid attention fusion and progressive refinement. By constructing a network structure including a coarse depth estimation branch and a depth refinement branch, it achieves effective fusion and depth completion of color images and sparse depth images. The coarse depth estimation branch uses the Swin-Unet network to extract multi-scale features and predict the initial depth image, while the depth refinement branch optimizes and refines the initial depth image through a hybrid attention fusion module and a recursive modulation module. The hybrid attention fusion module effectively fuses global and local features, while the recursive modulation module improves the efficiency and effectiveness of feature refinement. This invention fully utilizes the rich semantic information of color images, achieving coordinated processing of global and detailed depth information, significantly improving the accuracy and efficiency of depth completion, reducing algorithm complexity, and enhancing model performance and generalization ability. It provides a more efficient and accurate solution for depth perception tasks in fields such as autonomous driving.

[0039] Furthermore, by merging the color image with the sparse depth image, multi-scale features are extracted based on the Swin-Unet network, and the initial depth image is predicted using the depth prediction module (DP). This approach fully leverages the global feature learning capability and local feature extraction advantage of the Swin-Unet network, providing a more accurate and richer feature base for the depth completion task, thereby improving the quality of the initial depth image and laying a solid foundation for the subsequent depth refinement process.

[0040] Furthermore, in this invention, the depth prediction module DP amplifies the image resolution through interpolation upsampling, extracts local features using 3×3 convolution, performs downsampling through 3×3 strided convolution, and finally normalizes the depth prediction value to between 0 and 1 using the Sigmoid function. This design not only effectively extracts local detail features of the image but also introduces nonlinearity through the ReLU activation function, enhancing the model's expressive power and thus more accurately predicting the initial depth image, further improving the accuracy and effect of depth completion.

[0041] Furthermore, when the coarse depth estimation branch merges the color image and the sparse depth image, it uses the Concate function to combine them, adjusts the number of channels through a 1×1 convolution, and uses a 3×3 strided convolution to perform downsampling while extracting features. This method can effectively integrate the information from the two images, while reducing computational load and retaining key features, improving the model's computational efficiency and feature extraction capability, and providing higher-quality input data for subsequent depth estimation.

[0042] Furthermore, the depth refinement branch includes the ResNet50 image classifier network, the Hybrid Attention Fusion Module (HAF), the Recursive Modulation Module (RM), and the Depth Prediction Module (DP). This structural design fully utilizes the local feature extraction capabilities of ResNet50 and the global and local feature fusion capabilities of the HAF module. The RM module progressively refines the features, ultimately generating a high-quality, dense depth image. This branch design significantly improves the detail and accuracy of the depth image, further optimizing the depth completion effect.

[0043] Furthermore, by setting up three Hybrid Attention Fusion (HAF) modules and three Recursive Modulation (RM) modules, the degree of integration of global and local information of features can be gradually improved, optimizing the detail and accuracy of features. Meanwhile, the initial parameters of the ResNet50 image classifier network are determined based on the pre-training results of the ImageNet dataset, ensuring good initial performance when extracting color image features, thereby improving the feature extraction capability and depth completion quality of the entire depth refinement branch.

[0044] Furthermore, through three steps—local residual fusion, global self-attention fusion, and selective fusion—this invention enables deep fusion of image classifier network features and Swin-Unet network decoder features, generating two spatial attention maps and adaptively combining local and global features. This fusion method not only fully utilizes local detail information but also effectively captures global contextual information, significantly improving feature expressiveness and depth refinement, providing strong support for generating high-quality, dense depth images.

[0045] Furthermore, in the Recursive Modulation (RM) module, the use of a fully convolutional structure and a scale-wise progressive thinning method enables gradual feature optimization within a fewer iterations, avoiding error accumulation and improving the thinning effect. In addition, this module's design reduces computational complexity and improves computational efficiency, making it particularly suitable for processing large feature maps. This approach effectively enhances the quality and accuracy of depth images, further optimizing the performance of depth completion.

[0046] Furthermore, during the training of the dual-branch depth completion network, the loss of the coarse depth estimation branch and the loss of the depth refinement branch are summed to obtain the loss of the dual-branch depth completion network based on hybrid attention fusion and feature refinement, and a scale-invariant logarithmic scaling loss function is adopted. This loss function design can comprehensively consider the optimization of global structure and local details, enhance the network's sensitivity to changes in depth values, especially when dealing with scenes with large depth variations, further improving the overall quality of depth images and enhancing the model's generalization ability and depth completion accuracy.

[0047] Furthermore, the Adam gradient optimization algorithm was selected, and reasonable base learning rate, learning rate decay strategy, parameter values, and batch size were set. Simultaneously, RMSE (Recovery Mean Squared Error) was used as the criterion for judging the error magnitude when validating the network weight parameters. This training strategy effectively improves the network's training efficiency and convergence speed, ensuring that the network can better learn the features and patterns of depth images during training, thereby improving model performance and depth completion quality. Attached Figure Description

[0048] Figure 1 This is a flowchart of the depth map completion method based on dual-branch progressive fusion in an embodiment of the present invention;

[0049] Figure 2 This is a structural diagram of the depth graph completion network based on dual-branch progressive fusion in an embodiment of the present invention;

[0050] Figure 3 This is a DP structure diagram of the depth prediction module in an embodiment of the present invention;

[0051] Figure 4This is a structural diagram of the Hybrid Attention Fusion Module (HAF) in an embodiment of the present invention;

[0052] Figure 5 This is a structural diagram of the recursive modulation module (RM) in an embodiment of the present invention;

[0053] Figure 6 In this embodiment, a is the color image used for completion, and b is the sparse depth image of the LiDAR corresponding to the color image in this embodiment.

[0054] Figure 7 In Figure a, it is a comparative schematic diagram of the depth image completed on the KITTI test dataset in Comparative Example 1; in Figure b, it is a comparative schematic diagram of the depth image completed on the KITTI test dataset in Comparative Example 4; and in Figure c, it is a comparative schematic diagram of the depth image completed on the KITTI test dataset in Embodiment 1 of the present invention. Detailed Implementation

[0055] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0056] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0057] Please see Figure 1 As shown, this is a flowchart of the depth map completion method based on bi-branch progressive fusion in an embodiment of the present invention. The flowchart of the depth map completion method based on bi-branch progressive fusion in an embodiment of the present invention includes:

[0058] S1: Image acquisition, acquiring color images and sparse depth images;

[0059] S2: Construction of a dual-branch deep completion network, which is based on hybrid attention fusion and progressive refinement to construct a dual-branch deep completion network including a depth coarse estimation branch and a depth refinement branch;

[0060] S3: Training of the dual-branch deep completion network: The dual-branch deep completion network is trained based on the training set in the public deep completion dataset and the network weight parameters at the end of each training round are retained. The network weight parameters are verified based on the validation set in the public deep completion dataset, and the network weight parameters with the lowest error are selected as the optimal network weight parameters.

[0061] S4: Construction of the target dense depth image: Based on the test set in the publicly available depth completion dataset and the optimal network weight parameters, the dual-branch depth completion network is used to perform depth completion calculation to obtain the target dense depth image.

[0062] The method for acquiring color images is to select an RGB camera for acquisition. The RGB camera includes, but is not limited to, any one of a line scan camera, a hyperspectral camera, and an industrial inspection camera, which will not be elaborated here.

[0063] The method for acquiring sparse depth images is to use lidar.

[0064] Furthermore, the coarse depth estimation branch first merges the color image with the sparse depth image, then extracts multi-scale features of the merged image based on the Swin-Unet network, and predicts the initial depth image based on the depth prediction module DP.

[0065] The coarse depth estimation branch first merges the color image with the sparse depth image, providing a comprehensive input for subsequent feature extraction. Next, the Swin-Unet decoder extracts multi-scale features from the merged image. The Swin-Unet decoder combines the global feature learning capability of the Swin-Transformer with the local feature extraction advantages of CNNs. During multi-scale feature extraction, it captures both the global scene structure and dependencies while preserving local details, providing a rich feature base for the depth prediction module. This enables an effective combination of global and local features in the depth completion task. Finally, the depth prediction module (DP) uses these features to predict the initial depth image, achieving effective fusion of global and local features and providing a basic depth estimate for the depth completion task.

[0066] Furthermore, the depth prediction module DP first upscales and merges the image resolution based on interpolation upsampling, then extracts local features of the upscaled image through 3×3 convolution, and simultaneously downsamples while extracting features through 3×3 strided convolution. Finally, the final depth prediction value is normalized to between 0 and 1 through the Sigmoid function, where nonlinearity is introduced based on the ReLU activation function.

[0067] The depth prediction module (DP) is responsible for transforming the extracted multi-scale features into an initial depth image within the coarse depth estimation branch. It first upsamples and interpolates to enhance the resolution of the merged image, improving local feature extraction. Then, it uses 3×3 convolutions to further extract local features and enhance image detail. Simultaneously, 3×3 strided convolutions downsample while extracting features, reducing computation and preserving key features. The introduction of the ReLU activation function provides the model with non-linear expressive power, enabling it to handle complex feature relationships. Finally, the Sigmoid function normalizes the depth prediction values ​​to between 0 and 1, ensuring the output depth values ​​have a reasonable range. This module primarily focuses on local feature extraction and processing, but by combining it with global feature learning capabilities, it can predict depth images more accurately.

[0068] Furthermore, the method for merging the color image and the sparse depth image in the coarse depth estimation branch is as follows: first, the color image and the sparse depth image are merged based on the Concate function; then, the number of channels in the merged image is adjusted to a 1×1 convolution; and downsampling is performed while extracting features based on a 3×3 strided convolution, wherein the ReLU activation function is used as the activation function.

[0069] In the coarse depth estimation branch, when merging color and sparse depth images, the first step is to use the Concate function to merge them. Then, a 1×1 convolution is used to adjust the number of channels in the merged image, preparing for subsequent feature extraction. Next, a 3×3 strided convolution is used to downsample while extracting features, reducing computation and preserving important features. At the same time, the use of the ReLU activation function enhances the model's non-linear expressive power.

[0070] Furthermore, the deep refinement branch includes an image classifier network ResNet50, a hybrid attention fusion module HAF, a recursive modulation module RM, and a depth prediction module DP.

[0071] The depth refinement branch extracts semantic features from color images using the ResNet50 image classifier network. These features are then fused with those extracted by the Swin-Unet network decoder using the Hybrid Attention Fusion (HAF) module. The fused features are progressively refined using the Recursive Modulation (RM) module, and finally, the Depth Prediction (DP) module generates the final dense depth image. This process effectively improves the quality and accuracy of depth images.

[0072] Furthermore, in the deep refinement branch, the number of ResNet50 image classifier networks is selected from 1;

[0073] The number of Hybrid Attention Fusion (HAF) modules is selected from 3;

[0074] The number of recursive modulation modules (RMs) is selected from 3;

[0075] The number of depth prediction modules (DP) is selected from 1;

[0076] The initial parameters of the image classifier network ResNet50 are determined based on the pre-training results of the ImageNet dataset.

[0077] By setting up three Hybrid Attention Fusion (HAF) modules and three Recursive Modulation (RM) modules, it is helpful to ensure that the HAF and RM modules cooperate with each other. Based on the output of the three HAF modules, the three RM modules gradually improve the combination of global and local information of features and optimize the details and accuracy of features, ultimately generating high-quality dense depth images. Secondly, the initial parameters of the image classifier network ResNet50 are determined based on the pre-training results of the ImageNet dataset. This step ensures that ResNet50 has good initial performance when extracting color image features, thereby improving the feature extraction capability of the entire depth refinement branch. The pre-trained ResNet50 can better capture the semantic information in color images, providing more accurate input for subsequent feature fusion and depth refinement.

[0078] Furthermore, the Hybrid Attention Fusion Module (HAF) first performs local residual fusion and global self-attention fusion on the features of the image classifier network and the features extracted by the Swin-Unet network decoder, and finally performs selective fusion to generate two spatial attention maps. The spatial attention maps are then multiplied pixel-by-pixel with the output features of the local residual fusion and global self-attention fusion, and the multiplication results are summed to obtain the output features of the Hybrid Attention Fusion Module (HAF).

[0079] The Swin-Unet network decoder selects features from the first three stages, and the image classifier network selects features of the same scale.

[0080] The local residual fusion is used to fuse local features. The local residual fusion process includes: first, stitching the images output by the coarse depth estimation branch and the fine depth branch by using the Concate function; then, adjusting the number of channels by 1×1 convolution; and finally, processing by residual blocks, wherein the ReLU activation function is used as the activation function.

[0081] The global self-attention fusion is used to capture global features, and the global self-attention fusion is calculated based on the BiFormer cross self-attention calculation method;

[0082] The selective fusion first concatenates the results of the local residual fusion and the global self-attention fusion through the Concate function. Then, the number of channels is adjusted through a 1×1 convolution and the ReLU activation function is used as the activation function. Local features are extracted again through a 3×3 convolution and the ReLU activation function is used as the activation function. Then, the local features are refined again through a 3×3 convolution and two spatial attention maps are generated through the Sigmoid function.

[0083] In this invention, local residual fusion extracts local features through the concatenation function, 1×1 convolution, residual blocks, and ReLU activation function; global self-attention fusion captures global features through the BiFormer cross-self-attention calculation method; and selective fusion generates a spatial attention map through the concatenation function, 1×1 convolution, ReLU activation function, 3×3 convolution, and sigmoid function, adaptively combining local and global features. These steps work together to achieve effective fusion of global and local features, improving the expressive power and depth refinement of features.

[0084] Furthermore, the recursive modulation module (RM), based on the output of the hybrid attention fusion module (HAF), refines the fourth-stage features of the Swin-Unet network decoder downsampled to 1 / 8 of the original scale, including:

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] in, Represents the Adama product. This represents the Sigmoid function. ReLU represents a 3×3 convolution operation, and ReLU represents the ReLU activation function. This indicates a double interpolation upsampling operation. tanh Represents the hyperbolic tangent function. Indicates the first i During the secondary refinement process, the output features of the Hybrid Attention Fusion (HAF) module... Indicates the first i During the secondary refinement process, intermediate features are recursively modulated and refined. Indicates the first i During the secondary refinement process, recursive modulation refinement features are obtained. This represents the spatial domain two-dimensional attention map during the i-th refinement process. Indicates the first i The adaptive factor in the secondary refinement process is the initial recursive modulation feature, which is the fourth-stage feature of the Swin-Unet network decoder downsampled to 1 / 8 of the original scale.

[0090] in, This indicates a refinement of the recursive modulation features. Output features of the Hybrid Attention Fusion Module (HAF) Effective feature extraction is performed to generate a two-dimensional attention map in the spatial domain. ; Will Acting on ,accomplish Spatial domain enhancement, followed by a combination of 3×3 convolution and tanh function processing, produces modulated and refined intermediate features. ; right and Effective feature extraction is performed, and the adaptive update factor is calculated. ; use Acting on respectively and The results are summed and upsampled to obtain the next recursive modulation refinement features. ;

[0091] In the Recursive Modulation (RM) module, the output features of three Hybrid Attention Fusion (HAF) modules at different scales are used to progressively refine the features of the fourth stage of the Swin-Unet network decoder. Each refinement process is divided into two steps: first, spatial domain attention enhancement is used to improve the network's ability to process spatial details; then, the network further refines the feature processing by using the calculated adaptive update factor. In addition, the RM module adopts a fully convolutional network (FCN) structure. This design enables the RM module to efficiently process input feature maps of arbitrary size while maintaining the spatial structure of the input features, ensuring that the output feature map is spatially aligned with the input feature map. The fully convolutional structure eliminates fully connected layers, significantly reducing computational complexity and improving computational efficiency. It is particularly suitable for processing large feature maps. The core function of the RM module is to optimize features through scale-wise progressive refinement. This scale-wise refinement method can better capture multi-scale feature information, gradually improving the quality and accuracy of features. Each refinement operation is performed at a specific scale, recursively optimizing features and avoiding the error accumulation that can occur when processing features at all scales at once, thus improving the refinement effect. Furthermore, the design of the RM module allows it to achieve good depth completion results in fewer iterations. This not only reduces computation time and resource consumption but also avoids overfitting the training data during training, improving the model's generalization ability. Through the fully convolutional structure and scale-wise progressive refinement, the RM module can progressively optimize features in a small number of iterations, generating high-quality depth images, thus playing a crucial role in the depth refinement branch and significantly improving the performance and effect of depth completion.

[0092] Furthermore, during the training of the dual-branch deep completion network, the loss of the coarse depth estimation branch and the loss of the refined depth branch are summed to obtain the loss of the dual-branch deep completion network based on hybrid attention fusion and feature refinement, wherein the loss function adopts the scale-invariant logarithmic scaling loss function.

[0093] In the training process of the dual-branch depth completion network, the losses of the coarse depth estimation branch and the fine depth branch are summed to comprehensively consider the optimization of global structure and local details. The loss of the coarse depth estimation branch focuses on the approximate difference between the initial depth image and the true depth image, ensuring the accuracy of the global structure; while the loss of the fine depth branch focuses more on the accuracy of details and local structure in the refined depth image. This combination of losses allows the network to optimize both global and local features simultaneously during training, improving the overall quality of the depth image. In addition, the use of a scale-invariant logarithmic scaling loss function further enhances the optimization effect of the network. This loss function compresses the range of depth values ​​through logarithmic scaling, making the network more sensitive to changes in depth values, especially when dealing with scenes with large depth variations. At the same time, scale invariance makes the network adaptable to depth images of different scenes and scales, and the smoothing term in the loss function reduces the impact of global smoothing, allowing the network to focus more on optimizing local errors, thereby better capturing local details.

[0094] Furthermore, the Adam gradient optimization algorithm is selected during the training of the dual-branch deep completion network;

[0095] The base learning rate was set to 0.0001 when training the dual-branch deep completion network, and decreased by 10% every 5 epochs. The parameters... β 1 = 0.9 β 2 = 0.999, and use an attenuation rate of 0.0001;

[0096] When training the dual-branch deep completion network, the size of the network training parameter Batch is 16;

[0097] When validating the network weight parameters using the validation set of the publicly available deep completion dataset, the RMSE evaluation metric is selected to determine the magnitude of the error. Example 1

[0098] The depth map completion method based on bi-branch progressive fusion described in this invention is used to... Figure 6 a, Figure 6 The image in b is completed, where... Figure 6 'a' represents the color image used for completion in this embodiment of the invention. Figure 6 b is the sparse depth image of the lidar corresponding to the color image in the embodiment of the present invention;

[0099] S1: Use an RGB camera to acquire color images and a LiDAR to acquire sparse depth images.

[0100] S2: Please refer to Figure 2As shown, it is a structural diagram of a depth graph completion network based on dual-branch progressive fusion in an embodiment of the present invention. The embodiment of the present invention constructs a dual-branch depth completion network based on hybrid attention fusion, which includes a depth coarse estimation branch and a depth refinement branch.

[0101] The coarse depth estimation branch uses a combination of Concatenate function + 1×1 convolution + ReLU activation function + 3×3 strided convolution to merge the input color image and the sparse depth data from the LiDAR. Then, the Swin-Unet network is used to extract multi-scale features. Finally, the depth prediction module (DP) predicts the initial depth image. The structure of the depth prediction module (DP) is as follows: Figure 3 As shown, this is the structure diagram of the depth prediction module DP in an embodiment of the present invention. In Embodiment 1 of the present invention, the depth prediction module DP is composed of interpolation upsampling + 3×3 convolution + ReLU activation function + 3×3 strided convolution + Sigmoid function. The coarse depth estimation branch improves the global feature learning ability through the Swin-Transformer structure, while retaining the local feature extraction advantages of CNN.

[0102] The deep refinement branch comprises an image classifier network ResNet50, three Hybrid Attention Fusion (HAF) modules, three Recursive Modulation (RM) modules, and a Deep Prediction (DP) module. The ResNet50 image classifier network extracts color image features; its initial parameters are pre-trained on the ImageNet dataset. The HAF module fuses the features from the image classifier network with those from the Swin-Unet decoder network. The Swin-Unet decoder selects features from the first three stages, while the image classifier network selects features corresponding to the same scale. Fusing the features from the Swin-Unet decoder with those from the ResNet50 image classifier network improves the network's feature representation capabilities.

[0103] Please refer to Figure 4The diagram shown illustrates the Hybrid Attention Fusion Module (HAF) structure in this embodiment of the invention. The HAF comprises three steps: local residual fusion, global self-attention fusion, and selective fusion. Local residual fusion is composed of a concatenation function, a 1×1 convolution, a ReLU activation function, and a residual block. Global self-attention fusion uses the low-computational-complexity BiFormer cross-self-attention calculation method. Selective fusion uses a combination of a concatenation function, a 1×1 convolution, a ReLU activation function, a 3×3 convolution, a ReLU activation function, a 3×3 convolution, and a Sigmoid function to generate two spatial attention maps. These two spatial attention maps are multiplied pixel-by-pixel with the output features of local residual fusion and global self-attention fusion, respectively. The multiplication results are summed to obtain the output features of the HAF.

[0104] Please continue reading. Figure 2 As shown, the three recursive modulation modules (RMs) work together to progressively refine the fourth-stage features of the Swin-Unet network decoder, which are downsampled to 1 / 8 of the original scale, based on the outputs of the three hybrid attention fusion modules (HAFs).

[0105] Please see Figure 5 As shown, this is a structural diagram of the recursive modulation module (RM) in an embodiment of the present invention. In this embodiment, the recursive modulation module (RM) adopts a fully convolutional structure design. Through a one-time, scale-wise progressive feature refinement, the feature refinement operation of the recursive modulation module (RM) is as follows:

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] in, Represents the Adama product. This represents the Sigmoid function. ReLU represents a 3×3 convolution operation, and ReLU represents the ReLU activation function. This indicates a double interpolation upsampling operation. tanh Represents the hyperbolic tangent function. Indicates the first i During the secondary refinement process, the output features of the Hybrid Attention Fusion (HAF) module... Indicates the first i During the secondary refinement process, intermediate features are recursively modulated and refined. Indicates the first iDuring the secondary refinement process, recursive modulation refinement features are obtained. This represents the spatial domain two-dimensional attention map during the i-th refinement process. Indicates the first i The adaptive factor in the secondary refinement process is the initial recursive modulation feature, which is the fourth-stage feature of the Swin-Unet network decoder downsampled to 1 / 8 of the original scale.

[0111] S3: Following the KITTI official website rules, complete the partitioning of the training, test, and validation sets. The training dataset contains 86K images, the test dataset contains 3K images, and the validation dataset contains 4K images. During network training, the loss of the coarse depth estimation branch and the loss of the depth refinement branch are summed to obtain the loss of the dual-branch depth completion network based on hybrid attention fusion and feature refinement. The loss function is the scale-invariant logarithmic scaling loss function. The Adam gradient optimization algorithm is chosen for network training optimization, with a base learning rate of 0.0001, reduced by 10% every 5 epochs. Parameters β1=0.9, β2=0.999, and a decay rate of 0.0001. The network training batch size is 16. The RMSE evaluation metric is selected as the criterion for judging the error magnitude to determine the optimal network weight parameters.

[0112] S4: Using the test set and optimal network weight parameters from the public deep completion dataset, perform deep completion calculations on a dual-branch deep completion network based on hybrid attention fusion and feature refinement to obtain a dense depth image of the target.

[0113] Comparative Example 1

[0114] Using FCFRNet depth completion method Figure 6 a, Figure 6 Complete the image in b.

[0115] Comparative Example 2

[0116] Using GuideFormer depth completion method Figure 6 a, Figure 6 Complete the image in b.

[0117] Comparative Example 3

[0118] Using CFormer depth completion method Figure 6 a, Figure 6 Complete the image in b.

[0119] Comparative Example 4

[0120] Using the ImprovingDC depth completion method Figure 6 a, Figure 6Complete the image in b.

[0121] Test methods

[0122] The performance was evaluated according to the four performance metrics provided by the KITTI official website: RMSE, MAE, iRMSE, and iMAE. The test results are shown in Table 1. Under the PyTorch deep learning framework of the Ubuntu system, the hardware platform used four NVIDIA RTX2080ti graphics cards to implement GPU parallel computing in Example 1 and Comparative Examples 1-4. The number of parameters and FLOPS were compared, and the test results are shown in Table 1.

[0123] ;

[0124] As can be observed from Table 1, Example 1 has better RMSE, MAE, iRMSE, and iMAE than Comparative Examples 1-4, indicating that the performance indicators of the present invention are superior to those of Comparative Examples 1-4. Please also refer to... Figure 7 As shown, it is a comparative diagram of the depth images completed on the KITTI test dataset by Embodiment 1 of the present invention and Comparative Examples 1 and 4. It can be observed that the present invention still shows a clearer depth image result than Comparative Examples 1 and 4.

[0125] In summary, this invention achieves effective fusion and optimization of global and local features through the synergistic effect of a coarse depth estimation branch and a fine depth branch. This improves the accuracy and efficiency of depth image completion, reduces algorithm complexity, and enhances model performance and generalization ability. Experimental results on the KITTI test dataset demonstrate that the method of this invention outperforms existing typical depth completion methods in terms of both performance metrics and model efficiency, providing a more accurate and efficient depth image completion solution for depth perception tasks in fields such as autonomous driving.

[0126] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A depth map completion method based on bi-branch progressive fusion, characterized in that, include, Image acquisition, including the acquisition of color images and sparse depth images; The construction of a dual-branch deep completion network is based on hybrid attention fusion and progressive refinement to construct a dual-branch deep completion network including a depth coarse estimation branch and a depth refinement branch; The training of the dual-branch deep completion network is carried out by training the network on the training set in the public deep completion dataset and retaining the network weight parameters at the end of each training round. The network weight parameters are validated on the validation set in the public deep completion dataset, and the network weight parameters with the lowest error are selected as the optimal network weight parameters. The construction of the target dense depth image involves performing depth completion calculations on the dual-branch depth completion network based on the test set in the publicly available depth completion dataset and the optimal network weight parameters to obtain the target dense depth image. The coarse depth estimation branch first merges the color image with the sparse depth image, then extracts multi-scale features of the merged image based on the Swin-Unet network, and then predicts the initial depth image based on the depth prediction module DP. The depth refinement branch includes the image classifier network ResNet50, the hybrid attention fusion module HAF, the recursive modulation module RM, and the depth prediction module DP. During the depth refinement, the Hybrid Attention Fusion Module (HAF) first performs local residual fusion and global self-attention fusion on the features of the image classifier network and the features extracted by the Swin-Unet network decoder. Finally, it performs selective fusion to generate two spatial attention maps. The spatial attention maps are then multiplied pixel-by-pixel with the output features of the local residual fusion and global self-attention fusion, and the multiplication results are summed to obtain the output features of the Hybrid Attention Fusion Module (HAF). The recursive modulation module (RM) adopts a fully convolutional network structure, which removes the fully connected layers.

2. The depth map completion method according to claim 1, characterized in that, The depth prediction module DP first upscales and merges the image resolution based on interpolation and upsampling. Then, it extracts local features of the upscaled image through 3×3 convolution and downsamples while extracting features through 3×3 strided convolution. Finally, it normalizes the final depth prediction value to between 0 and 1 through the Sigmoid function, which introduces nonlinearity based on the ReLU activation function.

3. The depth map completion method according to claim 1, characterized in that, The method for merging the color image and the sparse depth image in the coarse depth estimation branch is as follows: first, the color image and the sparse depth image are merged based on the Concate function; then, the number of channels in the merged image is adjusted to a 1×1 convolution; and downsampling is performed while extracting features based on a 3×3 strided convolution, wherein the ReLU activation function is used as the activation function.

4. The depth map completion method according to claim 1, characterized in that, In the deep refinement branch, the number of ResNet50 image classifier networks is selected from 1; The number of Hybrid Attention Fusion (HAF) modules is selected from 3; The number of recursive modulation modules (RMs) is selected from 3; The number of depth prediction modules (DP) is selected from 1; The initial parameters of the image classifier network ResNet50 are determined based on the pre-training results of the ImageNet dataset.

5. The depth map completion method according to claim 1, characterized in that, The Swin-Unet network decoder selects features from the first three stages, and the image classifier network selects features of the same scale. The local residual fusion is used to fuse local features. The local residual fusion process includes: first, stitching the images output by the coarse depth estimation branch and the fine depth branch by using the Concate function; then, adjusting the number of channels by 1×1 convolution; and finally, processing by residual blocks, wherein the ReLU activation function is used as the activation function. The global self-attention fusion is used to capture global features, and the global self-attention fusion is calculated based on the BiFormer cross self-attention calculation method; The selective fusion first concatenates the results of the local residual fusion and the global self-attention fusion through the Concate function. Then, the number of channels is adjusted through a 1×1 convolution and the ReLU activation function is used as the activation function. Local features are extracted again through a 3×3 convolution and the ReLU activation function is used as the activation function. Then, the local features are refined again through a 3×3 convolution and two spatial attention maps are generated through the Sigmoid function.

6. The depth map completion method according to claim 1, characterized in that, The recursive modulation module (RM), based on the output of the hybrid attention fusion module (HAF), refines the fourth-stage features of the Swin-Unet network decoder downsampled to 1 / 8 of the original scale, including: ； in, Represents the Adama product. This represents the Sigmoid function. ReLU represents a 3×3 convolution operation, and ReLU represents the ReLU activation function. This indicates a double interpolation upsampling operation. tanh Represents the hyperbolic tangent function. Indicates the first i During the secondary refinement process, the output features of the Hybrid Attention Fusion (HAF) module... Indicates the first i During the secondary refinement process, intermediate features are recursively modulated and refined. Indicates the first i During the secondary refinement process, recursive modulation refinement features are obtained. Indicates the first i During the secondary refinement process, the spatial domain two-dimensional attention map, Indicates the first i The adaptive factor in the secondary refinement process is the initial recursive modulation feature, which is the fourth-stage feature of the Swin-Unet network decoder downsampled to 1 / 8 of the original scale.

7. The depth map completion method according to claim 1, characterized in that, During the training of the dual-branch deep completion network, the loss of the coarse depth estimation branch and the loss of the refined depth branch are summed to obtain the loss of the dual-branch deep completion network based on hybrid attention fusion and feature refinement. The loss function adopts the scale-invariant logarithmic scaling loss function.

8. The depth map completion method according to claim 1, characterized in that, In the training of the dual-branch deep completion network, the Adam gradient optimization algorithm is selected; The base learning rate was set to 0.0001 when training the dual-branch deep completion network, and decreased by 10% every 5 epochs. The parameters... β 1 = 0.9 β 2 = 0.999, and use an attenuation rate of 0.0001; When training the dual-branch deep completion network, the size of the network training parameter Batch is 16; When validating the network weight parameters using the validation set of the publicly available deep completion dataset, the RMSE evaluation metric is selected to determine the magnitude of the error.

Images