A sparse image depth completion method, system and device

Abstract

The present invention discloses a sparse image depth completion method, system and device. First, RGB image data of an image to be processed is obtained through a camera, and sparse depth data depth of the image to be processed is obtained through a laser radar; the RGB image data is used to generate semantic image data semantic of the image to be processed through a semantic segmentation network; then the RGB image data, semantic image data semantic and sparse depth data depth are input into a sparse image depth completion network to obtain a final depth completion result; the sparse image depth completion network of the present invention includes an RGB image feature extraction module, a semantic image feature extraction module, a sparse depth feature extraction module, a MAFF feature fusion module and a weighted fusion module; experiments show that this sparse image depth completion network has higher accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing and image enhancement, and relates to a sparse image depth completion method, system and device, and specifically to a sparse image depth completion method, system and device based on fusion of RGB images and semantic image guidance. Background Art

[0002] Estimating dense depth measurements is crucial in various 3D vision and robotics applications, such as augmented and mixed reality, scene reconstruction, autonomous driving, and obstacle avoidance. To obtain reliable depth estimates in outdoor scenes, measurements from a variety of sensors are used. The most commonly used sensors include RGB cameras in stereo setups, light detection and ranging (LiDAR), and time-of-flight cameras. Among these sensors, LiDAR is considered the most reliable and efficient, capable of providing accurate depth measurements in outdoor environments. However, the density of LiDAR depth measurements is low, with a significant amount of depth data missing. For example, the Velodyne HDL-64e LiDAR sensor used in mobile applications, also used in the KITTI dataset, generates depth maps containing valid depth values ​​for only 5.9% of the pixels. Such sparse depth maps cannot be directly used in the aforementioned applications. Therefore, estimating dense depth maps from sparse measurements is crucial. This is considered a challenging problem, as the measured depth values ​​only account for 5.9% of the complete depth map.

[0003] To address this issue, existing techniques employ deep learning-based methods to achieve dense depth completion. These methods utilize convolutional neural networks to combine sparse LiDAR data with different modalities, such as RGB images, affinity matrices, and surface normals. Huang et al. employed multi-scale features (https: / / ieeexplore.ieee.org / abstract / document / 8946876 / ), while Qiu et al. introduced surface normal information (https: / / openaccess.thecvf.c-om / content_CVPR_2019 / html / Qiu_Deep Li DAR_Deep_Surface_Normal_Guided_D-epth_Prediction_for_Outdoor_Scene_CVP R_2019_paper.html). These modalities act as guides and significantly aid in recovering missing depth values ​​from sparse maps. The idea is to actively fuse features across different modalities. Most existing methods employ a dual-branch network architecture for feature fusion. For example, DeepLiDAR, FusionNet, and PENet utilize an encoder-decoder architecture to perform early and late fusion between color images and LiDAR sparse depth maps to achieve dense depth completion. These methods use convolutional neural networks to combine sparse LiDAR data with different modalities, and the combination of early and late fusion between modalities enhances depth completion capabilities.

[0004] Gu et al. added an additional structural loss (https: / / ieeexplore.ieee.org / abstract / document / 9357967), and Chen et al. combined L2 loss and smooth L1 loss (https: / / openaccess.thecvf.com / content_ICCV_2019 / html / Chen_Learning_Joint_2D_3D_Representations_for_Depth_Completion_ICCV_2019_paper.html). In addition, Uhrig et al. used different sparse invariant convolutions (https: / / ieeexplore.ieee.org / abstract / document / 8374553 / ). Eldesokey et al. added uncertainty exploration (https: / / openaccess.thecvf.com / content_CVPR_2020 / html / Eldesokey_Uncertainty_Aware_CNNs_for_Depth_Completion_Uncertainty_from_Beginning_to_End_CVPR_2020_paper.html), and Tang et al. improved the multimodal fusion strategy (https: / / ieeexplore.ieee.org / abstract / document / 9286883), which further improved performance. Summary of the Invention

[0005] The purpose of this invention is to design a sparse image depth completion network guided by RGB image and semantic image data based on deep learning theory and methods. The three-branch feature extraction module is used to better extract the features of different modal data. The multimodal fusion module is used to better utilize the RGB image data features and semantic image data features to guide the sparse image depth completion to dense depth.

[0006] The technical solution adopted by the method of the present invention is: a sparse image depth completion method, comprising the following steps:

[0007] Step 1: Acquire RGB image data of the image to be processed through a camera, and acquire sparse depth data depth of the image to be processed through a lidar; the RGB image data is used to generate semantic image data semantic of the image to be processed through a semantic segmentation network;

[0008] Step 2: Input the RGB image data, semantic image data semantic and sparse depth data depth into the sparse image depth completion network to obtain the final depth completion result;

[0009] The sparse image depth completion network includes an RGB image feature extraction module, a semantic image feature extraction module, a sparse depth feature extraction module, a MAFF feature fusion module and a weighted fusion module;

[0010] The RGB image feature extraction module takes as input RGB image data and sparse depth data depth, and outputs the depth completion intermediate result C-depth and confidence weight C-confidence;

[0011] The semantic image feature extraction module takes as input semantic image data semantic, sparse depth data depth and C-depth, and outputs depth completion intermediate result S-depth and confidence weight S-confidence;

[0012] The sparse depth feature extraction module takes as input the sparse depth data depth, C-depth and S-depth, and outputs the depth completion intermediate result D-depth and the confidence weight D-confidence;

[0013] The MAFF feature fusion module is used to fuse semantic image features, RGB image features and sparse depth features in the semantic image feature extraction module and the sparse depth feature extraction module;

[0014] The weighted fusion module is used to weightedly fuse the outputs of the RGB image feature extraction module, the semantic image feature extraction module, and the sparse depth feature extraction module.

[0015] The technical solution adopted by the system of the present invention is: a sparse image depth completion system, including an information acquisition module and a depth completion module:

[0016] The information acquisition module is used to acquire RGB image data of the image to be processed through a camera and acquire sparse depth data depth of the image to be processed through a lidar; the RGB image data is used to generate semantic image data semantic of the image to be processed through a semantic segmentation network;

[0017] The depth completion module is used to input the RGB image data, the semantic image data semantic and the sparse depth data depth into the sparse image depth completion network to obtain the final depth completion result;

[0018] The sparse image depth completion network includes an RGB image feature extraction module, a semantic image feature extraction module, a sparse depth feature extraction module, a MAFF feature fusion module and a weighted fusion module;

[0019] The RGB image feature extraction module takes as input RGB image data and sparse depth data depth, and outputs the depth completion intermediate result C-depth and confidence weight C-confidence;

[0020] The semantic image feature extraction module takes as input semantic image data semantic, sparse depth data depth and C-depth, and outputs depth completion intermediate result S-depth and confidence weight S-confidence;

[0021] The sparse depth feature extraction module takes as input the sparse depth data depth, C-depth and S-depth, and outputs the depth completion intermediate result D-depth and the confidence weight D-confidence;

[0022] The MAFF feature fusion module is used to fuse semantic image features, RGB image features and sparse depth features in the semantic image feature extraction module and the sparse depth feature extraction module;

[0023] The weighted fusion module is used to weightedly fuse the outputs of the RGB image feature extraction module, the semantic image feature extraction module, and the sparse depth feature extraction module.

[0024] The technical solution adopted by the device of the present invention is: a sparse image depth completion device, comprising:

[0025] one or more processors;

[0026] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the sparse image depth completion method.

[0027] Based on a thorough analysis of the insufficient features of the depth completion model, this paper proposes a new depth completion model from three aspects: changing the network structure, adding semantic image data and semantic branches, and introducing a multimodal fusion module MAFF. Experiments show that this improved model has a higher prediction accuracy than the depth completion model with two branches and only RGB as the guide. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a diagram showing the structure of a sparse image depth completion network according to an embodiment of the present invention;

[0029] Figure 2This is a structural diagram of the MAFF feature fusion module according to an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of the sparse image depth completion network training according to an embodiment of the present invention;

[0031] Figure 4 Schematic diagram of depth completion results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0033] The present invention provides a sparse image depth completion method, comprising the following steps:

[0034] Step 1: Obtain RGB image data of the image to be processed through the camera, and obtain sparse depth data of the image to be processed through the lidar; the RGB image data is used to generate semantic image data of the image to be processed through the semantic segmentation network;

[0035] Step 2: Input the RGB image data, semantic image data semantic and sparse depth data depth into the sparse image depth completion network to obtain the final depth completion result;

[0036] Please see Figure 1 The sparse image depth completion network of this embodiment includes an RGB image feature extraction module, a semantic image feature extraction module, a sparse depth feature extraction module, a MAFF feature fusion module, and a weighted fusion module; the RGB image feature extraction module takes RGB image data and sparse depth data depth as input, and outputs a depth completion intermediate result C-depth and a confidence weight C-confidence; the semantic image feature extraction module takes semantic image data semantic, sparse depth data depth, and C-depth as input, and outputs a depth completion intermediate result S-depth and a confidence weight S-confidence; the sparse depth feature extraction module takes sparse depth data depth, C-depth, and S-depth as input, and outputs a depth completion intermediate result D-depth and a confidence weight D-confidence; the MAFF feature fusion module is used to fuse semantic image features, RGB image features, and sparse depth features in the semantic image feature extraction module and the sparse depth feature extraction module; and the weighted fusion module is used to perform weighted fusion of the outputs of the RGB image feature extraction module, the semantic image feature extraction module, and the sparse depth feature extraction module.

[0037] The network architecture of the RGB image data feature extraction module and the semantic image data feature extraction module of this embodiment is based on ResNet. The input of the RGB image data feature extraction module is the RGB image data obtained in the data preparation process. The data starts from the network input layer and passes through the 0th conventional convolution layer of the RGB image feature extraction module, the 1st residual convolution module, the 2nd residual convolution module, the 3rd residual convolution module, the 4th residual convolution module, the 5th residual convolution module, the 1st deconvolution module, the 2nd deconvolution module, the 3rd deconvolution module, the 4th deconvolution module, the 5th deconvolution module, the 6th residual convolution module, the 7th residual convolution module, the 8th residual convolution module, the 9th residual convolution module, the 10th residual convolution module, the 6th deconvolution module, the 7th deconvolution module, the 8th After the deconvolution module, the 9th deconvolution module, the 10th deconvolution module, and the 1st regular convolution output layer, the feature maps cd_0, cd_1, cd_2, cd_3, cd_4, cd_5, cd_6, cd_7, cd_8, cd_9, cd_10, cd_11, cd_12, cd_13, cd_14, cd_15, cd_16, cd_17, cd_18, cd_19, cd_20, cd_210, cd_225, cd_23, cd_24, cd_25, cd_26, cd_27, cd_28, cd_29, cd_30, cd_31, cd_32, cd_33, cd_34, cd_35, cd_36, cd_37, cd_38, cd_39, cd_40, cd_41, cd_42, cd_43, cd_44, cd_45, cd_46, cd_47, cd_48, cd_49, cd_50, cd_51, cd_52 608×160×64 deconvolution feature map d_cd_4, 1216×320×32 deconvolution feature map d_cd_5, 608×160×64 feature map cd_6, 304×80×128 feature map cd_7, 152×40×256 feature map cd_8, 76×20×512 feature map cd_9, 38×10×1024 feature map cd_10, 76×20×512 deconvolution feature map The product feature map d_cd_6, the deconvolution feature map d_cd_7 of 152×40×256, the deconvolution feature map d_cd_8 of 304×80×128, the deconvolution feature map d_cd_9 of 608×160×64, and the deconvolution feature map d_cd_10 of 1216×320×32 are added accordingly, cd_1 and d_cd_4, cd_2 and d_cd_3, cd_3 and d_cd_2, cd_4 and d_cd_1, and cd_0 and d_cd_5 are added accordingly to generate the first new feature map, the second new feature map, the third new feature map, the fourth new feature map, and the fifth new feature map of unchanged size. The first new feature map is added to cd_6, the second new feature map is added to cd_7, the third new feature map is added to cd_8, and the fourth new feature map is added to cd_9 to generate a sixth new feature map, a seventh new feature map, an eighth new feature map, and a ninth new feature map of unchanged size.The ninth new feature map is added to d_cd_6, the eighth new feature map is added to d_cd_7, the seventh new feature map is added to d_cd_8, the sixth new feature map is added to d_cd_9, and the fifth new feature map is added to d_cd_10 to generate the tenth new feature map, the eleventh new feature map, the twelfth new feature map, the fourteenth new feature map, and the fourteenth new feature map of the same size. The first to thirteenth feature maps are used as the fifth deconvolution module, the fourth deconvolution module, the third deconvolution module, and the second deconvolution module respectively. , the inputs of the 6th residual convolution module, the 7th residual convolution module, the 8th residual convolution module, the 9th residual convolution module, the 10th residual convolution module, the 6th deconvolution module, the 7th deconvolution module, the 8th deconvolution module, the 9th deconvolution module, and the 10th deconvolution module continue to participate in forward propagation; finally, the first regular convolution layer outputs a confidence weight CD-confidence of size 1216×352×1 and a C-depth of size 1216×352×1.

[0038] The input of the semantic image data feature extraction module of this embodiment is the output C-depth of the RGB feature extraction module and the semantic image in the data preparation process, which passes through the 0th conventional convolution layer of the semantic image data feature extraction module, the 1st fusion module, the 1st residual convolution module, the 2nd fusion module, the 2nd residual convolution module, the 3rd fusion module, the 3rd residual convolution module, the 4th fusion module, the 4th residual convolution module, the 5th fusion module, the 5th residual convolution module, the 1st deconvolution module, the 2nd deconvolution module, the 3rd deconvolution module, the 4th deconvolution module, the 5th deconvolution module, the 5th deconvolution module, the 6th deconvolution module, the 7th deconvolution module, the 8th deconvolution module, the 9th deconvolution module, the 10th deconvolution module, the 11th deconvolution module, the 12th deconvolution module, the 13th deconvolution module, the 14th deconvolution module, the 15th deconvolution module, the 16th deconvolution module, the 17th deconvolution module, the 18th deconvolution module, the 19th deconvolution module, the 20th deconvolution module, the 21st deconvolution module, the 22nd deconvolution module, the 23rd deconvolution module, the 24th deconvolution module, the 25th After the deconvolution module, the 6th residual convolution module, the 7th residual convolution module, the 8th residual convolution module, the 9th residual convolution module, the 10th residual convolution module, the 6th deconvolution module, the 7th deconvolution module, the 8th deconvolution module, the 9th deconvolution module, the 10th deconvolution module, and the 1st regular convolution output layer, the feature maps sd_0, sd_1 and sd_2 of sizes 1216×320×32 are obtained respectively. 1216×320×32 feature map F1, 608×160×64 feature map sd_1, 608×160×64 feature map F2, 304×80×128 feature map sd_2, 304×80×128 feature map F3, 152×40×256 feature map sd_3, 152×40×256 feature map F4, 76×20×512 feature map sd_4, 76×20×512 feature map F5, 76×20×512 feature map 10×18×128 024 feature map sd_5, 76×20×512 deconvolution feature map d_sd_1, 152×40×256 deconvolution feature map d_sd_2, 304×80×128 deconvolution feature map d_sd_3, 608×160×64 deconvolution feature map d_sd_4, 1216×320×32 deconvolution feature map d_sd_5, 608×160×64 feature map sd_6, 304×80×128 feature map sd_7, 152×40×256 The fourteenth new feature map generated by the RGB image feature extraction module is added to sd_0, the thirteenth new feature map is added to sd_1, the twelfth new feature map is added to sd_2, the eleventh new feature map is added to sd_3, and the tenth new feature map is added to sd_4.Generate new feature maps A, B, C, D, and E, all of unchanged size. Add new feature map E to d_sd_1, new feature map D to d_sd_2, new feature map C to d_sd_3, new feature map B to d_sd_4, and new feature map A to d_sd_5, respectively, to generate new feature maps F, G, H, I, and J, all of unchanged size. Add new feature map I to sd_6, new feature map H to sd_7, new feature map G to sd_8, and new feature map F to sd_9, respectively, to generate new feature maps K, L, M, and N, all of unchanged size. The new feature map N is added to d_sd_6, the new feature map M is added to d_sd_7, the new feature map L is added to d_sd_8, the new feature map K is added to d_sd_9, and the new feature map J is added to d_sd_10 to generate new feature maps O, P, Q, R, and S of unchanged size. New feature map A, feature map F6, new feature map B, feature map F7, new feature map C, feature map F8, new feature map D, feature map F9, new feature map E, feature map F10, new feature map F, new feature map G, new feature map H, new feature map I, new feature map J, new feature map K, new feature map L, new feature map M, new feature map N, feature map sd_10, new feature map O, new feature map P, new feature map Q, new feature map R, and new feature map S are added as the first fusion module, the first residual convolution module, the second fusion module, the second residual convolution module, the third fusion module, the third residual convolution module, the fourth fusion module, the fourth residual convolution module, the fifth fusion module, the fifth residual convolution module, and the sixth residual convolution module, respectively. The inputs of the first deconvolution module, the 7th deconvolution module, the 8th deconvolution module, the 9th deconvolution module, the 10th deconvolution module, the 7th residual convolution module, the 8th residual convolution module, the 9th residual convolution module, the 10th residual convolution module, the 6th deconvolution module, the 7th deconvolution module, the 8th deconvolution module, the 9th deconvolution module, and the 10th deconvolution module continue to participate in forward propagation; finally, the first regular convolution layer outputs a weight SD-confidence of size 1216×320×1 and an S-depth of size 1216×320×1. ,

[0039] The input of the sparse depth map data feature extraction module of this embodiment is the output C-depth of the RGB feature extraction module, the output S-depth of the semantic image feature extraction module, and the sparse depth image data, which passes through the 0th conventional convolution layer of the sparse depth image data feature extraction module, the 1st fusion module, the 1st residual convolution module, the 2nd residual convolution module, the 2nd fusion module, the 3rd residual convolution module, the 4th residual convolution module, the 3rd fusion module, the 5th residual convolution module, the 6th residual convolution module, the 4th fusion module, the 7th residual convolution module, the 8th residual convolution module, the 5th fusion module, the 9th residual convolution module, the 10th residual convolution module, the 1st deconvolution module, the 2nd deconvolution module, the 3rd deconvolution module, the 4th deconvolution module. Module, the 5th deconvolution module, the 6th residual convolution module, the 7th residual convolution module, the 8th residual convolution module, the 9th residual convolution module, the 10th residual convolution module, the 6th deconvolution module, the 7th deconvolution module, the 8th deconvolution module, the 9th deconvolution module, the 10th deconvolution module, and the 1st conventional convolution output layer respectively obtain the feature map d_0 of size 1216×320×32, the feature map F6 of size 1216×320×32, 608×160×64 feature map d_1, 608×160×64 feature map d_2, 608×160×64 feature map F7, 304×80×128 feature map d_3, 304×80×128 feature map d_4, 304×80×128 feature map F8, 152×40×256 feature map d_5, 152×40×256 feature map d_6, 152×40×256 feature map F9, 76×20×512 feature map d_7, 76×20×512 feature map d_8, 76×20×512 feature map F10, 38×10×1024 feature map d_9, 38×10×1024 feature map d_10, 76×20×512 deconvolution feature map d_d_1, 152×40×256 deconvolution feature map d_d_2, 304×80×128 deconvolution feature map d_d_3, 608×160×64 deconvolution feature map d_d_4, 1216×320×32 deconvolution feature map d_d_5, d_0 and new feature map S, d_2 and new feature map R, d_4 and new feature map Q, d_6 and new feature map P, d_8 is added to the new feature map O to generate new feature maps ①, ②, ③, ④, and ⑤ of unchanged size. New feature map ⑤ is added to d_d_1, ④, ③, d_d_2, ②, d_d_4, and ① to d_d_5 to generate new feature maps ⑥, ⑦, ⑧, and ⑨, which are unchanged in size. d_0, new feature map ①, d_1, d_2, new feature map ②, d_3, d_4, new feature map ③, d_5, d_6, new feature map ④, d_7, d_8, new feature map ⑤, d_9, d_10, feature map ⑥, new feature map ⑦, new feature map ⑧, new feature map ⑨, new feature map They are added as the 6th fusion module, the 1st residual convolution module, the 2nd residual convolution module, the 7th fusion module, the 3rd residual convolution module, the 4th residual convolution module, the 8th fusion module, the 5th residual convolution module, the 6th residual convolution module, the 9th fusion module, the 7th residual convolution module, the 8th residual convolution module, the 10th fusion module, the 9th residual convolution module, the 10th residual convolution module, the 1st deconvolution module, the 2nd deconvolution module, the 3rd deconvolution module, and the 4th deconvolution module respectively; the input of the 5th deconvolution module and the 1st regular convolution layer output layer continue to participate in forward propagation; finally, the first regular convolution layer outputs a confidence weight D-confidence of size 1216×320×1 and a D-depth of size 1216×320×1.

[0040] Please see Figure 2 The MAFF feature fusion module of this embodiment is composed of a sequentially connected splicing layer, a local attention layer and a global attention layer set in parallel, a Sigmoid layer, a local attention layer and a global attention layer set in parallel, and a Sigmoid layer; the local attention layer is composed of a regular convolution layer, a BN layer, a Leak_Relu activation function layer, a regular convolution layer, and a BN layer in sequence; the global attention layer is composed of a global pooling layer, a regular convolution layer, a BN layer, a Relu activation function layer, a regular convolution layer, and a BN layer in sequence.

[0041] The first RGB image feature (RGB), semantic image feature (semantic), sparse depth map feature (depth) splicing (Cat feature ) are input to the local attention module and the global attention module respectively, and then input to the Sigmoid layer to output Att map1 , the formula is as follows:

[0042] Fusion1=Cat feature *Att map1 +(1-Att map1 )dept;

[0043] The second fusion is input into the local attention module and the global attention module respectively, and then into the Sigmoid layer to output Att map2 The formula is as follows:

[0044] Fusion2=Cat feature *Att map2 +(1-Att map2 )dept.

[0045] The sparse image depth completion network in this embodiment is a trained sparse image depth completion network; see Figure 3 , its training process includes the following sub-steps:

[0046] Step 2.1: Obtain several RGB images. Each RGB image is passed through a semantic segmentation network to generate semantic image data. The RGB image data, semantic image data, sparse depth data depth, and true depth labels constitute the dataset.

[0047] The dataset is divided into training set, test set and validation set; both the training set and validation set contain RGB image data and corresponding semantic image data semantic, sparse depth data depth and true depth labels; the test set contains only RGB image data and sparse depth data depth;

[0048] The dataset used in this example to train the sparse image depth completion network is the KITTI open source dataset. KITTI (https: / / www.shapenet.org / ) is one of the most authoritative datasets in the field of depth completion. It contains more than 93,000 depth maps and their corresponding radar scan information and RGB image information. It also provides the corresponding camera parameters for each image. Therefore, the RGB image data information of this dataset can be used to implement depth completion guided by RGB image data.

[0049] The KITTI open source dataset provides RGB image data and corresponding sparse depth maps, which are obtained by projecting 3D lidar points onto corresponding image frames. The resolution of the RGB image data is 1241×376. In the depth map, approximately 5% of the pixels are valid (depth value > 0), while 16% of the pixels in the corresponding dense depth labels are valid. This dataset does not contain the corresponding semantic map, nor does it have the corresponding label data for semantic map training, so this embodiment uses a trained semantic segmentation model to convert the RGB map into semantic map data.

[0050] The dataset contains 86,000 training samples, 7,000 validation samples, and 1,000 test samples. The corresponding semantic images are generated by the HRNet network model, and each sample corresponds to 1 RGB image data, 1 sparse depth map data, 1 semantic image data, 1 depth label and camera parameters (5 samples and their corresponding camera parameters are randomly selected as input during training).

[0051] To eliminate the impact of abnormal data, this example uses the Numpy library (Python) to normalize the RGB image data, semantic image data, and sparse depth data, limiting the pixel values ​​and depth values ​​of these data to the range [0, 1]. To ensure data diversity, the four data types in each sample (RGB image data, semantic image data, sparse depth data, and true depth label data) are randomly cropped to the same size, unifying the image size to 1216×320.

[0052] Step 2.2: Set the loss function weight parameters, optimization method, learning rate and maximum number of iterations;

[0053] The loss function is L total :

[0054] L total =L(D)+αL(D cs )+βL(D csd );

[0055]

[0056] Among them, L(D) represents the main loss, L(D cs ) indicates that the depth completion intermediate result loss is one, L(D csa ) represents the loss of the intermediate result of depth completion; α and β are both hyperparameters, which are set to α=β=0.2 in the initial stage of training and decrease to 0 as the number of training rounds increases; P v Indicates the pixels with valid depth values ​​in the true depth labels of the training samples, p is a single pixel, represents the true depth label of the training sample, gt represents the actual depth label, D p Represents the prediction result, ||X|| represents the two-norm of X;

[0057] Step 2.3: Input the RGB image data, semantic image data, sparse depth data C-depth and true depth label data in the training set into the sparse image depth completion network for network training, and calculate the loss with the true depth label and backpropagate the gradient;

[0058] Step 2.4: In this example, the network model parameters are trained for 100 epochs. After each epoch, the model is validated on the validation set and the RMSE error is calculated. Training is terminated if the RMSE error does not decrease for 10 consecutive epochs.

[0059] In this embodiment, the training is performed for 10 rounds at a learning rate of 0.001, then the training is performed for 10 rounds at a learning rate of 0.0005, the training is performed for 10 rounds at a learning rate of 0.0001, and the training is performed for 10 rounds at a learning rate of 0.00001. The best performing model in the validation set is saved.

[0060] Step 2.5: The model with the lowest RMSE error in the validation set is considered the trained sparse image depth completion network. The model is then tested on the test set to evaluate its generalization ability.

[0061] This example evaluates the proposed 3D reconstruction model based on the KITTI dataset. The evaluation metrics used are RMSE, MAE, iRMSE, and iMAE (RMSE stands for root mean square error, MAE stands for mean absolute error, iRMSE stands for the root mean square of the inverse of the error between the true value and the predicted value, and iMAE stands for the mean absolute value of the inverse of the error between the true value and the predicted value). The model's actual performance in depth completion is also evaluated. The test set is divided using the dataset division strategy of Hu et al. (86,000 sets of data are used for training the model, 7,000 sets of data are used for validating the model, and 1,000 sets of data are used for testing the model). The RMSE error of the completion result is 758.549 mm, and the MAE is 207.171 mm.

[0062] In addition, this embodiment also compares the actual performance of these models in reconstructing shapes, such as Figure 4 (From left to right: RGB image data, discrete depth data, actual depth labels, and the model's completion results.) Compared to the discrete depth data, the model's completion results partially complete the depth values. Compared with the actual depth labels, the completed depth values ​​are effective and reliable.

[0063] Based on a thorough analysis of the insufficient features of the depth completion model, this paper changes the network structure, adds semantic image features and RGB images for simultaneous guidance, and adds a multimodal fusion module to fuse features of different modal data to more accurately guide depth completion. Experiments have shown that it has good prediction accuracy.

[0064] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A sparse image depth completion method, characterized in that: The following steps are involved: Step 1: Obtain RGB image data of the image to be processed through the camera, and obtain sparse depth data depth of the image to be processed through the lidar; The RGB image data is used to generate semantic image data of the image to be processed through a semantic segmentation network; Step 2: Input the RGB image data, semantic image data semantic and sparse depth data depth into the sparse image depth completion network to obtain the final depth completion result; The sparse image depth completion network includes an RGB image feature extraction module, a semantic image feature extraction module, a sparse depth feature extraction module, a MAFF feature fusion module and a weighted fusion module; The RGB image feature extraction module takes as input RGB image data and sparse depth data depth, and outputs the depth completion intermediate result C-depth and confidence weight C-confidence; The semantic image feature extraction module takes as input semantic image data semantic, sparse depth data depth and C-depth, and outputs depth completion intermediate result S-depth and confidence weight S-confidence; The sparse depth feature extraction module takes as input the sparse depth data depth, C-depth and S-depth, and outputs the depth completion intermediate result D-depth and the confidence weight D-confidence; The MAFF feature fusion module is used to fuse semantic image features, RGB image features and sparse depth features in the semantic image feature extraction module and the sparse depth feature extraction module; The weighted fusion module is used to weightedly fuse the outputs of the RGB image feature extraction module, the semantic image feature extraction module, and the sparse depth feature extraction module.

2. The sparse image depth completion method according to claim 1, characterized in that: The RGB image feature extraction module includes a 0th conventional convolution layer, a 1st residual convolution module, a 2nd residual convolution module, a 3rd residual convolution module, a 4th residual convolution module, a 5th residual convolution module, a 1st deconvolution module, a 2nd deconvolution module, a 3rd deconvolution module, a 4th deconvolution module, a 5th deconvolution module, a 6th residual convolution module, a 7th residual convolution module, an 8th residual convolution module, a 9th residual convolution module, a 10th residual convolution module, a 6th deconvolution module, a 7th deconvolution module, a 8th deconvolution module, a 9th deconvolution module, a 10th deconvolution module, The first conventional convolution output layer, each layer respectively obtains a feature map cd_0 of size 1216×320×32, a feature map cd_1 of size 608×160×64, a feature map cd_2 of size 304×80×128, a feature map cd_3 of size 152×40×256, a feature map cd_4 of size 76×20×512, a feature map cd_5 of size 38×10×1024, a deconvolution feature map d_cd_1 of size 76×20×512, a deconvolution feature map d_cd_2 of size 152×40×256, a deconvolution feature map d_cd_3 of size 304×80×128, a deconvolution feature map d_cd_4 of size 608×160×64, and a deconvolution feature map d_cd_5 of size 38×10×1024. Figure d_cd_4, 1216×320×32 deconvolution feature map d_cd_5, 608×160×64 feature map cd_6, 304×80×128 feature map cd_7, 152×40×256 feature map cd_8, 76×20×512 feature map cd_9, 38×10×1024 feature map cd_10, 76×20×512 deconvolution feature map d_cd_6, 152×40×256 deconvolution feature map d_cd_7, 304×80×128 deconvolution feature map d_cd_8, 608×160×64 deconvolution feature map d_cd_ 9. The deconvolution feature map d_cd_10 of 1216×320×32 is added accordingly, cd_1 and d_cd_4, cd_2 and d_cd_3, cd_3 and d_cd_2, cd_4 and d_cd_1, cd_0 and d_cd_5 to generate the first new feature map, the second new feature map, the third new feature map, the fourth new feature map, and the fifth new feature map of unchanged size; the first new feature map and cd_6, the second new feature map and cd_7, the third new feature map and cd_8, and the fourth new feature map and cd_9 are added accordingly to generate the sixth new feature map, the seventh new feature map, the eighth new feature map, and the ninth new feature map of unchanged size;The ninth new feature map is added to d_cd_6, the eighth new feature map is added to d_cd_7, the seventh new feature map is added to d_cd_8, the sixth new feature map is added to d_cd_9, and the fifth new feature map is added to d_cd_10 to generate the tenth new feature map, the eleventh new feature map, the twelfth new feature map, the fourteenth new feature map, and the fourteenth new feature map of the same size. The first to thirteenth feature maps are used as the fifth deconvolution module, the fourth deconvolution module, the third deconvolution module, the second deconvolution module, the sixth deconvolution module, and the ninth deconvolution module, respectively. The inputs of the residual convolution module, the 7th residual convolution module, the 8th residual convolution module, the 9th residual convolution module, the 10th residual convolution module, the 6th deconvolution module, the 7th deconvolution module, the 8th deconvolution module, the 9th deconvolution module, and the 10th deconvolution module continue to participate in forward propagation; finally, the first regular convolution layer outputs the confidence weight C-confidence of size 1216×352×1 and the depth completion intermediate result C-depth of size 1216×352×1.

3. The sparse image depth completion method according to claim 1, wherein: The semantic image feature extraction module includes a sequentially connected 0th conventional convolution layer, a 1st fusion module, a 1st residual convolution module, a 2nd fusion module, a 2nd residual convolution module, a 3rd fusion module, a 3rd residual convolution module, a 4th fusion module, a 4th residual convolution module, a 5th fusion module, a 5th residual convolution module, a 1st deconvolution module, a 2nd deconvolution module, a 3rd deconvolution module, a 4th deconvolution module, a 5th deconvolution module, a 6th residual convolution module, a 7th residual convolution module, an 8th residual convolution module, a 9th residual convolution module, a 10th residual convolution module, a 6th deconvolution module, a 7th deconvolution module, and a 8th deconvolution module. The convolution module, the 9th deconvolution module, the 10th deconvolution module and the 1st regular convolution output layer respectively obtain the feature map sd_0 of size 1216×320×32, the feature map F1 of size 1216×320×32, the feature map sd_1 of size 608×160×64, the feature map F2 of size 608×160×64, the feature map sd_2 of size 304×80×128, the feature map F3 of size 304×80×128, the feature map sd_3 of size 152×40×256, the feature map F4 of size 152×40×256, the feature map sd_4 of size 76×20×512, the feature map F5 of size 76×20×512 38×10×1024 feature map sd_5, 76×20×512 deconvolution feature map d_sd_1, 152×40×256 deconvolution feature map d_sd_2, 304×80×128 deconvolution feature map d_sd_3, 608×160×64 deconvolution feature map d_sd_4, 1216×320×32 deconvolution feature map d_sd_5, 608×160×64 feature map sd_6, 304×80×128 feature map sd_7, 152×40×256 feature map sd_8, 76×20×512 feature map sd_9, 38×10×1024 feature map sd_10, 76× The 20×512 deconvolution feature map d_sd_6, the 152×40×256 deconvolution feature map d_sd_7, the 304×80×128 deconvolution feature map d_sd_8, the 608×160×64 deconvolution feature map d_sd_9, and the 1216×320×32 deconvolution feature map d_sd_10 are added to the fourteenth new feature map generated by the RGB image feature extraction module and sd_0, the thirteenth new feature map and sd_1, the twelfth new feature map and sd_2, the eleventh new feature map and sd_3, and the tenth new feature map and sd_4, respectively, to generate new feature maps A, B, C, D, and E of unchanged size;The new feature map E is added to d_sd_1, the new feature map D is added to d_sd_2, the new feature map C is added to d_sd_3, the new feature map B is added to d_sd_4, and the new feature map A is added to d_sd_5 to generate new feature maps F, G, H, I, and J with unchanged sizes; the new feature map I is added to sd_6, H and sd_7, G and sd_8, and F and sd_9 to generate new feature maps K, L, M, and N with unchanged sizes; the new feature map The new feature map N and d_sd_6, the new feature map M and d_sd_7, the new feature map L and d_sd_8, the new feature map K and d_sd_9, and the new feature map J and d_sd_10 are added accordingly to generate new feature maps O, P, Q, R, and S of unchanged size; new feature maps A, F6, B, F7, C, F8, D, F9, E, F10, F, G, H, and I. , new feature map J, new feature map K, new feature map L, new feature map M, new feature map N, feature map sd_10, new feature map O, new feature map P, new feature map Q, new feature map R, and new feature map S are added as the first fusion module, the first residual convolution module, the second fusion module, the second residual convolution module, the third fusion module, the third residual convolution module, the fourth fusion module, the fourth residual convolution module, the fifth fusion module, the fifth residual convolution module, the sixth deconvolution module, the seventh deconvolution module, and the eighth deconvolution module respectively. The inputs of the 9th deconvolution module, the 10th deconvolution module, the 7th residual convolution module, the 8th residual convolution module, the 9th residual convolution module, the 10th residual convolution module, the 6th deconvolution module, the 7th deconvolution module, the 8th deconvolution module, the 9th deconvolution module, and the 10th deconvolution module continue to participate in forward propagation; finally, the first regular convolution layer outputs the weight SD-confidence of size 1216×320×1 and the depth completion intermediate result S-depth of size 1216×320×1.

4. The sparse image depth completion method according to claim 1, wherein: The sparse depth feature extraction module includes a sequentially connected 0th conventional convolution layer, a 1st fusion module, a 1st residual convolution module, a 2nd residual convolution module, a 2nd fusion module, a 3rd residual convolution module, a 4th residual convolution module, a 3rd fusion module, a 5th residual convolution module, a 6th residual convolution module, a 4th fusion module, a 7th residual convolution module, an 8th residual convolution module, a 5th fusion module, a 9th residual convolution module, a 10th residual convolution module, a 1st deconvolution module, a 2nd deconvolution module, a 3rd deconvolution module, a 4th deconvolution module, a 5th deconvolution module, a 6th residual convolution module, a 7th residual convolution module Convolution module, the 8th residual convolution module, the 9th residual convolution module, the 10th residual convolution module, the 6th deconvolution module, the 7th deconvolution module, the 8th deconvolution module, the 9th deconvolution module, the 10th deconvolution module, the 1st regular convolution output layer, each layer obtains a feature map d_0 of size 1216×320×32, a feature map F6 of size 1216×320×32, a feature map d_1 of size 608×160×64, a feature map d_2 of size 608×160×64, a feature map F7 of size 608×160×64, a feature map d_3 of size 304×80×128, a feature map d_4 of size 304 ×80×128 feature map F8, 152×40×256 feature map d_5, 152×40×256 feature map d_6, 152×40×256 feature map F9, 76×20×512 feature map d_7, 76×20×512 feature map d_8, 76×20×512 feature map F10, 38×10×1024 feature map d_9, 38×10×1024 feature map d_10, 76×20×512 deconvolution feature map d_d_1, 152×40×256 deconvolution feature map d_d_2, 304×80×128 deconvolution feature map d_d_3, 608×160 ×64 deconvolution feature map d_d_4, 1216×320×32 deconvolution feature map d_d_5, d_0 and new feature map S, d_2 and new feature map R, d_4 and new feature map Q, d_6 and new feature map P, d_8 and new feature map O are added respectively to generate new feature maps ①, ②, ③, ④ and ⑤ of unchanged size. New feature map ⑤ is added to d_d_1, new feature map ④ and d_d_2, new feature map ③ and d_d_3, new feature map ② and d_d_4, and new feature map ① and d_d_5 to generate new feature maps ⑥, ⑦, ⑧, ⑨ and ⑩ of unchanged size.d_0, new feature map ①, d_1, d_2, new feature map ②, d_3, d_4, new feature map ③, d_5, d_6, new feature map ④, d_7, d_8, new feature map ⑤, d_9, d_10, feature map ⑥, new feature map ⑦, new feature map ⑧, new feature map ⑨, and new feature map ⑩ are added as the 6th fusion module, the 1st residual convolution module, the 2nd residual convolution module, the 7th fusion module, the 3rd residual convolution module, the 4th residual convolution module, the 8th fusion module, the 5th residual convolution module, the 6th residual convolution module, the 9th The fusion module, the 7th residual convolution module, the 8th residual convolution module, the 10th fusion module, the 9th residual convolution module, the 10th residual convolution module, the 1st deconvolution module, the 2nd deconvolution module, the 3rd deconvolution module, and the 4th deconvolution module are all forward propagated. The inputs of the 5th deconvolution module and the output layer of the 1st regular convolution layer continue to participate in forward propagation. Finally, the first regular convolution layer outputs a confidence weight D-confidence of size 1216×320×1 and a depth completion intermediate result D-depth of size 1216×320×1.

5. The sparse image depth completion method according to claim 1, wherein: The MAFF feature fusion module consists of a sequentially connected splicing layer, a local attention layer and a global attention layer set in parallel, a Sigmoid layer, a local attention layer and a global attention layer set in parallel, and a Sigmoid layer; the local attention layer is sequentially composed of 1 regular convolution layer, 1 BN layer, 1 Leak_Relu activation function layer, 1 regular convolution layer and 1 BN layer; the global attention layer is sequentially composed of 1 global pooling layer, 1 regular convolution layer, 1 BN layer, Relu activation function layer, 1 regular convolution layer and 1 BN layer.

6. The sparse image depth completion method according to any one of claims 1 to 5, characterized in that: The sparse image depth completion network described in step 2 is a trained sparse image depth completion network; The training process includes the following sub-steps: Step 2.1: Obtain several RGB images. Each of the RGB images generates semantic image data semantic through a semantic segmentation network. The RGB image data, semantic image data semantic, sparse depth data depth and true depth labels constitute a dataset; The dataset is divided into a training set, a test set, and a validation set; the training set and the validation set both contain RGB image data and corresponding semantic image data semantic, sparse depth data depth, and true depth labels; The test set only contains RGB image data and sparse depth data depth; Step 2.2: Set the loss function weight parameters, optimization method, learning rate and maximum number of iterations; The loss function is L total : L total =L(D)+αL(D cs )+βL(D csd ); Among them, L(D) represents the main loss, L(D cs ) indicates that the depth completion intermediate result loss is one, L(D csd ) represents the loss of the intermediate result of depth completion; α and β are both hyperparameters, which are set to α=β=0.2 in the initial stage of training and decrease to 0 as the number of training rounds increases; P v Indicates the pixels with valid depth values ​​in the true depth labels of the training samples, p is a single pixel, represents the true depth label of the training sample, gt represents the actual depth label, D p Represents the prediction result, ||X|| represents the two-norm of X; Step 2.3: Input the RGB image data, semantic image data, sparse depth data C-depth and true depth label data in the training set into the sparse image depth completion network in turn for network training, and calculate the loss with the true depth label and back-propagate the gradient; Step 2.4: Set T epochs to train the network parameters. After each epoch, the network is validated on the validation set and the RMSE error is calculated. Training is stopped when the RMSE error does not decrease within R consecutive epochs. T and R are preset values. Step 2.5: The network with the smallest RMSE error in the validation set is used as the trained sparse image depth completion network.

7. A sparse image depth completion system, characterized by: Including information acquisition module and depth completion module: The information acquisition module is used to acquire RGB image data of the image to be processed through a camera and acquire sparse depth data depth of the image to be processed through a lidar; the RGB image data is used to generate semantic image data semantic of the image to be processed through a semantic segmentation network; The depth completion module is used to input RGB image data, semantic image data semantic and sparse depth data depth into the sparse image depth completion network to obtain the final depth completion result; The sparse image depth completion network includes an RGB image feature extraction module, a semantic image feature extraction module, a sparse depth feature extraction module, a MAFF feature fusion module and a weighted fusion module; The RGB image feature extraction module takes as input RGB image data and sparse depth data depth, and outputs the depth completion intermediate result C-depth and confidence weight C-confidence; The semantic image feature extraction module takes as input semantic image data semantic, sparse depth data depth and C-depth, and outputs depth completion intermediate result S-depth and confidence weight S-confidence; The sparse depth feature extraction module takes as input the sparse depth data depth, C-depth and S-depth, and outputs the depth completion intermediate result D-depth and the confidence weight D-confidence; The MAFF feature fusion module is used to fuse semantic image features, RGB image features and sparse depth features in the semantic image feature extraction module and the sparse depth feature extraction module; The weighted fusion module is used to weightedly fuse the outputs of the RGB image feature extraction module, the semantic image feature extraction module, and the sparse depth feature extraction module.

8. A sparse image depth completion device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the sparse image depth completion method according to any one of claims 1 to 6.

Images