Road scene depth completion method based on semantic information and color image guidance

Abstract

The application discloses a road scene depth completion method based on semantic information and color image guidance, comprising the following steps: a double-branch network composed of a color image guidance branch and a semantic guidance branch is designed, and semantic information is introduced into the network, so that road scene depth completion is carried out. By using the idea of multi-task learning, the backbone shared with the depth map prediction layer in the color image guidance branch is only needed to add a semantic segmentation layer, and the whole network does not need to be increased, so that the semantic information can be obtained. At the same time, the semantic label generated by the semantic segmentation layer is taken as input into the semantic guidance branch, so that the layer parameters can also be adaptively adjusted during network training. Through the integration of high-precision semantic information and RGB images, the sparse depth map provided by the laser radar is supplemented, and the accuracy and real-time performance of scene understanding in digital twin, virtual reality, digital infrastructure, intelligent transportation and the like are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning technology, and in particular to road scene depth image completion technology in intelligent transportation systems. More specifically, the present invention relates to a road scene depth completion method based on semantic information and color image guidance. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI), depth information has been widely applied in numerous fields, such as digital twins, virtual reality, digital infrastructure, and smart transportation. Acquiring dense depth information has become a key task in computer vision. However, existing depth sensors suffer from issues such as sparse data points and significant interference when acquiring depth maps, making them difficult to apply in real-world tasks. To address these issues, researchers have shifted their focus from sensor hardware to algorithmic improvements: simply acquiring sparse depth maps using LiDAR and then employing high-performance depth completion algorithms can yield satisfactory dense depth maps. However, obtaining high-quality dense depth completion results while conserving computational resources and improving efficiency remains a major challenge in the depth completion field. In recent years, deep learning techniques, particularly depth completion algorithms based on convolutional neural networks, have provided new solutions to this problem. However, these algorithms still have limitations in handling complex lighting conditions, occlusions, and sensor noise, and they do not fully utilize scene semantic information.

[0003] Most current methods use a dual-branch network architecture for feature fusion, utilizing an encoder-decoder structure to extract and fuse different orders of features between color images and LiDAR sparse depth maps. For example, the existing technology proposed by PENet (Towards Precise and Efficient Image Guided Depth Completion[C] / / 2021IEEEInternational Conference on Robotics and Automation(ICRA).2021:13656-13662.) designs two branches, color-dominant and depth-dominant, using the encoder-decoder to extract features in sequence, and deeply fuses the features at each layer of the encoder-decoder to achieve a full exchange of information. However, these methods rely heavily on RGB images to extract color-dominant information (such as object boundaries) to complete the sparse map. However, in reality, color images have irregular pixel values ​​due to shadows and reflections, and most of the time, this information is not provided. To address this issue, the existing technology proposed by SemAttNet (Towards Attention-based Semantic Aware Guided Depth Completion:arXiv:2204.13635[R].arXiv,2022.) proposes a three-branch network that leverages semantic information. This adds a semantic guidance branch to PENet, enabling the network to better identify object boundaries using semantic information and reducing the sensitivity of color image guidance methods to optical changes. However, this model requires additional semantically segmented images during inference and does not explore more efficient methods for integrating semantic information. Furthermore, the network structure in existing models has a large number of parameters, resulting in slow inference speed. Summary of the Invention

[0004] One objective of the present invention is to provide a road scene depth completion method based on semantic information and color image guidance. By integrating high-precision semantic information and RGB images, the sparse depth map provided by LiDAR is supplemented, thereby improving the accuracy and real-time performance of scene understanding in applications such as digital twins, virtual reality, digital infrastructure, and smart transportation. It solves the problems of contrast and shadows in RGB images used in existing depth completion methods, which make it impossible to fully utilize the semantic information of the image.

[0005] To solve the above technical problems, the present invention provides a road scene depth completion method based on semantic information and color image guidance, which includes designing a dual-branch network consisting of a color image guidance branch and a semantic guidance branch, and introducing semantic information into the network to complete the road scene depth. The method specifically includes the following steps:

[0006] Step S1: Construct the color image guidance branch of the dual-branch network, namely the CG branch, which includes the color image guidance branch backbone, the color depth map prediction layer, and the semantic segmentation layer;

[0007] Step S2: pre-training the color image guided branch of the dual-branch network;

[0008] Step S3: Fix the pre-trained color image guided branch backbone parameters and train the color depth map prediction layer in step S1;

[0009] Step S4: Construct a semantic guidance branch, i.e., the SG branch, and use the semantic labels generated by the semantic segmentation layer in step S1 as input to the semantic guidance branch;

[0010] Step S5: obtaining an attention-based feature fusion module based on the deep connection of the color image guidance branch and the semantic guidance branch;

[0011] Step S6: Fix the pre-trained parameters of the entire color image guidance branch and train the entire semantic guidance branch; then unfreeze the fixed color image guidance branch parameters to simultaneously train the color image guidance branch and the semantic guidance branch;

[0012] Step S7: Input the features of the color image guided branch and the semantic guided branch into the attention-based feature fusion module to output a refined fused deep feature map, so that the color-dominated features in the CG branch decoder are fused into the corresponding encoder features of the SG branch;

[0013] Step S8: Apply CSPN++ to improve the dense depth map generated by the two-branch network.

[0014] Preferably, in step S1, the color image guidance branch aligns and connects the color image and the sparse depth map as input, and outputs a dense depth map and a confidence map;

[0015] Among them, in the color image guided branch backbone, we define C B ∈C and D B ∈D, where B is the batch size, C represents the color image, and D represents the depth image;

[0016] CG branch will be X cg As input, it is first encoded into a hidden representation As shown in the following formula 1, it is then decoded into a hidden representation of the color depth map and semantic joint features based on the following formula 2 φ cg =f(W cg X cg +b cg )#(1);φcs =g(V cg φ cg +c cg )#(2);

[0017] Among them, W cg and V cg Represents the weight matrix of the encoder and decoder respectively, and the variable b cg and c cg represents the encoding and decoding bias values, f(.) and g(.) represent the activation function;

[0018] In the process of generating color depth maps guided by color images, φ cs Generate a confidence map of batch size B according to the following formula 3 and color depth map

[0019] Among them, W cd Represents the weight matrix for generating the color depth layer, variable c cd Represents the bias value for generating the color depth map, and h(.) represents the activation function;

[0020] In the semantic segmentation layer, let the number of semantic segmentation categories be nc, and generate the semantic segmentation label map S according to formula (4) B ∈R B×nc×H×W , S B =p(V sl φ cs +b sl )#(4);

[0021] Among them, V sl Represents the weight matrix for generating semantic labels, variable b sl To generate the bias value of the semantic label, p(.) represents the activation function.

[0022] Preferably, in step S2, during training, the input is a color image and a sparse depth map, the output is a semantic segmentation result of the semantic segmentation layer, and the pre-training error is calculated using cross entropy loss.

[0023] Preferably, in step S3, use Loss is used to estimate the training error of the final color depth map prediction layer:

[0024]

[0025] in, The depth map D is used to guide the branch prediction for the color image. gt is the real depth map.

[0026] Preferably, in step S4, first the semantic label probability map S output by the CG branch isB As input, data dimension reduction is performed according to formula 8, from nc dimension to 3 dimension,

[0027] in, is the semantic label probability map after dimensionality reduction, B is the batch size, V dr Represents the dimensionality reduction weight matrix, variable b dr is the bias value of dimensionality reduction, p(.) represents the activation function;

[0028] The output has the same structure as the CG branch.

[0029] Preferably, in step S5, in the semantic branch X sg During the decoding process, each layer has a deep connection with the intermediate feature map of the color image guidance branch. Let F RGB ∈R C×H×W , F Sem ∈R C×H×W is the intermediate feature map of the input color image guidance and semantic branch, F = [F RGB ; F Sem ]∈R 2C×H×W Represents their deep connection, and the channel attention weight is calculated according to Formula 11,

[0030]

[0031] in, represents the channel attention weight of F, σ represents the sigmoid function, Represents the weight matrix layer of the multilayer perceptron. The parameter r controls the number of learnable parameters in the multilayer perceptron. and Represent the maximum pooling feature and average pooling feature respectively;

[0032] Applying the channel attention weights to F, we get and make The product F′ is then input into the spatial attention module as shown in Equation 12:

[0033] in, represents the spatial attention weight, σ represents the sigmoid function, and Denote the maximum pooling feature and the average pooling feature respectively; the spatial attention weight is applied to F′ to obtain the final output of the feature fusion module Make

[0034] Preferably, in step S6, use The loss is used to estimate the training error of the final semantic guidance branch output depth map: Among them, D sg is the depth map predicted by the SG branch, D gt is the real depth map.

[0035] Preferably, in step S7, the fused depth map D f As shown below,

[0036] Among them, D cg and D sg represents the depth map predicted by the CG branch and the SG branch, and Represent the learned confidence maps of the depth maps predicted by the CG branch and the SG branch.

[0037] Preferably, in step S8, the fused depth map is given Embed it into a hidden representation The propagation steps of CSPN++ with Atrous convolution are shown in Equation 14 below,

[0038]

[0039] Preferably, when constructing a two-branch network, the backbone and semantic segmentation layer of one branch are also used to predict semantic information, specifically generating a semantic segmentation label map based on the sparse depth map and the RGB map, generating a semantic segmentation pseudo ground truth based on the existing model, and converting the RGB image in the KITTI depth completion dataset into a semantic label map.

[0040] The innovation and optimization of the structural design of the present invention enable the depth completion algorithm of the present application to not only surpass traditional methods in accuracy, but also achieve significant improvements in computational efficiency and resource utilization. It provides more accurate and efficient depth perception capabilities for fields such as digital twins, virtual reality, digital infrastructure, and smart transportation, and has at least the following beneficial effects:

[0041] 1. This application introduces semantic information into the network by designing a dual-branch network consisting of a color image-guided branch and a semantic-guided branch. Compared to earlier methods, this application incorporates semantic information into the traditional depth-guided branch to reduce depth value differences near object boundaries. Ablation experiments demonstrate that the inclusion of semantic information significantly improves the accuracy of depth completion tasks.

[0042] 2. This application utilizes the idea of ​​multi-task learning. By sharing the backbone with the depth map prediction layer in the color image guidance branch, semantic information can be obtained by simply adding a semantic segmentation layer without adding the entire network. At the same time, the semantic labels generated by the semantic segmentation layer are input into the semantic guidance branch, so that the parameters of this layer can also be adaptively adjusted during network training. Compared with existing methods that use semantic information, the network proposed in this application has smaller parameters, faster inference speed, and higher efficiency.

[0043] 3. This application also proposes a training process that enables the depth completion network that requires semantic information to obtain semantic information even when faced with the lack of semantic labels, thereby guiding training. Specifically, for the KITTI depth completion dataset that does not provide semantic labels, the most advanced methods of other semantic segmentation benchmarks can be used, combined with the color images of the depth completion dataset, to generate pseudo ground truth of semantic labels. Combined with the idea of ​​multi-task learning and the multi-stage training method of this application, the depth completion network can utilize semantic information without the need to input semantic labels during reasoning.

[0044] 4. Based on validation set accuracy, the proposed method outperforms most methods and demonstrates comparable performance on other evaluation metrics. Furthermore, the algorithm's average runtime is only 0.14 seconds, demonstrating high efficiency and further demonstrating its feasibility in practical applications.

[0045] Other advantages, objectives and features of the present invention will be reflected in part from the following description and will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Detailed comparison of the experimental results of the present invention on the KITTI depth completion test set;

[0047] Figure 2 This is a schematic diagram of the network structure of the present invention;

[0048] Figure 3 Comparison of semantic label graphs used in this application corresponding to the present invention and early research in the prior art;

[0049] Figure 4 Schematic diagram of the attention-based feature fusion structure of the present invention. DETAILED DESCRIPTION

[0050] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below with reference to the accompanying drawings so that those skilled in the art can implement it according to the description.

[0051] It should be noted that the experimental methods described in the following embodiments are conventional methods unless otherwise specified, and the reagents and materials are commercially available unless otherwise specified; in the description of the present invention, the terms "horizontal", "longitudinal", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention.

[0052] This application formulates the image-guided depth completion problem as a three-stage task. In the first stage, the study uses a single-branch backbone and semantic segmentation layer to predict semantic information; in the second stage, the study uses a full two-branch network to generate dense depth maps; and in the third stage, CSPN++ and Atrous convolution are used to refine the dense depth maps. Figure 2 The network structure studied in this application is shown.

[0053] This application discloses a road scene depth completion method based on semantic information and color image guidance, comprising the following steps:

[0054] Step 1: Generate semantic segmentation label maps based on sparse depth maps and RGB images. The KITTI depth completion dataset does not provide semantic label maps corresponding to RGB images and lidar point clouds, so this study generated semantic segmentation pseudo ground truth based on existing models. Considering that semantic segmentation is usually performed only using RGB, this study uses the pre-trained model of InternImage-XL, which ranks first on the Cityscape semantic segmentation benchmark, to convert RGB images in the KITTI depth completion dataset into semantic label maps. The semantic label maps generated by this study are more accurate than the semantic label maps used in earlier studies using semantic segmentation, as shown in the attached figure. Figure 3 shown.

[0055] Step 2: Construct a color-guided branch (CGBranch) of a dual-branch network. The color-guided branch aims to learn important color cues for dense depth completion, learn the discontinuity of object boundaries, and maintain smoothness inside the object. The CG branch takes the color image and the sparse depth map as input and outputs a dense depth map. This branch consists of an encoder-decoder network with jump connections. The encoder consists of 10 ResNet blocks and the decoder consists of one convolution layer and five transposed convolution layers for upsampling. The output of this branch includes a dense depth map and a confidence map. Although the dense depth map output by the branch is noisy, it provides a basis for learning the structural information of the scene in other branches.

[0056] (1) Color image guided branch backbone

[0057] definition C B ∈C and D B ∈ D. Where B is the batch size, C represents the color image, and D represents the depth image.

[0058] CG branch will be X cg As input, it is first encoded into a hidden representation (As shown in Equation 1), it is then decoded into a hidden representation of the color depth map and the semantic joint feature (color-semantic) based on Equation 2

[0059] φ cg =f(W cg X cg +b cg )#(1)

[0060] φ cs =g(V cg φ cg +c cg )#(2)

[0061] Among them, W cg and V cg Represents the weight matrix of the encoder and decoder respectively, and the variable b cg and c cg represents the encoding and decoding bias values, and f(.) and g(.) represent the activation functions.

[0062] (2) Color depth map prediction layer

[0063] In the process of generating the color depth map guided by the color image, φ cs Generate a confidence map of batch size B according to Formula 3 and color depth map

[0064]

[0065] Among them, W cd Represents the weight matrix for generating the color depth layer, variable c cd Represents the bias value for generating the color depth map, and h(.) represents the activation function.

[0066] (3) Semantic segmentation layer

[0067] In the semantic segmentation layer, let the number of semantic segmentation categories be nc, and generate the semantic segmentation label map (semantic labels) S according to formula (4) B ∈R B×nc×H×W .

[0068] S B =p(V sl φ cs +b sl )#(4)

[0069] Among them, V sl Represents the weight matrix for generating semantic labels, variable b sl To generate the bias value of the semantic label, p(.) represents the activation function.

[0070] Step 3: Pre-train the color image guided branch of the two-branch network. The input is the color image and the sparse depth map, and the output is the semantic segmentation result of the semantic segmentation layer. The cross entropy loss is used to calculate the error of this pre-training, and the loss of pixel i is defined as:

[0071]

[0072] Where i is the index of the pixel, y gt is the true value, is the logarithmic softmax value of the predicted probability of pixel i. The logarithmic softmax function is defined as:

[0073]

[0074] Step 4: Fix the pre-trained color image guide branch backbone parameters and train the color depth map prediction layer. Loss Estimation. Since the ground truth contains invalid depth values, this study only considers pixels with valid depth values ​​when calculating the loss.

[0075]

[0076] in, The depth map D is used to guide the branch prediction for the color image. gtis the real depth map.

[0077] Step 5: Construct the semantic-guided branch (SG Branch). First, the semantic label probability map S output by the CG branch B As input, data dimensionality reduction is performed according to Formula 8, from nc dimensions to 3 dimensions, so that each input is more matched in dimension.

[0078]

[0079] in, is the semantic label probability map after dimensionality reduction, B is the batch size, V dr Represents the dimensionality reduction weight matrix, variable b dr is the bias value for dimensionality reduction, and p(·) represents the activation function.

[0080] The output structure is the same as that of the CG branch, and the definition For batch size B, and D B ∈D. The matrix X sg As the input of the SG branch, it is encoded into the hidden representation φ sg ∈R B ×1024×H×W (As shown in Equation 9). Then φ sg Decoded into confidence map C sg ∈R B×1×H×W and semantic depth map D sg ∈R B×1×H×W (As shown in formula 10).

[0081]

[0082] Among them, W cg and V cg Represents the weight matrix of the encoder and decoder, variable b sg and c sg represents the encoding and decoding bias values, and f(.) and g(.) represent the activation functions.

[0083] Step 6: In semantic branch X sg During the decoding process, each layer has a deep connection with the intermediate feature map of the color image guidance branch, see the attached Figure 4 Let F RGB ∈R C×H×W , F Sem ∈R C×H×W is the intermediate feature map of the input color image guidance and semantic branch, F = [F RGB ; F Sem ]∈R 2C×H×WDenotes their deep connection. The channel attention weight is calculated according to formula (11).

[0084]

[0085] in, represents the channel attention weight of F, σ represents the sigmoid function, Represents the weight matrix layer of a Multilayer Perceptron (MLP). The parameter r controls the number of learnable parameters in the MLP. and Represent the maximum pooling features and average pooling features respectively.

[0086] Applying the channel attention weights to F, we get and make The product F′ is then input into the spatial attention module as shown in Equation 12:

[0087]

[0088] in, represents the spatial attention weight, σ represents the sigmoid function, and Denote the maximum pooling feature and the average pooling feature respectively. The spatial attention weight is applied to F′ to obtain the final output of the feature fusion module Make

[0089] Step 7: Fix the pre-trained parameters of the entire color image guided branch and train the entire semantic guided branch. Then unfreeze the fixed color image guided branch parameters and train the color image guided branch and the semantic guided branch at the same time. The loss is used to estimate the training error of the final semantic guidance branch output depth map:

[0090]

[0091] Among them, D sg is the depth map predicted by the SG branch, D gt is the real depth map.

[0092] Step 8: Input the features of the color image guided branch and the semantic guided branch CG branch and SG branch into the attention-based feature fusion module to output a refined fusion depth feature map, so that the color-dominated features in the CG branch decoder are fused into the corresponding encoder features of the SG branch.

[0093] Among them, D fDenotes the fused depth map, D cg and D sg represents the depth map predicted by CG and SG branches, and Represent the learned confidence maps of the depth maps predicted by the CG and SG branches.

[0094] Step 9: Apply CSPN++ to improve the dense depth map generated by the two-branch network. Given the fused depth map Embed it into a hidden representation Equation 14 shows the propagation step of CSPN++ with Atrous convolution applied.

[0095]

[0096] Based on the validation set accuracy, the proposed method outperforms most methods and demonstrates comparable performance on other evaluation metrics. Table 1 shows the comparison results with other existing methods. Furthermore, the algorithm's average runtime is only 0.14 seconds, demonstrating its high efficiency and further demonstrating its feasibility in practical applications.

[0097] Table 1 Quantitative comparison of this algorithm with other algorithms

[0098]

[0099]

[0100] The above method Sparse2Dense (gd) is quoted from MA F, CAVALHEIRO GV, KARAMAN S. Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR andMonocular Camera[C] / / 2019International Conference on Robotics and Automation(ICRA).Montreal,QC,Canada:IEEE,2019:3288-3295.

[0101] The above method NConv-CNN-L2 is cited in ELDESOKEY A, FELSBERG M, KHAN F S. Confidence Propagation through CNNs for Guided Sparse Depth Regression[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(10): 2423-2436.

[0102] The above method CrossGuidence is cited in LEE S, LEE J, KIM D, et al. Deep Architecture With Cross Guidance Between Single Image and Sparse LiDAR Data for DepthCompletion[J]. IEEE Access, 2020, 8: 79801-79810.

[0103] The above method PwP is cited in XU Y, ZHU X, SHI J, et al. Depth Completion from SparseLiDAR Data with Depth-Normal Constraints[M].arXiv, 2019.

[0104] The above method DSPN is cited in XU Z, YIN H, YAO J. Deformable spatial propagation network for depth completion[M].arXiv,2020.

[0105] The above method FuseNet is cited in CHEN Y, YANG B, LIANG M, et al. Learning Joint 2D-3D Representations for Depth Completion[C] / / 2019 IEEE / CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019: 10022-10031.

[0106] The above method GuideNet is cited in TANG J, TIAN FP, FENG W, et al. Learning Guided Convolutional Network for Depth Completion[J]. IEEE Transactions on Image Processing, 2021, 30: 1116-1129.

[0107] In addition, the experimental results of this algorithm on the KITTI depth completion test set and the detailed comparison with other algorithms are shown in the attached Figure 1 shown.

[0108] It will be understood that the present invention is described by way of some embodiments, and those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and embodiments. They can be fully applied to various fields suitable for the present invention, and further modifications can be easily implemented by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A road scene depth completion method based on semantic information and color image guidance, characterized by: This involves designing a dual-branch network consisting of a color image-guided branch and a semantic-guided branch, and introducing semantic information into the network to perform road scene depth completion. The specific steps include: Step S1: Construct the color image guidance branch of the dual-branch network, namely the CG branch, which includes the color image guidance branch backbone, the color depth map prediction layer, and the semantic segmentation layer; Step S2: pre-training the color image guided branch of the dual-branch network; Step S3: Fix the pre-trained color image guided branch backbone parameters and train the color depth map prediction layer in step S1; Step S4: Construct a semantic guidance branch, i.e., the SG branch, and use the semantic labels generated by the semantic segmentation layer in step S1 as input to the semantic guidance branch; Step S5: obtaining an attention-based feature fusion module based on the deep connection of the color image guidance branch and the semantic guidance branch; Step S6: Fix the pre-trained parameters of the entire color image guidance branch and train the entire semantic guidance branch; then unfreeze the fixed color image guidance branch parameters to simultaneously train the color image guidance branch and the semantic guidance branch; Step S7: Input the features of the color image guided branch and the semantic guided branch into the attention-based feature fusion module to output a refined fused deep feature map, so that the color-dominated features in the CG branch decoder are fused into the corresponding encoder features of the SG branch; Step S8: Apply CSPN++ to improve the dense depth map generated by the two-branch network; In step S1, the color image guidance branch aligns and connects the color image and the sparse depth map as input, and outputs a dense depth map and a confidence map; Among them, in the color image guided branch backbone, we define for and Where B is the batch size, C represents the color image, and D represents the depth image; CG branch will As input, it is first encoded into a hidden representation , as shown in the following formula 1, and then decoded into the hidden representation of the color depth map and semantic joint features based on the following formula 2 , in, and Represents the weight matrix of the encoder and decoder respectively, and the variables and Represents the encoding and decoding bias values, and represents the activation function; In the process of generating color image guided color depth map, The batch size is generated according to the following formula 3: Confidence map of and color depth map , in, Represents the weight matrix for generating the color depth layer, variable Indicates the bias value for generating a color depth map, represents the activation function; In the semantic segmentation layer, let the number of semantic segmentation categories be nc, and generate the semantic segmentation label map according to formula 4 ; in, Represents the weight matrix for generating semantic labels, variable To generate the bias value of the semantic label, Represents the activation function.

2. The road scene depth completion method based on semantic information and color image guidance according to claim 1, characterized in that: In step S2, during training, the input is a color image and a sparse depth map, the output is the semantic segmentation result of the semantic segmentation layer, and the pre-training error is calculated using cross entropy loss.

3. The road scene depth completion method based on semantic information and color image guidance according to claim 1, characterized in that: In step S3, use Loss is used to estimate the training error of the final color depth map prediction layer: ; in, Depth map for color image guided branch prediction, is the real depth map.

4. The road scene depth completion method based on semantic information and color image guidance according to claim 1, characterized in that: In step S4, the semantic label probability map output by the CG branch is first As input, data dimension reduction is performed according to formula 8, from nc dimension to 3 dimension, ; in, is the semantic label probability map after dimensionality reduction, B is the batch size, represents the dimensionality reduction weight matrix, variable is the bias value for dimensionality reduction, represents the activation function; The output has the same structure as the CG branch.

5. The road scene depth completion method based on semantic information and color image guidance according to claim 1, characterized in that: In step S5, in the semantic branch During the decoding process, each layer has a deep connection with the intermediate feature map of the color image guidance branch. is the intermediate feature map of the input color image guidance and semantic branch, Represents their deep connection, and the channel attention weight is calculated according to Formula 11, in, represents the channel attention weight of F, represents the sigmoid function, Represents the weight matrix layer of the multilayer perceptron. The parameter r controls the number of learnable parameters in the multilayer perceptron. and Represent the maximum pooling feature and average pooling feature respectively; Applying the channel attention weights to F, we get , and make ; Then multiply the product Input spatial attention module, as shown in Equation 12: in, represents the spatial attention weight, represents the sigmoid function, and Represent the maximum pooling feature and the average pooling feature respectively; apply the spatial attention weight to To obtain the final output of the feature fusion module , making .

6. The road scene depth completion method based on semantic information and color image guidance according to claim 1, characterized in that: In step S6, use The loss is used to estimate the training error of the final semantic guidance branch output depth map: in, is the depth map predicted by the SG branch, is the real depth map.

7. The road scene depth completion method based on semantic information and color image guidance according to claim 1, characterized in that: In step S7, the fused depth map As shown below, ;in, and represents the depth map predicted by the CG branch and the SG branch, and Represent the learned confidence maps of the depth maps predicted by the CG branch and the SG branch.

8. The method for road scene depth completion based on semantic information and color image guidance according to claim 7, characterized in that: In step S8, given the fused depth map , embed it into the hidden representation ; The CSPN++ propagation steps using Atrous convolution are shown in Equation 14 below, 。 9. The method for road scene depth completion based on semantic information and color image guidance according to claim 1, characterized in that: When building a two-branch network, it also includes using the backbone and semantic segmentation layer of one branch to predict semantic information. Specifically, a semantic segmentation label map is generated based on the sparse depth map and RGB map. The semantic segmentation pseudo ground truth is generated based on the existing model, and the RGB images in the KITTI depth completion dataset are converted into semantic label maps.

Images