Road scene depth completion method based on semantic information and color image guidance

Abstract

The invention discloses a road scene depth completion method based on semantic information and color image guidance, and the method comprises the steps: designing a double-branch network composed of a color image guidance branch and a semantic guidance branch, introducing semantic information into the network, and carrying out the depth completion of a road scene. According to the method, a multi-task learning thought is utilized, a backbone is shared with a depth map prediction layer in a color image guide branch, and semantic information can be obtained only by adding a semantic segmentation layer without adding a whole network. Meanwhile, a semantic label generated by the semantic segmentation layer serves as input to enter a semantic guidance branch, so that parameters of the layer can be adaptively adjusted during network training. According to the method, the sparse depth map provided by the laser radar is supplemented by integrating high-precision semantic information and the RGB image, so that the accuracy and the real-time performance of scene understanding in applications such as digital twinning, virtual reality, digital capital construction and intelligent transportation are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a road scene depth image completion technology in an intelligent transportation system. More specifically, the present invention relates to a road scene depth completion method based on semantic information and color image guidance. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, depth information has been widely used in many fields, such as digital twins, virtual reality, digital infrastructure, smart transportation, etc. How to obtain dense depth information has become a key task in the current computer vision field. However, existing depth sensors have problems such as too sparse data points and obvious external interference when obtaining depth maps, which makes them difficult to apply in real tasks. In order to solve these problems, researchers have turned their attention from sensor hardware to algorithm improvements: as long as sparse depth maps are obtained through lidar, and high-performance depth completion algorithms are used, satisfactory dense depth maps can be obtained. However, obtaining high-quality dense depth completion results while saving computing resources and improving efficiency is still a major challenge in the field of depth completion. In recent years, deep learning technology, especially depth completion algorithms based on convolutional neural networks, has provided new solutions to this problem. However, these algorithms still have shortcomings in dealing with complex lighting conditions, occlusions, and sensor noise, and do not make full use of scene semantic information.

[0003] Most current methods use a dual-branch network architecture for feature fusion, using an encoder-decoder structure to extract and fuse different orders between color images and LiDAR sparse depth maps. For example, the prior art proposed by PENet (Towards Precise and Efficient Image Guided Depth Completion [C] / / 2021 IEEE International Conference on Robotics and Automation (ICRA). 2021: 13656-13662.) designs two branches, color-dominant and depth-dominant, and uses the codec to extract features in sequence, and deeply fuses the features at each layer of the codec to obtain a full exchange of information. However, these methods rely heavily on RGB images to extract color-dominant information (such as object boundaries) to complete sparse maps, but in fact, color images have irregular pixel values ​​due to shadows and reflections, and most of the time do not provide this information. To address this problem, the existing technology proposed by SemAttNet (Towards Attention-based Semantic Aware Guided Depth Completion:arXiv:2204.13635[R].arXiv,2022.) proposed a three-branch network that uses semantic information. On the basis of PENet, a semantic guidance branch was added, which enables the network to use semantic information to better identify object boundaries and reduce the sensitivity of color image guidance methods to optical changes. However, this model also needs to input additional semantic segmentation images during reasoning, and does not explore more efficient ways to fuse semantic information. In addition, the network structure has huge parameters in existing models, and the reasoning speed is slow. Summary of the invention

[0004] An objective of the present invention is to provide a road scene depth completion method based on semantic information and color image guidance. By integrating high-precision semantic information and RGB images, the sparse depth map provided by LiDAR is supplemented, thereby improving the accuracy and real-time performance of scene understanding in applications such as digital twins, virtual reality, digital infrastructure, and smart transportation. The existing depth completion method solves the problems of contrast and shadow in RGB images, which makes it impossible to fully utilize the semantic information of the image.

[0005] In order to solve the above technical problems, the present invention provides a road scene depth completion method based on semantic information and color image guidance, including designing a dual-branch network consisting of a color image guidance branch and a semantic guidance branch, and introducing semantic information into the network, so as to complete the road scene depth; specifically, the method includes the following steps:

[0006] Step S1: constructing a color image guided branch of a dual-branch network, namely, a CG branch, including a color image guided branch backbone, a color depth map prediction layer, and a semantic segmentation layer;

[0007] Step S2: pre-training the color image guided branch of the dual-branch network;

[0008] Step S3: fix the pre-trained color image guided branch backbone parameters and train the color depth map prediction layer in step S1;

[0009] Step S4: construct a semantic guidance branch, i.e., the SG branch, and use the semantic labels generated by the semantic segmentation layer in step S1 as input to enter the semantic guidance branch;

[0010] Step S5: obtaining an attention-based feature fusion module according to the deep connection of the color image guided branch and the semantic guided branch;

[0011] Step S6: fix the pre-trained parameters of the entire color image guided branch and train the entire semantic guided branch; then unfreeze the fixed color image guided branch parameters to simultaneously train the color image guided branch and the semantic guided branch;

[0012] Step S7: Input the features of the color image guided branch and the semantic guided branch into the attention-based feature fusion module to output a refined fused deep feature map so that the color-dominated features in the CG branch decoder are fused into the corresponding encoder features of the SG branch;

[0013] Step S8: Apply CSPN++ to improve the dense depth map generated by the two-branch network.

[0014] Preferably, in step S1, the color image guiding branch aligns and connects the color image and the sparse depth map as input, and the output includes a dense depth map and a confidence map;

[0015] Among them, in the color image guided branch backbone, we define C B ∈C and D B ∈D, where B is the batch size, C represents the color image, and D represents the depth image;

[0016] CG branch will be X cg As input, it is first encoded into a hidden representation As shown in the following formula 1, it is then decoded into a hidden representation of the color depth map and semantic joint features based on the following formula 2 φ cg =f(W cg X cg +b cg )#(1);φcs =g(V cg φ cg +c cg )#(2);

[0017] Among them, W cg and V cg Represents the weight matrix of the encoder and decoder respectively, and the variable b cg and c cg represents the encoding and decoding bias values, f(.) and g(.) represent the activation functions;

[0018] In the process of generating color image-guided color depth map, φ cs Generate a confidence map with a batch size of B according to the following formula 3 and color depth map

[0019] Among them, W cd Represents the weight matrix for generating the color depth layer, variable c cd represents the bias value for generating the color depth map, and h(.) represents the activation function;

[0020] In the semantic segmentation layer, let the number of semantic segmentation categories be nc, and generate the semantic segmentation label map S according to formula (4): B ∈R B×nc×H×W , S B = p(V sl φ cs +b sl )#(4);

[0021] Among them, V sl Represents the weight matrix for generating semantic labels, variable b sl To generate the bias value of the semantic label, p(.) represents the activation function.

[0022] Preferably, in step S2, during training, the input is a color image and a sparse depth map, the output is a result of semantic segmentation of the semantic segmentation layer, and the pre-training error is calculated using cross entropy loss.

[0023] Preferably, in step S3, using Loss is used to estimate the training error of the final color depth map prediction layer:

[0024]

[0025] in, The depth map D is the color image-guided branch prediction. gt The real depth map.

[0026] Preferably, in step S4, firstly, the semantic label probability map S output by the CG branch isB As input, data dimension reduction is performed according to formula 8, from nc dimension to 3 dimension,

[0027] in, is the semantic label probability map after dimensionality reduction, B is the batch size, V dr represents the dimension reduction weight matrix, variable b dr is the bias value of dimensionality reduction, p(.) represents the activation function;

[0028] The output has the same structure as the CG branch.

[0029] Preferably, in step S5, in the semantic branch X sg During the decoding process, each layer has a deep connection with the intermediate feature map of the color image guidance branch. Let F RGB ∈R C×H×W , F Sem ∈R C×H×W is the intermediate feature map of the input color image guidance and semantic branch, F = [F RGB ; F Sem ]∈R 2C×H×W Represents their deep connection, and the channel attention weight is calculated according to Formula 11,

[0030]

[0031] in, represents the channel attention weight of F, σ represents the sigmoid function, represents the weight matrix layer of the multilayer perceptron. The parameter r controls the number of learnable parameters in the multilayer perceptron. and Represent the maximum pooling feature and the average pooling feature respectively;

[0032] Applying the channel attention weights to F, we get and make The product F′ is then input into the spatial attention module, as shown in Equation 12:

[0033] in, represents the spatial attention weight, σ represents the sigmoid function, and Denote the maximum pooling feature and the average pooling feature respectively; the spatial attention weight is applied to F′ to obtain the final output of the feature fusion module Make

[0034] Preferably, in step S6, using The loss is used to estimate the training error of the final semantic guidance branch output depth map: Among them, D sg is the depth map predicted by the SG branch, D gt The real depth map.

[0035] Preferably, in step S7, the fused depth map D f As shown below,

[0036] Among them, D cg and D sg represents the depth map predicted by the CG branch and the SG branch, and Represent the learned confidence maps of the depth maps predicted by the CG branch and the SG branch.

[0037] Preferably, in step S8, given the fused depth map Embed it into a hidden representation The CSPN++ propagation steps using Atrous convolution are shown in Equation 14,

[0038]

[0039] Preferably, when constructing a dual-branch network, it also includes using the backbone and semantic segmentation layer of one branch to predict semantic information, specifically generating a semantic segmentation label map based on the sparse depth map and the RGB map, generating a semantic segmentation pseudo ground truth based on the existing model, and converting the RGB image in the KITTI depth completion dataset into a semantic label map.

[0040] The innovation and optimization of the structural design of the present invention make the depth completion algorithm of the present application not only surpass the traditional method in accuracy, but also achieve significant improvements in computing efficiency and resource utilization, providing more accurate and efficient depth perception capabilities for digital twins, virtual reality, digital infrastructure, smart transportation and other fields, including at least the following beneficial effects:

[0041] 1. This application introduces semantic information into the network by designing a dual-branch network consisting of a color image guided branch and a semantic guided branch. Compared with earlier methods, this application adds semantic information to the traditional depth guided branch to reduce the depth value difference near the object boundary, and proves in ablation experiments that the introduction of semantic information significantly improves the accuracy of the depth completion task.

[0042] 2. This application uses the idea of ​​multi-task learning. By sharing the backbone with the depth map prediction layer in the color image guidance branch, semantic information can be obtained by only adding a semantic segmentation layer without adding the entire network. At the same time, the semantic labels generated by the semantic segmentation layer enter the semantic guidance branch as input, so that the parameters of this layer can also be adaptively adjusted during network training. Compared with existing methods that use semantic information, the network parameters proposed in this application are smaller, the reasoning speed is faster, and the efficiency is higher.

[0043] 3. This application also proposes a training process that enables the deep completion network that requires semantic information to obtain semantic information even when faced with the lack of semantic labels, thereby guiding training. Specifically, for the KITTI deep completion dataset that does not provide semantic labels, the most advanced methods of other semantic segmentation benchmarks can be used, combined with the color images of the deep completion dataset, to generate pseudo ground truth of semantic labels. Combined with the idea of ​​multi-task learning and the multi-stage training method of this application, the deep completion network can use semantic information without the need to input semantic labels during reasoning.

[0044] 4. According to the accuracy of the validation set, the method proposed in this application is superior to most methods and shows comparable performance in other evaluation indicators. In addition, the average running time of the algorithm in this application is only 0.14s, which is relatively efficient, further proving its feasibility in practical applications.

[0045] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Detailed comparison of the experimental results of the present invention on the KITTI depth completion test set;

[0047] Figure 2 This is a schematic diagram of the network structure of the present invention;

[0048] Figure 3 Comparison of the semantic label graphs used in the present application corresponding to the present invention and the early studies in the prior art;

[0049] Figure 4 Schematic diagram of the attention-based feature fusion structure of the present invention. DETAILED DESCRIPTION

[0050] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement it according to the description.

[0051] It should be noted that the experimental methods described in the following embodiments are conventional methods unless otherwise specified, and the reagents and materials are commercially available unless otherwise specified; in the description of the present invention, the terms "lateral", "longitudinal", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0052] This application formulates the image-guided depth completion problem as a three-stage task. In the first stage, the study uses a one-branch backbone and semantic segmentation layer to predict semantic information; in the second stage, the study uses a full two-branch network to generate dense depth maps; and in the third stage, CSPN++ and Atrous convolution are used to refine the dense depth maps. Figure 2 The network structure studied in this application is shown.

[0053] The present application discloses a road scene depth completion method based on semantic information and color image guidance, comprising the following steps:

[0054] Step 1: Generate semantic segmentation label maps based on sparse depth maps and RGB images. The KITTI depth completion dataset does not provide semantic label maps corresponding to RGB images and lidar point clouds, so this study generated semantic segmentation pseudo ground truth based on existing models. Considering that semantic segmentation is usually performed only using RGB, this study uses the pre-trained model of InternImage-XL, which ranks first on the Cityscape semantic segmentation benchmark, to convert RGB images in the KITTI depth completion dataset into semantic label maps. The semantic label maps generated by this study are more accurate than those used in earlier studies using semantic segmentation, as shown in the attached figure. Figure 3 shown.

[0055] Step 2: Construct the Color-Guided Branch (CGBranch) of the dual-branch network. The color-guided branch aims to learn important color cues for dense depth completion, learn the discontinuity of object boundaries, and maintain smoothness inside the object. The CG branch takes the color image and the sparse depth map as input and outputs a dense depth map. This branch consists of an encoder-decoder network with jump connections. The encoder consists of 10 ResNet blocks and the decoder consists of one convolutional layer and five transposed convolutional layers for upsampling. The output of this branch includes a dense depth map and a confidence map. Although the dense depth map output by the branch is noisy, it provides a basis for learning the structural information of the scene in other branches.

[0056] (1) Color image-guided branch backbone

[0057] definition C B ∈C and D B ∈D. Where B is the batch size, C represents the color image, and D represents the depth image.

[0058] CG branch will be X cg As input, it is first encoded into a hidden representation (As shown in Formula 1), and then based on Formula 2, it is decoded into a hidden representation of the color depth map and the semantic joint feature (color-semantic)

[0059] φ cg =f(W cg X cg +b cg )#(1)

[0060] φ cs =g(V cg φ cg +c cg )#(2)

[0061] Among them, W cg and V cg Represents the weight matrix of the encoder and decoder respectively, and the variable b cg and c cg represents the encoding and decoding bias values, and f(.) and g(.) represent the activation functions.

[0062] (2) Color depth map prediction layer

[0063] In the process of generating the color depth map guided by the color image, φ cs Generate a confidence map with a batch size of B according to Formula 3 and color depth map

[0064]

[0065] Among them, W cd Represents the weight matrix for generating the color depth layer, variable c cd represents the bias value for generating the color depth map, and h(.) represents the activation function.

[0066] (3) Semantic Segmentation Layer

[0067] In the semantic segmentation layer, let the number of semantic segmentation categories be nc, and generate the semantic segmentation label map (semantic labels) S according to formula (4): B ∈R B×nc×H×W .

[0068] S B = p(V sl φ cs +b sl )#(4)

[0069] Among them, V sl Represents the weight matrix for generating semantic labels, variable b sl To generate the bias value of the semantic label, p(.) represents the activation function.

[0070] Step 3: Pre-train the color image guided branch of the dual-branch network. The input is the color image and the sparse depth map, and the output is the semantic segmentation result of the semantic segmentation layer. The error of this pre-training is calculated using the cross entropy loss, and the loss of pixel i is defined as:

[0071]

[0072] Where i is the index of the pixel, y is gt is the true value, is the logarithmic softmax value of the predicted probability of pixel i. The logarithmic softmax function is defined as:

[0073]

[0074] Step 4: Fix the pre-trained color image guide branch backbone parameters and train the color depth map prediction layer. Loss Estimation. Since the ground truth contains invalid depth values, this study only considers pixels with valid depth values ​​when calculating the loss.

[0075]

[0076] in, The depth map D is the color image-guided branch prediction. gtThe real depth map.

[0077] Step 5: Construct a semantic-guided branch (SG Branch). First, convert the semantic label probability map S output by the CG branch into B As input, data dimension reduction is performed according to Formula 8, from nc dimensions to 3 dimensions, so that each input is more matched in dimension.

[0078]

[0079] in, is the semantic label probability map after dimensionality reduction, B is the batch size, V dr represents the dimension reduction weight matrix, variable b dr is the bias value for dimensionality reduction, and p(·) represents the activation function.

[0080] The output is also the same as the CG branch structure, defining For batch size B, and D B ∈D. The matrix X sg As the input of the SG branch, it is encoded into the hidden representation φ sg ∈R B×1024×H×W (As shown in Equation 9). Then φ sg Decoded into the confidence graph C sg ∈R B×1×H×W and semantic depth map D sg ∈R B×1×H×W (As shown in formula 10).

[0081]

[0082] Among them, W cg and V cg Represents the weight matrix of the encoder and decoder, variable b sg and c sg represents the encoding and decoding bias values, and f(.) and g(.) represent the activation functions.

[0083] Step 6: In semantic branch X sg During the decoding process, each layer has a deep connection with the intermediate feature map of the color image guidance branch, see Appendix Figure 4 Let F RGB ∈R C×H×W , F Sem ∈R C×H×W is the intermediate feature map of the input color image guidance and semantic branch, F = [F RGB ; F Sem ]∈R 2C×H×W Denotes their deep connection. The channel attention weight is calculated according to formula (11).

[0084]

[0085] in, represents the channel attention weight of F, σ represents the sigmoid function, Represents the weight matrix layer of a multilayer perceptron (MLP). The parameter r controls the number of learnable parameters in the multilayer perceptron. and Represent the maximum pooling feature and the average pooling feature respectively.

[0086] Applying the channel attention weights to F, we get and make The product F′ is then input into the spatial attention module, as shown in Equation 12:

[0087]

[0088] in, represents the spatial attention weight, σ represents the sigmoid function, and Denote the maximum pooling feature and the average pooling feature respectively. The spatial attention weight is applied to F′ to obtain the final output of the feature fusion module Make

[0089] Step 7: Fix the pre-trained color image guided branch parameters and train the semantic guided branch. Then unfreeze the fixed color image guided branch parameters and train the color image guided branch and the semantic guided branch at the same time. The loss is used to estimate the training error of the final semantic guidance branch output depth map:

[0090]

[0091] Among them, D sg is the depth map predicted by the SG branch, D gt The real depth map.

[0092] Step 8: Input the features of the color image guided branch and the semantic guided branch CG branch and SG branch into the attention-based feature fusion module to output a refined fused deep feature map so that the color-dominated features in the CG branch decoder are fused into the corresponding encoder features of the SG branch.

[0093] Among them, D f Denotes the fused depth map, D cg and Dsg represents the depth map predicted by the CG and SG branches, and Learned confidence maps representing the depth maps predicted by the CG and SG branches.

[0094] Step 9: Apply CSPN++ to improve the dense depth map generated by the two-branch network. Given the fused depth map Embed it into a hidden representation Equation 14 shows the CSPN++ propagation step with Atrous convolution applied.

[0095]

[0096] According to the accuracy of the validation set, the method proposed in this application is superior to most methods and shows comparable performance on other evaluation indicators. The comparison results with other existing methods are shown in Table 1. In addition, the average running time of this algorithm is only 0.14s, which is relatively efficient, further proving its feasibility in practical applications.

[0097] Table 1 Quantitative comparison of this algorithm with other algorithms

[0098]

[0099]

[0100] The above method Sparse2Dense (gd) is quoted from MA F, CAVALHEIRO GV, KARAMAN S. Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR andMonocular Camera[C] / / 2019International Conference on Robotics and Automation(ICRA).Montreal,QC,Canada:IEEE,2019:3288-3295.

[0101] The above method NConv-CNN-L2 is cited in ELDESOKEY A, FELSBERG M, KHAN F S. Confidence Propagation through CNNs for Guided Sparse Depth Regression[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(10): 2423-2436.

[0102] The above method CrossGuidence is cited in LEE S, LEE J, KIM D, et al. Deep Architecture With Cross Guidance Between Single Image and Sparse LiDAR Data for Depth Completion[J]. IEEE Access, 2020, 8: 79801-79810.

[0103] The above method PwP is cited in XU Y, ZHU X, SHI J, et al. Depth Completion from SparseLiDAR Data with Depth-Normal Constraints[M].arXiv, 2019.

[0104] The above method DSPN is cited in XU Z, YIN H, YAO J. Deformable spatial propagation network for depth completion [M]. arXiv, 2020.

[0105] The above method FuseNet is cited in CHEN Y, YANG B, LIANG M, et al. Learning Joint 2D-3D Representations for Depth Completion[C] / / 2019 IEEE / CVF International Conference on Computer Vision(ICCV). Seoul, Korea(South): IEEE, 2019:10022-10031.

[0106] The above method GuideNet is cited in TANG J, TIAN FP, FENG W, et al. Learning Guided Convolutional Network for Depth Completion[J]. IEEE Transactions on Image Processing, 2021, 30: 1116-1129.

[0107] In addition, the experimental results of this algorithm on the KITTI depth completion test set and the detailed comparison with other algorithms are shown in the attached Figure 1 shown.

[0108] It is understood that the present invention is described by some embodiments, and those skilled in the art are aware that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation mode, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, other modifications can be easily implemented, so without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to the specific details and the illustrations shown and described here.

Claims

1. A road scene depth completion method based on semantic information and color image guidance, characterized in that: The method includes designing a dual-branch network consisting of a color image-guided branch and a semantic-guided branch, and introducing semantic information into the network to perform road scene depth completion. The specific steps include: Step S1: constructing a color image guided branch of a dual-branch network, namely, a CG branch, including a color image guided branch backbone, a color depth map prediction layer, and a semantic segmentation layer; Step S2: pre-training the color image guided branch of the dual-branch network; Step S3: fix the pre-trained color image guided branch backbone parameters and train the color depth map prediction layer in step S1; Step S4: construct a semantic guidance branch, i.e., the SG branch, and use the semantic labels generated by the semantic segmentation layer in step S1 as input to enter the semantic guidance branch; Step S5: obtaining an attention-based feature fusion module according to the deep connection of the color image guided branch and the semantic guided branch; Step S6: fix the pre-trained parameters of the entire color image guided branch and train the entire semantic guided branch; then unfreeze the fixed color image guided branch parameters to simultaneously train the color image guided branch and the semantic guided branch; Step S7: Input the features of the color image guided branch and the semantic guided branch into the attention-based feature fusion module to output a refined fused deep feature map so that the color-dominated features in the CG branch decoder are fused into the corresponding encoder features of the SG branch; Step S8: Apply CSPN++ to improve the dense depth map generated by the two-branch network.

2. The road scene depth completion method based on semantic information and color image guidance as claimed in claim 1, characterized in that: In step S1, the color image guiding branch aligns and connects the color image and the sparse depth map as input, and outputs a dense depth map and a confidence map; Among them, in the color image guided branch backbone, we define C B ∈C and D B ∈D, where B is the batch size, C represents the color image, and D represents the depth image; CG branch will be X cg As input, it is first encoded into a hidden representation As shown in the following formula 1, it is then decoded into a hidden representation of the color depth map and semantic joint features based on the following formula 2 φ cg =f(w cg x cg +b cg )#(1);φ cs =g(V cg φ cg +c cg )#(2); Among them, W cg and V cg Represents the weight matrix of the encoder and decoder respectively, and the variable b cg and c cg represents the encoding and decoding bias values, f(.) and g(.) represent the activation functions; In the process of generating color image-guided color depth map, φ cs Generate a confidence map with a batch size of B according to the following formula 3 and color depth map Among them, W cd Represents the weight matrix for generating the color depth layer, variable c cd represents the bias value for generating the color depth map, and h(.) represents the activation function; In the semantic segmentation layer, let the number of semantic segmentation categories be nc, and generate the semantic segmentation label map S according to formula (4): B ∈R B ×nc×H×W , S B = p(V sl φ cs +b sl )#(4); Among them, V sl Represents the weight matrix for generating semantic labels, variable b sl To generate the bias value of the semantic label, p(.) represents the activation function.

3. The road scene depth completion method based on semantic information and color image guidance as claimed in claim 1, characterized in that: In step S2, during training, the input is a color image and a sparse depth map, the output is the semantic segmentation result of the semantic segmentation layer, and the pre-training error is calculated using the cross entropy loss.

4. The road scene depth completion method based on semantic information and color image guidance as claimed in claim 1, characterized in that: In step S3, use Loss is used to estimate the training error of the final color depth map prediction layer: in, The depth map D is the depth map predicted by the color image guide branch. gt The real depth map.

5. The road scene depth completion method based on semantic information and color image guidance as claimed in claim 2, characterized in that: In step S4, firstly, the semantic label probability map S output by the CG branch is B As input, data dimension reduction is performed according to formula 8, from nc dimension to 3 dimension, in, is the semantic label probability map after dimensionality reduction, B is the batch size, V dr represents the dimension reduction weight matrix, variable b dr is the bias value of dimensionality reduction, p(.) represents the activation function; The output has the same structure as the CG branch.

6. The road scene depth completion method based on semantic information and color image guidance as claimed in claim 1, characterized in that: In step S5, in the semantic branch X sg During the decoding process, each layer has a deep connection with the intermediate feature map of the color image guidance branch. Let F RGB ∈R C×H×W , F Sem ∈R C×H×W is the intermediate feature map of the input color image guidance and semantic branch, F = [F RGB ; F Sem ]∈R 2C×H×W Represents their deep connection, and the channel attention weight is calculated according to Formula 11, in, represents the channel attention weight of F, σ represents the sigmoid function, represents the weight matrix layer of the multilayer perceptron. The parameter r controls the number of learnable parameters in the multilayer perceptron. and Represent the maximum pooling feature and the average pooling feature respectively; Applying the channel attention weights to F, we get and make The product F′ is then input into the spatial attention module, as shown in Equation 12: in, represents the spatial attention weight, σ represents the sigmoid function, and Denote the maximum pooling feature and the average pooling feature respectively; the spatial attention weight is applied to F′ to obtain the final output of the feature fusion module Make 7. The road scene depth completion method based on semantic information and color image guidance as claimed in claim 1, characterized in that: In step S6, use The loss is used to estimate the training error of the final semantic guidance branch output depth map: Among them, D sg is the depth map predicted by the SG branch, D gt The real depth map.

8. The road scene depth completion method based on semantic information and color image guidance as claimed in claim 1, characterized in that: In step S7, the fused depth map D f As shown below, Among them, D cg and D sg represents the depth map predicted by the CG branch and the SG branch, and Represent the learned confidence maps of the depth maps predicted by the CG branch and the SG branch.

9. The road scene depth completion method based on semantic information and color image guidance as claimed in claim 8, characterized in that: In step S8, given the fused depth map Embed it into a hidden representation The CSPN++ propagation steps using Atrous convolution are shown in Equation 14, 10. The method for road scene depth completion based on semantic information and color image guidance according to claim 1, characterized in that: When building a dual-branch network, it also includes using the backbone and semantic segmentation layer of one branch to predict semantic information. Specifically, a semantic segmentation label map is generated based on the sparse depth map and RGB map. The semantic segmentation pseudo ground truth is generated based on the existing model, and the RGB images in the KITTI depth completion dataset are converted into semantic label maps.

Images