Road-scene depth completion method based on guidance of semantic information and color image

Abstract

Disclosed in the present invention is a road-scene depth completion method based on the guidance of semantic information and a color image. The method comprises: designing a dual-branch network consisting of a color-image-guided branch and a semantic-guided branch, and introducing semantic information into the network, so as to perform road-scene depth completion. The idea of multi-task learning is used, and a backbone is shared with a depth map prediction layer in the color-image-guided branch, such that the semantic information can be obtained simply by adding a semantic segmentation layer, without the need to add an entire network. Moreover, semantic labels generated by the semantic segmentation layer are fed into the semantic-guided branch as input, such that parameters of the layer can also be adaptively adjusted during network training. By integrating high-precision semantic information and an RGB image, the present invention completes a sparse depth map provided by LiDAR, thereby improving the accuracy and real-time performance of scene understanding in applications such as digital twin, virtual reality, digital infrastructure and intelligent transportation.

Description

Road scene depth completion method based on semantic information and color image guidance TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, and particularly to a road scene depth image completion technology in an intelligent transportation system. More specifically, the present application relates to a road scene depth completion method based on semantic information and color image guidance. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence technology, depth information has been widely applied in many fields, such as digital twin, virtual reality, digital infrastructure, smart transportation, etc. How to obtain dense depth information has become a key task in the field of computer vision. However, existing depth sensors have problems such as too sparse data points, obvious interference from the outside world, etc. when obtaining depth maps, which makes it difficult to apply in real-world tasks. In order to solve these problems, researchers have turned their attention from sensor hardware to algorithm improvement: as long as a sparse depth map is obtained by a lidar, and a high-performance depth completion algorithm is used, a satisfactory dense depth map can be obtained. However, obtaining high-quality dense depth completion results while saving computing resources and improving efficiency is still a major challenge in the field of depth completion. In recent years, deep learning technology, especially depth completion algorithms based on convolutional neural networks, has provided a new solution to this problem. However, these algorithms still have deficiencies in dealing with complex lighting conditions, occlusions and sensor noise, and do not fully utilize scene semantic information.

[0003] And the current method mostly adopts a double-branch network architecture to perform feature fusion, and uses an encoder-decoder structure to perform feature extraction and fusion in different orders between a color image and a LiDAR sparse depth map. For example, the prior art (Towards Precise and Efficient Image Guided Depth Completion[C] / / 2021IEEE International Conference on Robotics and Automation (ICRA). 2021: 13656-13662.) proposed by PENet designs two branches of color-dominant and depth-dominant, extracts features in sequence by using an encoder-decoder, and performs depth fusion of the features at each layer of the encoder-decoder to obtain sufficient exchange of information. However, these methods rely heavily on RGB images to extract color-dominant information (such as object boundaries) to complete the sparse map, but in fact, due to the influence of shadows and reflections, color images have irregular pixel values and do not provide this information most of the time. To solve this problem, the prior art (Towards Attention-based Semantic Aware Guided Depth Completion: arXiv: 2204.13635[R]. arXiv, 2022.) proposed by SemAttNet uses a three-branch network to utilize semantic information, and adds a semantic guidance branch based on PENet, so that the network can better identify object boundaries using semantic information, and reduce the sensitivity of color image guidance methods to optical changes. However, this model still needs to input an additional semantic segmentation image during inference, and does not explore a more efficient way of fusing semantic information, and the network structure has a huge number of parameters in the existing model and slow inference speed. SUMMARY

[0004] An object of the present application is to provide a road scene depth completion method based on semantic information and color image guidance, which integrates high-precision semantic information and RGB images to supplement the sparse depth map provided by the LiDAR, thereby improving the accuracy and real-time performance of scene understanding in digital twin, virtual reality, digital infrastructure, intelligent transportation and other applications, and solving the problem that the RGB images used by existing depth completion methods have contrast and shadow problems, which makes it impossible to fully utilize the semantic information of the image.

[0005] To solve the above technical problems, the present application provides a road scene depth completion method based on semantic information and color image guidance, which comprises a double-branch network composed of a color image guidance branch and a semantic guidance branch, and introduces semantic information in the network to perform road scene depth completion; specifically comprising the following steps:

[0006] Step S1: constructing a color image guided branch of the dual-branch network, i.e., a CG branch, including a color image guided branch backbone, a color depth map prediction layer, and a semantic segmentation layer;

[0007] Step S2: pre-training the color image guided branch of the dual-branch network;

[0008] Step S3: fixing the pre-trained color image guided branch backbone parameters, and training the color depth map prediction layer in step S1;

[0009] Step S4: constructing a semantic guided branch, i.e., a SG branch, taking the semantic label generated by the semantic segmentation layer in step S1 as input into the semantic guided branch;

[0010] Step S5: obtaining an attention-based feature fusion module according to the depth connection of the color image guided branch and the semantic guided branch;

[0011] Step S6: fixing the pre-trained parameters of the entire color image guided branch, training the entire semantic guided branch, and then unfreezing the fixed parameters of the color image guided branch to simultaneously train the color image guided branch and the semantic guided branch;

[0012] Step S7: inputting the features of the color image guided branch and the semantic guided branch into the attention-based feature fusion module to output fine fusion depth feature maps, so that the color-dominant features in the CG branch decoder are fused into the corresponding encoder features of the SG branch;

[0013] Step S8: applying CSPN++ to improve the dense depth map generated by the dual-branch network.

[0014] Preferably, in step S1, the color image guided branch takes the aligned connection of the color image and the sparse depth map as input, and outputs a dense depth map and a confidence map.

[0015] In the color image guided branch backbone, the following definitions are made: as the C B ∈C and D B ∈D are spliced, where B is the batch size, C represents the color image, and D represents the depth image.

[0016] The CG branch takes X cg as input, which is first encoded into a hidden representation as shown in the following formula 1, and then decoded into a hidden representation of color depth map and semantic joint features based on the following formula 2 φ cg =f(W cg X cg +b cg )#(1);φcs =g(V cg φ cg +c cg )#(2);

[0017] Among them, w cg and V cg Let b represent the weight matrices of the encoder and decoder, respectively. cg and c cg The values ​​represent the encoding and decoding biases, and f(.) and g(.) represent the activation functions.

[0018] In the process of generating a color depth map guided by a color image, φ cs Generate a confidence graph with a batch size of B according to the following formula 3. and color depth map

[0019] Among them, W cd The variable c represents the weight matrix used to generate the color depth layer. cd The bias value for generating the color depth map is represented by h(.), and the activation function is represented by h(.).

[0020] In the semantic segmentation layer, let the number of semantic segmentation categories be nc, and generate the semantic segmentation label map S according to formula (4). B ∈R B×nc×H×W S B =p(V sl φ cs +b sl )#(4);

[0021] Among them, V sl The weight matrix for generating semantic labels, variable b sl The bias value for generating semantic labels is p(.), which represents the activation function.

[0022] Preferably, in step S2, during training, the input is a color image and a sparse depth map, and the output is the semantic segmentation result of the semantic segmentation layer. The error of the pre-training is calculated using cross-entropy loss.

[0023] Preferably, in step S3, the following is used: The loss is used to estimate the training error of the final color depth map prediction layer:

[0024] in, Depth map for guiding branch prediction of color images, D gt This is a true depth map.

[0025] Preferably, in step S4, the semantic label probability map S output by the CG branch is first... BAs input, data dimensionality reduction is performed according to formula 8, from nc dimensions to 3 dimensions,

[0026] wherein, is the semantic label probability map after dimensionality reduction, B is the batch size, V dr represents the dimensionality reduction weight matrix, variable b dr is the dimensionality reduction bias value, p(.) represents the activation function.

[0027] The output has the same structure as the CG branch.

[0028] Preferably, in step S5, in the semantic branch X sg During decoding, each layer is depth-connected with the intermediate feature map of the color image guided branch, and let F RGB ∈R C×H×W , F Sem ∈R C×H×W is the input color image guided and semantic branch intermediate feature map, F = [F RGB ; F Sem ] ∈R 2C×H×W represents their depth connection, and the channel attention weight is calculated according to formula 11,

[0029] wherein, represents the channel attention weight of F, and σ represents the sigmoid function, represents the weight matrix layer of the multi-layer perceptron, and the parameter r controls the number of learnable parameters in the multi-layer perceptron, and represent the maximum pooling feature and the average pooling feature, respectively;

[0030] The channel attention weight is applied to F to obtain and make Then the product F' is input into the spatial attention module, as shown in formula 12:

[0031] wherein, represents the spatial attention weight, and σ represents the sigmoid function, and represent the maximum pooling feature and the average pooling feature, respectively; the spatial attention weight is applied to F' to obtain the final output of the feature fusion module so that

[0032] Preferably, in step S6, the loss is used to estimate the training error of the final semantic guided branch output depth map: wherein, Dsg D is the depth map predicted by the SG branch, gt D is the real depth map.

[0033] Preferably, in step S7, the fused depth map D f As shown below,

[0034] Wherein D cg And D sg Indicate the depth map predicted by the CG branch and the SG branch, And Indicate the learning confidence map of the depth map predicted by the CG branch and the SG branch.

[0035] Preferably, in step S8, the given fused depth map D Embedding the hidden representation The CSPN++ propagation step applying Atrous convolution is shown in the following formula 14,

[0036] Preferably, when constructing the dual-branch network, it also includes using the backbone and semantic segmentation layer of one branch to predict semantic information, specifically generating a semantic segmentation label map according to the sparse depth map and the RGB map, and generating a semantic segmentation pseudo ground truth according to an existing model, and converting the RGB image in the KITTI depth completion dataset into a semantic label map.

[0037] The innovation and optimization of the structure design of the application make the depth completion algorithm of the application not only surpass the traditional method in accuracy, but also significantly improve the calculation efficiency and resource utilization, and provide more accurate and efficient depth perception capability for the fields of digital twin, virtual reality, digital infrastructure, smart transportation, etc., at least including the following beneficial effects:

[0038] 1. The application comprises a dual-branch network composed of a color image guided branch and a semantic guided branch, and introduces semantic information in the network. Compared with the early method, the application adds semantic information in the traditional depth guided branch to reduce the difference of depth values near the object boundary, and proves in the ablation experiment that the introduction of semantic information significantly improves the accuracy of the depth completion task.

[0039] 2. The application uses the idea of multi-task learning, shares the backbone with the depth map prediction layer in the color image guided branch, only needs to add a semantic segmentation layer, and does not need to increase the whole network, so as to obtain semantic information. At the same time, the semantic label generated by the semantic segmentation layer is input into the semantic guided branch, so that the parameters of the layer can also be adaptively adjusted during network training. Compared with the existing method using semantic information, the network parameter amount of the application is smaller, the inference speed is faster, and the efficiency is higher.

[0040] 3.The application also proposes a training process, which can enable the deep completion network requiring semantic information to obtain semantic information when facing the absence of semantic labels, thereby guiding the training.Specifically, for the KITTI depth completion dataset without providing semantic labels, the most advanced method of other semantic segmentation benchmarks can be used in combination with the color image of the depth completion dataset to generate pseudo ground truth of semantic labels.Combined with the idea of multi-task learning and the method of multi-stage training of the application, the depth completion network can use semantic information without the need to input semantic labels during inference.

[0041] 4.According to the accuracy of the verification set, the method proposed in the application is better than most methods, and shows comparable performance on other evaluation indicators.The average running time of the algorithm is only 0.14s, which is relatively high in efficiency, further proving its feasibility in practical applications.

[0042] Other advantages, objects and features of the present application will be apparent from the following description, and will be understood by those skilled in the art. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a detailed comparison of the experimental results of the application on the KITTI depth completion test set;

[0044] Figure 2 is a schematic diagram of the network structure of the application;

[0045] Figure 3 is a comparison of the corresponding semantic label graphs of the application and the early research in the prior art;

[0046] Figure 4 is a schematic diagram of the attention-based feature fusion structure of the application. DETAILED DESCRIPTION

[0047] In order to better understand the purpose, structure and function of the application, the application will be further described in detail below in combination with the drawings, so that those skilled in the art can implement it according to the description.

[0048] It should be noted that the experimental methods described in the following embodiments are all conventional methods unless otherwise specified, and the reagents and materials can be obtained from commercial channels unless otherwise specified;In the description of the application, the terms "lateral", "longitudinal", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the application and simplifying the description, and does not indicate or imply that the device or element must have a particular orientation, be constructed and operated in a particular orientation, therefore it cannot be understood as a limitation on the application.

[0049] The present application formulates the image-guided depth completion problem as a three-stage task. In the first stage, the study uses a backbone with a branch of semantic segmentation layers to predict semantic information; in the second stage, the study uses a complete dual-branch network to generate dense depth maps; and in the third stage, the dense depth map is refined using CSPN++ and Atrous convolution. FIG. 2 shows the network structure of the present study.

[0050] The present application discloses a road scene depth completion method based on semantic information and color image guidance, comprising the following steps:

[0051] Step 1: Generate semantic segmentation label map according to sparse depth map and RGB image. The KITTI depth completion dataset does not provide semantic label maps corresponding to RGB images and laser radar point clouds, so the present study generates a semantic segmentation pseudo ground truth according to existing models. Considering that semantic segmentation is usually performed using only RGB, the present study adopts the pre-trained model of InternImage-XL, which ranks first on the Cityscape semantic segmentation benchmark, to convert RGB images in the KITTI depth completion dataset into semantic label maps. The semantic label maps generated by the present study have higher accuracy than the semantic label maps used in early studies that employed semantic segmentation, as shown in FIG. 3.

[0052] Step 2: Construct the color image guidance branch (CG Branch) of the dual-branch network. The color image guidance branch aims to learn important color cues for dense depth completion, learn the discontinuity of object boundaries, and maintain the smoothness of the object interior. The CG branch connects the aligned color image and sparse depth map as input and outputs a dense depth map. This branch is composed of an encoder-decoder network with skip connections, the encoder consists of 10 ResNet blocks, and the decoder consists of a convolutional layer and five transpose convolutional layers for upsampling. The output of this branch includes a dense depth map and a confidence map. Although the dense depth map output by the branch has noise, it provides a basis for learning the structural information of the scene in other branches.

[0053] (1) Color image guidance branch backbone

[0054] Definition For C B ∈C and D B ∈D are concatenated. Where B is the batch size, C represents the color image, and D represents the depth image.

[0055] The CG branch takes X cg as input, which is first encoded into a hidden representation (As formula 1), then based on formula 2, is decoded into a color depth map and a hidden representation of color-semantic joint features

[0056] φ cg =f(W cg X cg +b cg )#(1)

[0057] φ cs =g(V cg φ cg +c cg )#(2)

[0058] where w cg and v cg represent the weight matrix of the encoder, the decoder respectively, the variable b cg and c cg represent the encoding and decoding bias values, f(·) and g(.) represent the activation function.

[0059] (2) Color depth map prediction layer

[0060] In the process of generating color image guided color depth map, φ cs According to formula 3, a confidence map and a color depth map

[0061] where W cd represent the weight matrix of the color depth map generation layer, the variable c cd represent the bias value of the color depth map generation, h(.) represent the activation function.

[0062] (3) Semantic segmentation layer

[0063] In the semantic segmentation layer, let the number of semantic segmentation categories be nc, according to formula (4), the semantic label map S B ∈R B×nc×H×W .

[0064] S B =p(V sl φ cs +b sl )#(4)

[0065] where V sl represent the weight matrix of the semantic label generation, the variable b sl is the bias value of the semantic label generation, p(.) represents the activation function.

[0066] Step 3: Pre-training the color image guided branch of the dual-branch network. The input is the color image and sparse depth map, and the output is the result of semantic segmentation of the semantic segmentation layer. The error of this pre-training is calculated with cross-entropy loss, and the loss of pixel i is defined as:

[0067] where i is the index of the pixel, y gt is the true value, is the log-Softmax value of the predicted probability of pixel i. The log-Softmax function is defined as:

[0068] Step 4: Fix the backbone parameters of the pre-trained color image guided branch, and train the color depth map prediction layer with loss estimation. Since the ground truth contains invalid depth values, only pixels with valid depth values are considered when calculating the loss.

[0069] where, is the depth map predicted by the color image guided branch, D gt is the true depth map.

[0070] Step 5: Construct the semantic guided branch (SG Branch). First, input the semantic label probability map S B output by the CG branch, and perform data dimension reduction according to formula 8, from nc dimensions to 3 dimensions, so that the dimensions of each input are more matched.

[0071] where, is the reduced semantic label probability map, B is the batch size, V dr represents the dimension reduction weight matrix, and b dr is the dimension reduction bias value, and p(.) represents the activation function.

[0072] The output is also the same structure as the CG branch, defined as is the concatenation of B and D sg ∈D under batch size B. The matrix X sg is input to the SG branch, and is encoded into hidden representation φ B×1024×H×W ∈R sg (as formula 9). Then φ sg is decoded into confidence map C B×1×H×W ∈R sg and semantic depth map D B×1×H×W ∈R cg (as formula 10).

[0073] Among them, W cg and V cg The weight matrix represents the encoder and decoder, and variable b is... sg and c sg The values ​​represent the encoding and decoding biases, and f(·) and g(.) represent the activation functions.

[0074] Step 6: In semantic branch X sg During the decoding process, each layer has a deep connection with the intermediate feature map of the color image guiding branch, as shown in Figure 4. Let F... RGB ∈R C×H×W F Sem ∈R C×H×W For the intermediate feature maps guiding and semantic branches of the input color image, F = [F RGB ;F Sem ]∈R 2C×H×W This indicates their deep connection. Channel attention weights are calculated according to formula (11).

[0075] in, F represents the channel attention weights, and σ represents the sigmoid function. This represents the weight matrix layer of a Multilayer Perceptron (MLP). The parameter r controls the number of learnable parameters in the MLP. and These represent the max pooling feature and the average pooling feature, respectively.

[0076] Applying the channel attention weights to F, we obtain And make Then the product F′ is input into the spatial attention module, as shown in Equation 12:

[0077] in, The spatial attention weights are represented by σ, which represents the sigmoid function. and Let F' and F' represent the max-pooling and average-pooling features, respectively. Spatial attention weights are applied to F' to obtain the final output of the feature fusion module. Make

[0078] Step 7: Fix the pre-trained parameters of the entire color image guidance branch and train the entire semantic guidance branch. Then unfreeze the fixed parameters of the color image guidance branch and train both the color image guidance branch and the semantic guidance branch simultaneously. (The same process is repeated.) The loss is used to estimate the training error of the final semantically guided branch output depth map:

[0079] where D sg is the depth map predicted by the SG branch, D gt is the ground truth depth map.

[0080] Step 8: The features of the color-guided branch and the semantic-guided branch CG branch and SG branch are input into the attention-based feature fusion module to output fine fusion depth feature maps, so that the color-guided features in the CG branch decoder are fused into the corresponding encoder features of the SG branch,

[0081] where D f represents the fused depth map, D cg and D sg represent the depth maps predicted by the CG and SG branches, and represent the learning confidence maps of the depth maps predicted by the CG and SG branches.

[0082] Step 9: Apply CSPN++ to improve the dense depth map generated by the dual-branch network. Given the fused depth map embed it into the hidden representation Equation 14 shows the CSPN++ propagation step applying Atrous convolution.

[0083] According to the accuracy of the verification set, the method proposed in the present application is superior to most methods, and has comparable performance on other evaluation indicators. The comparison results of the present application with other methods are shown in Table 1. And the average running time of the algorithm is only 0.14s, which is relatively efficient, further proving its feasibility in practical application.

[0084] Table 1 Quantitative comparison of the algorithm of the present application and other algorithms

[0085] The above method Sparse2Dense(gd) is introduced in MAF, CAVALHEIRO G V, KARAMAN S. Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera [C] / / 2019 International Conference on Robotics and Automation (ICRA). Montreal, QC, Canada: IEEE, 2019: 3288-3295.

[0086] The method NConv-CNN-L2 is introduced in EL DESSOKEY A, FELSBERG M, KHAN F S. Confidence Propagation through CNNs for Guided Sparse Depth Regression[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(10): 2423-2436.

[0087] The method CrossGuidence is introduced in LEE S, LEE J, KIM D, et al. Deep Architecture With Cross Guidance Between Single Image and Sparse LiDAR Data for Depth Completion[J]. IEEE Access, 2020, 8: 79801-79810.

[0088] The method PwP is introduced in XU Y, ZHU X, SHI J, et al. Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints[M]. arXiv, 2019.

[0089] The method DSPN is introduced in XU Z, YIN H, YAO J. Deformable spatial propagation network for depth completion[M]. arXiv, 2020.

[0090] The method FuseNet is introduced in CHEN Y, YANG B, LIANG M, et al. Learning Joint 2D-3D Representations for Depth Completion[C] / / 2019IEEE / CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019: 10022-10031.

[0091] The method GuideNet is introduced in TANG J, TIAN F P, FENG W, et al. Learning Guided Convolutional Network for Depth Completion[J]. IEEE Transactions on Image Processing, 2021, 30: 1116-1129.

[0092] In addition, the experimental results of the algorithm of the present application on the KITTI depth completion test set and the details of other algorithms are shown in FIG. 1.

[0093] It can be understood that the present application is described by some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. Although the embodiments of the present application have been disclosed as above, it is not limited to only the application listed in the specification and the embodiments, and it can be fully applied to various fields suitable for the present application, and additional modifications can be easily realized by those skilled in the art, and therefore the present application is not limited to specific details and the figures shown and described herein, and the general concept defined by the claims and the equivalent scope.

Claims

1. A method for road scene depth completion guided by semantic information and color image, characterized in that, The method comprises the following steps: Step S1: constructing a color image guided branch of the dual-branch network, i.e., a CG branch, including a color image guided branch backbone, a color depth map prediction layer and a semantic segmentation layer; Step S2: pre-training the color image guided branch of the dual-branch network; Step S3: fixing the parameters of the pre-trained color image guided branch backbone, and training the color depth map prediction layer in step S1; Step S4: constructing a semantic guided branch, i.e., an SG branch, using the semantic label generated by the semantic segmentation layer in step S1 as input into the semantic guided branch; Step S5: obtaining an attention-based feature fusion module according to the depth connection of the color image guided branch and the semantic guided branch; Step S6: fixing the parameters of the entire pre-trained color image guided branch, training the entire semantic guided branch, and then unfreezing the fixed parameters of the color image guided branch to simultaneously train the color image guided branch and the semantic guided branch; Step S7: inputting the features of the color image guided branch and the semantic guided branch into the attention-based feature fusion module to output fine fusion depth feature maps, so that the color-dominated features in the CG branch decoder are fused into the corresponding encoder features of the SG branch; Step S8: applying the CSPN++ to improve the dense depth map generated by the dual-branch network. 2.The method of claim 1, wherein, In step S1, the color image guided branch takes the aligned connection of the color image and the sparse depth map as input, and outputs a dense depth map and a confidence map. wherein, in the color image guiding branch backbone, the definition For D B ∈ C and D B stitching of ∈ D, where B is batch size, C denotes color images, and D denotes depth images; The CG branch will X cg As input, is first encoded into a hidden representation The following Equation 1, and then based on the following Equation 2, is decoded into a hidden representation of color depth map and semantic joint features φ cg = f(W cg X cg + b cg ) # (1) ; φ cs = g(V cg φ cg + c cg ) # (2) ; where W cg and V cg denote the weight matrices of the encoder and decoder, respectively, and the variables b cg and c cg denote the encoding and decoding bias values, and f(·) and g(·) denote the activation functions; In generating a color depth map guided by a color image, φ cs A confidence map of batch size B is generated according to the following equation 3 and color depth map where W cd denotes the weight matrix for generating the color depth layer, and variable c cd denotes the bias value for generating the color depth map, and h(.) denotes the activation function; In the semantic segmentation layer, let the number of semantic segmentation categories be nc, and generate a semantic segmentation label map S according to formula (4) B ∈R B×nc×H×W , S B = p(V sl φ cs +b sl ) # (4) ; where V sl represents a weight matrix for generating semantic labels, variable b sl is a bias value for generating semantic labels, and p(.) represents an activation function. 3.The method of claim 1, wherein, In step S2, during training, the input is the color image and the sparse depth map, and the output is the semantic segmentation result of the semantic segmentation layer, and the cross-entropy loss is used to calculate the error of the pre-training. 4.The method of claim 1, wherein, In said step S3, the loss is used to estimate the training error of the final color depth map prediction layer: ​ wherein A depth map is provided for color image guided branch prediction. gt is a real depth map. 5.The method of road scene depth completion guided by semantic information and color image based on claim 2, In the step S4, first, the semantic label probability map S output by the CG branch is obtained B As input, data dimensionality reduction is performed according to formula 8, from nc dimensions to 3 dimensions, wherein for the reduced dimension semantic label probability map, B is the batch size, V dr denotes the reduced dimension weight matrix, variable b dr is the reduced dimension bias value, p(.) denotes the activation function; The output has the same structure as the CG branch. 6.The method of road scene depth completion guided by semantic information and color image based on claim 1, In the step S5, in the semantic branch X sg In the decoding process, each layer is connected with the intermediate feature map of the color image guided branch in depth, and is denoted as F RGB ∈R C×H×W , F Sem ∈R C×H×W is the intermediate feature map of the input color image guided branch and the semantic branch, F = [F RGB ;F Sem ]∈R 2C×H×W represents the depth connection thereof, and the channel attention weight is calculated according to formula 11, wherein, channel attention weights representing F, and σ denotes a sigmoid function, representing a layer of weight matrices of a multilayer perceptron, the parameter r controls the number of learnable parameters in the multilayer perceptron, and Max-pooled features and average-pooled features are represented by Max and Avg, respectively. applying the weights of the channel attention to F to obtain and such that The product F' is then input to the spatial attention module as shown in equation 12: wherein, denotes the spatial attention weight, and σ denotes the sigmoid function, and respectively represent the max-pooling feature and the average-pooling feature; apply the spatial attention weight to F' to obtain the final output of the feature fusion module such that 7. The method for road scene depth completion guided by semantic information and color image based on claim 1, characterized in that, In the step S6, the loss is used to estimate the training error of the final semantic guided branch output depth map: where D is the predicted depth map for the SG branch, D sg is the real depth map. gt is the real depth map. 8.The method of claim 1, wherein, In the step S7, the fused depth map D f As follows, where D cg and D sg represent the depth map of CG branch and SG branch prediction, and The learning confidence map of the depth map predicted by the CG branch and the SG branch is represented by C.

9. The method for road scene depth completion guided by semantic information and color image based on claim 8, characterized in that, In the step S8, the fused depth map is given Embedding it into the hidden representation The CSPN++ propagation step with Atrous convolution is given by Equation 14, 10. The method of claim 1, wherein, When constructing the dual-branch network, the backbone of one branch and the semantic segmentation layer are used to predict semantic information, specifically, the semantic segmentation label map is generated according to the sparse depth map and the RGB image, and the semantic segmentation pseudo ground truth is generated according to the existing model, and the RGB image in the KITTI depth completion dataset is converted into a semantic label map.

Images