A depth estimation system and method that integrates semantic information from lidar and vision

Abstract

This invention belongs to the field of autonomous driving technology, specifically relating to a depth estimation system and method that integrates semantic information from LiDAR and vision. It includes a first-level network and a second-level network. In the first-level network, RGB image features and radar point cloud features are fused using a dual-encoder-single-decoder structure to generate a sparse depth map. In the second-level network, semantic information obtained from RGB image segmentation is introduced to implement hierarchical constraints. Combined with channel feature adaptation and attention mechanisms, the sparse depth map is processed to obtain a dense depth map, thus generating the final depth map. During this process, a multi-constraint loss function composed of LiDAR branch loss, sparse depth map loss, and final prediction result loss is designed to optimize the system and guide the network to perform more accurate depth estimation. Compared with existing technologies, this invention obtains more comprehensive, accurate, and efficient depth estimation results.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically relating to a depth estimation system and method that integrates semantic information from LiDAR and vision. Background Technology

[0002] Depth estimation is a key technology in modern computer science that enables autonomous systems to understand and perceive three-dimensional spatial information, thereby enabling precise interaction and operation in complex environments. This technology is widely used in fields such as augmented reality (AR), robot navigation, and autonomous driving.

[0003] Currently, the main methods for depth estimation can be broadly categorized into two main types:

[0004] The first type of method involves directly measuring the depth data of a scene using hardware devices. Commonly used hardware devices include active sensors such as radar and lidar. These sensors typically acquire detailed depth point cloud data by emitting laser beams or electromagnetic waves and relying on the reflection signals from the surfaces of surrounding objects. This allows them to construct accurate 3D surface models and depth data. However, these devices are usually limited to specific application scenarios and are relatively expensive.

[0005] The second type of method uses monocular depth estimation techniques to directly recover the depth information of the environment from a single image. By analyzing features and patterns in the image, the depth estimation algorithm can estimate the depth estimation model of the distance from each pixel to the camera in an RGB image. Its advantage is that it does not rely on expensive hardware and has broad application potential. However, in some practical application scenarios, it is difficult to obtain a large amount of real depth data as training samples; at the same time, its performance will be significantly affected by insufficient lighting conditions (such as at night or in indoor environments) or severe weather (such as heavy rain, dense fog, or snowstorms).

[0006] The two common depth estimation methods mentioned above each have their own advantages and limitations. In recent years, researchers have begun to try new methods that combine hardware measurement with monocular depth estimation algorithms.

[0007] In autonomous driving applications, road conditions and scene structures are relatively fixed. During operation, vehicles primarily rely on depth information of target areas, such as lane lines, surrounding vehicles, and pedestrians. The dependence on depth information of non-target areas, such as the sky, buildings, and vegetation, is relatively low. Existing applications mainly focus on improving overall performance and reducing depth estimation errors through network structure improvements, without fully considering the specificities of autonomous driving scenarios and the differences between practical application needs and theoretical research.

[0008] Therefore, this study aims to develop a depth estimation method that fully considers the specific characteristics of autonomous driving scenarios and the differences between practical application needs and theoretical research, in order to obtain more comprehensive, accurate, and reliable depth estimation results. This will enable autonomous driving systems to understand the three-dimensional structure of the surrounding environment more accurately, thereby improving the reliability of autonomous driving systems. Summary of the Invention

[0009] The purpose of this invention is to provide a depth estimation system and method that integrates semantic information from LiDAR and vision. It fully considers the special characteristics of autonomous driving scenarios and practical application needs, utilizes LiDAR and visible light camera data, combines deep learning technology and hierarchical constraint technology, and uses semantic segmentation to integrate semantic information into network training to obtain more comprehensive, accurate and profound depth estimation results.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A depth fusion estimation system integrating semantic information from lidar and vision includes a first-level network and a second-level network.

[0012] The first-level network includes a first encoder and a first decoder. The first encoder is a dual encoder comprising an image branch network and a radar point cloud branch network. The image branch network is used to extract image features from RGB image data. The radar point cloud branch network includes a sparse pre-mapping module and a residual block connected after the sparse pre-mapping module. The sparse pre-mapping module is used to extract a first point cloud feature from the radar point cloud data, and the residual block is used to extract a second point cloud feature, i.e., the final point cloud feature, from the first point cloud feature. The first decoder is used to receive the image features provided by the image branch network and the point cloud features provided by the radar branch network, and fuse the image features and point cloud features to generate a sparse depth map.

[0013] The second-level network includes a semantic segmentation module, a second encoder, and a second decoder. The semantic segmentation module is used to perform semantic segmentation on RGB image data to obtain semantic information. The second encoder receives sparse depth maps and semantic segmentation information, and obtains fused features by concatenating the sparse depth maps and semantic segmentation information. The second decoder receives the fused features and decodes them to generate a dense depth map, and generates the final depth prediction map based on the dense depth map.

[0014] Furthermore, the image branching network is a residual network RseNet34.

[0015] Furthermore, the sparse pre-mapping module includes five sequentially connected SPM structures, whose inputs are radar point cloud data and an initial binary observation mask, and whose output is the first point cloud feature; the initial binary observation mask is generated based on the radar point cloud data, and subsequent binary observation masks are determined by performing a max pooling operation on the output of the previous SPM structure; each SPM structure includes five sequentially connected sparse invariant convolutional layers, whose convolutional kernels are 7×7, 5×5, 3×3, 3×3, and 1×1, respectively, with the first four having 16 channels and the last having 1 channel.

[0016] Furthermore, the first decoder includes four sequentially connected up-projection (UpProj) modules.

[0017] Furthermore, the second encoder includes a feature adaptation module and a multi-layer fusion module. The input of the feature adaptation module is the sparse depth map output by the first decoder, which is used to extract features from the sparse depth map and output the extracted features to the first input of the multi-fusion module. The second input of the multi-fusion module receives semantic information provided by the semantic segmentation module and uses a channel attention mechanism to weightedly fuse the semantic information and the features extracted by the feature adaptation module to obtain fused features.

[0018] Furthermore, the second decoder includes four up-projection UpProj modules and a bilinear interpolation layer connected after the four up-projection UpProj modules.

[0019] Furthermore, the loss function used for optimization in the aforementioned depth estimation system that integrates semantic information from LiDAR and vision is:

[0020] Loss=λ1L LiDAR +λ2L Mid +λ3L Final

[0021] Where λ1, λ2, and λ3 are hyperparameters λ1L LIDAR The loss function for the radar branch network; λ²L Mid λ3L represents the loss of the first-level network.Final This represents the loss in the final prediction result.

[0022] Furthermore, the semantic segmentation module uses a pre-trained eepLabV3+ model to perform semantic segmentation.

[0023] A depth estimation method that integrates semantic information from lidar and vision includes the following steps:

[0024] Step 1: Use a residual network to extract image features from the RGB image data;

[0025] Step 2: Use the lidar branch network to extract point cloud features from the lidar point cloud data. The lidar branch network includes a sparse pre-mapping module and a residual block connected after the sparse pre-mapping module. The sparse pre-mapping module is used to extract the first point cloud feature from the lidar point cloud data, and the residual block is used to extract the second point cloud feature from the first point cloud feature, which is the final point cloud feature.

[0026] Step 3: Use the semantic segmentation module to perform semantic segmentation on the RGB image data to obtain semantic segmentation information;

[0027] Steps 1 through 3 have no specific order;

[0028] Step 4: Perform cross-modal fusion of image features and point cloud features to obtain fused features;

[0029] Step 5: Based on the fused features, use the first decoder to generate a sparse depth map;

[0030] Step 6: Use the second encoder to concatenate the semantic segmentation information and the sparse depth map, and then input the concatenation into the second decoder to obtain the dense depth map through decoding.

[0031] Step 7: Generate the final depth prediction map based on the dense depth map.

[0032] Furthermore, step 7 also includes visualization processing of the final depth prediction map.

[0033] This invention provides a LiDAR and visual depth fusion estimation system and method that integrates semantic information. In the first-level network, RGB image features and radar point cloud features are fused through a dual-encoder-single-decoder structure to generate a sparse depth map. In the second-level network, semantic information obtained from RGB image segmentation is introduced to achieve hierarchical constraints. Combined with channel feature adaptation and attention mechanisms, the sparse depth map is processed to obtain a dense depth map, which is then used to generate the final depth map. During this process, a multi-constraint loss function composed of LiDAR branch loss, sparse depth map loss, and final prediction result loss is designed to optimize the system and guide the network to perform more accurate depth estimation. Compared with existing technologies, this invention obtains more comprehensive, accurate, and precise depth estimation results. Attached Figure Description

[0034] Figure 1 This is an example of a lidar and visual depth fusion estimation system architecture that integrates semantic information;

[0035] Figure 2 This is a diagram of the RseNet34 architecture used in the image branching network of the embodiment;

[0036] Figure 3 This is a schematic diagram of the sparse pre-mapping module (SPM) structure in the radar branch network of an embodiment.

[0037] Figure 4 This is a schematic diagram of sparse invariant convolution in the sparse pre-mapping module of the embodiment;

[0038] Figure 5 This is a schematic diagram of the UpProj module in the decoder of the first-level network in the embodiment;

[0039] Figure 6 This is a schematic diagram of the encoder structure in the second-level network of the embodiment;

[0040] Figure 7 This is a visualization and analysis diagram of the final depth map of the embodiment, where a is the RGB input; b is the output of the unipolar network; and c is the output after incorporating semantic segmentation information. Detailed Implementation

[0041] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0042] like Figure 1 As shown in the figure, this embodiment provides a lidar and visual depth fusion estimation system that integrates semantic information, including a first-level network and a second-level network.

[0043] The first-level network consists of a first encoder and a first decoder. The first encoder includes an image branch network and a radar point cloud branch network.

[0044] Image branching networks are used to extract image features from RGB image data. The aim is to process RGB images using this structure to provide scene information for depth estimation, supporting the transformation of depth information from sparse LiDAR point clouds to dense depth information. For example... Figure 2 As shown, the image branching network in this embodiment uses the residual network RseNet34, which has two training paths. The main path consists of stacked traditional convolutional layers, while the side branches are skip connections. The main branch is formed by adding a series of convolutional layers, side branches, and the processed main branch, followed by another activation function. This allows the network to learn the residual between the input and output, rather than directly learning the output, thus alleviating the gradient vanishing problem in deep networks and enabling the network to successfully train deeper structures.

[0045] The radar point cloud branch network consists of a sparse pre-mapping module and residual blocks connected after the sparse pre-mapping module. The sparse pre-mapping module is used to extract the first point cloud feature from the radar point cloud data. The residual block is used to extract the second point cloud feature, i.e. the final point cloud feature, from the first point cloud feature.

[0046] The formula for a standard convolution operation is as follows:

[0047]

[0048] The convolution kernel size is 2k+1, the weight is w, and the bias is b. The standard convolution operation described above is effective when the input data is complete, non-sparse, and continuous. However, when the input data is sparse radar point cloud data, traditional convolution operations cannot effectively handle this incomplete information, destroying the original discreteness of the data and turning it into continuous data.

[0049] Sparse invariant convolution, on the other hand, explicitly considers sparsity by calculating the effective pixels and appropriately normalizing the output. Its calculation formula is as follows:

[0050]

[0051] Where o is the data mask, when the input matrix x u,v When effective u,v The value is 1 when the input matrix x is 1. u,v When invalid, o u,v It is 0; a small ∈ was added to the denominator to avoid the case where there is no input like x. u+i,v+i When observed, the filter location is divided by zero. Simultaneously, sparse invariant convolution determines subsequent observation masks by utilizing maximum pooling operations to maintain tracking of the valid input information state and make it available for the next layer of the network.

[0052] Therefore, the sparse pre-mapping module in this embodiment consists of five sequentially connected SPM structures. Its input is radar point cloud data and an initial binary observation mask, and its output is the first point cloud feature. The initial binary observation mask is generated based on the radar point cloud data, and subsequent binary observation masks are determined by performing a max-pooling operation on the output of the previous SPM structure. Figures 3-4 As shown, each SPM structure consists of five sparse invariant convolutional layers connected in sequence, with convolutional kernels of 7×7, 5×5, 3×3, 3×3, and 1×1, respectively. The first four have 16 channels, and the last one has 1 channel.

[0053] The first decoder receives image features from the image branch network and point cloud features from the radar branch network, and fuses the image features and point cloud features to generate a sparse depth map. In this embodiment, the first decoder consists of four sequentially connected up-projection (UpProj) modules. Figure 5 As shown, the structure of each UpProj module is as follows.

[0054] First, the input information for each feature map is a feature map F0 with a resolution of W*H. This feature map is first passed through an inverse pooling layer consisting of 2×2 convolutional kernels to double its size, resulting in a 2W*2H feature map F1. Then, F1 is further processed by convolutional layers and the non-linear activation function ReLU. Skip connections are used to... Figure 5 The skip connection passes F1 through convolutional kernels of size 5 and 3, then passes it through an additional convolutional kernel of size 5. The two different feature maps calculated by F1 are then directly added together to obtain a new feature F2, which is the output of the residual module in the diagram. Finally, the 2W*2H resolution F2 is processed with the ReLU activation function to increase the network's non-linearity. F2 is then mapped to a W*H feature map using a 3×3 convolution, and finally, bilinear upsampling is used to restore the feature map to 2W*2H.

[0055] The second-level network includes a semantic segmentation module, a second encoder, and a second decoder. The semantic segmentation module obtains semantic segmentation information for the driving scenario by performing semantic segmentation on RGB image data. The semantic segmentation module uses existing semantic segmentation techniques, such as the DeepLabV3+ model, to perform semantic segmentation on RGB images. The segmentation process includes:

[0056] 1) The semantic information in the application scenario is classified into importance levels according to the requirements. The purpose of the classification is to determine the target area and the non-target area. In this embodiment, the semantic information importance level classification in the driving scenario is shown in Table 1, where importance level 0 is the non-target area and importance levels 1 and 2 are the target areas.

[0057] Table 1. Classification of the Importance Levels of Semantic Information

[0058]

[0059] 2) Based on the importance level classification results, assign a category label to each pixel in the RGB image to obtain semantic segmentation results and realize semantic understanding of each pixel in the RGB image.

[0060] In this embodiment, if the semantic segmentation result is used directly, jagged noise will be generated in the boundary region, making it easy to overlook the use of surrounding information in pixel-level classification. To solve this problem, enhance the boundary between target and non-target areas, and improve the accuracy of target region prediction, this embodiment processes the semantic segmentation output by using a sliding window and pooling operations.

[0061] The second encoder receives a sparse depth map D and semantic segmentation information S. It obtains a fused feature F by concatenating D and S at the channel scale. Its network structure includes a feature adaptation module and a multi-layer fusion module. The feature adaptation module takes a sparse depth map with residual connections as input (which is the output of the first decoder) and extracts features from the sparse depth map to obtain feature F. d The first input to the multi-fusion module is F. d The second input is S, which is fused with features extracted from the sparse depth map and semantic information using a channel attention mechanism to obtain F. The specific operation is as follows: Figure 6 As shown:

[0062] The feature adaptation module, referencing the residual blocks in the ResNet network, first convolves D through a 1×1 convolutional layer, then processes it in two paths. One path is used by the feature refinement mapping module to obtain feature f1(Conv(D)), and the other path, after convolution, is added to f1(Conv(D)) and then output. The formula is:

[0063] Yi=ReLU(Conv(S)+f1(Conv(D)))

[0064] Where Fi and Yi represent the input and output of the i-th layer, Conv represents the convolution operation, and f1 represents the feature refinement mapping operation.

[0065] The specific operation of the multi-fusion module is as follows:

[0066] (1) Semantic information and decoder features processed by the feature adaptation module are concatenated at the channel scale.

[0067] (2) The concatenated result undergoes global pooling, followed by convolutional layers with a kernel size of 1*1.

[0068] The system uses ReLU activation function, a 1x1 convolutional layer, and a Sigmoid activation function, with the output being weights.

[0069] (3) The final output is:

[0070] y=σ(f2(Global[Yi,FSi])⊙Yi+Fsi

[0071] Where Global represents global pooling, σ represents the Sigmoid activation function, f2 represents the specific convolution operation, ⊙ represents the pixel-level multiplication operator used for multiplying two matrices, FSi represents the encoder features, and Yi represents the processed decoder output.

[0072] The second decoder takes fused features as input and outputs a predicted depth map. It first decodes the fused features to generate a dense depth map, and then generates the final depth prediction map based on the dense depth map. The structure of the second decoder in this embodiment is the same as that of the first decoder, and will not be described again here.

[0073] The depth estimation system for LiDAR and vision fusion that integrates semantic information in this embodiment uses the following loss function for optimization:

[0074] Loss=λ1L LiDAR +λ2L Mid +λ3L Final

[0075] Where λ1, λ2, and λ3 are hyperparameters λ1L LIDAR The loss function for the radar branch network; λ²L Mid λ3L represents the loss of the first-level network. Final This represents the loss in the final prediction result.

[0076] A depth estimation method that integrates semantic information from lidar and vision includes the following steps:

[0077] Step 1: Use a residual network to extract image features from the RGB image data;

[0078] Step 2: Use the lidar branch network to extract point cloud features from the lidar point cloud data;

[0079] The Sparse Pre-Mapping (SPM) module and residual blocks extract features from the point cloud data to obtain point cloud features.

[0080] Step 3: Use a semantic segmentation network to perform semantic segmentation on the RGB image data to obtain semantic segmentation information;

[0081] Steps 1 through 3 have no specific order;

[0082] Step 4: Perform cross-modal fusion of image features and point cloud features to obtain fused features;

[0083] Step 5: Based on the fused features, use the first decoder to generate a sparse depth map;

[0084] Step 6: After concatenating the semantic segmentation information and the sparse depth map at the channel scale, input them into the codec and obtain the dense depth map through encoding and decoding.

[0085] Step 7: Generate the final depth prediction map based on the dense depth map, and perform visualization processing on the final depth prediction map. Figure 7 This is a visualization and analysis diagram of the final depth map schematic of the embodiment, where a is the RGB input; b is the output of the unipolar network; and c is the output after incorporating semantic segmentation information. Figure 7 As can be seen, compared with the original single-level network, the lidar and vision fusion depth estimation system and method that fuses semantic information in this embodiment recovers more accurate detail information in the selected area, making the target object and the background more distinct, and at the same time, the depth prediction of the edge of the target area and the non-target area is more accurate.

[0086] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A lidar and visual depth fusion estimation system integrating semantic information, comprising a first-level network and a second-level network, characterized in that: The first-level network includes a first encoder and a first decoder. The first encoder is a dual encoder, including an image branch network and a radar point cloud branch network. The image branch network is used to extract image features from RGB image data. The radar point cloud branch network includes a sparse pre-mapping module and residual blocks connected after the sparse pre-mapping module. The sparse pre-mapping module is used to extract the first point cloud feature from the radar point cloud data, and the residual blocks are used to extract the second point cloud feature, i.e., the final point cloud feature, from the first point cloud feature. The first decoder receives image features provided by the image branch network and point cloud features provided by the radar branch network, and fuses the image features and point cloud features to generate a sparse depth map. The first decoder includes four sequentially connected up-projection modules. The input information for each module is a feature F0 with a resolution of W*H, which first passes through an inverse pooling layer composed of 2×2 convolutional kernels to process the feature map. The size is doubled to obtain a 2W*2H feature map F1. F1 is then further processed through convolutional layers and the ReLU nonlinear activation function. Skip connections are used to pass F1 through convolutional kernels of sizes 5 and 3, followed by an additional convolutional kernel of size 5. The feature maps calculated using these two different kernels are then directly added together to obtain a new feature map F2, which is the output of the residual module. Finally, the ReLU activation function is applied to F2 with a resolution of 2W*2H to enhance the network's nonlinearity. A 3×3 convolution is used to map F2 to a feature map of size WH, and finally, bilinear upsampling is used to restore it to a resolution of 2W*2H. The second-level network includes a semantic segmentation module, a second encoder, and a second decoder. The semantic segmentation module performs semantic segmentation on RGB image data to obtain semantic information. The second encoder receives a sparse depth map and semantic segmentation information, and obtains fused features by concatenating the sparse depth map and semantic segmentation information. The second encoder includes a feature adaptation module and a multi-layer fusion module. The feature adaptation module takes the sparse depth map output by the first decoder as input, extracts features from the sparse depth map, and outputs the extracted features to the first input of the multi-layer fusion module. The second input of the multi-layer fusion module receives semantic information provided by the semantic segmentation module, and uses a channel attention mechanism to weightedly fuse the semantic information and the features extracted by the feature adaptation module to obtain fused features. The second decoder receives the fused features and decodes them to generate a dense depth map, and then generates the final depth prediction map based on the dense depth map.

2. The lidar and visual depth fusion estimation system according to claim 1, characterized in that: The image branching network is a residual network RseNet34.

3. The lidar and visual depth fusion estimation system according to claim 2, characterized in that: The sparse pre-mapping module includes five sequentially connected SPM structures, whose inputs are radar point cloud data and initial binary observation mask, and whose outputs are the first point cloud features. The initial binary observation mask is generated based on radar point cloud data, and the subsequent binary observation mask is determined by performing a max pooling operation on the output of the previous SPM structure. Each SPM structure includes five sparse invariant convolutional layers connected in sequence, with convolutional kernels of 7×7, 5×5, 3×3, 3×3, and 1×1, respectively. The first four have 16 channels, and the last one has 1 channel.

4. The lidar and visual depth fusion estimation system according to claim 3, characterized in that: The second decoder includes four up-projection modules and a bilinear interpolation layer connected after the four up-projection modules.

5. A lidar and visual depth fusion estimation system according to any one of claims 1 to 4, characterized in that: The aforementioned depth estimation system that fuses semantic information with LiDAR and vision uses the following loss function for optimization: Loss=λ1L LiDAR +λ2L Mid +λ3L Final Where λ1, λ2, and λ3 are hyperparameters λ1L LIDAR The loss function for the radar branch network; λ²L Mid λ3L represents the loss of the first-level network. Final This represents the loss in the final prediction result.

6. A lidar and visual depth fusion estimation system according to any one of claims 1 to 4, characterized in that: The semantic segmentation module uses a pre-trained eepLabV3+ model to perform semantic segmentation.

7. A depth estimation method that fuses semantic information from lidar and vision, characterized in that, Includes the following steps: Step 1: Use a residual network to extract image features from the RGB image data; Step 2: Use the lidar branch network to extract point cloud features from the lidar point cloud data. The lidar branch network includes a sparse pre-mapping module and a residual block connected after the sparse pre-mapping module. The sparse pre-mapping module is used to extract the first point cloud feature from the lidar point cloud data, and the residual block is used to extract the second point cloud feature from the first point cloud feature, which is the final point cloud feature. Step 3: Use the semantic segmentation module to perform semantic segmentation on the RGB image data to obtain semantic segmentation information; Step 4: Perform cross-modal fusion of image features and point cloud features to obtain fused features; Step 5: Based on the fused features, a sparse depth map is generated using the first decoder. The first decoder includes four sequentially connected upprojection modules. The input information of each module is a feature F0 with a resolution of W*H. First, the feature map size is doubled by an inverse pooling layer composed of 2×2 convolutional kernels, resulting in a 2W*2H feature map F1. Then, F1 is further processed by convolutional layers and the nonlinear activation function ReLU. After passing F1 through convolutional kernels of sizes 5 and 3 using skip connections, F1 is passed through an additional convolutional kernel of size 5. Then, the feature maps calculated by these two different convolutional kernels are directly added together to obtain a new feature map F2, which is the output of the residual module. Finally, the ReLU activation function is applied to F2 with a resolution of 2W*2H to enhance the nonlinearity of the network. F2 is mapped to a feature map of size WH using a 3×3 convolution, and finally, it is restored to a resolution of 2W*2H by bilinear upsampling. Step 6: The semantic segmentation information and the sparse depth map are concatenated using the second encoder and then input into the second decoder to obtain a dense depth map. The second encoder includes a feature adaptation module and a multi-layer fusion module. The feature adaptation module takes the sparse depth map output by the first decoder as input and extracts features from the sparse depth map. The extracted features are then output to the first input of the multi-fusion module. The second input of the multi-fusion module receives the semantic information provided by the semantic segmentation module and uses a channel attention mechanism to weightedly fuse the semantic information and the features extracted by the feature adaptation module. Step 7: Generate the final depth prediction map based on the dense depth map; In the above steps, steps 1 through 3 are not in any particular order.

8. The depth estimation method for LiDAR and vision fusion based on semantic information as described in claim 7, characterized in that, Step 7 also includes visualizing the final depth prediction map.

Images