A Point Cloud Image Depth Completion Method Based on Cascade Feature Interaction

Abstract

The present invention discloses a method for depth completion of point cloud images based on cascaded feature interaction, belonging to the field of autonomous driving, including: obtaining three-dimensional point clouds and two-dimensional RGB images of an autonomous driving scene, constructing an encoder according to a plurality of residual modules, constructing a decoder according to a plurality of upsampling modules, respectively constructing neural networks for point clouds and images, constructing a plurality of cascaded feature interaction modules between the neural networks of point clouds and images, constructing a dual-branch neural network model of point clouds and images with feature interaction, inputting the three-dimensional point clouds and two-dimensional RGB images of the scene into the dual-branch neural network model of point clouds and images with feature interaction, outputting a dense depth map of the scene, and fusing the depth maps of the scene output by the two branches by using the method of confidence map weighting to obtain a depth map with higher credibility. Compared with other models based on image and point cloud fusion, this method has better depth perception performance when taking images and low-beam lidar point clouds as inputs.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and more specifically, to a method for depth completion of point cloud images based on cascaded feature interaction. Background Art

[0002] Depth perception is a very basic and important perception technology in an autonomous driving system. Its purpose is to obtain accurate and dense depth information of the surrounding scene. Based on the obtained dense depth information, many high-level perception tasks in autonomous driving, such as semantic segmentation, object detection, three-dimensional scene reconstruction, etc., can achieve a great degree of performance improvement. At present, autonomous driving mainly relies on two sensors, cameras and lidar, for depth perception. Each of the two sensors, cameras and lidar, has its own advantages and disadvantages. The image data collected by the camera sensor can obtain rich texture and color information of the scene, but it is greatly affected by lighting conditions. The point cloud data collected by the lidar sensor can obtain accurate depth information of the scene and is not affected by lighting, but the point cloud data is very sparse and cannot provide enough effective information.

[0003] There are existing depth perception schemes based on pure images and depth perception schemes based on images and lidar point clouds in the prior art.

[0004] In the depth perception scheme based on pure images, there is mainly a depth perception scheme for monocular depth estimation. Monocular depth estimation, as the name implies, is to estimate the distance of each pixel in the image relative to the shooting source by using one or the only perspective RGB image. The monocular depth estimation method based on supervised learning directly uses a two-dimensional image as the input and a depth map as the output, and uses the ground truth depth map as the supervision information to train the depth model; in addition, due to the high difficulty of obtaining depth label data, there are currently many algorithms based on unsupervised models, that is, only using the binocular image data collected by two cameras for joint training. Among them, the binocular data can predict each other to obtain the corresponding disparity data, and then evolve according to the relationship between the disparity and the depth, or regard the corresponding problem of each pixel point in the binocular image as a stereo matching problem for training.

[0005] In the depth perception scheme based on images and lidar point clouds, considering the advantages and disadvantages of cameras and lidar sensors, the current autonomous driving perception system usually adopts a multi-sensor perception fusion scheme. By fusing the data of the two sensors, the advantages of the two types of sensor data are complemented to improve the depth perception ability. According to the fusion stage, the existing heterogeneous multi-sensor fusion perception schemes can be divided into three fusion methods: early fusion, mid-term fusion, and late fusion. Among them, early fusion, also known as data-level fusion, is to fuse the two types of perception data at the raw data level. The main idea is to use the coordinate transformation relationship between the camera and the lidar to project the three-dimensional point cloud onto the two-dimensional image plane, and then fuse it by means of channel splicing or direct addition. The main problem of this fusion scheme is that it is difficult to align the data of the two heterogeneous sensors in space, resulting in a relatively coarse fusion granularity and poor fusion effect. Late fusion, also known as result-level fusion, mainly aims to perform perception tasks on the data of the two sensors respectively, and then fuse the perception results of the two sensors. This scheme is very simple to implement, but its disadvantages are also obvious. It does not fully realize the information interaction and complementary advantages between the two types of modal data, and the fusion effect is also limited. Sometimes the fusion result is even worse than the perception result in the case of a single perception modality. Mid-term fusion, also known as feature-level fusion, extracts features from the two types of perception data respectively, and then fuses the extracted features. The advantage is that a network can be designed for a single perception modality data to fully extract features, but there are also disadvantages. It also cannot effectively realize the full interaction of the two types of perception data.

[0006] Currently, the depth estimation methods based on pure images can be divided into traditional methods, machine learning-based methods, and deep learning-based methods. Traditional methods are based on binocular or multi-view images and use stereo matching technology. The triangulation method is used to convert the parallax information between the two images into depth information to estimate the scene depth information from the images. The disadvantages of this method are difficult matching in weak texture areas and repetitive texture areas, large computational load, sensitive to light, and limited application range. For monocular image depth estimation based on machine learning, the Markov Random Field (MRF) is used to construct a probability graph model for depth relationships, and the image depth estimation is realized by minimizing the energy function. The disadvantages of this method are low algorithm accuracy, long time consumption, many artificial assumptions, and cumbersome processing process. It is difficult to estimate the correct scene depth information and cannot be applied to actual scenarios. Deep learning-based methods are also the most commonly used methods at present. By inputting RGB images, the model is trained to learn the mapping relationship from the image to the depth map. The disadvantage of this method is that the performance of the model depends heavily on the data quality. Therefore, in occasions with poor lighting conditions such as at night and in tunnels, the model performance will decline severely.

[0007] The solution based on the fusion of images and lidar point clouds is the mainstream solution for depth perception in current autonomous driving, which overcomes the shortcomings of the pure-image-based solution. In the current depth perception technology for point cloud and image fusion, although the pre-fusion-based solution can retain the original information of the data to the greatest extent, it is difficult to achieve fine-grained spatial alignment and fusion of heterogeneous perception data in the existing technology, often resulting in poor fusion effects. The post-fusion-based solution fuses the perception results of the two sensors at the decision-making level and is easy to implement. However, due to the limitations of each sensor, there is a lack of interaction between the two modalities and the complementary advantages between the two modalities cannot be realized. Therefore, the fusion effect is not good, and sometimes the perception effect is even worse because the perception results of the two sensors are contradictory. Currently, the more commonly used fusion solution is multi-modal fusion perception based on the feature layer. The advantage of this method is that it does not need to consider the spatial alignment of the data. However, the current implementations of various feature-layer fusion technologies are still not fine enough in terms of fusion granularity. They often regard one modality as auxiliary supplementary information for the other modality or simply fuse them by addition, resulting in insufficient interaction and insufficient fusion between the two modalities. Summary of the Invention

[0008] In order to solve the problems of insufficient depth perception accuracy and poor fusion effect of heterogeneous perception data existing in the above solutions, and to achieve fine-grained fusion and sufficient interaction of the data of the two sensors, namely point cloud and image, the present invention proposes a dual-branch heterogeneous perception data cascade interaction network, which fuses the corresponding features of the two modalities at multiple scales and inputs the fused features into the branch networks corresponding to their respective modalities, improving the information richness and depth perception ability of the two branch networks. In addition, the idea of an auxiliary task is introduced. By introducing an image reconstruction task, the model is guided to learn the scene structure information in the image, making the output depth map structure information more complete. Finally, through the confidence map, the high-confidence depth values in the output depth maps of the two branch networks are used as the final model output to obtain the fusion perception result.

[0009] An embodiment of the present invention provides a method for depth completion of point cloud and image based on cascade feature interaction, including:

[0010] Obtain the three-dimensional point cloud of the autonomous driving scene and the two-dimensional RGB image of the scene;

[0011] Construct two encoders for feature extraction of the three-dimensional point cloud of the scene and the two-dimensional RGB image of the scene according to the residual modules of multiple Resnet34;

[0012] Construct two decoders for feature reduction of the three-dimensional point cloud of the scene and the two-dimensional RGB image of the scene according to multiple upsampling modules;

[0013] Connect the encoder and decoder of the three-dimensional point cloud extraction and reduction branch to construct a three-dimensional point cloud branch neural network;

[0014] Connect the encoder and decoder of the scene 2D RGB image extraction and restoration branch to construct a scene 2D RGB image branch neural network;

[0015] Set the corresponding levels of the residual modules of the two encoders in the scene 3D point cloud branch neural network and the scene 2D RGB image branch neural network;

[0016] Construct multiple cascaded feature interaction modules. The input of each cascaded feature interaction module is connected to the output of the corresponding level of the residual modules of the two encoders, and the output of each cascaded feature interaction module is connected to the next corresponding level of the two encoders to construct a point cloud and image dual-branch neural network model for feature interaction;

[0017] Input the scene 3D point cloud and the scene 2D RGB image into the point cloud and image dual-branch neural network model for feature interaction, and output the scene depth map;

[0018] Fuse the scene depth maps by using the method of confidence map weighting to obtain a new scene depth map.

[0019] Preferably, both of the two encoders for feature extraction of the scene 3D point cloud and the scene 2D RGB image include five cascaded residual modules, and both of the two decoders for feature restoration of the scene 3D point cloud and the scene 2D RGB image include five cascaded upsampling modules. In the encoder for feature extraction of the scene 3D point cloud, the convolutional neural network of the residual module adopts a sparse convolutional neural network with a convolution kernel of 3x3; in the encoder for feature extraction of the scene 2D RGB image, the convolutional neural network of the residual module adopts a standard convolutional neural network with a convolution kernel of 3x3.

[0020] Preferably, both the 3D point cloud branch neural network and the scene 2D RGB image branch neural network include multiple different convolutional layers, pooling layers, activation layers, transposed convolutional layers, and cross-scale feature connection layers.

[0021] Preferably, each decoder includes five cascaded upsampling modules, and each upsampling module includes a transposed convolution, a batch normalization layer, and a pooling layer.

[0022] Preferably, the number of cascaded feature interaction modules is five, and each of the cascaded feature interaction modules includes a 1x1 convolution, three dilated convolutions with dilation rates of 1, 2, and 4 respectively, and a 1x1 convolution;

[0023] The output of the last cascaded feature interaction module is used as the input of the first upsampling layer of the point cloud and image dual-branch neural network model for feature interaction.

[0024] Preferably, it further includes:

[0025] Taking the reconstructed image output by the last upsampling module as an auxiliary task, calculate the difference between the reconstructed image and the input scene two-dimensional RGB image according to the L2 loss function, and train the model to learn the structural information of the image.

[0026] Preferably, the L2 loss function includes:

[0027]

[0028] where D i represents the depth value at the i-th position of the predicted depth map, represents the depth value at the i-th position of the ground truth depth map.

[0029] Preferably, it also includes training the point cloud and image double-branch neural network model for feature interaction, which includes:

[0030] Taking the point cloud and image data pair as the training data set;

[0031] Performing enhancement processing on the images in the data set, and the enhancement processing includes flipping processing, cropping processing, brightness adjustment, normalization processing, and conversion to a tensor map processing;

[0032] Initializing the model parameters with a random Gaussian distribution;

[0033] Setting the loss function for model training and the loss function for the reconstructed image, adding the two loss functions, and setting their respective coefficients. Taking minimizing the loss function as the optimization goal, training the model through the gradient update strategy to obtain the optimal model parameters.

[0034] Preferably, by using the method of weighting with a confidence map to fuse the scene maps, a new scene depth map is obtained, including:

[0035] Respectively obtaining two estimated depth values at the corresponding positions of the two scene depth maps output by the cloud and image double-branch neural network;

[0036] Calculating the confidence levels of the two estimated depth values at the corresponding positions of the two scene depth maps;

[0037] Respectively calculating the product of the confidence level and the depth value at the corresponding positions of the two scene depth maps to obtain a picture to be fused;

[0038] Adding the depth values at the corresponding positions of the pictures to be fused output by the point cloud and image double-branch neural network respectively, and fusing them into the scene depth map to obtain a new scene depth map.

[0039] The embodiment of the present invention provides a point cloud image depth completion method based on cascaded feature interaction. Compared with the prior art, its beneficial effects are as follows:

[0040] The proposed method for depth completion of point cloud images based on cascaded feature interaction greatly improves the interaction degree of two heterogeneous perception data through multi-scale fine-grained fusion of point cloud image features, realizes the complementary advantages of the two perception data, and enriches the information content of the two branches by re-inputting the fused features into the corresponding branch networks, thus enhancing the perception ability of the two branch networks. Finally, the outputs of the two branch networks are fused, and the depth value with a higher confidence at the corresponding position of the two output depth maps is taken as the depth value of the final output depth map. In addition, the outputs of the two modalities are independent of each other and do not rely on a certain branch, which improves the robustness of the model to noise. Finally, compared with other models based on image and point cloud fusion, our model has better performance when taking images and low-beam lidar point clouds as inputs, which also proves that our model can be applied to resource-constrained devices with only cameras and low-cost low-beam lidar. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 FIG. is a model structure diagram of a method for depth completion of point cloud images based on cascaded feature interaction according to an embodiment of the present invention;

[0042] Figure 2 FIG. is a structure diagram of a cascaded feature interaction module proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0044] Embodiment:

[0045] Referring to Figures 1 - 2 , this example provides a method for depth completion of point cloud images based on cascaded feature interaction. By taking advantage of the characteristics that images have color and texture information but are affected by light, while point clouds are not affected by light but have sparse information, a multi-scale fine-grained fusion model is constructed to achieve the full fusion and complementary advantages of image and point cloud data, greatly enhancing the perception ability of small objects. In this example, by introducing an image reconstruction auxiliary task, the model is guided to learn the structural information in the image, making the object contours in the output depth map more complete. By using the method of confidence map weighting, the output depth maps of the two branches are fused to obtain a more reliable depth map.

[0046] Step1: Build a multi-scale dual-branch neural network model using the residual module of Resnet34;

[0047] Step101: The encoder parts of both branch networks consist of five residual blocks, and the decoder parts of both consist of five upsampling modules.

[0048] Step102: The two branch networks include multiple different convolutional layers, pooling layers, activation layers, transposed convolutional layers, and cross-scale feature connections. Among them, in the encoder part of the point cloud branch, the convolutional networks of the five residual blocks use sparse convolutions, and in the encoder part of the image branch, the convolutional networks of the five residual blocks use standard convolutional networks. The kernel size of all convolutional neural networks is 3x3.

[0049] Step103: Each upsampling module in the decoder consists of a transposed convolution, a batch normalization layer, and a pooling layer.

[0050] Step104: Between the two branch networks, there are five cascaded feature interaction modules. The cascaded feature interaction modules are composed of a 1x1 convolution, dilated convolutions with dilation rates of 1, 2, and 4 respectively, and a 1x1 convolution in the top-down construction order. The input is the feature maps of the corresponding levels of the two branch networks.

[0051] Step105: Five upsampling modules are connected after the last cascaded feature interaction module. Each upsampling module consists of a convolutional network layer, a normalization layer, and an activation layer. The output of the last upsampling module is the reconstructed input image. The reconstructed image is used as an auxiliary task, and the L2 loss function is used to calculate the difference between the reconstructed image and the input RGB image to train the model to learn the structural information of the image.

[0052] Working process: Each network of the dual-branch network has an output depth map. By calculating the confidence levels of the two estimated depth values at the corresponding positions of the two depth maps, multiplying the confidence levels by the depth values of the corresponding depth maps respectively, and adding them together, the depth value at the corresponding position of the final output depth map is obtained.

[0053] Step2: The images in the dataset are enhanced. The enhancement operations include flipping, cropping, brightness, etc., and then normalized, and finally converted into tensor form to obtain a training dataset that is convenient for deep learning convolutional neural networks to process.

[0054] Step 3: The real-scenario autonomous driving dataset used in this example is the KITTI 2015 depth estimation and depth completion dataset, including the left and right images of a binocular camera, lidar point clouds, and ground truth depth maps. The images input into the model are cropped to a resolution of HxW of 325x1216. To accelerate the training speed of the model, the input images are normalized with zero mean in this example. Before starting the training, the model parameters are initialized with a random Gaussian distribution. Sufficient randomness can enhance the performance of the model. The specific parameter settings during the training of this example are as follows:

[0055] Parameter name Parameter value Batch size 16 Input image resolution (H x W) 352x1216 Number of training epochs 30 Learning rate 1e-4 Effective depth value range (unit: m) 0-80

[0056] Step 4: According to the dual-branch network model we built, set the loss function for model training and the loss function for image reconstruction, add the two loss functions, and set their respective coefficients. With the goal of minimizing the loss function, train the model through a gradient update strategy to obtain the optimal model parameters.

[0057] The loss function used in this example is the L2 loss function: , where D i represents the depth value at the i-th position of the predicted depth map, and represents the depth value at the i-th position of the ground truth depth map. In this example, the Adam optimizer is used to optimize the model parameters to achieve the goal of minimizing the loss function. The optimization process of the Adam algorithm can be summarized as follows: At each iteration, the learning rate of each parameter is dynamically adjusted using the sample mean estimate and sample squared mean estimate of the gradient, so that the parameter update during training is relatively stable and the model gradient can decrease steadily.

[0058] Step 5: Input the point cloud and image data pair into the dual-branch network model to obtain the finally output depth map.

[0059] The method for depth completion of point cloud image fusion based on cascaded feature fusion in this example is based on a convolutional neural network. It uses a network specifically designed for two types of perceptual data to extract the features of the two modalities respectively, and uses a cascaded feature network to achieve fine-grained fusion of multi-scale point cloud image features, which fully improves the interaction degree of the two modalities, enhances the depth perception ability of the two branch networks, and improves the robustness of the model to noise. After introducing the image reconstruction task and guiding the model to learn the scene structure information in the image through the image reconstruction task, the output depth map of our model has better integrity on the object contour. Through comparison on the KITTI2015 depth completion and depth estimation tasks, our model has achieved the best performance on the depth estimation task and also achieved competitive performance on the depth completion task. In the robustness experiment of introducing Gaussian noise, our model has also obtained strong competitive results, which proves the practicability of the present invention.

[0060] The above are only several specific embodiments of the present invention disclosed. Those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the present invention. However, the embodiments of the present invention are not limited thereto, and any changes that can be thought of by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for depth completion of point cloud images based on cascaded feature interaction, characterized in that Including: Obtaining a three-dimensional point cloud and a two-dimensional RGB image of an autonomous driving scenario; Constructing two encoders for feature extraction of the three-dimensional point cloud and the two-dimensional RGB image of the scenario according to the residual modules of multiple cascaded Resnet34; Constructing two decoders for feature restoration of the three-dimensional point cloud and the two-dimensional RGB image of the scenario according to multiple cascaded upsampling modules; Connecting the output of the encoder of the three-dimensional point cloud extraction and restoration branch to the input of the decoder to construct a three-dimensional point cloud branch neural network; Connecting the output of the encoder of the two-dimensional RGB image extraction and restoration branch to the input of the decoder to construct a two-dimensional RGB image branch neural network; Setting each layer of the residual modules of the two encoders in the three-dimensional point cloud branch neural network and the two-dimensional RGB image branch neural network in one-to-one correspondence; Building a feature interaction module in sequence according to a 1x1 convolution, three dilated convolutions with dilation rates of 1, 2, and 4 respectively, and a 1x1 convolution, and cascading them to obtain multiple cascaded feature interaction modules. The input of each cascaded feature interaction module is connected to the corresponding layer output of the residual modules of the two encoders, and the output of each cascaded feature interaction module is connected to the next corresponding layer of the two encoders to construct a point cloud and image dual-branch neural network model for feature interaction; Inputting the three-dimensional point cloud and the two-dimensional RGB image of the scenario into the point cloud and image dual-branch neural network model for feature interaction, and outputting two depth maps of the scenario; Fusing the two depth maps of the scenario by using the method of weighting with a confidence map to obtain a new depth map of the scenario.

2. A method for depth completion of point cloud and image based on cascaded feature interaction according to claim 1, wherein the two encoders for feature extraction of the three-dimensional point cloud and the two-dimensional RGB image of the scenario both include five cascaded residual modules, and the two decoders for feature restoration of the three-dimensional point cloud and the two-dimensional RGB image of the scenario both include five cascaded upsampling modules. In the encoder for feature extraction of the three-dimensional point cloud, the convolutional neural network of the residual module adopts a sparse convolutional neural network with a convolutional kernel of 3x3; in the encoder for feature extraction of the two-dimensional RGB image, the convolutional neural network of the residual module adopts a standard convolutional neural network with a convolutional kernel of 3x3.

3. The method for depth completion of point cloud images based on cascaded feature interaction according to claim 1, wherein Both the three-dimensional point cloud branch neural network and the two-dimensional RGB image branch neural network include multiple different convolutional layers, pooling layers, activation layers, transposed convolutional layers, and cross-scale feature connection layers.

4. A method for depth completion of point cloud images based on cascaded feature interaction, characterized in that Each decoder includes five cascaded upsampling modules, and each upsampling module includes a transposed convolution, a batch normalization layer, and a pooling layer.

5. The method for depth completion of point cloud images based on cascaded feature interaction according to claim 4, wherein, The number of cascaded feature interaction modules is five; The output of the last cascaded feature interaction module is used as the input of the first upsampling layer of the point cloud and image dual-branch neural network model for feature interaction.

6. The method for depth completion of point cloud images based on cascaded feature interaction according to claim 4, characterized in that Also including: Taking the reconstructed image output by the last upsampling module as an auxiliary task, calculating the gap between the reconstructed image and the input two-dimensional RGB image of the scenario according to the L2 loss function, and training the model to learn the structural information of the image.

7. The method for depth completion of point cloud images based on cascaded feature interaction according to claim 6, characterized in that, The L2 loss function includes: Among them, D i represents the depth value at the i-th position of the predicted depth map, and represents the depth value at the i-th position of the ground truth depth map.

8. A method for depth completion of point cloud images based on cascaded feature interaction according to claim 7, characterized in that It also includes the training of a point cloud and image dual-branch neural network model for feature interaction, which includes: Using the point cloud and image data pair as the training data set; Performing enhancement processing on the images in the data set. The enhancement processing includes flipping processing, cropping processing, brightness adjustment, normalization processing, and conversion to a tensor graph; Initializing the model parameters with a random Gaussian distribution; Setting the loss function for model training and the loss function for the reconstructed image, adding the two loss functions, and setting their respective coefficients. Taking the minimization of the loss function as the optimization objective, training the model through a gradient update strategy to obtain the optimal model parameters.

9. A method for depth completion of point cloud images based on cascaded feature interaction, characterized in that The method of fusing two scene depth maps by using a confidence map weighting to obtain a new scene depth map includes: Respectively obtaining two estimated depth values at corresponding positions of the two scene depth maps output by the cloud and image dual-branch neural network; Calculating the confidence levels of the two estimated depth values at corresponding positions of the two scene depth maps; Respectively calculating the product of the confidence level and the depth value at corresponding positions of the two scene depth maps to obtain a picture to be fused; Adding the depth values at corresponding positions of the picture to be fused output by the point cloud and image dual-branch neural network respectively and fusing them into the scene depth map to obtain a new scene depth map.

Images