Cross-modal driving scene laser point cloud segmentation method

Abstract

The invention discloses a cross-modal driving scene laser point cloud segmentation method, which is characterized in that a pre-trained single-modal laser radar segmentation network is used as a teacher network to assist in supervising and training a cross-modal point cloud segmentation network with a point cloud branch and an image branch at the same time; bidirectional feature fusion and transmission are carried out between a point cloud branch and an image branch through a bidirectional feature fusion completion module, defective image features are completed, and the method specifically comprises the steps of teacher network construction and training, cross-modal student network construction, joint training, feedforward reasoning and the like. Compared with the prior art, the method has the advantages that the point cloud segmentation precision is remarkably improved on a driving scene data set, meanwhile, the robustness of point cloud segmentation is improved, and the problem that part of laser radar points cannot acquire corresponding pixels due to the view field range difference of the vehicle-mounted laser radar and the surround-view camera is solved; the corresponding relation between the point cloud and the image is difficult to maintain after data enhancement, and the segmentation precision is reduced due to the lack of image features.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-modal driving scene laser point cloud segmentation method based on single-to-multimodal knowledge distillation. Background Art

[0002] Semantic segmentation plays a vital role in large-scale outdoor scene understanding and has been widely used in autonomous driving, robotics and related fields. In recent years, academia and industry have made great efforts to achieve accurate and reliable segmentation results using different sensors, such as camera images and LiDAR point clouds. Although each modality has its own advantages, it also inevitably faces unique challenges. For example, camera images provide rich color and texture information, but there is ambiguity in depth and they are sensitive to illumination changes. LiDAR point clouds contain accurate depth and geometry information and are robust to environmental changes, but lack color descriptors. In addition, the density of point clouds drops sharply with increasing distance.

[0003] Combining LiDAR point clouds and images for robust semantic segmentation shows great potential. However, the heterogeneity between the two modalities (e.g., density, field of view) poses challenges in establishing a bijective mapping between each point and pixel. This modality alignment problem introduces new challenges in network design and data processing for cross-modal methods, specifically: 1) LiDAR points projected outside the image plane face the problem of missing modality; 2) During model training, the complexity of maintaining geometric consistency limits the application of many data augmentation techniques, thus affecting the model's inference accuracy and generalization ability; 3) The missing image modality can seriously affect the model's inference accuracy, resulting in a decrease in the model's robustness.

[0004] To address the above challenges, current research methods focus on removing point clouds that do not appear in the camera's field of view during network training and reasoning, or using complex Transformer modules to complete missing image features. The first method will cause blind spots in network predictions, leading to potential safety hazards, while the second method, although it can work, will bring huge graphics memory overhead and reasoning delays, and cannot meet the real-time requirements of autonomous driving scenarios.

[0005] In summary, it is a key issue to be solved urgently to develop a cross-modal point cloud segmentation method that can segment lidar point clouds outside the camera's field of view, can freely use multiple data augmentation methods during training, and can rely on a single point cloud modality to give reasonable segmentation results when the image modality is missing. Summary of the invention

[0006] The purpose of the present invention is to design a cross-modal driving scene laser point cloud segmentation method for the shortcomings of the prior art, using a pre-trained single-mode laser radar segmentation network as a teacher network, and assisting in the supervision of training a cross-modal point cloud segmentation network composed of a point cloud branch, an image branch, and a bidirectional feature fusion and completion module, wherein the bidirectional feature fusion and completion module is used to realize the bidirectional fusion and transmission of features and complete the missing image features. The method uses the color images taken by the vehicle-mounted laser radar and the surround camera, and obtains the point cloud segmentation result through a segmentation network with a point cloud branch and an image branch. It effectively solves the problem that some laser radar points cannot obtain corresponding pixels due to the difference in the field of view of the vehicle-mounted laser radar and the surround camera in the driving scene, it is difficult to maintain the correspondence between the point cloud and the image after data enhancement, and the segmentation accuracy decreases due to the lack of image features. The accuracy of point cloud segmentation is significantly improved on the driving scene data set, while the robustness of point cloud segmentation is improved. The method is simple, effective, and has good application prospects and commercial development value.

[0007] The object of the present invention is achieved by: a laser point cloud segmentation method for a cross-modal driving scene, which is characterized in that the method specifically includes:

[0008] Step 1: Dataset preparation and preprocessing

[0009] Collect a large number of lidar point clouds and surround view images taken by the on-board lidar and camera in driving scenes, denoted as P and I respectively, annotate the lidar point cloud with point-by-point semantic labels, denoted as y, and surround view images do not need to be labeled. Calibrate the camera intrinsic parameters, the transformation matrix of the lidar and the camera relative to the vehicle.

[0010] Step 2: Build and train the LiDAR segmentation network

[0011] Based on three-dimensional sparse convolution, an encoder-decoder structure of the lidar point cloud segmentation network is constructed. The lidar point cloud is subjected to data enhancement such as multi-frame fusion and instance enhancement, denoted as data enhancement A, and randomly rotated, flipped and scaled, denoted as data enhancement B. The data is then input into the lidar point cloud segmentation network, and the network is supervised by point-by-point true value semantic labels.

[0012] Step 3: Build a cross-modal point cloud segmentation network based on the bidirectional feature fusion completion module

[0013] A cross-modal point cloud segmentation network is constructed, which includes a point cloud branch, an image branch, and a bidirectional feature fusion and completion module. The point cloud branch is a lidar point cloud segmentation model with an encoder-decoder structure constructed by three-dimensional sparse convolution. The image branch is an image segmentation model with an encoder-decoder structure constructed by a two-dimensional convolution operator. The bidirectional feature fusion and completion module consists of several convolutional layers and multi-layer perceptrons, which can complete the point cloud features that are missing matching image features, and fuse the input matching point cloud features and image features and return them to the corresponding point cloud and image branches.

[0014] Step 4: Joint training

[0015] 4-1: Input lidar point cloud and multi-frame surround view images;

[0016] 4-2: After data enhancement A and data re-enhancement B, the lidar point cloud is input into the lidar segmentation network to obtain encoder features and prediction results;

[0017] 4-3: After the data enhancement B and its corresponding surround view images are randomly deleted, a fixed number of images are input into the cross-modal point cloud segmentation network based on the bidirectional feature fusion completion module to obtain the point cloud branch encoder features and cross-modal prediction results;

[0018] 4-4: Use the LiDAR point cloud to annotate point-by-point semantic labels, as well as the encoder features and prediction results of the LiDAR segmentation network obtained in step 4-2 to supervise the point cloud branch encoder features and cross-modal prediction results obtained in step 4-3, thereby training the cross-modal point cloud segmentation network based on the bidirectional feature fusion completion module.

[0019] Step 5: Feedforward Inference

[0020] The lidar point cloud of the driving scene and its corresponding surround view image are input into the trained cross-modal point cloud segmentation network, and the segmentation result of the lidar point cloud is obtained in a feed-forward manner.

[0021] Compared with the prior art, the present invention significantly improves the accuracy of point cloud segmentation on driving scene data sets while improving the robustness of point cloud segmentation. It solves the problems that some lidar points cannot obtain corresponding pixels due to the difference in the field of view of the vehicle-mounted lidar and the surround camera, it is difficult to maintain the correspondence between the point cloud and the image after data enhancement, and the segmentation accuracy decreases due to the missing image features. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flowchart for training LiDAR point cloud segmentation network;

[0023] Figure 2 Flowchart for training a cross-modal point cloud segmentation network;

[0024] Figure 3 This is the joint training flow chart;

[0025] Figure 4 is the reasoning flow chart;

[0026] Figure 5 The experimental results for the benchmark method comparison are shown in Figure 2.

[0027] Figure 6 This is the experimental result diagram for the missing image modality comparison;

[0028] Figure 7 Correction map to obtain image segmentation effect. DETAILED DESCRIPTION

[0029] In order to facilitate the understanding of the present invention, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0030] Example 1

[0031] The specific steps of a cross-modal driving scene laser point cloud segmentation based on single-to-multimodal knowledge distillation are as follows:

[0032] Step 1: Dataset preparation and preprocessing

[0033] Collect a large number of lidar point clouds and surround view images taken by the on-board lidar and camera in driving scenes, denoted as P and I respectively, annotate the lidar point cloud with point-by-point semantic labels, denoted as y, and surround view images do not need to be labeled. Calibrate the camera intrinsic parameters, the transformation matrix of the lidar and the camera relative to the vehicle.

[0034] The present invention first uses the vehicle-mounted lidar point cloud P of a large number of driving scenes to perform data enhancements such as multi-frame fusion and instance enhancement, which are difficult to ensure the consistency of the point cloud image, as well as random rotation, flipping and scaling of the lidar point cloud P that will not affect the consistency of the point cloud image, to train a lidar point cloud segmentation network (lidar point cloud segmentation model). The purpose is to learn the knowledge brought by all the above data enhancements through the lidar point cloud segmentation model, so as to provide supervision for the subsequent training of a cross-modal point cloud segmentation network based on the bidirectional feature fusion and completion module.

[0035] Step 2: Build and train the teacher network (LiDAR point cloud segmentation network)

[0036] Based on three-dimensional sparse convolution, an encoder-decoder structure of the lidar point cloud segmentation network is constructed. The lidar point cloud P is enhanced by point cloud data A through multi-frame fusion and instance enhancement, and then the point cloud data is enhanced again by randomly rotated, flipped and scaled point cloud data B and input into the lidar point cloud segmentation network. The network is supervised by point-by-point true value semantic labels.

[0037] See also Figure 1 ,The specific training process of the lidar point cloud segmentation network is as follows:

[0038] A0: Use data enhancement methods such as multi-frame fusion and instance enhancement (data enhancement A) to perform a series of transformations on the input lidar point cloud P, aiming to enhance the integrity and generalization of model features;

[0039] A1: A series of transformations are performed on the LiDAR point cloud P output by A0 using data enhancement methods such as random rotation around the z-axis, horizontal flipping along the x-axis or y-axis, and random scaling with the coordinate center as the origin (data enhancement B), aiming to enhance the generalization ability of the model features;

[0040] A2: Use 3D sparse convolution instead of traditional 2D convolution to build a 3D point cloud encoder similar to ResNet-18 to extract features of the input 3D point cloud P;

[0041] A3: Referring to the classic encoder-decoder structure, a 3D point cloud decoder is built using 3D sparse convolution, which accepts the 3D point cloud features output by the 3D point cloud encoder, gradually restores its size, and obtains the segmentation result of the 3D point cloud;

[0042] A4: Use the true value label to supervise the output results, and the loss function is the cross entropy loss function (CrossEntropy) and the Lovasz-Softmax loss function.

[0043] Step 3: Construct a cross-modal student network (cross-modal point cloud segmentation network)

[0044] A cross-modal point cloud segmentation network is constructed, which includes a point cloud branch, an image branch, and a bidirectional feature fusion and completion module. The point cloud branch is a lidar point cloud segmentation model with an encoder-decoder structure constructed by three-dimensional sparse convolution; the image branch is an image segmentation model with an encoder-decoder structure constructed by a two-dimensional convolution operator; the bidirectional feature fusion and completion module is composed of several convolutional layers and a multi-layer perceptron, which completes the point cloud features that are missing matching image features, and fuses the input matching point cloud features and image features and returns them to the corresponding point cloud and image branches.

[0045] The cross-modal point cloud segmentation network based on the bidirectional feature fusion and completion module constructed by the present invention includes three parts: a point cloud branch, an image branch, and a bidirectional feature fusion and completion module. The point cloud branch is a lidar point cloud segmentation model of an encoder-decoder structure constructed by three-dimensional sparse convolution; the image branch is an image segmentation model of an encoder-decoder structure constructed by a two-dimensional convolution operator; the bidirectional feature fusion and completion module includes feature fusion modules from point cloud to image and image to point cloud, and a missing image feature repair module; the feature fusion module is intended to perform feature fusion on cross-modal features and return them to their respective branches to improve feature expression capabilities; the feature repair module is intended to use point cloud features to estimate corresponding image features to achieve the purpose of repairing missing image features.

[0046] See also Figure 2 , the specific training of the cross-modal student point cloud segmentation network (cross-modal point cloud segmentation network) is as follows:

[0047] B0: The input point cloud is transformed through data enhancement B, and the surround image corresponding to the input point cloud is randomly deleted to enhance the network's adaptability to missing image features;

[0048] B1: Input the point cloud and surround image into the point cloud encoder and image encoder respectively to obtain point cloud features and image features;

[0049] B2: Input the point cloud and image features into the bidirectional feature fusion and completion module for feature fusion and completion, aiming to enhance the expression ability of the two modal features through feature fusion. The process is as follows:

[0050] 1) Fusion of point cloud features into image features, specifically expressed by the following formulas (a) to (b):

[0051]

[0052] Among them, F P and F I They represent the input point cloud and image features respectively, Conv, Cat and σ represent the convolution layer with a convolution kernel of 1×1, feature connection and Sigmoid activation function respectively.

[0053] 2) Input the point cloud features into the feature patching module to predict the corresponding image features as the image features corresponding to the missing image feature points. For points with corresponding image features, the mean square error loss is calculated between their image features and the image features output by the patching module using the following formula (c) to supervise the patching module:

[0054]

[0055] Among them, B, F P2I , and They represent the mask of the points in the field of view, the features output by the patching module, and F′ I The sampled point cloud corresponds to image features and point-wise multiplication operators.

[0056] 3) The image features are integrated into the point cloud features, which is specifically expressed by the following formulas (d) to (e):

[0057]

[0058] Among them, w′ is the image feature fusion weight, F fuse is the fused point cloud feature, Mlp represents the multi-layer perceptron, Represents the image features corresponding to the point cloud. For points within the camera's field of view, the image features are obtained from F′ I For points that are not in the camera's field of view, their image features come from the image features generated by the feature patching module;

[0059] B3: Input the point cloud and image features after feature fusion and repair into the point cloud decoder and image decoder, gradually restore the size of the point cloud and image, and obtain the segmentation results of the point cloud and image;

[0060] B4: Based on the projection position of the point cloud on the image, sample the part corresponding to the true value of the point cloud from the image segmentation result;

[0061] B5: Use the point cloud truth label to supervise the segmentation results of the point cloud and image, and the loss function is the cross entropy loss function and the Lovasz-Softmax loss function.

[0062] Step 4: Joint training

[0063] The trained teacher point cloud semantic segmentation network (lidar point cloud segmentation network) will supervise the feature maps and prediction results of the student cross-modal point cloud semantic segmentation network.

[0064] See also Figure 3 , the specific steps of joint training are as follows:

[0065] C0: The feature map output by the teacher network encoder will be compared with the feature map output by the student network point cloud branch encoder to calculate the point-wise mean square error loss to supervise the student network, and is expressed by the following formula (f):

[0066]

[0067] Among them, F &,7 and F &,；They respectively represent the feature maps output by the point cloud branch encoder in the cross-modal point cloud segmentation network and the feature maps output by the lidar point cloud segmentation network encoder.

[0068] C1: The point cloud prediction results output by the teacher network will be calculated by the Kullback-Leibler divergence loss function with the point cloud prediction results output by the student network point cloud branch decoder to supervise the student network, which is expressed by the following formula (g):

[0069]

[0070] in, and p ； They respectively represent the segmentation prediction results output by the point cloud branch in the cross-modal point cloud segmentation network and the segmentation prediction results output by the lidar point cloud segmentation network.

[0071] The training loss function of the student network is expressed by the following formula (h):

[0072]

[0073] in, and y represent the segmentation prediction result and semantic segmentation truth value output by the image branch in the cross-modal point cloud segmentation network, respectively.

[0074] Step 5: Feedforward Inference

[0075] The lidar point cloud P of the driving scene and its corresponding surround view image I are input into the above-trained cross-modal point cloud segmentation network, and the segmentation result of the lidar point cloud P is obtained in a feed-forward manner.

[0076] See also Figure 4 , given the lidar point cloud and surround view image, the image decoder can be omitted during reasoning to obtain the point cloud segmentation result. The specific reasoning steps are as follows:

[0077] D0: Input the lidar point cloud and surround view image into the point cloud encoder and image encoder respectively to obtain point cloud and image features;

[0078] D1: Input the point cloud and image features into the bidirectional feature fusion and patching module for feature fusion and patching of missing image features;

[0079] D2: Input the point cloud feature map into the point cloud decoder to gradually restore the scale and obtain the prediction result.

[0080] This example uses the academic dataset nuScenes, which contains 1,000 driving scenes, including 700 scenes for training data, 150 scenes for verification data, and 150 scenes for test data. Each driving scene consists of 20s of continuous image frames, and all images are captured by 6 on-board cameras, covering the surround view. This example uses 700 scenes of training data and 150 scenes of verification data divided in a standard way, and uses an RTX A6000 as hardware support.

[0081] See also Figure 3 , the lidar point cloud is input into the teacher network, the lidar point cloud and surround image are input into the student network, the feature map and prediction results output by the teacher network will supervise the student network, and after the student network obtains the supervision of the teacher network and the true value label, it combines the information of the point cloud and image to output the point cloud segmentation prediction result.

[0082] See also Figure 4 ,This embodiment is tested in nuScenes. During the test, the lidar point cloud and the surround image are input, the model feedforward reasoning is performed, and the segmentation result of the lidar point cloud is output.

[0083] See also Figure 5 By comparing the segmentation accuracy (mIoU is the accuracy index), model size (#Params(M) is the model size index) and frame-by-frame reasoning time (Latency(ms) is the reasoning time index) with the benchmark method, the effectiveness and efficiency of the present invention are proved. The present invention has a smaller model size and reasoning latency at the same accuracy, and has a higher reasoning accuracy at the same model size and reasoning latency, which proves that the present invention can effectively utilize the complementary information of the two modalities to accurately and efficiently segment the lidar point cloud.

[0084] See also Figure 6 By comparing the segmentation accuracy in the case of gradual loss of image modalities, it is proved that the present invention has a certain robustness to the loss of image modalities. nuScenes has 6 surround images corresponding to each frame of laser radar, and randomly deletes 1 to 6 images during reasoning to simulate the situation in which some or even all cameras are damaged in actual scenes. As the number of images involved in reasoning decreases, the network reasoning accuracy steadily decreases, and finally reaches the same accuracy as the single point cloud modality network reasoning accuracy, which proves that the present invention has a certain robustness to the loss of image modalities.

[0085] See also Figure 7 After obtaining the corresponding perspective image, the present invention can correct the segmentation result according to the information in the image and improve the segmentation accuracy, which fully proves that the present invention can effectively utilize the complementary information of the two modalities to accurately and efficiently segment the lidar point cloud.

[0086] The present invention uses color images taken by vehicle-mounted laser radar and surround-view cameras to obtain point cloud segmentation results through a segmentation network with point cloud branches and image branches. A pre-trained single-modality laser radar segmentation network is used as a teacher network to assist in the supervised training of a cross-modal point cloud segmentation network with both point cloud branches and image branches, wherein the point cloud branch and the image branch of the cross-modal point cloud segmentation network are bidirectionally fused and transferred through a bidirectional feature fusion and completion module to complete the missing image features.

[0087] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.

Claims

1. A cross-modal driving scene laser point cloud segmentation method, characterized in that: A method based on single-to-multimodal knowledge distillation is adopted. The pre-trained single-modal LiDAR segmentation network is used as the teacher network to assist in the supervised training of a cross-modal point cloud segmentation network with both point cloud branches and image branches. The point cloud branch and the image branch are bidirectionally fused and transferred through a bidirectional feature fusion and completion module, and the missing image features are completed to achieve cross-modal driving scene laser point cloud segmentation. The specific steps are as follows: Step 1: Dataset preparation and preprocessing Collect the lidar point cloud P and surround view image I taken by the on-board lidar and camera in the driving scene, annotate the lidar point cloud with point-by-point semantic labels y, calibrate the camera intrinsic parameters, and the transformation matrix of the lidar and camera relative to the vehicle; Step 2: Build and train the LiDAR segmentation network Based on 3D sparse convolution, an encoder-decoder structure of lidar point cloud segmentation network is constructed. The lidar point cloud P is enhanced by multi-frame fusion and instance enhancement point cloud data A, and then the point cloud data is enhanced by random rotation, flipping and scaling B and input into the lidar point cloud segmentation network. The network is supervised by point-by-point true value semantic labels. Step 3: Build a cross-modal point cloud segmentation network based on the bidirectional feature fusion completion module A cross-modal point cloud segmentation network is constructed, which includes a point cloud branch, an image branch, and a bidirectional feature fusion completion module. The point cloud branch is a lidar point cloud segmentation model with an encoder-decoder structure constructed by three-dimensional sparse convolution; the image branch is an image segmentation model with an encoder-decoder structure constructed by a two-dimensional convolution operator; the bidirectional feature fusion completion module is composed of several convolutional layers and a multi-layer perceptron, which completes the point cloud features that are missing matching image features, and returns the input matching point cloud features and image features to the corresponding point cloud and image branches after fusion; Step 4: Joint training 4-1: Input the lidar point cloud P and multi-frame surround image I; 4-2: After the laser radar point cloud P is enhanced by point cloud data A and point cloud data re-enhancement B, it is input into the laser radar segmentation network to obtain the encoder features and prediction results; 4-3: After the data enhancement B of the lidar point cloud P and the surround view image I corresponding to the lidar point cloud P are randomly deleted, they are input into the cross-modal point cloud segmentation network based on the bidirectional feature fusion completion module to obtain the point cloud branch encoder features and cross-modal prediction results; 4-4: Use the LiDAR point cloud P to annotate point-by-point semantic labels, as well as encoder features and prediction results to supervise the point cloud branch encoder features and cross-modal prediction results, so as to train the cross-modal point cloud segmentation network based on the bidirectional feature fusion completion module; Step 5: Feedforward Inference The lidar point cloud P of the driving scene and its corresponding surround view image I are input into the above-trained cross-modal point cloud segmentation network, and the segmentation result of the lidar point cloud P is obtained in a feed-forward manner.

Images