An unstructured road segmentation method based on point cloud and image feature fusion

Abstract

The application provides a non-structured road segmentation method based on point cloud and image feature fusion. First, a data set of a non-structured road scene is constructed by using an automatic driving simulator CARLA. Then, a 3D backbone network for training and reasoning is designed according to the characteristics of the point cloud data, and a 2D network for auxiliary training is designed according to the characteristics of the image data. Then, knowledge distillation is used for point cloud-image feature fusion, on the basis of retaining the original 3D feature knowledge, fully utilizing the texture and color features of the 2D features as prior information for auxiliary training, and using multi-modal features to enhance the 3D backbone network. Finally, the model is trained by using the self-built non-structured road scene data set, and the non-structured road segmentation is carried out. The application effectively solves the problem of the current scarcity of non-structured road scene data sets, and effectively extracts and fuses the complementary multi-modal features, further improving the accuracy and efficiency of the non-structured road segmentation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and vehicle traffic, and relates to a semantic segmentation method for unstructured roads, in particular to an unstructured road segmentation method based on fusion of point cloud and image features. BACKGROUND

[0002] The ultimate goal of the development of automatic driving technology is to not need human attention in any environment, and effective segmentation of drivable roads is one of the key tasks to achieve this goal. Existing researches are mostly concentrated on structured road scenes with clear lane lines, regular road surfaces and clear markers, while for unstructured road scenes, there are problems such as blurred road boundaries and irregular road shapes, thus bringing great challenges to road segmentation.

[0003] At present, laser radar and RGB camera are two commonly used different modal sensors in automatic driving data collection, which are used to obtain different modal data. Due to the great difference between different modal data, most existing methods use single modal data collected by a single sensor as input. However, the limited features of single modal data are difficult to support further improvement of road segmentation accuracy, so it is necessary to fuse multi-modal features. The 3D point cloud data collected by laser radar contains rich structure information and is not affected by weather and light, but can only capture sparse and textureless data; the 2D image data collected by RGB camera contains rich texture and color information, but lacks three-dimensional features and is unreliable in weak light conditions. Therefore, it is necessary to design a method to effectively fuse point cloud features and image features to improve the effect of road segmentation in unstructured scenes.

[0004] In recent years, the method of deep learning has provided a new idea for solving the problem of unstructured road segmentation, and a deep neural network is designed and the model is trained by a large amount of data to enhance the feature expression ability. However, most of the open source automatic driving data sets are structured road scenes in urban background, and almost do not involve unstructured road scenes such as damaged roads and post-disaster scenes. Therefore, it is particularly important to construct a targeted unstructured road scene data set according to the segmentation task. The use of automatic driving simulator can effectively solve the problem of scarcity of unstructured road scene data, and the simulator can simulate various scenes and set multiple sensors, which can provide strong support for the performance of unstructured road segmentation task. SUMMARY

[0005] To solve the above problems, the application discloses a kind of unstructured road segmentation method based on point cloud and image feature fusion, corresponding feature extraction network is designed for the different features of multi-modal data, and multi-modal features are fused, and the model is trained using self-built unstructured road scene dataset, to further improve the accuracy and efficiency of unstructured road semantic segmentation in practical application.

[0006] To achieve the above purpose, the application provides the following technical solutions:

[0007] The unstructured road segmentation method based on point cloud-image feature fusion includes the following steps:

[0008] (1) construct unstructured road scene dataset

[0009] The dataset for unstructured road scene is constructed, an unstructured road scene is built using an automatic driving simulator CARLA, and the collected data is processed according to the KITTI dataset format. The unstructured road scene dataset includes RGB images (1242x512), image semantic segmentation labels, point cloud data, point cloud semantic segmentation labels, and image point cloud calibration files.

[0010] (2) design 3D main network

[0011] A 3D main network based on point cloud-voxel is designed, and the input is 3D point cloud data collected by a laser radar. The network can be used for training and reasoning. The specific implementation mode includes the following sub-steps:

[0012] Substep 1: design voxel branch network, which converts point cloud into voxel for operation, and uses the traditional encoder-decoder structure of semantic segmentation. Specifically, it includes:

[0013] ①Design voxel convolution module. The module is composed of 3D convolution layer, batch normalization layer and LeakyReLU activation function layer. The 3D convolution layer uses a convolution kernel size of 2 and a step size of 2.

[0014] ②Design voxel deconvolution module. The module is composed of 3D convolution layer, batch normalization layer and LeakyReLU activation function layer. The 3D convolution layer uses a convolution kernel size of 3 and a step size of 1.

[0015] ③Design standard residual module. The module is composed of 3D convolution layer and batch normalization layer, wherein the convolution kernel size is 3 and the step size is 1.

[0016] ④According to ① and ③ in substep 1, design down-sampling module. The module is composed of a voxel convolution module in ① and two residual modules in ③.

[0017] ⑤Design the up-sampling module according to ② and ③ in sub-step 1. The module is composed of a voxel deconvolution module in ② and two residual modules in ③.

[0018] ⑥Combine the different types of network modules involved according to the modules in ①-⑤ in the above sub-step 1 to design the voxel branch network model. Through comparison of the accuracy and efficiency of the model under different network architectures, the optimal network is as follows:

[0019] Point cloud preprocessing: First, standardize the point cloud data, convert all points to a local coordinate system with the barycenter as the origin, and normalize these points to a unit sphere, and convert the point coordinates to [0, 1]. Then rasterize the normalized point cloud into a three-dimensional voxel form.

[0020] Down-sampling layer 1: According to the down-sampling module designed in ④ in step 1, first use 32 convolution kernels of size 2 to convolve the original voxel, then perform batch normalization and LeakyReLU activation. Then use 32 convolution kernels of size 3 to convolve, and perform batch normalization. Finally, after LeakyReLU activation, the original voxel is converted into a 32-dimensional feature vector.

[0021] Down-sampling layer 2: According to the down-sampling module designed in ④ in step 1, first use 64 convolution kernels of size 2 to convolve the original voxel, then perform batch normalization and LeakyReLU activation. Then use 64 convolution kernels of size 3 to convolve, and perform batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 64-dimensional feature vector.

[0022] Down-sampling layer 3: According to the down-sampling module designed in ④ in step 1, first use 128 convolution kernels of size 2 to convolve the original voxel, then perform batch normalization and LeakyReLU activation. Then use 128 convolution kernels of size 3 to convolve, and perform batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 128-dimensional feature vector.

[0023] Down-sampling layer 4: According to the down-sampling module designed in ④ in step 1, first use 256 convolution kernels of size 2 to convolve the original voxel, then perform batch normalization and LeakyReLU activation. Then use 256 convolution kernels of size 3 to convolve, and perform batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 256-dimensional feature vector.

[0024] Down-sampling layer 5: Repeat the operation of down-sampling layer 4 to obtain a more accurate 256-dimensional feature vector.

[0025] Upsampling layer 1: according to the design of the upsampling module in step 1⑤, first use 256 convolution kernels with size 3 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation. Fuse the obtained feature vector with the down-sampling layer 4, then use 128 convolution kernels with size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 128-dimensional feature vector.

[0026] Upsampling layer 2: according to the design of the upsampling module in step 1⑤, first use 128 convolution kernels with size 3 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation. Fuse the obtained feature vector with the down-sampling layer 3, then use 64 convolution kernels with size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 64-dimensional feature vector.

[0027] Upsampling layer 3: according to the design of the upsampling module in step 1⑤, first use 64 convolution kernels with size 3 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation. Fuse the obtained feature vector with the down-sampling layer 2, then use 32 convolution kernels with size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 32-dimensional feature vector.

[0028] Upsampling layer 4: according to the design of the upsampling module in step 1⑤, first use 32 convolution kernels with size 3 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation. Fuse the obtained feature vector with the down-sampling layer 1, then use 32 convolution kernels with size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, a more accurate 32-dimensional feature vector is obtained.

[0029] Substep 2: design a point cloud fusion branch network, which directly operates on point cloud. Design upsampling layer and down-sampling layer for point cloud sequence. Specifically, it includes:

[0030] ① Design a point cloud convolution module, with 3 convolution operations, all designed as fully connected structure.

[0031] ② Design a point cloud-voxel fusion module to map points and voxels in the point cloud sequence. First, construct a hash table for all active points in the voxel tensor, then traverse all points in the point cloud tensor, and use each point voxelization coordinate as a key to query the corresponding index in the voxel tensor.

[0032] ③ According to the modules in substep 2① and ② above, combine different types of network modules to design a point cloud fusion branch network model. By comparing the accuracy and efficiency of the model under different network architectures, the optimal network is obtained as follows:

[0033] Point cloud down-sampling layer 1: using the point cloud convolution module in sub-step 2, using full connection, batch normalization, and ReLU activation function, the point cloud sequence is converted into a 32-dimensional feature vector.

[0034] Point cloud down-sampling layer 2: using the point cloud convolution module in sub-step 2, using full connection, batch normalization, and ReLU activation function, the point cloud sequence is converted into a 64-dimensional feature vector.

[0035] Point cloud-voxel fusion layer 1: using the voxel-point cloud fusion module in sub-step 2, the down-sampling layer 5 in sub-step 1 is converted and superimposed with the point cloud down-sampling layer 2 to obtain a 256-dimensional feature vector.

[0036] Point cloud up-sampling layer 1: using full connection, batch normalization, and ReLU activation function to operate on the point cloud-voxel fusion layer 1, the point cloud sequence is converted into a 64-dimensional feature vector.

[0037] Point cloud-voxel fusion layer 2: using the voxel-point cloud fusion module in sub-step 2, the up-sampling layer 2 in sub-step 1 is converted and superimposed with the point cloud up-sampling layer 1 to obtain a 32-dimensional feature vector.

[0038] Point cloud up-sampling layer 2: using full connection, batch normalization, and ReLU activation function to operate on the point cloud-voxel fusion layer 2, the point cloud sequence is converted into a 64-dimensional feature vector.

[0039] Point cloud-voxel fusion layer 3: using the voxel-point cloud fusion module in sub-step 2, the up-sampling layer 4 in sub-step 1 is converted and superimposed with the point cloud up-sampling layer 2 to obtain a 32-dimensional feature vector, i.e. 3D feature, denoted as F 3D .

[0040] (3) Design of 2D auxiliary training network

[0041] The designed auxiliary semantic segmentation network is a 2D image network, and the input is the 2D image data collected by the RGB camera. This network is only used for auxiliary training, and adopts the traditional encoder-decoder structure of semantic segmentation. Image-point cloud mapping is performed using perspective projection to convert pixel-level features into point-level features. The specific implementation includes the following sub-steps:

[0042] Sub-step 1: design a 2D feature extraction network using an encoder-decoder structure to process RGB images for auxiliary training

[0043] ① Design a 2D convolution layer. Use a conventional 2D convolution with a kernel size of 3 and a stride of 1.

[0044] ②Design the pooling layer. Max pooling is adopted, and the sampling size is 2 and the step size is 2.

[0045] ③Design the dropout layer. The dropout operation is adopted, and the activation values of certain neurons are stopped working with a certain probability in the forward propagation process in the model training stage, so as to improve the generalization ability of the model.

[0046] ④Design the 2D deconvolution layer. The layer is used to restore the feature map to the original picture size, and the conventional 2D convolution is adopted, the kernel size is 3, and the step size is 2.

[0047] ⑤According to the modules of ①-⑤ in the above substep 1, different types of network layers involved are combined to design a 2D auxiliary training network. Through comparison of the accuracy and efficiency of the model under different network architectures, the optimal network is as follows:

[0048] Down-sampling layer 1: according to the design of ①-③ in the substep 1, 32 convolution kernels with a size of 3 and a step size of 1 are used to convolve the input RGB image. Batch normalization is performed on the obtained feature map, and then a pooling kernel with a size of 2 and a step size of 2 is used to perform a max pooling operation to obtain a 32-dimensional feature map.

[0049] Down-sampling layer 2: according to the design of ①-③ in the substep 1, 64 convolution kernels with a size of 3 and a step size of 1 are used to convolve the 32-dimensional feature map in the down-sampling layer 1. Batch normalization is performed on the obtained feature map, and then a dropout operation is performed, and then a pooling kernel with a size of 2 and a step size of 2 is used to perform a max pooling operation to obtain a 64-dimensional feature map.

[0050] Down-sampling layer 3: according to the design of ①-③ in the substep 1, 128 convolution kernels with a size of 3 and a step size of 1 are used to convolve the 64-dimensional feature map in the down-sampling layer 2. Batch normalization is performed on the obtained feature map, and then a dropout operation is performed, and then a pooling kernel with a size of 2 and a step size of 2 is used to perform a max pooling operation to obtain a 128-dimensional feature map.

[0051] Down-sampling layer 4: according to the design of ①-③ in the substep 1, 256 convolution kernels with a size of 3 and a step size of 1 are used to convolve the 128-dimensional feature map in the down-sampling layer 3. Batch normalization is performed on the obtained feature map, and then a dropout operation is performed, and then a pooling kernel with a size of 2 and a step size of 2 is used to perform a max pooling operation to obtain a 256-dimensional feature map.

[0052] Upsampling layer 1: according to the design of ②-④ in substep 1, use 128 convolution kernels with a size of 3 and a step of 2 to do deconvolution on the 256-dimensional feature map in down-sampling layer 4. Perform batch normalization processing on the obtained feature map, then perform dropout operation, and add the obtained feature map and the feature map output in down-sampling layer 3 pixel by pixel to obtain a 128-dimensional feature map.

[0053] Upsampling layer 2: according to the design of ②-④ in substep 1, use 64 convolution kernels with a size of 3 and a step of 2 to do deconvolution on the 128-dimensional feature map in upsampling layer 1. Perform batch normalization processing on the obtained feature map, then perform dropout operation, and add the obtained feature map and the feature map output in down-sampling layer 2 pixel by pixel to obtain a 64-dimensional feature map.

[0054] Upsampling layer 3: according to the design of ②-④ in substep 1, use 32 convolution kernels with a size of 3 and a step of 2 to do deconvolution on the 64-dimensional feature map in upsampling layer 2. Perform batch normalization processing on the obtained feature map, then perform dropout operation, and add the obtained feature map and the feature map output in down-sampling layer 1 pixel by pixel to obtain a 32-dimensional feature map.

[0055] Substep 2: design a pixel-to-point mapping module. The present application uses perspective projection to calculate the point-to-pixel mapping between the point cloud and the image, converts the pixel-level features of the RGB image into point-level features, and obtains point-level 2D features, denoted as F 2D , which facilitates subsequent feature fusion.

[0056] Let p i =(x i ,y i ,z i ) represent the coordinates of the 3D point cloud, and p' i =(u i ,v i ) represent the coordinates of the point cloud projected onto the 2D image. Let K and T represent the intrinsic matrix and extrinsic matrix of the camera, respectively. The conversion relationship of the 3D point cloud projected onto the 2D image is as follows:

[0057]

[0058] (4) Design a 2D feature to 3D feature knowledge distillation network

[0059] Using knowledge distillation, on the basis of preserving the original 3D feature knowledge, fully utilizing the texture and color features of the 2D features as prior information to assist training, and using multi-modal feature to enhance the 3D backbone network. The specific implementation mode includes the following substeps:

[0060] Sub-step 1: Design feature approximation module. For convenience of expression, the 3D features extracted in (2) and (3) are denoted as F 3D , and the 2D features are denoted as F 2D . F 3D is input into a multi-layer perception (MLP) to obtain features similar to F 2D , denoted as F 3D’ , thereby reducing the difference between the 3D features and the 2D features.

[0061] Sub-step 2: Design knowledge distillation network. The features F 3D’ obtained in sub-step 1 are divided into two paths, one of which is concatenated with the 2D features to implement multi-modal feature fusion, obtaining 3D2D fusion features, denoted as F 3D2D ; the other is connected back to the 3D branch. The two branches are processed respectively, and the specific operations are as follows:

[0062] ① F 3D2D is input into a multi-layer perception, and the obtained result is input into another multi-layer perception and activated by a Sigmoid function. The obtained weight is multiplied by the output result of the previous multi-layer perception, and then element-wise added with the pure 2D feature F 2D to obtain enhanced 3D2D fusion features, denoted as F 3D2De . The obtained result is input into a fully connected layer to output fusion segmentation results, and the loss function of the segmentation results will affect the features F 3D’ through back propagation.

[0063] ② F 3D’ is element-wise added with the pure 3D feature F 3D to obtain enhanced 3D features, denoted as F 3De . The obtained result is input into a fully connected layer to output 3D segmentation results.

[0064] ③ The KL divergence between the fusion segmentation results in ① and ② and the 3D segmentation results in ② is calculated to make the 3D segmentation results approach the fusion segmentation results, thereby strengthening the 3D backbone network.

[0065] (5) Network model training

[0066] According to the designed network, the initial learning rate is set to 0.01, the dropout rate is set to 0.5, the batch size is set to 2, and the epoch is set to 64. The self-built unstructured road scene dataset and the KITTI dataset are used for mixed training. First, the KITTI dataset is used to pre-train the model to obtain pre-training weights, and then the unstructured road scene dataset is used for further training to obtain the final network weights.

[0067] The present application has the following advantages and beneficial effects:

[0068] 1、 The present application aims at the problem of less unstructured environment public data set, uses an automatic driving simulation simulator CARLA to build an unstructured road scene, and processes the collected data according to the KITTI data set format to provide support for model training.

[0069] 2、 The present application considers the characteristics of unstructured road scenes and the needs of practical applications, and uses a combination of point cloud sequences and voxels at the 3D data coding level to design the network, ensuring that the network has high precision under the premise of real-time.

[0070] 3、 The present application fully utilizes the structure information of 3D point cloud data and the texture and color information of 2D image data, and fuses multi-modal features. During training, a 2D network is used to assist a 3D backbone network, and during inference, only the 3D backbone network is used, which effectively improves the semantic segmentation effect and also ensures the efficiency of model inference. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is a method flowchart of the present application.

[0072] Figure 2 is a 3D backbone network and 2D auxiliary training network design flowchart of the present application.

[0073] Figure 3 is a 2D feature to 3D feature knowledge distillation network of the present application. DETAILED DESCRIPTION

[0074] The ultimate goal of the development of autonomous driving technology is to not need human attention in any environment, and effective segmentation of drivable roads is one of the key tasks to achieve this goal. Existing researches are mostly concentrated on structured road scenes with clear lane lines, regular road surfaces and clear markers, but for unstructured road scenes, there are problems such as blurred road boundaries and irregular road shapes, which bring great challenges to road segmentation.

[0075] Currently, LiDAR and RGB camera are two commonly used different modal sensors in autonomous driving data collection to obtain different modal data. Due to the large difference between different modal data, most existing methods use single modal data collected by a single sensor as input. However, the limited features of single modal data are difficult to support further improvement of road segmentation accuracy, so it is necessary to fuse multi-modal features. The 3D point cloud data collected by LiDAR contains rich structure information and is not affected by weather and light, but can only capture sparse and textureless data; the 2D image data collected by RGB camera contains rich texture and color information, but lacks three-dimensional features and is unreliable in weak light conditions. Therefore, it is necessary to design an effective method to fuse point cloud features and image features to improve the effect of road segmentation in unstructured scenes.

[0076] In recent years, the method of deep learning has provided a new idea for solving the problem of unstructured road segmentation. A deep neural network is designed and a large amount of data is used to train the model to enhance the feature expression ability. However, most of the open source autonomous driving data sets are structured road scenes in urban background, and almost no unstructured road scenes such as damaged roads and post-disaster scenes are involved. Therefore, it is particularly important to construct a targeted unstructured road scene data set according to the segmentation task. The use of autonomous driving simulator can effectively solve the problem of scarcity of unstructured road scene data. The simulator can simulate various scenes and set up various sensors, which can provide strong support for the unstructured road segmentation task.

[0077] To solve the above problems, the present application discloses a non-structured road segmentation method based on point cloud-image feature fusion. This method uses multi-modal data, designs a 3D backbone network for 3D point cloud features, which is used for model training and inference; a 2D auxiliary training network is designed for 2D image features, which is only used for auxiliary model training. In addition, a 2D feature to 3D feature knowledge distillation network is also designed, which fully utilizes the texture and color features of 2D features as prior information to assist training and enhances the 3D backbone network with multi-modal features. The network is trained using open source data sets and self-built unstructured road data sets to obtain network parameters, thereby realizing more efficient and accurate unstructured road segmentation.

[0078] The non-structured road segmentation method based on point cloud-image feature fusion provided by the present application specifically includes the following steps:

[0079] (1) Constructing an unstructured road scene data set

[0080] The unstructured road refers to a road lacking or without artificial markers, and since there are few existing open-source data sets containing such scenes, an unstructured road scene is built by using an automatic driving simulator CARLA, point cloud data is collected by using a laser radar in the simulator, visible light image data is collected by using a camera in the simulator, and the collected data is processed according to the KITTI data set format.

[0081] The application constructs a data set for an unstructured road scene, including RGB images (1242*512), semantic segmentation labels of images, point cloud data, semantic segmentation labels of point clouds, and image point cloud calibration files.

[0082] (2) Designing a 3D main network

[0083] The main semantic segmentation network designed in the application is a point cloud-voxel network, and the input is 3D point cloud data collected by a laser radar, and the network can be used for training and reasoning.

[0084] The point cloud-voxel network is composed of a point cloud branch and a voxel branch. For the point cloud branch, the point cloud is directly processed, which can efficiently capture high-resolution information and extract fine-grained features of the scene. The branch does not design convolution operation, which can effectively reduce the cost of random access memory, but also loses the ability of proximity modeling. For the voxel branch, the original point cloud data is first converted into a voxel grid with low resolution, and then the grid is convolved. In view of the sparsity of point cloud in the unstructured road scene, the branch can capture the proximity information between point clouds while ensuring the processing speed, and is good for extracting coarse-grained features of the scene. The point cloud-voxel network designed in the application for the unstructured road scene complements the fusion of the point cloud-based method and the voxel-based method, and takes into account the accuracy and efficiency of the algorithm. The specific implementation mode of the 3D main network of the application includes the following sub-steps:

[0085] Sub-step 1: Designing a voxel branch network, the branch converts point cloud into voxel for operation, adopts a traditional encoder-decoder structure of semantic segmentation, and designs four down-sampling encoding layers and four up-sampling decoding layers. At the same time, the up-sampling layer and the down-sampling layer are connected across layers, so that the features recovered by the up-sampling network are fused with more original features, and different scale features are fused to achieve better segmentation effect. Specifically, it includes:

[0086] ① Design a voxel convolution module. The module is composed of a 3D convolution layer, a batch normalization layer, and a LeakyReLU activation function layer. The 3D convolution layer is used for feature extraction, with a convolution kernel size of 2 and a step size of 2, allowing the network to extract both fine-grained features in the scene and consider the processing speed of convolution operations. The batch normalization layer is used to solve the problem of difficult gradient descent during training, making it easier and more stable to train deep network models and improving network generalization. The LeakyReLU activation function layer is a variant of the ReLu activation function, used to enhance model classification capabilities.

[0087] ② Design a voxel deconvolution module. The module is composed of a 3D convolution layer, a batch normalization layer, and a LeakyReLU activation function layer. When performing 3D convolution, considering the sparsity of unstructured scene features, a convolution kernel size of 3 and a step size of 1 are used, which to some extent increases the receptive field while also reducing the complexity of the model as much as possible. The batch normalization layer and the LeakyReLU activation function layer have similar functions as in sub-step 1.

[0088] ③ Design a standard residual module. The module is composed of a 3D convolution layer and a batch normalization layer, with a convolution kernel size of 3 and a step size of 1. The residual module is connected after the voxel convolution module and the voxel deconvolution module, improving network training efficiency.

[0089] ④ Design a down-sampling module according to sub-step 1 in ① and ③. The module is composed of a voxel convolution module in ① and two residual modules in ③.

[0090] ⑤ Design an up-sampling module according to sub-step 1 in ② and ③. The module is composed of a voxel deconvolution module in ② and two residual modules in ③.

[0091] ⑥ Combine different types of network modules involved in the above sub-step 1 in ①-⑤ to design a voxel branch network model. By comparing the accuracy and efficiency of the model under different network architectures, the optimal network is obtained as follows:

[0092] Point cloud preprocessing: First, standardize the point cloud data and convert all points to a local coordinate system with the center of gravity as the origin. Then normalize these points to a unit sphere and convert the point coordinates to [0, 1]. Then rasterize the normalized point cloud and convert it to a three-dimensional voxel form.

[0093] Down-sampling layer 1: According to the down-sampling module designed in step 1 in ④, first use 32 convolution kernels with a size of 2 to convolve the original voxel, then perform batch normalization and LeakyReLU activation. Then use 32 convolution kernels with a size of 3 to convolve and perform batch normalization. Finally, after LeakyReLU activation, the original voxel is converted into a 32-dimensional feature vector.

[0094] Down-sampling layer 2: According to the design of the down-sampling module in step 1 (IV), first use 64 convolution kernels of size 2 to do convolution on the original voxel, and then do batch normalization and LeakyReLU activation. Then use 64 convolution kernels of size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 64-dimensional feature vector.

[0095] Down-sampling layer 3: According to the design of the down-sampling module in step 1 (IV), first use 128 convolution kernels of size 2 to do convolution on the original voxel, and then do batch normalization and LeakyReLU activation. Then use 128 convolution kernels of size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 128-dimensional feature vector.

[0096] Down-sampling layer 4: According to the design of the down-sampling module in step 1 (IV), first use 256 convolution kernels of size 2 to do convolution on the original voxel, and then do batch normalization and LeakyReLU activation. Then use 256 convolution kernels of size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 256-dimensional feature vector.

[0097] Down-sampling layer 5: Repeat the operation of down-sampling layer 4 to get a more accurate 256-dimensional feature vector.

[0098] Up-sampling layer 1: According to the design of the up-sampling module in step 1 (V), first use 256 convolution kernels of size 3 to do convolution on the original voxel, and then do batch normalization and LeakyReLU activation. Fuse the obtained feature vector with the down-sampling layer 4, then use 128 convolution kernels of size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 128-dimensional feature vector.

[0099] Up-sampling layer 2: According to the design of the up-sampling module in step 1 (V), first use 128 convolution kernels of size 3 to do convolution on the original voxel, and then do batch normalization and LeakyReLU activation. Fuse the obtained feature vector with the down-sampling layer 3, then use 64 convolution kernels of size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 64-dimensional feature vector.

[0100] Up-sampling layer 3: According to the design of the up-sampling module in step 1 (V), first use 64 convolution kernels of size 3 to do convolution on the original voxel, and then do batch normalization and LeakyReLU activation. Fuse the obtained feature vector with the down-sampling layer 2, then use 32 convolution kernels of size 3 to do convolution, and do batch normalization. Finally, after LeakyReLU activation, the voxel vector is converted into a 32-dimensional feature vector.

[0101] Upsampling layer 4: According to the upsampling module designed in step 1, first use 32 convolution kernels with a size of 3 to convolve the original voxel, then perform batch normalization and LeakyReLU activation. Fuse the resulting feature vector with the down-sampling layer 1, then use 32 convolution kernels with a size of 3 to convolve, and perform batch normalization. Finally, after LeakyReLU activation, a more accurate 32-dimensional feature vector is obtained.

[0102] Sub-step 2: Design a point cloud fusion branch network, which directly operates on point cloud. Design an upsampling layer and a down-sampling layer for point cloud sequence. Fuse the features once in the down-sampling stage and twice in the up-sampling stage to ensure that the network can recover high-precision features during up-sampling to achieve better segmentation results. Specifically, it includes:

[0103] ① Design a point cloud convolution module, with a convolution number of 3, all designed as a fully connected structure. Since point cloud is a one-dimensional sequence, in order to model the fine-grained single point feature, this paper uses a common and effective method, which directly uses a fully connected layer to extract the feature of a single point. This high-resolution single point information is crucial for supplementing the information based on coarse-grained voxels.

[0104] ② Design a point cloud-voxel fusion module. This module is used to map the points of the point cloud sequence and the voxels. Since the time complexity of ordinary point cloud-voxel conversion methods is large, it cannot meet the real-time application, therefore, a hash table is used to speed up the point cloud-voxel encoding and decoding. First, a hash table is constructed for all active points in the voxel tensor, which can be completed in O(n) time. Then, traverse all points in the point cloud tensor, use the voxelization coordinates of each point as the key to query the corresponding index in the voxel tensor, which takes a total of O(m) time. Therefore, the total feature conversion time is O(m+n). This method only needs to establish a hash table during point cloud voxelization, and then do feature conversion, which can greatly improve the efficiency.

[0105] ③ According to the modules in sub-step 2① and ②, combine different types of network modules to design a point cloud fusion branch network model. By comparing the accuracy and efficiency of the model under different network architectures, the optimal network is as follows:

[0106] Point cloud down-sampling layer 1: Use the point cloud convolution module in sub-step 2① to convert the point cloud sequence into a 32-dimensional feature vector using full connection, batch normalization and ReLU activation function.

[0107] Point cloud down-sampling layer 2: Use the point cloud convolution module in sub-step 2① to convert the point cloud sequence into a 64-dimensional feature vector using full connection, batch normalization and ReLU activation function.

[0108] Point cloud-voxel fusion layer 1: the voxel-point cloud fusion module in substep 2 is used to transform the downsampling layer 5 in substep 1 and superimpose it with the point cloud downsampling layer 2 to obtain a 256-dimensional feature vector.

[0109] Point cloud upsampling layer 1: the point cloud-voxel fusion layer 1 is operated by using full connection, batch normalization and ReLU activation function to convert the point cloud sequence into a 64-dimensional feature vector.

[0110] Point cloud-voxel fusion layer 2: the voxel-point cloud fusion module in substep 2 is used to transform the upsampling layer 2 in substep 1 and superimpose it with the point cloud upsampling layer 1 to obtain a 32-dimensional feature vector.

[0111] Point cloud upsampling layer 2: the point cloud-voxel fusion layer 2 is operated by using full connection, batch normalization and ReLU activation function to convert the point cloud sequence into a 64-dimensional feature vector.

[0112] Point cloud-voxel fusion layer 3: the voxel-point cloud fusion module in substep 2 is used to transform the upsampling layer 4 in substep 1 and superimpose it with the point cloud upsampling layer 2 to obtain a 32-dimensional feature vector, i.e. 3D feature, denoted as F 3D .

[0113] (3) Design a 2D auxiliary training network

[0114] The auxiliary semantic segmentation network designed in the application is a 2D image network, and the input is 2D image data collected by an RGB camera. The network is only used for auxiliary training. The network adopts the traditional encoder-decoder structure of semantic segmentation, uses a residual network with two-dimensional convolution as an encoder, uses a fully convolutional neural network as a decoder, and then performs semantic segmentation on the feature map obtained by a linear classifier. At the same time, image-point cloud mapping is performed by using perspective projection to convert pixel-level features into point-level features. The specific implementation mode of the auxiliary semantic segmentation network designed in the application includes the following substeps:

[0115] Substep 1: design a 2D feature extraction network, adopt an encoder-decoder structure, process an RGB image, and use it for auxiliary training

[0116] ① Design a 2D convolution layer. The layer is used for image feature extraction, adopts a conventional 2D convolution, and the convolution kernel size is 3 and the step size is 1.

[0117] ② Design a pooling layer. The layer is used to increase the receptive field of the network to realize feature extraction of larger targets. The application adopts maximum pooling as the pooling layer of the network in view of the unstructured scene characteristics, and the sampling size is 2 and the step size is 2.

[0118] ③Design dropout layer. The present application is aimed at the sparse characteristics of the target in the unstructured scene, and dropout operation is adopted to stop the activation value of certain neurons from working in the forward propagation process in the model training stage, so as to not only reduce the model parameters to a certain extent, but also prevent the model from overfitting and improve the generalization ability of the model.

[0119] ④Design 2D deconvolution layer. The layer is used to restore the feature map to the original picture size, and adopts the conventional 2D convolution with a convolution kernel size of 3 and a step size of 2.

[0120] ⑤According to the modules of ①-⑤ in the above sub-step 1, different types of network layers involved are combined to design a 2D auxiliary training network. Through comparison of the accuracy and efficiency of the model under different network architectures, the optimal network is as follows:

[0121] Down-sampling layer 1: according to the design of ①-③ in the sub-step 1, 32 convolution kernels with a size of 3 and a step size of 1 are used to convolve the input RGB image. Batch normalization is performed on the obtained feature map, and then a pooling kernel with a size of 2 and a step size of 2 is used to perform maximum pooling operation to obtain a 32-dimensional feature map.

[0122] Down-sampling layer 2: according to the design of ①-③ in the sub-step 1, 64 convolution kernels with a size of 3 and a step size of 1 are used to convolve the 32-dimensional feature map in the down-sampling layer 1. Batch normalization is performed on the obtained feature map, then dropout operation is performed, and then a pooling kernel with a size of 2 and a step size of 2 is used to perform maximum pooling operation to obtain a 64-dimensional feature map.

[0123] Down-sampling layer 3: according to the design of ①-③ in the sub-step 1, 128 convolution kernels with a size of 3 and a step size of 1 are used to convolve the 64-dimensional feature map in the down-sampling layer 2. Batch normalization is performed on the obtained feature map, then dropout operation is performed, and then a pooling kernel with a size of 2 and a step size of 2 is used to perform maximum pooling operation to obtain a 128-dimensional feature map.

[0124] Down-sampling layer 4: according to the design of ①-③ in the sub-step 1, 256 convolution kernels with a size of 3 and a step size of 1 are used to convolve the 128-dimensional feature map in the down-sampling layer 3. Batch normalization is performed on the obtained feature map, then dropout operation is performed, and then a pooling kernel with a size of 2 and a step size of 2 is used to perform maximum pooling operation to obtain a 256-dimensional feature map.

[0125] Upsampling layer 1: according to the design of ②-④ in substep 1, use 128 convolution kernels with a size of 3 and a step of 2 to do deconvolution on the 256-dimensional feature map in the downsampling layer 4. Perform batch normalization processing on the obtained feature map, then perform dropout operation, and add the obtained feature map and the feature map output in the downsampling layer 3 pixel by pixel to obtain a 128-dimensional feature map.

[0126] Upsampling layer 2: according to the design of ②-④ in substep 1, use 64 convolution kernels with a size of 3 and a step of 2 to do deconvolution on the 128-dimensional feature map in the upsampling layer 1. Perform batch normalization processing on the obtained feature map, then perform dropout operation, and add the obtained feature map and the feature map output in the downsampling layer 2 pixel by pixel to obtain a 64-dimensional feature map.

[0127] Upsampling layer 3: according to the design of ②-④ in substep 1, use 32 convolution kernels with a size of 3 and a step of 2 to do deconvolution on the 64-dimensional feature map in the upsampling layer 2. Perform batch normalization processing on the obtained feature map, then perform dropout operation, and add the obtained feature map and the feature map output in the downsampling layer 1 pixel by pixel to obtain a 32-dimensional feature map.

[0128] Substep 2: design a pixel-to-point mapping module. The present application uses perspective projection to calculate the point-to-pixel mapping between the point cloud and the image, converts the pixel-level features of the RGB image into point-level features, obtains point-level 2D features, and is denoted as F 2D , which facilitates subsequent feature fusion.

[0129] Let p i =(x i ,y i ,z i ) represent the coordinates of the 3D point cloud, p' i =(u i ,v i ) represent the coordinates of the point cloud projected onto the 2D image, K and T represent the intrinsic matrix and extrinsic matrix of the camera respectively, and the conversion relationship of the 3D point cloud projected onto the 2D image is as follows:

[0130]

[0131] (4) Design a 2D feature to 3D feature knowledge distillation network

[0132] The present invention utilizes knowledge distillation, and on the basis of retaining the original 3D feature knowledge, fully utilizes the texture and color features of 2D features as prior information to assist training, and utilizes multimodal features to enhance the 3D backbone network. During the training process, the designed 3D network branch and 2D network branch are used to generate semantic segmentation predictions respectively, and are supervised by pure 3D labels. During the reasoning process, the 2D auxiliary training network branch is not involved, and only the 3D network is used for reasoning. Compared with the traditional fusion method of simply splicing features, the fusion network based on knowledge distillation designed by the present invention only needs to input 3D data during reasoning, and does not require additional processing of 2D data, which can effectively reduce the computational burden of the network in practical applications. The specific implementation method of the knowledge distillation network from 2D features to 3D features designed by the present invention includes the following sub-steps:

[0133] Sub-step 1: Design a feature approximation module. Since the backbone network designs of 3D networks and 2D networks are different, the 3D features and 2D features extracted in (2) and (3) cannot be directly fused. The present invention designs a feature approximation module to prevent the 3D features from being contaminated by the fused features during the distillation process, thereby better achieving distillation. Figure 2 As shown, for the convenience of expression, the 3D features extracted in (2) and (3) are expressed as F 3D , 2D features are represented as F 2D 。 3D Input a multi-layer perceptron (MLP) and get the same as F 2D Similar features, denoted as F 3D’ , thereby reducing the difference between 3D features and 2D features.

[0134] Sub-step 2: Design a knowledge distillation network. 3D’ It is divided into two paths. One path is spliced ​​with 2D features to achieve multimodal feature fusion, and the 3D2D fusion feature is obtained, which is expressed as F 3D2D ; The other branch is connected back to the 3D branch. The two branches are processed separately. The specific operations are as follows:

[0135] ①F 3D2D Input into a multi-layer perceptron, input the result into another multi-layer perceptron, and activate it with the Sigmoid function, multiply the weight obtained by the output of the previous multi-layer perceptron, and then multiply it with the pure 2D feature F 2D Perform element-wise addition to obtain enhanced 3D2D fusion features, denoted as F 3D2De The obtained result is input into the fully connected layer and the fused segmentation result is output. The loss function of the segmentation result will affect the feature F through back propagation. 3D’ .

[0136] ②F 3D’with pure 3D features F 3D Element-level addition is performed to obtain enhanced 3D features, denoted as F 3De The obtained result is input into a full connection layer to output a 3D segmentation result.

[0137] ③Calculate the KL divergence between the fusion segmentation result in ① and ② in sub-step 2 and the 3D segmentation result, so that the 3D segmentation result approximates to the fusion segmentation result, thereby strengthening the 3D backbone network.

[0138] (5) Network model training

[0139] According to the network designed in the application, the initial learning rate is set to 0.01, the dropout rate is set to 0.5, the batchsize is set to 2, the epoch is set to 64, the unstructured road scene dataset constructed by the application is mixed with the KITTI dataset for training, the model is pre-trained by using the KITTI dataset to obtain pre-training weights, and the unstructured road scene dataset constructed by the application is used for further training to obtain final network weights.

Claims

1. An unstructured road segmentation method based on point cloud and image feature fusion, characterized in that, Specifically comprising the following steps: (1) Constructing an unstructured road scene dataset A dataset for an unstructured road scene is constructed, an unstructured road scene is built using an autonomous driving simulator CARLA, and the collected data is processed according to the KITTI dataset format; the unstructured road scene dataset includes RGB images, image semantic segmentation labels, point cloud data, point cloud semantic segmentation labels, and image point cloud calibration files; (2) Designing a 3D backbone network A 3D backbone network based on point cloud-voxel is designed, and the input is 3D point cloud data collected by a laser radar; the network can be used for training and inference; the specific implementation mode includes the following sub-steps: Sub-step 1: Design a voxel branch network, which converts point cloud into voxel for operation, and adopts an encoder-decoder structure; specifically comprising: ① Design a voxel convolution module; the module is composed of a 3D convolution layer, a batch normalization layer, and a LeakyReLU activation function layer, wherein the 3D convolution layer uses a convolution kernel size of 2 and a step size of 2; ② Design a voxel deconvolution module; the module is composed of a 3D convolution layer, a batch normalization layer, and a LeakyReLU activation function layer, wherein the 3D convolution layer uses a convolution kernel size of 3 and a step size of 1; ③ Design a standard residual module; the module is composed of a 3D convolution layer and a batch normalization layer, wherein the convolution kernel size is 3 and the step size is 1; ④ Design a down-sampling module according to ① and ③ in sub-step 1; the module is composed of a voxel convolution module in ① and two residual modules in ③; ⑤ Design an up-sampling module according to ② and ③ in sub-step 1; the module is composed of a voxel deconvolution module in ② and two residual modules in ③; ⑥ According to the modules in ①-⑤ in the above sub-step 1, combine different types of network modules to design a voxel branch network model; by comparing the accuracy and efficiency of the model under different network architectures, the optimal network is as follows: Point cloud preprocessing: first, standardize the point cloud data, convert all points to a local coordinate system with the center of gravity as the origin, and normalize these points to a unit sphere, and convert the point coordinates to [0, 1]; then rasterize the normalized point cloud into a three-dimensional voxel form; Down-sampling layer 1: according to the down-sampling module designed in step 1, first use 32 convolution kernels with a size of 2 to convolve the original voxel, then perform batch normalization and LeakyReLU activation; then use 32 convolution kernels with a size of 3 to convolve, and perform batch normalization; finally, after LeakyReLU activation, the original voxel is converted into a 32-dimensional feature vector; Down-sampling layer 2: according to the down-sampling module designed in step 1, first use 64 convolution kernels with a size of 2 to convolve the original voxel, then perform batch normalization and LeakyReLU activation; then use 64 convolution kernels with a size of 3 to convolve, and perform batch normalization; finally, after LeakyReLU activation, the voxel vector is converted into a 64-dimensional feature vector; Down-sampling layer 3: according to the design of the down-sampling module in step 1 (IV), first use 128 convolution kernels with a size of 2 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation; then use 128 convolution kernels with a size of 3 to do convolution, and do batch normalization; finally, after LeakyReLU activation, the voxel vector is converted into a 128-dimensional feature vector; Down-sampling layer 4: according to the design of the down-sampling module in step 1 (IV), first use 256 convolution kernels with a size of 2 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation; then use 256 convolution kernels with a size of 3 to do convolution, and do batch normalization; finally, after LeakyReLU activation, the voxel vector is converted into a 256-dimensional feature vector; Down-sampling layer 5: repeat the operation of down-sampling layer 4 to get a more accurate 256-dimensional feature vector; Up-sampling layer 1: according to the design of the up-sampling module in step 1 (V), first use 256 convolution kernels with a size of 3 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation; fuse the obtained feature vector with the down-sampling layer 4, then use 128 convolution kernels with a size of 3 to do convolution, and do batch normalization; finally, after LeakyReLU activation, the voxel vector is converted into a 128-dimensional feature vector; Up-sampling layer 2: according to the design of the up-sampling module in step 1 (V), first use 128 convolution kernels with a size of 3 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation; fuse the obtained feature vector with the down-sampling layer 3, then use 64 convolution kernels with a size of 3 to do convolution, and do batch normalization; finally, after LeakyReLU activation, the voxel vector is converted into a 64-dimensional feature vector; Up-sampling layer 3: according to the design of the up-sampling module in step 1 (V), first use 64 convolution kernels with a size of 3 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation; fuse the obtained feature vector with the down-sampling layer 2, then use 32 convolution kernels with a size of 3 to do convolution, and do batch normalization; finally, after LeakyReLU activation, the voxel vector is converted into a 32-dimensional feature vector; Up-sampling layer 4: according to the design of the up-sampling module in step 1 (V), first use 32 convolution kernels with a size of 3 to do convolution on the original voxel, then do batch normalization and LeakyReLU activation; fuse the obtained feature vector with the down-sampling layer 1, then use 32 convolution kernels with a size of 3 to do convolution, and do batch normalization; finally, after LeakyReLU activation, a more accurate 32-dimensional feature vector is obtained; Sub-step 2: design a point cloud fusion branch network, which directly operates on the point cloud, and design up-sampling layers and down-sampling layers for the point cloud sequence; specifically including: ① Design a point cloud convolution module, with a convolution number of 3, all designed as a fully connected structure; ②Design a point cloud-voxel fusion module for mapping points of the point cloud sequence and voxels; first, construct a hash table for all active points in the voxel tensor, and then traverse all points in the point cloud tensor, use the voxel coordinates of each point as the key to query the corresponding index in the voxel tensor; ③According to the modules in ① and ② in the above substep 2, combine different types of network modules involved, and design a point cloud fusion branch network model; by comparing the accuracy and efficiency of the model under different network architectures, the optimal network is as follows: Point cloud downsampling layer 1: use the point cloud convolution module in ① of substep 2, use full connection, batch normalization and ReLU activation function to convert the point cloud sequence into a 32-dimensional feature vector; Point cloud downsampling layer 2: use the point cloud convolution module in ① of substep 2, use full connection, batch normalization and ReLU activation function to convert the point cloud sequence into a 64-dimensional feature vector; Point cloud-voxel fusion layer 1: use the voxel-point cloud fusion module in ② of substep 2 to convert the downsampling layer 5 in substep 1, and stack it with the point cloud downsampling layer 2 to obtain a 256-dimensional feature vector; Point cloud upsampling layer 1: use full connection, batch normalization and ReLU activation function to operate on the point cloud-voxel fusion layer 1, and convert the point cloud sequence into a 64-dimensional feature vector; Point cloud-voxel fusion layer 2: use the voxel-point cloud fusion module in ② of substep 2 to convert the upsampling layer 2 in substep 1, and stack it with the point cloud upsampling layer 1 to obtain a 32-dimensional feature vector; Point cloud upsampling layer 2: use full connection, batch normalization and ReLU activation function to operate on the point cloud-voxel fusion layer 2, and convert the point cloud sequence into a 64-dimensional feature vector; Point cloud - voxel fusion layer 3: using the voxel - point cloud fusion module in substep 2②, the up-sampling layer 4 in substep 1 is converted and superimposed with the point cloud up-sampling layer 2 to obtain a 32-dimensional feature vector, i.e. a 3D feature, denoted as F 3D ; (3) Design a 2D auxiliary training network The designed auxiliary semantic segmentation network is a 2D image network, which inputs 2D image data collected by an RGB camera. The network adopts an encoder-decoder structure and is only used for auxiliary training. Perspective projection is used for image-point cloud mapping to convert pixel-level features into point-level features. The specific implementation includes the following substeps: Substep 1: Design a 2D feature extraction network using an encoder-decoder structure to process RGB images for auxiliary training; ①Design a 2D convolution layer; use a conventional 2D convolution with a kernel size of 3 and a step size of 1; ②Design a pooling layer; use maximum pooling with a sampling size of 2 and a step size of 2; ③Design a dropout layer; use dropout operation to stop the activation of certain neurons with a certain probability during the forward propagation process in the model training stage, to improve the generalization ability of the model; ④Design a 2D deconvolution layer; this layer is used to restore the feature map to the original image size, using a conventional 2D convolution with a kernel size of 3 and a step size of 2; ⑤According to the modules in ①-⑤ of the above substep 1, combine different types of network layers involved, and design a 2D auxiliary training network; by comparing the accuracy and efficiency of the model under different network architectures, the optimal network is as follows: Down-sampling layer 1: according to the design in 1-1-1 to 1-1-3, 32 convolution kernels with a size of 3 and a step of 1 are used to convolve the input RGB image; the obtained feature map is batch normalized, and then a pooling kernel with a size of 2 and a step of 2 is used to perform a maximum pooling operation to obtain a 32-dimensional feature map; Down-sampling layer 2: according to the design in 1-1-1 to 1-1-3, 64 convolution kernels with a size of 3 and a step of 1 are used to convolve the 32-dimensional feature map in the down-sampling layer 1; the obtained feature map is batch normalized, then a dropout operation is performed, and then a pooling kernel with a size of 2 and a step of 2 is used to perform a maximum pooling operation to obtain a 64-dimensional feature map; Down-sampling layer 3: according to the design in 1-1-1 to 1-1-3, 128 convolution kernels with a size of 3 and a step of 1 are used to convolve the 64-dimensional feature map in the down-sampling layer 2; the obtained feature map is batch normalized, then a dropout operation is performed, and then a pooling kernel with a size of 2 and a step of 2 is used to perform a maximum pooling operation to obtain a 128-dimensional feature map; Down-sampling layer 4: according to the design in 1-1-1 to 1-1-3, 256 convolution kernels with a size of 3 and a step of 1 are used to convolve the 128-dimensional feature map in the down-sampling layer 3; the obtained feature map is batch normalized, then a dropout operation is performed, and then a pooling kernel with a size of 2 and a step of 2 is used to perform a maximum pooling operation to obtain a 256-dimensional feature map; Up-sampling layer 1: according to the design in 1-1-2 to 1-1-4, 128 convolution kernels with a size of 3 and a step of 2 are used to deconvolve the 256-dimensional feature map in the down-sampling layer 4; the obtained feature map is batch normalized, then a dropout operation is performed, and then the obtained feature map is added to the feature map output from the down-sampling layer 3 pixel by pixel to obtain a 128-dimensional feature map; Up-sampling layer 2: according to the design in 1-1-2 to 1-1-4, 64 convolution kernels with a size of 3 and a step of 2 are used to deconvolve the 128-dimensional feature map in the up-sampling layer 1; the obtained feature map is batch normalized, then a dropout operation is performed, and then the obtained feature map is added to the feature map output from the down-sampling layer 2 pixel by pixel to obtain a 64-dimensional feature map; Up-sampling layer 3: according to the design in 1-1-2 to 1-1-4, 32 convolution kernels with a size of 3 and a step of 2 are used to deconvolve the 64-dimensional feature map in the up-sampling layer 2; the obtained feature map is batch normalized, then a dropout operation is performed, and then the obtained feature map is added to the feature map output from the down-sampling layer 1 pixel by pixel to obtain a 32-dimensional feature map; Sub-step 2: design a pixel-to-point mapping module; adopt perspective projection to calculate the point-to-pixel mapping between the point cloud and the image, convert the pixel-level features of the RGB image into point-level features, obtain the point-level 2D features, denoted as F 2D , for the subsequent feature fusion; Use p i =(x i ,y i ,z i ) represents the coordinates of the 3D point cloud, and p' i =(u i ,v i ) represents the coordinates of the point cloud projected onto the 2D image, and K and T represent the intrinsic parameter matrix and extrinsic parameter matrix of the camera respectively. The conversion relationship from the 3D point cloud projection to the 2D image is as follows: (4) Design of 2D feature to 3D feature knowledge distillation network By using knowledge distillation, the texture and color features of the 2D feature are fully utilized as prior information to assist training, and the multi-modal feature is used to enhance the 3D backbone network on the basis of retaining the original 3D feature knowledge; the specific implementation manner includes the following sub-steps: Sub-step 1: design a feature approximation module; for convenience of expression, the 3D features extracted in (2) and (3) are denoted as F 3D , and the 2D features are denoted as F 2D ; F 3D is input into a multi-layer perception MLP to obtain features similar to F 2D , denoted as F 3D’ , so as to reduce the difference between the 3D features and the 2D features; Sub-step 2: design a knowledge distillation network; the features F obtained in sub-step 1 are input into the knowledge distillation network to obtain a feature F 3D’ is divided into two paths, one of which is spliced with the 2D feature to realize multi-modal feature fusion, and the other is connected back to the 3D branch; the two paths are processed respectively, and the specific operation is as follows: 3D2D ​ 3D2D input into a multi-layer perceptron, the resulting output is input into another multi-layer perceptron and activated by a Sigmoid function, the resulting weight is multiplied by the output of the previous multi-layer perceptron, and then multiplied by the pure 2D feature F 2D Element-wise addition is performed to obtain an enhanced 3D2D fusion feature, denoted as F 3D2De ; the resulting output is input into a fully connected layer to output a fusion segmentation result, and the loss function of the segmentation result will affect the feature F 3D’ ;​ ②F 3D’ is added to pure 3D feature F 3D element level, to obtain an enhanced 3D feature, denoted as F 3De ; the result is input into a fully connected layer to output a 3D segmentation result; ③Calculate the KL divergence between the fusion segmentation result in ① and ② and the 3D segmentation result in sub-step 2, so as to make the 3D backbone network approach the fusion segmentation result, thereby strengthening the 3D backbone network; (5) Network model training According to the designed network, the initial learning rate is set to 0.01, the dropout rate is set to 0.5, the batch size is set to 2, the epoch is set to 64, the self-built unstructured road scene data set and the KITTI data set are used for mixed training, the model is pre-trained using the KITTI data set to obtain the pre-training weight, and then the unstructured road scene data set is used for further training to obtain the final network weight.

Images