A semantic segmentation method and system for vehicle-mounted laser point clouds assisted by visible light and infrared images

Abstract

This invention relates to the field of point cloud processing and application technology, and particularly to a method and system for semantic segmentation of vehicle-mounted laser point clouds assisted by visible light and infrared images. The method extracts modal features from the vehicle-mounted laser point cloud through a point cloud branch, modal features from the visible light image through a visible light image branch, and modal features from the infrared image through an infrared image branch. In the visible light-infrared-point cloud fusion branch, modal feature selection and an adaptive gating fusion module are used to fuse different modal features. An auxiliary prediction branch is used, based on a knowledge distillation strategy, to transfer knowledge from the fusion branch to the point cloud branch, enhancing the predictive ability of the point cloud branch. During model training, the model is optimized by comprehensively considering the loss functions of the point cloud branch, the fusion branch, and the knowledge distillation stage. This invention effectively improves the performance of vehicle-mounted laser point cloud semantic segmentation in complex scenes by jointly utilizing the complementary advantages of visible light and infrared images.

Description

Technical Field

[0001] This invention relates to the field of point cloud processing and application technology, and in particular to a method and system for semantic segmentation of vehicle-mounted laser point clouds assisted by visible light infrared images. Background Technology

[0002] Semantic segmentation of vehicle-mounted LiDAR point clouds is a key research area for understanding outdoor scenes in autonomous driving. The rapid development of mobile LiDAR, coupled with the public availability of numerous autonomous driving data benchmarks, has significantly propelled research into semantic segmentation of vehicle-mounted LiDAR point clouds for autonomous driving. Modern autonomous vehicles are typically equipped with accurately calibrated cameras and LiDAR sensors. Compared to sparse and textureless point cloud data, visible light images acquired by visible light cameras contain rich, dense color and fine-grained texture information. Therefore, many recent works have focused on how to effectively utilize the beneficial complementary information of visible light images to improve the performance of semantic segmentation of vehicle-mounted LiDAR point clouds.

[0003] While these methods have made significant progress, the inherent limitations of visible light cameras still restrict their applicability in complex scenes. For example, under adverse imaging conditions such as fog, haze, and poor lighting, the quality of visible light images inevitably degrades, resulting in the loss of dense color and fine-grained texture information. Therefore, how to improve the semantic segmentation performance of vehicle-mounted laser point clouds under adverse imaging conditions remains an open problem. In contrast, while infrared images cannot effectively provide color and texture information, they can reflect thermal radiation information in a scene, highlighting objects with surface temperatures higher than the surrounding environment, and maintaining stable performance under different lighting conditions. In recent years, infrared images have been increasingly used as supplementary data to solve complex urban driving scene perception problems, driving the development of visible light-infrared image 2D semantic segmentation. However, current research is limited to using visible light images to enhance point cloud semantic segmentation. How to effectively combine the complementary advantages of visible light and infrared images to improve the performance of vehicle-mounted laser 3D point cloud semantic segmentation is a very meaningful and promising research area, but this research direction is currently largely unexplored. Summary of the Invention

[0004] This invention aims to address the limitations of existing technologies in semantic segmentation of vehicle-mounted laser point clouds, which suffer from performance degradation of visible light images under adverse conditions and insufficient utilization of thermal radiation information from infrared images, resulting in inadequate segmentation performance in complex scenes. This invention proposes a semantic segmentation method and system for vehicle-mounted laser point clouds assisted by visible light and infrared images. By jointly utilizing the complementary advantages of visible light and infrared images, the performance of semantic segmentation of vehicle-mounted laser point clouds in complex scenes is effectively improved.

[0005] To achieve the above objectives, the technical solution adopted is:

[0006] This invention provides a method for semantic segmentation of vehicle-mounted laser point clouds assisted by visible light and infrared images, comprising the following steps:

[0007] Construct point cloud branches, visible light image branches, and infrared image branches. Extract modal features of vehicle-mounted laser point clouds through the point cloud branches, extract modal features of visible light images through the visible light image branches, and extract modal features of infrared images through the infrared image branches.

[0008] In the visible light-infrared-point cloud fusion branch, different modal features are fused through modal feature selection and adaptive gating fusion modules;

[0009] By utilizing the auxiliary prediction branch and based on the knowledge distillation strategy, the knowledge of the fusion branch is transferred to the point cloud branch, thereby enhancing the prediction capability of the point cloud branch.

[0010] During model training, the model is optimized by comprehensively considering the loss functions of point cloud branch, fusion branch, and knowledge distillation.

[0011] According to the visible-infrared image-assisted semantic segmentation method for vehicle-mounted laser point clouds of the present invention, the point cloud branch adopts the SPVCNN model, extracts point cloud features through a sparse voxel convolution module, and retains the point-by-point features output from different decoding stages in the encoder-decoder architecture.

[0012] According to the present invention, the visible light image branch and the infrared image branch both adopt the pre-trained ResNet34 model, wherein the infrared image is copied into a three-channel input through a single channel, and both images are randomly cropped to a size of 480×320.

[0013] According to the visible-infrared image-assisted semantic segmentation method for vehicle-mounted laser point clouds of the present invention, the modal feature selection further includes:

[0014] Map the same 3D laser point onto the 2D coordinates of the visible light image and the infrared image respectively, and retrieve the projection point that is simultaneously located in the visible light image and the infrared image, i.e., the overlapping point P. O ;

[0015] Based on the overlapping point P O The extreme values ​​of the two-dimensional coordinates are used to determine the random cropping range. Visible light and infrared images are randomly cropped at a size of 480×320. Based on the cropped images, overlapping points are selected to obtain P′. O And set an overlap point ratio threshold τ to ensure that there are a sufficient number of overlap points in the overlap area after cropping;

[0016] If the overlap ratio P′ in the cropping region of visible light and infrared images O / P O If the value is below the threshold τ, random cropping is performed again until the overlap ratio of the two images is greater than or equal to the threshold τ.

[0017] For each overlapping point in the cropped region, the three-dimensional coordinates of the overlapping point are projected onto the visible light image and the infrared image respectively to obtain the corresponding two-dimensional coordinates. The two-dimensional coordinates are used to accurately locate the pixel-level features of the visible light and infrared images, and the aligned point-by-point features are obtained for feature fusion.

[0018] According to the visible-infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method of the present invention, the implementation of the adaptive gating fusion module further includes:

[0019] Different modal features are mapped using a multilayer perceptron (MLP) to reduce intermodal discrepancies.

[0020] The mapped features are concatenated as follows Let Cat(·) be the fused feature of the k-th decoding stage, and let Cat(·) be the feature concatenation operation. For the point cloud branch, the point-by-point features of the k-th decoding stage are... For the k-th decoding stage of the visible light image branch, the pixel-wise features are... For the k-th decoding stage of the infrared image branch, the pixel-by-pixel feature is used.

[0021] Weights are generated using the sigmoid activation function. Adjust the fusion features according to element-wise product.

[0022] According to the visible light infrared image-assisted semantic segmentation method for vehicle-mounted laser point clouds of the present invention, the auxiliary prediction branch is further implemented through a knowledge distillation strategy, specifically including:

[0023] Utilizing fusion features and point cloud features Generate semantic scores respectively and

[0024] Using the KL divergence loss function This forces the point cloud branches to approximate the prediction results of the fusion branches.

[0025] According to the visible-infrared image-assisted semantic segmentation method for vehicle-mounted laser point clouds of the present invention, the overall loss function for model training is further as follows:

[0026]

[0027] Among them, l PThe cross-entropy loss of the point cloud branches, Let be the cross-entropy loss function for the k-th decoding stage of the point cloud branch. Let the cross-entropy loss function be the cross-entropy loss function for the k-th decoding stage of the fusion branch. Let λ be the knowledge distillation loss for the k-th decoding stage, and λ be the weighting coefficient of the knowledge distillation loss.

[0028] According to the present invention, the semantic segmentation method for vehicle-mounted laser point clouds assisted by visible light and infrared images further demonstrates that, after training, only the point cloud branch is retained for inference, without the participation of visible light and infrared images, thereby achieving semantic segmentation of vehicle-mounted laser point clouds.

[0029] Furthermore, the present invention also provides a visible-infrared image-assisted vehicle-mounted laser point cloud semantic segmentation system for implementing the above-mentioned visible-infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method. The system includes:

[0030] The feature extraction module is used to construct point cloud branches, visible light image branches, and infrared image branches. It extracts modal features of vehicle-mounted laser point clouds through the point cloud branches, modal features of visible light images through the visible light image branches, and modal features of infrared images through the infrared image branches.

[0031] The feature fusion module is used to fuse different modal features in the visible light-infrared-point cloud fusion branch through modal feature selection and adaptive gating fusion module;

[0032] The auxiliary prediction module is used to utilize the auxiliary prediction branch and, based on the knowledge distillation strategy, transfer the knowledge of the fusion branch to the point cloud branch to enhance the prediction capability of the point cloud branch.

[0033] The model training module is used to optimize the model during the model training process by comprehensively considering the loss function of point cloud branch, fusion branch and knowledge distillation.

[0034] According to the visible light infrared image-assisted vehicle-mounted laser point cloud semantic segmentation system of the present invention, the system further includes an inference module for inference by retaining only the point cloud branches to achieve semantic segmentation of the vehicle-mounted laser point cloud.

[0035] The beneficial effects achieved by adopting the above technical solution are:

[0036] This invention aims to enhance semantic segmentation of vehicle-mounted laser point clouds by jointly utilizing visible light and infrared images, proposing a visible light and infrared image-assisted semantic segmentation method for vehicle-mounted laser point clouds—RTPSegNet. First, the model employs three independent branches, preserving modality specificity during the feature representation learning stages of point cloud, visible light image, and infrared image, respectively. Subsequently, a dedicated visible light-infrared-point cloud fusion branch is designed, including modality feature selection and adaptive gating fusion modules to promote intermodal interaction. Finally, multimodal knowledge is learned through a knowledge distillation strategy, and the overall predictive capability of the point cloud branch is enhanced using an auxiliary prediction branch. Experimental results demonstrate the effectiveness of this invention in jointly utilizing complementary information from point cloud, visible light image, and infrared image, fully validating that introducing additional infrared image modalities can effectively enhance the performance of vehicle-mounted laser point cloud semantic segmentation, showcasing the significant value and potential of this research direction. Furthermore, the performance of this method surpasses previous state-of-the-art single-modal point cloud methods and visible light image-point cloud dual-modal methods. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. The drawings are merely illustrative of some embodiments of the present invention and are not intended to limit the scope of the present invention to all embodiments.

[0038] Figure 1 This is a flowchart illustrating the visible light infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method according to an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram of the RTPSeg three-modal vehicle-mounted laser point cloud semantic segmentation dataset of the present invention, which is a visible light-infrared image-point cloud trimodal dataset.

[0040] Figure 3 This is a partial visualization of the RTPSegNet-L baseline and RTPSegNet on the RTPSeg dataset, according to an embodiment of the present invention. Detailed Implementation

[0041] The exemplary solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art.

[0042] This embodiment discloses a visible light infrared image-assisted semantic segmentation method for vehicle-mounted laser point clouds, such as... Figure 1 As shown, it includes the following steps:

[0043] Step S101: Construct point cloud branch, visible light image branch and infrared image branch. Extract modal features of vehicle-mounted laser point cloud through point cloud branch, extract modal features of visible light image through visible light image branch, and extract modal features of infrared image through infrared image branch.

[0044] (1) Point cloud branches

[0045] When constructing the point cloud branch, SPVCNN is used as the point cloud branch of RTPSegNet. SPVCNN is a model specifically designed to solve the semantic segmentation task of point clouds in vehicle-mounted laser systems. SPVCNN extracts point cloud features by introducing an efficient sparse voxel convolution (SPVConv) module, balancing the advantages of points and voxels, reducing the computational burden in large-scale scenes, and mitigating the loss of detail caused by coarse voxelization. The encoder-decoder architecture adopted by SPVCNN facilitates the subsequent visible light-infrared-point cloud fusion branch. This method fully integrates the public structure of SPVCNN into RTPSegNet and retains its key hyperparameters. By maintaining the original training settings of SPVCNN, the independence of the point cloud branch and the specificity of the point cloud modalities in RTPSegNet are guaranteed.

[0046] During training, the point cloud branch is directly supervised by point-by-point truth semantic labels and uses the cross-entropy loss function l. P Optimization will be performed. Next, the outputs of the K different decoding layers in the decoding stage will be analyzed. Recorded for subsequent visible light-infrared-point cloud feature fusion, where n represents the number of points, k represents the level index of the decoding stage, and D... k This represents the feature dimension of the k-th decoding stage. These outputs... It uses the SPVConv module and upsamples to point-by-point features at each stage. This setup allows the point cloud branch of RTPSegNet to independently complete point cloud semantic segmentation tasks even without the participation of other branches.

[0047] (2) Visible light image branch

[0048] When constructing the visible light image branch, a pre-trained model ResNet34 is used as the visible light image branch of RTPSegNet for subsequent training, thus effectively utilizing the initial capabilities of the pre-trained model to extract and learn image features. The visible light image branch is not directly supervised by any supervisory signal; its main purpose is to extract visible light image features and use them for visible light-infrared-point cloud feature fusion. Similar to the point cloud branch, the corresponding outputs are recorded at different decoding stages of the visible light image branch. Among them, H R W represents the height of the feature map in the k-th decoding stage. RD represents the width of the feature map in the k-th decoding stage. k This represents the channel dimension of the feature map in the k-th decoding stage, and is used for subsequent visible light-infrared-point cloud feature fusion. These outputs... It consists of pixel-by-pixel features at each stage after progressive upsampling.

[0049] (3) Infrared Image Branch

[0050] Considering the two-dimensional nature of infrared images, the pre-trained model ResNet34 is still used as the infrared image branch of RTPSegNet for training when constructing the infrared image branch. To meet the standard input requirements of the two-dimensional encoder, the single-channel infrared image is copied to the three-channel (…). Similarly, pixel-by-pixel infrared image features from different decoding stages are used. Record it for subsequent multimodal feature fusion.

[0051] To reduce computational cost and enhance feature representation, both visible light and infrared images were randomly cropped to a size of 480×320 as model input. Furthermore, random cropping ensures that the same image sample provides different input samples in different training rounds, thus significantly enriching the diversity of training samples.

[0052] Step S102: In the visible light-infrared-point cloud fusion branch, different modal features are fused through modal feature selection and adaptive gating fusion modules.

[0053] (1) Modal feature selection

[0054] Due to differences between different sensors, precise pixel-by-pixel alignment between visible light and infrared images is difficult to achieve, and differences in spatial resolution and field of view also exist, posing a challenge to the direct fusion of visible light and infrared images. However, during multimodal data acquisition, the LiDAR is time-stamped and coordinate-system-unified with both the visible light and infrared cameras, enabling the establishment of a 3D-to-2D pixel-by-pixel projection correspondence between the visible light and infrared images. Next, the 3D points of the LiDAR point cloud and their corresponding projection points on the visible light and infrared images are used as the basic units for multimodal feature fusion, thereby solving the pixel misalignment problem between the visible light and infrared images.

[0055] ① Due to the differences in the field of view of different sensors, multimodal fusion can only be performed on three-dimensional points located within the intersection of the field of view of the three sensors. Therefore, the same laser three-dimensional point... Mapping to two-dimensional coordinates on the visible light image and infrared image respectively, retrieving and retaining the projection point that simultaneously lies in both the visible light image and the infrared image, i.e., the overlap point P. O=P[i], where i is the projection point index. ② For two different types of sensors, the 3D to 2D projection correspondence is different, and the 2D coordinates of the 3D point on the two images are also different. Therefore, take the overlapping point P. O The extreme values ​​of the horizontal and vertical coordinates on the two types of images are used to determine the random cropping range for the visible light image and the infrared image, respectively. ③ After determining the cropping range, random cropping is performed with a size of 480×320. The overlapping points are then filtered based on the cropped image, which can be represented as P′. O =P′ R ∩P′ T =P[i′], P′ R Let P' represent the set of three-dimensional points corresponding to the cropped visible light image. T This represents the set of three-dimensional points corresponding to the cropped infrared image, to ensure overlap of points P′. O The number of overlapping points should not be too small. A threshold τ for the overlap point ratio is set to remove overlapping regions with too few overlapping points. Specifically, if the overlap point ratio P′ O / P O If the overlap ratio is less than the threshold τ, both the visible light image and the infrared image will be randomly cropped again until the overlap ratio of the two images is greater than or equal to the threshold, thus ensuring that the overlapping area of ​​the cropped image is large enough and the number of overlapping points is sufficient.

[0056] After determining the final overlap point, the three-dimensional coordinates of the overlap point are projected onto the visible light and infrared images to obtain two-dimensional coordinates. These two-dimensional coordinates are then used to precisely locate pixel-level features in the visible light and infrared images, thereby obtaining strictly aligned point-by-point features from different modalities for visible light-infrared-point cloud modal feature fusion. Specifically, different modal features can be represented as point cloud features. Visible light image features Infrared image features Where m represents the number of points used for multimodal feature fusion. When locating features using the two-dimensional coordinates of the projected points, the two-dimensional coordinates are rounded down for pixel localization.

[0057] (2) Adaptive gating fusion

[0058] Since visible light-infrared-point cloud feature fusion involves interactions between three different modalities, the potentially large modal gap poses a challenge to the fusion process. Therefore, to narrow the modal gap, a dedicated MLP layer is first deployed as a pointwise feature mapper for each modality. By mapping modal features, features from different modalities are made closer to each other in the embedding space. Next, while reducing the modal gap, the projected features from different modalities are directly concatenated for feature fusion, yielding the fused features at each stage. As shown in the following formula:

[0059]

[0060] in, Let Cat(·) be the fused feature of the k-th decoding stage, and let Cat(·) be the feature concatenation operation. For the point cloud branch, the point-by-point features of the k-th decoding stage are... For the k-th decoding stage of the visible light image branch, the pixel-wise features are... This represents the pixel-by-pixel features of the k-th decoding stage in the infrared image branch.

[0061] Due to the dynamic complexity and variability of the external environment, the modal advantages and contributions vary under different perception conditions. For example, the quality of visible light images degrades significantly at night, while infrared images can highlight objects with surface temperatures higher than their surroundings. Conversely, visible light images provide rich texture and color information during the day, outperforming infrared images. Therefore, to adaptively and effectively utilize complementary information from different modalities, enabling the model to highlight beneficial modal contributions while suppressing modal information that might interfere with model training, this invention designs a feature-level adaptive gating fusion module to dynamically adjust the contribution of each modality, which can be expressed as:

[0062]

[0063] Where σ represents the sigmoid activation function, This represents element-wise multiplication. Therefore, the adaptive gating fusion module effectively enhances the expressive power of multimodal fusion features by reducing modal gaps and adjusting modal contributions.

[0064] Step S103: Using the auxiliary prediction branch, based on the knowledge distillation strategy, the knowledge of the fusion branch is transferred to the point cloud branch to enhance the prediction ability of the point cloud branch.

[0065] The overlapping point P′ is obtained by fusing visible light, infrared, and point cloud multimodal features. O Point-by-point fusion features Next, a classifier to be trained is deployed, and then features are fused point by point. Input to the classifier to obtain overlapping points P′ O semantic score Where N cls This represents the number of semantic categories. Classifiers in different modal branches and at different stages are directly supervised by pointwise ground truth semantic labels and pass through a multi-stage cross-entropy loss function. Optimization was performed. This was done in conjunction with single-modal point cloud features. In comparison, fusion features Integrating beneficial complementary information from different modalities facilitates more robust point-by-point prediction and classification. However, the multimodal feature fusion process and the fusion-based classification process can only be applied to overlapping points P′. O .

[0066] To enhance the prediction results of the entire point cloud through multimodal feature fusion, this method further constructs an auxiliary prediction branch based on a knowledge distillation strategy, thereby achieving knowledge transfer from the visible light-infrared-point cloud fusion branch to the point cloud branch. Specifically, similarly, for multi-stage single-modal point cloud features... Deploy a separate classifier to be trained to obtain the corresponding semantic score. Similarly, the point cloud branches and classifiers at different stages are directly supervised by point-by-point ground truth semantic labels and through a multi-stage cross-entropy loss function. Optimization is then performed. Next, during network training, the prediction outputs of different stages of the point cloud branches are forced to be optimized. The corresponding predicted output of the imitation and approximation of the multimodal fusion branch The loss function of this knowledge distillation process can be expressed as:

[0067]

[0068] Wherein, KL(·‖·) represents the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions.

[0069] Through this knowledge distillation process, the point cloud branch can effectively learn beneficial multimodal knowledge from the multimodal fusion branch. Therefore, the auxiliary prediction branch further enhances the semantic segmentation performance of the point cloud branch by making the overall prediction of the point cloud branch approximate a more discriminative distribution.

[0070] Step S104: During model training, the model is optimized by comprehensively considering the loss functions of point cloud branch, fusion branch and knowledge distillation.

[0071] The overall loss function for model training is:

[0072]

[0073] Among them, l P The cross-entropy loss of the point cloud branches, Let be the cross-entropy loss function for the k-th decoding stage of the point cloud branch. Let the cross-entropy loss function be the cross-entropy loss function for the k-th decoding stage of the fusion branch. Let λ be the knowledge distillation loss for the k-th decoding stage, and λ be the weighting coefficient of the knowledge distillation loss.

[0074] Step S105: After training, only the point cloud branch is retained for inference, without the participation of visible light images and infrared images, to achieve semantic segmentation of the vehicle-mounted laser point cloud.

[0075] During training, RTPSegNet learns beneficial complementary information from different modalities. After training, only the point cloud branch of RTPSegNet is retained for inference, eliminating the need for visible light and infrared images. This allows all points to be predicted, regardless of whether they are located within the intersection of the fields of view of different sensors. Furthermore, retaining only the point cloud branch for inference effectively reduces model inference latency, thus better meeting the requirements of real-time inference.

[0076] Corresponding to the above method, this embodiment also discloses a visible light infrared image-assisted vehicle-mounted laser point cloud semantic segmentation system, which includes:

[0077] The feature extraction module is used to construct point cloud branches, visible light image branches, and infrared image branches. It extracts modal features of vehicle-mounted laser point clouds through the point cloud branches, modal features of visible light images through the visible light image branches, and modal features of infrared images through the infrared image branches.

[0078] The feature fusion module is used to fuse different modal features in the visible light-infrared-point cloud fusion branch through modal feature selection and adaptive gating fusion module;

[0079] The auxiliary prediction module is used to utilize the auxiliary prediction branch and, based on the knowledge distillation strategy, transfer the knowledge of the fusion branch to the point cloud branch to enhance the prediction capability of the point cloud branch.

[0080] The model training module is used to optimize the model during the model training process by comprehensively considering the loss functions of point cloud branches, fusion branches, and knowledge distillation.

[0081] The inference module is used to perform inference by retaining only the point cloud branches, thereby achieving semantic segmentation of the vehicle-mounted laser point cloud.

[0082] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings, tables, and specific examples. The RTPSeg dataset, a three-modal vehicle-mounted laser point cloud semantic segmentation dataset containing visible light-infrared images and point clouds, is used as the experimental data. RTPSeg is the first vehicle-mounted laser point cloud semantic segmentation dataset that simultaneously includes visible light and infrared images. A schematic diagram of the dataset is shown below. Figure 2As shown in Table 1, RTPSeg contains over 3,000 synchronized frames collected by visible light cameras, infrared cameras, and LiDAR in Guangming District, Shenzhen, China. The dataset includes daytime and nighttime scenes in both urban and rural areas, providing over 240 million point-by-point semantic annotations for 18 semantic categories in autonomous driving. The annotation details are shown in Table 2. The training set, test set, and scene divisions are shown in Table 2.

[0083] Table 1

[0084]

[0085]

[0086] Table 2

[0087]

[0088] 1. Extracting different modal features using independent branches of the model

[0089] 1.1 First, point cloud modal features are extracted using point cloud branching. For point clouds... Inputting this into the point cloud branch yields the outputs of K different decoding layers in the decoding stage. Point cloud features used for subsequent multimodal feature fusion.

[0090] 1.2 Next, based on the projection correspondence between the two images, calculate the two-dimensional coordinates of the projection points on the two images respectively;

[0091] 1.3 Based on the calculated two-dimensional coordinates, for the training sample pair consisting of a visible light image (1920×1080×3) and an infrared image (1280×1024×1), random cropping is performed. For both types of images, only a cropping region of size 480×320 is retained. Cropping is completed when the overlap ratio is greater than the ratio threshold τ, and the overlap point P′ of the cropped image is determined. O ;

[0092] 1.4 The cropped visible light image (480×320×3) is input into the visible light image branch to extract visible light image features, and the output corresponding to different decoding stages is obtained. Visible light image features used for subsequent multimodal feature fusion;

[0093] 1.5 The cropped infrared image (480×320×1) is copied from a single channel to three channels (480×320×3), and then input into the infrared image branch to extract infrared image features, obtaining the output corresponding to different decoding stages. Infrared image features used for subsequent multimodal feature fusion.

[0094] 2. Multimodal feature fusion using visible light-infrared-point cloud fusion branch

[0095] 2.1 First, using the overlapping point P′ O The index and two-dimensional projected coordinates are used to determine different modal features, including point cloud features, through point-by-point alignment. Visible light image features Infrared image features

[0096] 2.2 Next, the different modal features aligned point by point are input into the adaptive gating mechanism module for multimodal feature fusion to obtain the overlapping point P′. O Point-by-point fusion features

[0097] 3. Enhance point cloud branch task performance by utilizing auxiliary prediction branches.

[0098] 3.1 First, the point cloud features The data is fed into the classifiers corresponding to different stages of the point cloud branch to obtain the point cloud branch prediction scores.

[0099] 3.2 Next, we will fuse the features. The data is fed into the classifiers corresponding to different stages to obtain the prediction scores of the fusion branch.

[0100] 3.3 Finally, the overlapping point prediction outputs of different stages of the point cloud branches are forced. The prediction output corresponds to the overlap point of the multimodal branches.

[0101] 4. Model Training and Inference

[0102] 4.1 Point cloud branching is directly supervised by point-by-point truth semantic labels and through the cross-entropy loss function l P Optimization is performed. Simultaneously, classifiers deployed at different stages are directly supervised by point-by-point ground truth semantic labels and optimized using a multi-stage cross-entropy loss function. Optimize;

[0103] 4.2 The visible light image branch and the infrared image branch are not directly supervised by the supervision signal, but their extracted features are input to the multimodal feature fusion branch, thus receiving indirect supervision. Therefore, the classifiers at different stages of different modal branches and the multimodal feature fusion branch are supervised by point-by-point truth semantic labels and through a multi-stage cross-entropy loss function. Optimize;

[0104] 4.3 During network training, force the overlapping points of different stages of the point cloud branches to predict the output. The corresponding prediction output of the multimodal branch that mimics and approximates the overlap point

[0105] 4.4 After the model is trained, only the point cloud branch of RTPSegNet is retained for inference, eliminating the need for visible light and infrared images. This allows all points to be predicted, regardless of whether they are located within the intersection of the fields of view of different sensors. Furthermore, retaining only the point cloud branch for inference effectively reduces model inference latency, thus better meeting the requirements of real-time inference.

[0106] 1. Simulation conditions: An Intel Core (R) Xeon (R) Gold 6330 CPU, an NVIDIA GeForce RTX 4090 GPU, 24GB of GPU video memory, and 64GB of RAM were used. This invention was developed and implemented using Python and PyTorch libraries.

[0107] 2. Simulation settings:

[0108] The model was trained for 200 iterations using the SGD optimizer and cosine scheduler with a learning rate of 0.024, and momentum and weight decay were configured to 0.9 and 0.0001, respectively. During random pruning, the overlap ratio threshold τ was set to 0.7. During knowledge distillation, the loss function weight λ was set to 0.5. Class Intersection over Union (IoU) and mean Intersection over Union (mIoU) were used as evaluation metrics.

[0109] The visible light and infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method RTPSegNet in this invention can be flexibly transformed into a point cloud single-modal model RTPSegNet-L, a visible light image-point cloud dual-modal model RTPSegNet-LC, and an infrared image-point cloud dual-modal model RTPSegNet-LT by deleting the corresponding modal branches.

[0110] 3. Simulation results:

[0111] First, we tested various scene scenarios on the RTPSeg dataset using four different versions of RTPSegNet, including daytime, nighttime, and mixed scenarios. The results are shown in Table 3. It can be seen that compared to the version using only point clouds, other versions of RTPSegNet, aided by visible light or infrared images, achieve better performance (mIoU) in all scene scenarios. In all scenarios, the infrared image modality improves the point cloud model performance better than the visible light image modality, highlighting the crucial complementary effect of infrared images on point cloud modalities and its significant value. Furthermore, compared to other versions, the RTPSegNet method for vehicle-mounted laser point cloud semantic segmentation, aided by visible light and infrared images, achieved the highest performance in all scene scenarios, demonstrating the importance of simultaneously utilizing the complementary information from visible light and infrared images to improve point cloud semantic segmentation performance.

[0112] Table 3

[0113]

[0114] Next, through designed experiments, several advanced point cloud semantic segmentation methods were compared on mixed scenes of the RTPSeg dataset, including point cloud unimodal methods and visible light image-point cloud multimodal fusion methods, as well as the baseline model RTPSegNet and its different versions. Point cloud unimodal methods included the voxel-based MinkowskiNet, the projection-based SqueezeSegV3, and the point-based PolarNet and SphereFormer. Visible light image-point cloud multimodal fusion methods included 2DPASS and MSeg3D. For 2DPASS and MSeg3D, their unimodal versions were also compared. Simultaneously, infrared images were extended to three channels to replace the visible light image inputs for 2DPASS and MSeg3D, thus enabling a comparison of simplified infrared image-point cloud multimodal fusion methods for 2DPASS and MSeg3D. The quantitative comparison results are shown in Table 4. It can be seen that the performance improvement of the simplified infrared image-point cloud multimodal fusion methods for 2DPASS and MSeg3D further validates the effectiveness of infrared image modalities. However, since these are all bimodal methods, they cannot effectively utilize information from all three modalities simultaneously, especially for the additional infrared image modality. In contrast, the RTPSegNet of this invention achieved the highest mIoU (70.07%) among all compared methods, validating its effectiveness in jointly utilizing complementary information from point cloud, visible light image, and infrared image modalities. With the assistance of visible light and infrared images, the RTPSegNet of this invention can significantly improve the segmentation accuracy of imbalanced hard categories such as cars, bicycles, motorcyclists, and trucks. Since these categories are key road targets that require attention in autonomous driving, this demonstrates the research potential and application value of visible light-infrared image-assisted vehicle-mounted laser point cloud semantics. Furthermore, other versions of the RTPSegNet of this invention also showed relatively better performance than similar methods.

[0115] Table 4

[0116]

[0117] Then, the prediction results of RTPSegNet are visualized to qualitatively verify the effectiveness of RTPSegNet. Figure 3The visualization results of the RTPSegNet-L baseline and RTPSegNet on the RTPSeg dataset are shown. It can be seen that due to the unimodal bias of the RTPSegNet-L baseline, RTPSegNet-L incorrectly predicts some targets; for example, a distant car is incorrectly identified as a truck, a fence is incorrectly identified as a building due to geometric similarity, and a distant motorcycle is incorrectly predicted as tall vegetation. By effectively utilizing the complementary information between visible light images, infrared images, and point clouds, RTPSegNet can correct the significant misprediction caused by the unimodal bias of the RTPSegNet-L baseline, demonstrating the beneficial effect of multimodal fusion. These corrections ultimately improve the point cloud semantic segmentation performance of RTPSegNet, showing consistency between the results of qualitative and quantitative comparisons. Furthermore, the bird's-eye view projection results show that although multimodal feature fusion can only be performed at projection points located at the intersection of the three different sensor field-of-view angles, RTPSegNet can still correct the prediction results of 3D points outside the field-of-view angles. Therefore, the multimodal fusion branch and auxiliary prediction branch in this invention can effectively improve the overall performance of the point cloud branch of RTPSegNet.

[0118] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0120] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0121] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0122] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A semantic segmentation method for vehicle-mounted laser point clouds assisted by visible light and infrared images, characterized in that, Includes the following steps: Construct point cloud branches, visible light image branches, and infrared image branches. Extract modal features of vehicle-mounted laser point clouds through the point cloud branches, extract modal features of visible light images through the visible light image branches, and extract modal features of infrared images through the infrared image branches. In the visible light-infrared-point cloud fusion branch, different modal features are fused through modal feature selection and an adaptive gating fusion module. The modal feature selection specifically includes: mapping the same laser 3D point to 2D coordinates on both the visible light and infrared images, and retrieving projection points simultaneously located in both the visible light and infrared images, i.e., overlapping points. Based on overlapping points The extreme values ​​of the two-dimensional coordinates are used to determine the random cropping range. Visible light and infrared images are randomly cropped at a size of 480×320. The overlapping points are then selected based on the cropped images. And set an overlap point ratio threshold. Ensure there are a sufficient number of overlapping points within the cropped overlapping area; if the ratio of overlapping points in the cropped areas of visible light and infrared images is... Below the threshold Then, random cropping is performed again until the overlap ratio of the two images is greater than or equal to the threshold. For each overlapping point in the cropping region, the three-dimensional coordinates of the overlapping point are projected onto the visible light image and the infrared image respectively to obtain the corresponding two-dimensional coordinates. The two-dimensional coordinates are used to accurately locate the pixel-level features of the visible light and infrared images, and the aligned point-by-point features are obtained for feature fusion. By utilizing the auxiliary prediction branch and based on the knowledge distillation strategy, the knowledge of the fusion branch is transferred to the point cloud branch, thereby enhancing the prediction capability of the point cloud branch. During model training, the model is optimized by comprehensively considering the loss functions of point cloud branch, fusion branch, and knowledge distillation.

2. The visible light infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method according to claim 1, characterized in that, The point cloud branch uses the SPVCNN model, which extracts point cloud features through a sparse voxel convolution module and retains the point-by-point features output from different decoding stages in the encoder-decoder architecture. .

3. The visible-infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method according to claim 1, characterized in that, Both the visible light image branch and the infrared image branch use a pre-trained ResNet34 model. The infrared image is copied from a single channel to a three-channel input, and both images are randomly cropped to a size of 480×320.

4. The visible light infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method according to claim 1, characterized in that, The implementation of the adaptive gating fusion module includes: Different modal features are mapped using a multilayer perceptron (MLP). The mapped features are concatenated as follows , For the fusion features of the k-th decoding stage, For feature splicing operations, For the point cloud branch, the point-by-point features of the k-th decoding stage are... For the k-th decoding stage of the visible light image branch, the pixel-wise features are... For the k-th decoding stage of the infrared image branch, the pixel-by-pixel feature is used. Weights are generated using the sigmoid activation function. And adjust the fusion features according to element-wise product. .

5. The visible-infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method according to claim 4, characterized in that, The auxiliary prediction branch is implemented through a knowledge distillation strategy, specifically including: Utilizing fusion features and point cloud features Generate semantic scores respectively and ; Using the KL divergence loss function This forces the point cloud branches to approximate the prediction results of the fusion branches.

6. The visible-infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method according to claim 1, characterized in that, The overall loss function for model training is: ,in, The cross-entropy loss of the point cloud branches, Let be the cross-entropy loss function for the k-th decoding stage of the point cloud branch. Let cross-entropy loss function be the cross-entropy loss function for the k-th decoding stage of the fusion branch. The knowledge distillation loss for the k-th decoding stage is... This represents the weighting coefficient for knowledge distillation loss.

7. The visible-infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method according to claim 1, characterized in that, After training, only the point cloud branch is retained for inference, without the need for visible light and infrared images, thus achieving semantic segmentation of the vehicle-mounted laser point cloud.

8. A vehicle-mounted laser point cloud semantic segmentation system assisted by visible light and infrared images, characterized in that, For implementing the visible light infrared image-assisted vehicle-mounted laser point cloud semantic segmentation method as described in any one of claims 1-7, the system comprises: The feature extraction module is used to construct point cloud branches, visible light image branches, and infrared image branches. It extracts modal features of vehicle-mounted laser point clouds through the point cloud branches, modal features of visible light images through the visible light image branches, and modal features of infrared images through the infrared image branches. The feature fusion module is used to fuse different modal features in the visible light-infrared-point cloud fusion branch through modal feature selection and adaptive gating fusion module; The auxiliary prediction module is used to utilize the auxiliary prediction branch and, based on the knowledge distillation strategy, transfer the knowledge of the fusion branch to the point cloud branch to enhance the prediction capability of the point cloud branch. The model training module is used to optimize the model during the model training process by comprehensively considering the loss function of point cloud branch, fusion branch and knowledge distillation.

9. The visible-infrared image-assisted vehicle-mounted laser point cloud semantic segmentation system according to claim 8, characterized in that, The system also includes an inference module, which is used to perform inference by retaining only the point cloud branches to achieve semantic segmentation of the vehicle-mounted laser point cloud.

Images