Laser-radar image-oriented semantic segmentation method based on learnable prototypes

Abstract

The application relates to the technical field of image semantic segmentation, in particular to a laser radar image-oriented semantic segmentation method based on a learnable prototype, which comprises the following steps: inputting a pair of laser radar images and near-infrared images into laser radar processing branches and near-infrared processing branches of a double-branch feature extraction module to obtain laser radar features at different levels and near-infrared features at different levels; performing feature splicing on the laser radar features at different levels and the near-infrared features at different levels through a multi-level feature fusion module to obtain joint multi-modal features; calculating the cosine similarity between the joint multi-modal features and each learnable prototype through a learnable prototype embedding module, determining a matching learnable prototype, and fusing the matching learnable prototype with the joint multi-modal features to obtain fused features; inputting the fused features into a dilated convolution pyramid pooling module to obtain high-level fused features; and performing semantic segmentation on the high-level fused features through a decoder to obtain a semantic segmentation result.

Description

Technical Field

[0001] The embodiments of this application relate to the field of image semantic segmentation technology, and in particular to a semantic segmentation method based on a learnable prototype for LiDAR images. Background Technology

[0002] LiDAR imagery is a product of surface information acquired using LiDAR (Light Detection and Ranging) technology. Its core technology involves measuring target distances using laser pulses to ultimately generate images or data products containing three-dimensional spatial information. Platforms equipped with LiDAR devices (aircraft, satellites, drones) actively emit laser pulses. These pulses reach the ground (or object) and are reflected back to the receiver. By calculating the round-trip time (or phase difference) of the pulses, combined with the platform's position and attitude data, the three-dimensional coordinates of each laser point are precisely calculated. A large number of these laser points form "point cloud data," which is then processed through denoising, classification, and interpolation to generate visualizations such as digital surface models (DSMs), digital elevation models (DEMs), and orthophotos. These visualizations constitute LiDAR imagery.

[0003] Semantic segmentation of LiDAR imagery refers to the processing of point cloud data or its derived images (such as depth images and reflectance intensity images) acquired by LiDAR, assigning a semantic label to each LiDAR point or pixel to achieve automatic identification and classification of surface (or object) categories. Its core objective is to enable machines to "understand" the three-dimensional scene structure described by LiDAR imagery and to clearly define the spatial distribution and category attributes of different objects.

[0004] In recent years, deep convolutional neural networks (DCNNs) have brought new methods for processing and applying large amounts of LiDAR images, especially in semantic segmentation. Compared with traditional methods, deep learning-based semantic segmentation methods are more effective in analyzing the spatial and semantic features of LiDAR images. Mainstream DCNN-based semantic segmentation techniques can be divided into two main categories. One category is pixel-based segmentation. For example, DCNN extracts high-dimensional semantic features from raw pixels, MCNN utilizes multi-scale learning to capture contextual information at different levels, RotNet enhances the model's generalization ability through rotation-invariant structures, and networks like PT-GID can automatically select training samples from the target domain to achieve semi-supervised learning. The other category is object-based segmentation. This method combines DCNN with object-level image analysis frameworks to obtain more accurate boundary determination. For example, OCNN designs an object-based image analysis framework to obtain more accurate boundary information.

[0005] However, with the introduction of DCNN, the semantic segmentation of LiDAR images has encountered the following problems.

[0006] First, the sparsity and irregularity of point cloud data. Point cloud data is discrete and disordered, with large differences in the density of points in different regions (for example, fewer points for distant objects and more points for nearby objects), making it difficult for traditional DCNN-based semantic segmentation techniques to process directly.

[0007] Second, there is a class imbalance. Most points in the scene belong to the "background" (such as the ground), while the number of small target points (such as utility poles, pedestrians, etc.) is very small, which leads to low accuracy of the model in recognizing small targets.

[0008] Third, interference from complex scenes. In urban scenes, buildings and trees can obstruct the view, and it is difficult to distinguish objects of similar materials (such as metal fences and vehicles).

[0009] Fourth, the computational cost is high. Massive point cloud data (such as tens of millions of points in a single aerial scan) places extremely high demands on the storage and computing power of the model, which leads to high costs.

[0010] Fifth, labeled data is scarce. Semantic segmentation of point cloud data requires a large number of manually labeled 3D point cloud labels as a reference. Labeling is costly and time-consuming, which limits the training effect of the model. Summary of the Invention

[0011] To address the aforementioned technical issues, embodiments of this application propose a semantic segmentation method for lidar images based on learnable prototypes. This method utilizes a dual-branch feature extraction framework to extract shallow and deep features from lidar and near-infrared images. It also employs learnable prototypes to establish semantic category representations. By introducing a dual-modal feature fusion mechanism, the method establishes the interrelationship between lidar and near-infrared images, fusing information from both modalities. Through the collaborative work of learnable prototypes and multimodal features, the accuracy and efficiency of semantic segmentation for lidar images are significantly improved.

[0012] To achieve the above objectives, embodiments of this application propose a semantic segmentation method based on learnable prototypes for LiDAR images, implemented using multimodal images. The method includes: acquiring paired LiDAR and near-infrared images; inputting the LiDAR and near-infrared images into the LiDAR processing branch and near-infrared processing branch of a dual-branch feature extraction module, performing feature extraction at different levels to obtain LiDAR features and near-infrared features at different levels; using a multi-level feature fusion module, based on a cross-attention mechanism, concatenating the LiDAR and near-infrared features at different levels to obtain joint multimodal features; and using learnable prototypes... The embedding module initializes a set of learnable prototypes, each representing a semantic category. It calculates the cosine similarity between the joint multimodal features and each learnable prototype, selecting the learnable prototype with the highest cosine similarity as the matching learnable prototype corresponding to the joint multimodal features. The joint multimodal features and the matching learnable prototypes are then fused to obtain fused features. These fused features are input into a dilated convolutional pyramid pooling module to capture information from different receptive fields, enhancing contextual understanding and yielding advanced fused features. Finally, the advanced fused features are upsampled using bilinear interpolation by a decoder to restore them to the same size as the LiDAR image, and semantic segmentation is performed to obtain the semantic segmentation result.

[0013] To achieve the above objectives, embodiments of this application also propose an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a learnable prototype-based semantic segmentation method for LiDAR images as described above.

[0014] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a semantic segmentation method for lidar images based on a learnable prototype, as described above.

[0015] This application proposes a learnable prototype-based semantic segmentation method for LiDAR imagery, which fully leverages the advantages of both LiDAR and near-infrared (NIIR) imagery, providing unique advantages for understanding and analyzing surface scenes. LiDAR imagery provides high-resolution visual information, including point clouds, micro-topographic details, and terrain undulations, while NIIR imagery provides stable radar echo information unaffected by weather and lighting conditions. The combination of these two imagery modalities enables surface monitoring in complex environments (such as at night or in severe weather), which is of significant value for applications such as environmental monitoring and disaster assessment. To fuse these two imagery modalities, this application designs a multimodal semantic segmentation model consisting of a bi-branch feature extraction module, a multi-level feature fusion module, a learnable prototype embedding module, a dilated convolutional pyramid pooling module, and a decoder to perform the semantic segmentation task. The bi-branch feature extraction module can effectively extract different levels of LiDAR and NIIR features. The multi-level feature fusion module, based on a cross-attention mechanism, concatenates features from different levels of LiDAR and near-infrared technologies to obtain joint multi-modal features, effectively establishing the interrelationship between the two modalities of the image. The learnable prototype embedding module effectively establishes semantic category representations using learnable prototypes. The introduction of dilated convolutional pyramid pooling modules expands the receptive region, effectively enhancing the extraction and utilization of contextual information. Through the collaborative work of learnable prototypes and multi-modal features, the multi-modal semantic segmentation model significantly improves the accuracy and efficiency of semantic segmentation.

[0016] Optionally, the LiDAR processing branch and the near-infrared processing branch of the dual-branch feature extraction module have the same structure, both consisting of a ResNet50 and five convolutional blocks in series. Each convolutional block consists of a series of convolutional layers, BN normalization layers, ReLU activation function layers and max pooling layers.

[0017] Let the five convolutional blocks represent the first to fifth levels, then the lidar features and near-infrared features at different levels can be expressed by the following formulas:

[0018] ;

[0019] ;

[0020] ;

[0021] in, Represents lidar imagery. Indicates near-infrared image, ResNet50 represents the processing branch of LiDAR. and These represent the first branch of lidar processing. Convolution operations and convolution weights for each convolutional block and They represent the first The level and the first Features of LiDAR at various levels ResNet50 represents the near-infrared processing branch. and These represent the first branch of near-infrared processing. Convolution operations and convolution weights for each convolutional block and They represent the first The level and the first Near-infrared features at various levels.

[0022] Optionally, based on the cross-attention mechanism, features from different levels of lidar features and different levels of near-infrared features are stitched together to obtain joint multimodal features, including:

[0023] For the LiDAR characteristics at each level Applying a 1×1 convolution, we obtain query vectors with the same number of channels. Key vector Sum value vector ;

[0024] For the Near-infrared features at various levels Similarly, applying a 1×1 convolution yields a query vector with the same number of channels. Key vector Sum value vector ;

[0025] based on and Calculate the first The self-attention characteristics of LiDAR at each level, and based on and Calculate the first Near-infrared self-attention features at various levels;

[0026] No. The self-attention characteristics of lidar at each level and the first level The near-infrared self-attention features at each level are expressed by the following formulas:

[0027] ;

[0028] ;

[0029] in, This represents the Softmax function. This indicates the calculation of the Kronecker product, indicated by a superscript. This indicates the transpose operation. Indicates the first The self-attention characteristics of lidar at various levels, Indicates the first Near-infrared self-attention features at various levels;

[0030] based on and Calculate the first The near-infrared mutual attention characteristics of lidar at each level, and based on and Calculate the first Mutual attention characteristics of near-infrared lidar at multiple levels;

[0031] No. The near-infrared mutual attention characteristics of lidar at each level and the first The mutual attention characteristics of near-infrared lidar at each level are expressed by the following formulas:

[0032] ;

[0033] ;

[0034] in, This indicates the calculation of the Hadama product. Indicates the first The near-infrared mutual attention characteristics of lidar at various levels. Indicates the first Mutual attention characteristics of near-infrared lidar at multiple levels;

[0035] Based on the following formula, , , and Calculate the first Joint features at each level:

[0036] ;

[0037] in, , , and They represent , , and The joint weighting coefficients, Indicates the first Joint features at each level;

[0038] The dimensions of the joint features at different levels are unified, and the unified joint features are then concatenated along the channel dimension to obtain the joint multimodal features. .

[0039] Alternatively, assume that the total number of semantic categories is Then it is initialized through a learnable prototype embedding module. There are 3 learnable prototypes, each representing a semantic category, and each learnable prototype has 1 dimension. , record the A learnable prototype is , ;

[0040] Calculate the cosine similarity between the joint multimodal features and each learnable prototype. Use the learnable prototype with the highest cosine similarity as the matching learnable prototype corresponding to the joint multimodal features. Fuse the joint multimodal features and the matching learnable prototypes to obtain the fused features, including:

[0041] The cosine similarity between the joint multimodal features and each learnable prototype is calculated using the following formula:

[0042] ;

[0043] in, Indicates modulo, Indicates joint multimodal features and the first Cosine similarity between learnable prototypes;

[0044] The learnable prototype with the highest cosine similarity is used as... The corresponding matching learnable prototype The following formula is used to... and By performing fusion, the fusion characteristics are obtained:

[0045] ;

[0046] in, The preset fusion weight coefficients, This indicates the fusion feature.

[0047] Optionally, the fused features are input into a dilated convolutional pyramid pooling module to capture information from different receptive fields, enhance contextual understanding, and obtain advanced fused features, including:

[0048] Will The input is fed into the dilated convolutional pyramid pooling module, where it is processed sequentially through a 1×1 convolutional layer, a BN normalization layer, and a ReLU activation function layer. Multiple dilated convolutional branches are then used with parallel dilated convolutional layers of different dilation rates to capture information from different receptive fields, enhancing contextual understanding without reducing resolution. The outputs of all dilated convolutional branches are then concatenated and processed sequentially through a 1×1 convolutional layer, a BN normalization layer, a ReLU activation function layer, and a dropout layer to finally obtain the advanced fusion features. .

[0049] Optionally, the decoder consists of two convolutional layers and one softmax layer;

[0050] The first convolutional layer is used to upsample the high-level fusion features based on bilinear interpolation to restore them to the same size as the LiDAR image. The kernel size of the first convolutional layer is 3×3, the stride is 1, the padding is 1, and the number of output channels is 256.

[0051] The second convolutional layer is used for semantic segmentation based on the output of the first convolutional layer. It only changes the number of channels but not the spatial dimension, and obtains the predicted probability for each semantic category. The size of the convolutional kernel of the second convolutional layer is 1×1, and the number of output channels is the same as the number of semantic categories.

[0052] The Softmax layer is used to obtain the semantic segmentation result based on the predicted probability for each semantic category output by the second convolutional layer.

[0053] Optionally, the dual-branch feature extraction module, multi-level feature fusion module, learnable prototype embedding module, dilated convolution pyramid pooling module, and decoder together constitute the multimodal semantic segmentation model. The training set used to train the multimodal semantic segmentation model is constructed through the following steps:

[0054] Acquire a large number of paired lidar sample images and near-infrared sample images;

[0055] By using a fixed step size and fixed size image cropping method, each pair of LiDAR sample images and near-infrared sample images are cropped and segmented to obtain several pairs of training samples, which are then labeled to form the original training set; where the labeled tags are used to represent semantic categories.

[0056] Penalty weights are calculated for each semantic category in the original training set to obtain the corresponding penalty weights for each semantic category. The corresponding penalty weights are applied to each semantic category to avoid the problem of data imbalance, resulting in a balanced training set. The balanced training set is then used to train the multimodal semantic segmentation model.

[0057] The penalty weight calculation process is expressed by the formula:

[0058] ;

[0059] ;

[0060] in, Indicates belonging to the first The proportion of pixels in each semantic category to the total number of pixels. Indicates the first Penalty weights for each semantic category before normalization. This represents the maximum value of the penalty weights for each semantic category before normalization. Indicates the first The normalized penalty weights for each semantic category.

[0061] Optionally, the training of the multimodal semantic segmentation model is implemented based on the deep learning framework PyTorch. The Adam algorithm is used for optimization during training. The initial learning rate is set to 0.001. When the error rate stops decreasing, the current learning rate is divided by 10, and learning continues based on the new learning rate. The loss function is cross-entropy loss.

[0062] After obtaining the trained multimodal semantic segmentation model, it was tested using novel pairs of LiDAR and near-infrared images that the multimodal semantic segmentation model had never seen before, and the performance was evaluated based on evaluation metrics including accuracy, IoU and kappa coefficient.

[0063] Once the trained multimodal semantic segmentation model passes the performance evaluation, it is deployed to perform semantic segmentation tasks. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0065] Figure 1 This is a flowchart of a semantic segmentation method based on a learnable prototype for LiDAR images provided in one embodiment of this application;

[0066] Figure 2 This is a schematic diagram of the structure of a multimodal semantic segmentation model provided in one embodiment of this application;

[0067] Figure 3 This is a schematic diagram of the structure of a convolution block provided in one embodiment of this application;

[0068] Figure 4 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0070] One embodiment of this application proposes a learnable prototype-based semantic segmentation method for LiDAR images, implemented based on multimodal images, and applied to electronic devices. The electronic device can be a terminal or a server; this embodiment and subsequent embodiments will use a server as an example. The implementation details of the learnable prototype-based semantic segmentation method for LiDAR images proposed in this embodiment are described below. These details are provided for ease of understanding only and are not essential for implementing this solution.

[0071] The specific process of the semantic segmentation method based on learnable prototypes for LiDAR images proposed in this embodiment can be as follows: Figure 1 As shown, it includes:

[0072] Step 11: Acquire paired lidar images and near-infrared images. Input the lidar images and near-infrared images into the lidar processing branch and near-infrared processing branch of the dual-branch feature extraction module to perform feature extraction at different levels, thereby obtaining lidar features and near-infrared features at different levels.

[0073] In the specific implementation, paired LiDAR images and near-infrared images are the basis for semantic segmentation. The server acquires paired LiDAR images and near-infrared images captured facing the target scene, and inputs the LiDAR images and near-infrared images into the LiDAR processing branch and near-infrared processing branch of the dual-branch feature extraction module, respectively, to perform feature extraction at different levels (the LiDAR processing branch performs feature extraction at different levels on the LiDAR images, and the near-infrared processing branch performs feature extraction at different levels on the near-infrared images), resulting in LiDAR features and near-infrared features at different levels.

[0074] In one example, a multi-branch feature extraction module, a multi-level feature fusion module, a learnable prototype embedding module, a dilated convolutional pyramid pooling module, and a decoder together constitute a multimodal semantic segmentation model. The specific structure of the multimodal semantic segmentation model is as follows: Figure 2 As shown, the LiDAR processing branch and near-infrared processing branch of the dual-branch feature extraction module have the same structure, both consisting of a cascaded ResNet50 and five convolutional blocks. The specific structure of the convolutional block is shown below. Figure 3 As shown, each convolutional block consists of a series of convolutional layers, BN normalization layers, ReLU activation function layers, and max pooling layers.

[0075] In one example, let the five convolutional blocks be the first to the fifth levels. Then, the lidar features at different levels and the near-infrared features at different levels can be expressed by the following formula:

[0076] ;

[0077] ;

[0078] ;

[0079] in, Represents lidar imagery. Indicates near-infrared image, ResNet50 represents the processing branch of LiDAR. and These represent the first branch of lidar processing. Convolution operations and convolution weights for each convolutional block and They represent the first The level and the first Features of LiDAR at various levels ResNet50 represents the near-infrared processing branch. and These represent the first branch of near-infrared processing. Convolution operations and convolution weights for each convolutional block and They represent the first The level and the first Near-infrared features at various levels.

[0080] In one example, the ResNet50 output features are 64×64×64 in size. The five convolutional blocks each have a different downsample (True or False) parameter to indicate whether to add a max-pooling layer to reduce the spatial size of the features. The first convolutional block has its downsample parameter set to False, resulting in an output feature size of 64×64×256. The second convolutional block has its downsample parameter set to True, resulting in an output feature size of 32×32×512. The third convolutional block has its downsample parameter set to False, resulting in an output feature size of 32×32×1024. The fourth convolutional block has its downsample parameter set to False, resulting in an output feature size of 32×32×2048. The fifth convolutional block uses a 1×1 kernel with a stride of 1 and zero padding, resulting in an output feature size of 32×32×256.

[0081] Step 12: Using a multi-level feature fusion module, based on a cross-attention mechanism, feature stitching is performed on lidar features and near-infrared features at different levels to obtain joint multi-modal features.

[0082] In the specific implementation, the lidar features and near-infrared features extracted by the dual-branch feature extraction module at different levels will enter the multi-level feature fusion module. The multi-level feature fusion module, based on the cross-attention mechanism, performs feature stitching on lidar features and near-infrared features at different levels to obtain joint multimodal features.

[0083] In one example, for the first LiDAR characteristics at each level In this regard, the multi-level feature fusion module applies a 1×1 convolution to obtain a query vector with the same number of channels. Key vector Sum value vector For the first Near-infrared features at various levels In contrast, the multi-level feature fusion module also applies a 1×1 convolution to it, resulting in a query vector with the same number of channels. Key vector Sum value vector Subsequently based on and Calculate the first The self-attention characteristics of LiDAR at each level, and based on and Calculate the first Near-infrared self-attention features at various levels.

[0084] No. The self-attention characteristics of lidar at each level and the first level The near-infrared self-attention features at each level can be expressed by the following formulas:

[0085] ;

[0086] ;

[0087] in, This represents the Softmax function. This indicates the calculation of the Kronecker product, indicated by a superscript. This indicates the transpose operation. Indicates the first The self-attention characteristics of lidar at various levels, Indicates the first Near-infrared self-attention features at various levels.

[0088] In addition to self-attention, the multi-level feature fusion module will also apply mutual attention, that is, based on... and Calculate the first The near-infrared mutual attention characteristics of lidar at each level, and based on and Calculate the first The mutual attention features of near-infrared lidar at multiple levels. Mutual attention can capture long-range dependencies between different modalities, effectively model the context, and treat the entire image as the observation range, enhancing the contextual information between long-distance pixels.

[0089] No. The near-infrared mutual attention characteristics of lidar at each level and the first The mutual attention characteristics of near-infrared lidar at each level can be expressed by the following formulas:

[0090] ;

[0091] ;

[0092] in, This indicates the calculation of the Hadama product. Indicates the first The near-infrared mutual attention characteristics of lidar at various levels. Indicates the first Mutual attention characteristics of near-infrared lidar at multiple levels.

[0093] Subsequently, the multi-level feature fusion module uses the following formula, based on , , and Calculate the first Joint features at each level:

[0094] ;

[0095] in, , , and They represent , , and The joint weighting coefficients, Indicates the first Joint features at each level.

[0096] Finally, the multi-level feature fusion module unifies the dimensions of the joint features at different levels, and then concatenates the dimension-unified joint features along the channel dimension to obtain the joint multimodal features. .

[0097] Step 13: Initialize a set of learnable prototypes through the learnable prototype embedding module. Each learnable prototype represents a semantic category. Calculate the cosine similarity between the joint multimodal features and each learnable prototype. Use the learnable prototype with the highest cosine similarity as the matching learnable prototype corresponding to the joint multimodal features. Fuse the joint multimodal features and the matching learnable prototypes to obtain the fused features.

[0098] In its implementation, the multi-level feature fusion module inputs the joint multimodal features into the learnable prototype embedding module. The learnable prototype embedding module initializes a set of learnable prototypes, each of which represents a semantic category. The cosine similarity between the joint multimodal features and each learnable prototype is calculated. The learnable prototype with the highest cosine similarity is taken as the matching learnable prototype corresponding to the joint multimodal features. The joint multimodal features and the matching learnable prototypes are fused to obtain fused features, ensuring that the multimodal semantic segmentation model can accurately extract ground feature information from different modalities.

[0099] In one example, suppose the total number of semantic categories is Therefore, it is necessary to initialize through a learnable prototype embedding module. There are 3 learnable prototypes, each representing a semantic category, and each learnable prototype has 1 dimension. , record the A learnable prototype is , .

[0100] The learnable prototype embedding module calculates the cosine similarity between the joint multimodal features and each learnable prototype using the following formula:

[0101] ;

[0102] in, Indicates modulo, Indicates joint multimodal features and the first Cosine similarity between learnable prototypes.

[0103] In calculation After calculating the cosine similarity between each learnable prototype, the learnable prototype embedding module uses the learnable prototype with the highest cosine similarity as... The corresponding matching learnable prototype The following formula is used to... and By performing fusion, the fusion characteristics are obtained:

[0104] ;

[0105] in, The preset fusion weight coefficients, This indicates the fusion feature.

[0106] Step 14: Input the fused features into the dilated convolutional pyramid pooling module to capture information from different receptive fields, enhance the level of contextual understanding, and obtain advanced fused features.

[0107] In its implementation, the learnable prototype embedding module inputs the fused features into the dilated convolutional pyramid pooling module, capturing information from different receptive fields, enhancing contextual understanding, and obtaining advanced fused features. The dilated convolutional pyramid pooling module can effectively compensate for the limitations of convolutional networks, which is very useful for processing terrain features with various scale variations (such as roads, buildings, forests, etc.).

[0108] In one example After being input into the dilated convolution pyramid pooling module, the data passes through a 1×1 convolutional layer, a BN normalization layer, and a ReLU activation function layer in sequence. It then enters multiple dilated convolutional branches, where it is processed by parallel dilated convolutional layers with different dilation rates to capture information from different receptive fields, enhancing contextual understanding without reducing resolution. Subsequently, the outputs of all dilated convolutional branches are concatenated and then processed again through a 1×1 convolutional layer, a BN normalization layer, a ReLU activation function layer, and a dropout layer for final processing, ultimately yielding high-level fusion features. .

[0109] Step 15: Upsample the advanced fusion features based on bilinear interpolation using the decoder to restore them to the same size as the LiDAR image, and then perform semantic segmentation to obtain the semantic segmentation result.

[0110] In the specific implementation, the dilated convolutional pyramid pooling module inputs the high-level fusion features into the decoder for final decoding processing. The decoder upsamples the high-level fusion features based on bilinear interpolation to restore them to the same size as the LiDAR image, and performs semantic segmentation to obtain the semantic segmentation result.

[0111] In one example, the decoder consists of two convolutional layers and a softmax layer.

[0112] The first convolutional layer is used to upsample the high-level fusion features based on bilinear interpolation to restore them to the same size as the LiDAR image. The kernel size of the first convolutional layer is 3×3, the stride is 1, the padding is 1, and the number of output channels is 256.

[0113] The second convolutional layer is used for semantic segmentation based on the output of the first convolutional layer. It only changes the number of channels but not the spatial dimension, and obtains the predicted probability for each semantic category. The size of the convolutional kernel of the second convolutional layer is 1×1, and the number of output channels is the same as the number of semantic categories.

[0114] The Softmax layer is used to obtain the semantic segmentation result based on the predicted probability for each semantic category output by the second convolutional layer.

[0115] In one example, when training a multimodal semantic segmentation model, it is necessary to acquire a large number of pairs of LiDAR sample images and near-infrared sample images. By using an image cropping method with a fixed step size and a fixed size, each pair of LiDAR sample images and near-infrared sample images is cropped and segmented to obtain several pairs of training samples, which are then labeled to form the original training set. The labels used to label the training samples are used to represent the semantic category.

[0116] It is important to note that the data for each semantic category in the original training set may be imbalanced. Therefore, it is necessary to calculate the penalty weight for each semantic category in the original training set separately to obtain the penalty weight corresponding to each semantic category. Apply the corresponding penalty weight to each semantic category to avoid the problem of data imbalance and obtain a balanced training set. The balanced training set is then used to train the multimodal semantic segmentation model.

[0117] In one example, the penalty weight calculation process can be expressed by the formula:

[0118] ;

[0119] ;

[0120] in, Indicates belonging to the first The proportion of pixels in each semantic category to the total number of pixels. Indicates the first Penalty weights for each semantic category before normalization. This represents the maximum value of the penalty weights for each semantic category before normalization. Indicates the first The normalized penalty weights for each semantic category can be understood as follows: The lower the semantic category, the higher the penalty weight will be.

[0121] In one example, training a multimodal semantic segmentation model can be implemented using the deep learning framework PyTorch. During training, the Adam algorithm is used for optimization, with the initial learning rate set to 0.001. When the error rate stops decreasing, the current learning rate is divided by 10, and training continues based on the new learning rate. The loss function can be cross-entropy loss.

[0122] In one example, after obtaining the trained multimodal semantic segmentation model, the server will test it using novel pairs of LiDAR and near-infrared images that the model has never seen before, evaluating its performance based on metrics including accuracy, IoU, and kappa coefficient. If the trained multimodal semantic segmentation model passes the performance evaluation, it will be deployed to perform semantic segmentation tasks.

[0123] This embodiment proposes a semantic segmentation method based on learnable prototypes for LiDAR imagery, which fully leverages the advantages of both LiDAR and near-infrared (NIIR) imagery, providing unique advantages for understanding and analyzing surface scenes. LiDAR imagery provides high-resolution visual information, including point clouds, micro-topographic details, and terrain undulations, while NIIR imagery provides stable radar echo information unaffected by weather and lighting conditions. The combination of these two imagery modalities enables surface monitoring in complex environments (such as at night or in severe weather), which is of significant value for applications such as environmental monitoring and disaster assessment. To fuse these two imagery modalities, this application designs a multimodal semantic segmentation model consisting of a bi-branch feature extraction module, a multi-level feature fusion module, a learnable prototype embedding module, a dilated convolutional pyramid pooling module, and a decoder to perform the semantic segmentation task. The bi-branch feature extraction module can effectively extract different levels of LiDAR and NIIR features. The multi-level feature fusion module, based on a cross-attention mechanism, concatenates features from different levels of LiDAR and near-infrared technologies to obtain joint multi-modal features, effectively establishing the interrelationship between the two modalities of the image. The learnable prototype embedding module effectively establishes semantic category representations using learnable prototypes. The introduction of dilated convolutional pyramid pooling modules expands the receptive region, effectively enhancing the extraction and utilization of contextual information. Through the collaborative work of learnable prototypes and multi-modal features, the multi-modal semantic segmentation model significantly improves the accuracy and efficiency of semantic segmentation.

[0124] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0125] Another embodiment of this application provides an electronic device with the following structure: Figure 4 As shown, it includes: at least one processor 21; and a memory 22 communicatively connected to the at least one processor 21; wherein the memory 22 stores instructions executable by the at least one processor 21, the instructions being executed by the at least one processor 21 to enable the at least one processor 21 to perform a semantic segmentation method for lidar images based on a learnable prototype as described in the above method embodiment.

[0126] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0127] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0128] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a semantic segmentation method for lidar images based on a learnable prototype as described in the above method embodiments.

[0129] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0130] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A semantic segmentation method based on learnable prototypes for lidar images, implemented based on multimodal images, characterized in that, The method includes: Acquire pairs of lidar images and near-infrared images, and input the lidar images and near-infrared images into the lidar processing branch and near-infrared processing branch of the dual-branch feature extraction module to perform feature extraction at different levels, thereby obtaining lidar features at different levels and near-infrared features at different levels. Through a multi-level feature fusion module, based on a cross-attention mechanism, features from different levels of lidar and near-infrared features are stitched together to obtain joint multi-modal features. A set of learnable prototypes is initialized through the learnable prototype embedding module. Each learnable prototype represents a semantic category. The cosine similarity between the joint multimodal features and each learnable prototype is calculated. The learnable prototype with the highest cosine similarity is taken as the matching learnable prototype corresponding to the joint multimodal features. The joint multimodal features and the matching learnable prototypes are fused to obtain the fused features. The fused features are input into the dilated convolutional pyramid pooling module to capture information from different receptive fields, enhance the level of contextual understanding, and obtain advanced fused features; The high-level fusion features are upsampled using bilinear interpolation by the decoder to restore them to the same size as the LiDAR image, and then semantic segmentation is performed to obtain the semantic segmentation result.

2. The semantic segmentation method based on learnable prototypes for lidar images according to claim 1, characterized in that, The LiDAR processing branch and the near-infrared processing branch of the dual-branch feature extraction module have the same structure. They are both composed of a ResNet50 and five convolutional blocks in series. Each convolutional block consists of a series of convolutional layers, BN normalization layers, ReLU activation function layers and max pooling layers. Let the five convolutional blocks represent the first to fifth levels, then the lidar features and near-infrared features at different levels can be expressed by the following formulas: ； ； ； in, Represents lidar imagery. Indicates near-infrared image, ResNet50 represents the processing branch of LiDAR. and These represent the first branch of lidar processing. Convolution operations and convolution weights for each convolutional block and They represent the first The level and the first Features of LiDAR at various levels ResNet50 represents the near-infrared processing branch. and These represent the first branch of near-infrared processing. Convolution operations and convolution weights for each convolutional block and They represent the first The level and the first Near-infrared features at various levels.

3. The semantic segmentation method based on learnable prototypes for lidar images according to claim 2, characterized in that, Based on the cross-attention mechanism, features from different levels of lidar features and near-infrared features are stitched together to obtain joint multimodal features, including: For the LiDAR characteristics at each level Applying a 1×1 convolution, we obtain query vectors with the same number of channels. Key vector Sum value vector ; For the Near-infrared features at various levels Similarly, applying a 1×1 convolution yields a query vector with the same number of channels. Key vector Sum value vector ; based on and Calculate the first The self-attention characteristics of LiDAR at each level, and based on and Calculate the first Near-infrared self-attention features at various levels; No. The self-attention characteristics of lidar at each level and the first level The near-infrared self-attention features at each level are expressed by the following formulas: ； ； in, This represents the Softmax function. This indicates the calculation of the Kronecker product, indicated by a superscript. This indicates the transpose operation. Indicates the first The self-attention characteristics of lidar at various levels, Indicates the first Near-infrared self-attention features at various levels; based on and Calculate the first The near-infrared mutual attention characteristics of lidar at each level, and based on and Calculate the first Mutual attention characteristics of near-infrared lidar at multiple levels; No. The near-infrared mutual attention characteristics of lidar at each level and the first The mutual attention characteristics of near-infrared lidar at each level are expressed by the following formulas: ； ； in, This indicates the calculation of the Hadama product. Indicates the first The near-infrared mutual attention characteristics of lidar at various levels. Indicates the first Mutual attention characteristics of near-infrared lidar at multiple levels; Based on the following formula, , , and Calculate the first Joint features at each level: ； in, , , and They represent , , and The joint weighting coefficients, Indicates the first Joint features at each level; The dimensions of the joint features at different levels are unified, and the unified joint features are then concatenated along the channel dimension to obtain the joint multimodal features. .

4. A semantic segmentation method based on learnable prototypes for lidar images according to claim 3, characterized in that, Assume the total number of semantic categories is Then it is initialized through a learnable prototype embedding module. There are 3 learnable prototypes, each representing a semantic category, and each learnable prototype has 1 dimension. , record the A learnable prototype is , ; Calculate the cosine similarity between the joint multimodal features and each learnable prototype. Use the learnable prototype with the highest cosine similarity as the matching learnable prototype corresponding to the joint multimodal features. Fuse the joint multimodal features and the matching learnable prototypes to obtain the fused features, including: The cosine similarity between the joint multimodal features and each learnable prototype is calculated using the following formula: ； in, Indicates modulo, Indicates joint multimodal features and the first Cosine similarity between learnable prototypes; The learnable prototype with the highest cosine similarity is used as... The corresponding matching learnable prototype The following formula is used to... and By performing fusion, the fusion characteristics are obtained: ； in, The preset fusion weight coefficients, This indicates the fusion feature.

5. A semantic segmentation method based on learnable prototypes for lidar images according to claim 4, characterized in that, The fused features are input into the dilated convolutional pyramid pooling module to capture information from different receptive fields, enhance contextual understanding, and obtain high-level fused features, including: Will The input is fed into the dilated convolutional pyramid pooling module, where it is processed sequentially through a 1×1 convolutional layer, a BN normalization layer, and a ReLU activation function layer. Multiple dilated convolutional branches are then used with parallel dilated convolutional layers of different dilation rates to capture information from different receptive fields, enhancing contextual understanding without reducing resolution. The outputs of all dilated convolutional branches are then concatenated and processed sequentially through a 1×1 convolutional layer, a BN normalization layer, a ReLU activation function layer, and a dropout layer to finally obtain the advanced fusion features. .

6. A semantic segmentation method based on learnable prototypes for lidar images according to claim 5, characterized in that, The decoder consists of two convolutional layers and one softmax layer; The first convolutional layer is used to upsample the high-level fusion features based on bilinear interpolation to restore them to the same size as the LiDAR image. The kernel size of the first convolutional layer is 3×3, the stride is 1, the padding is 1, and the number of output channels is 256. The second convolutional layer is used for semantic segmentation based on the output of the first convolutional layer. It only changes the number of channels but not the spatial dimension, and obtains the predicted probability for each semantic category. The size of the convolutional kernel of the second convolutional layer is 1×1, and the number of output channels is the same as the number of semantic categories. The Softmax layer is used to obtain the semantic segmentation result based on the predicted probability for each semantic category output by the second convolutional layer.

7. A semantic segmentation method for lidar images based on learnable prototypes according to any one of claims 1 to 6, wherein a dual-branch feature extraction module, a multi-level feature fusion module, a learnable prototype embedding module, a dilated convolutional pyramid pooling module, and a decoder together constitute a multimodal semantic segmentation model, and the training set used for training the multimodal semantic segmentation model is constructed through the following steps: Acquire a large number of paired lidar sample images and near-infrared sample images; Using a fixed-step, fixed-size image cropping method, each pair of LiDAR sample images and near-infrared sample images is cropped and segmented to obtain several pairs of training samples, which are then labeled to form the original training set; among them, The labels used in the annotations are used to represent semantic categories; Penalty weights are calculated for each semantic category in the original training set to obtain the corresponding penalty weights for each semantic category. The corresponding penalty weights are applied to each semantic category to avoid the problem of data imbalance, resulting in a balanced training set. The balanced training set is then used to train the multimodal semantic segmentation model. The penalty weight calculation process is expressed by the formula: ； ； in, Indicates belonging to the first The proportion of pixels in each semantic category to the total number of pixels. Indicates the first Penalty weights for each semantic category before normalization. This represents the maximum value of the penalty weights for each semantic category before normalization. Indicates the first The normalized penalty weights for each semantic category.

8. The semantic segmentation method based on learnable prototype for LiDAR images according to claim 7, wherein the training of the multimodal semantic segmentation model is implemented based on the deep learning framework PyTorch, and the Adam algorithm is used for optimization during training. The initial learning rate is set to 0.

001. When the error rate stops decreasing, the current learning rate is divided by 10, and learning continues based on the new learning rate. The loss function is cross-entropy loss. After obtaining the trained multimodal semantic segmentation model, it was tested using novel pairs of LiDAR and near-infrared images that the multimodal semantic segmentation model had never seen before, and the performance was evaluated based on evaluation metrics including accuracy, IoU and kappa coefficient. Once the trained multimodal semantic segmentation model passes the performance evaluation, it is deployed to perform semantic segmentation tasks.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform a learnable prototype-based semantic segmentation method for lidar images as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a semantic segmentation method for lidar images based on a learnable prototype as described in any one of claims 1 to 7.

Images