A remote sensing semantic segmentation method fusing optical images and lidar point clouds

Abstract

The application discloses a remote sensing semantic segmentation method fusing optical images and laser radar point clouds, and comprises the following steps: acquiring multiple three-dimensional laser radar point cloud data of a target region; processing each three-dimensional laser radar point cloud data by adopting a kernel point convolution network to obtain laser radar point cloud data features of each level of a branch encoder; performing bird's-eye view projection on the laser radar point cloud data features to obtain laser radar features of each level of each branch encoder; acquiring an optical image of the target region, and performing multi-level feature extraction on the optical image to obtain RGB features of the optical image; performing feature fusion processing on the laser radar features and the matched RGB features by an L-AAM module to obtain fused RGB features; performing upsampling on the RGB features of each level after fusion respectively, and splicing the sampling results of each level to obtain a semantic segmentation result. The method avoids information waste of the three-dimensional laser radar point cloud data.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition and machine learning technology, and in particular, it is a remote sensing semantic segmentation method that integrates optical images and lidar point clouds. Background Technology

[0002] With the continuous development of satellite technology, remote sensing satellite imagery has been widely used in various fields. Many applications require semantic segmentation of remote sensing satellite imagery, which involves classifying different features within the image into different categories. This process is usually automated and relies on technologies such as computer vision and machine learning.

[0003] In recent years, significant progress has been made in this field with the emergence of deep learning and convolutional neural networks (CNNs). Various CNN-based models for semantic segmentation of 2D images have been developed, such as fully convolutional networks (FCNs), U-Net, PSPNet, and SegNet. However, these CNN-based networks do not focus on the correlation between global and local features. To overcome this limitation, ViT applied the Transformer architecture, originally developed for natural language processing, to the field of computer vision, and subsequently made a series of improvements, including SegFormer, SwinTransformer, and Mask2Former, which have shown significant improvements in segmentation accuracy.

[0004] In addition to 2D image segmentation, semantic segmentation of 3D LiDAR point cloud data has also developed rapidly due to the emergence of 3D convolution, such as PointNet and Kernel Point Convolution (KPConv). These methods have made significant progress in processing 3D point cloud data.

[0005] The fusion of 2D optical images and 3D LiDAR point cloud data has been shown to improve the accuracy of semantic segmentation. Optical images provide rich spectral information, while 3D LiDAR point cloud data provides depth information. Traditional aerial image and LiDAR data fusion uses a digital surface model (DSM) or other information obtained from the LiDAR point cloud and fuses the optical image by adding a depth channel. However, adding DSM information from the 3D LiDAR point cloud results in some information loss.

[0006] Existing methods for remote sensing image land cover classification using LiDAR fusion optical images typically employ image data along with single-channel depth information, such as DSM, DEM, nDSM, etc. However, this approach fails to fully utilize the richer information from LiDAR point cloud data.

[0007] Therefore, how to effectively avoid the waste of information in 3D lidar point cloud data has become a key research issue. Summary of the Invention

[0008] In view of the above problems, the present invention provides a remote sensing semantic segmentation method that integrates optical images and lidar point clouds to at least solve some of the above technical problems.

[0009] This invention provides a remote sensing semantic segmentation method that fuses optical images and lidar point clouds, comprising the following steps:

[0010] S1. Acquire multiple 3D LiDAR point cloud data of the target area;

[0011] S2. A kernel-point convolutional network is used to process each of the three-dimensional lidar point cloud data to obtain the lidar point cloud data features of each level of the encoder in this branch.

[0012] S3. Project a bird's-eye view onto the features of the lidar point cloud data to obtain the lidar features of each level of each branch encoder.

[0013] S4. Obtain the optical image of the target area, and perform multi-level feature extraction on the optical image to obtain the RGB features of the optical image;

[0014] S5. The lidar features and the matching RGB features are fused using the L-AAM module to obtain the fused RGB features;

[0015] S6. Upsample the RGB features after fusion at each level, and concatenate the sampling results of each level to obtain the semantic segmentation result.

[0016] Furthermore, in step S4, the Transformer module is used to extract multi-level features from the optical image.

[0017] Further, step S5 specifically includes:

[0018] S51. Connect the lidar features and the RGB features in the channel dimension;

[0019] S52. Input the concatenated features as input features into the L-AAM module;

[0020] S53. In the L-AAM module, the input features in the laser radar feature enhancement channel dimension are used;

[0021] S54. The enhanced input features are processed through a self-attention mechanism to obtain an attention matrix;

[0022] S55. Based on the enhanced input features and the attention matrix, slice the data to obtain enhanced RGB features, and use the enhanced RGB features as the fused RGB features.

[0023] Further, step S53 specifically includes:

[0024] S531. The input features are compressed using global pooling to obtain a compression result;

[0025] S532. The compression result is processed through multiple fully connected layers to obtain an excitation result;

[0026] S533. The excitation result is processed by the Sigmoid function, and the processing result of the excitation result is applied to the input feature by point-by-point multiplication to obtain the original feature;

[0027] S534. Slice the original features to half the original feature dimension, and concatenate the slicing result with the lidar features to obtain the enhanced input features.

[0028] Further, step S54 specifically includes:

[0029] Fine-grained linear transformation is performed on the enhanced input features based on a self-attention mechanism;

[0030] Based on the transformation results, the attention matrix is ​​obtained through the softmax function.

[0031] Furthermore, in step S6, a multilayer perceptron is used to upsample the RGB features fused at each level to obtain semantic labels.

[0032] Compared with the prior art, the remote sensing semantic segmentation method that integrates optical images and lidar point clouds described in this invention has the following beneficial effects: Since the lidar features obtained by kernel point convolution processing and projection are sparse, this invention directly integrates the regions of interest of RGB features and lidar features through the L-AAM module, effectively avoiding the waste of information in the three-dimensional lidar point cloud data.

[0033] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0034] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0035] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0036] Figure 1 This is a schematic diagram of the framework of the remote sensing semantic segmentation method that fuses optical images and lidar point clouds, as provided in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the L-AAM module framework provided in an embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of the classification results of the GRSS18 dataset provided in an embodiment of the present invention. Detailed Implementation

[0039] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0040] See Figure 1 As shown, this embodiment of the invention provides a remote sensing semantic segmentation method that fuses optical images and lidar point clouds, specifically including the following steps:

[0041] S1. Acquire multiple 3D LiDAR point cloud data of the target area;

[0042] S2. A kernel-point convolutional network is used to process each 3D LiDAR point cloud data to obtain the LiDAR point cloud data features of each level of the encoder in this branch.

[0043] S3. Project a bird's-eye view onto the features of the lidar point cloud data to obtain the lidar features of each level of each branch encoder.

[0044] S4. Acquire the optical image of the target area and perform multi-level feature extraction on the optical image to obtain the RGB features of the optical image;

[0045] S5. The lidar features and the matching RGB features are fused using the L-AAM module to obtain the fused RGB features.

[0046] S6. Upsample the RGB features after fusion at each level, and concatenate the sampling results of each level to obtain the semantic segmentation result.

[0047] The steps S1-S6 described above are merely for illustrating the method provided in the embodiments of the present invention and do not limit the order of execution of the steps.

[0048] The steps described above will be explained in detail below.

[0049] In step S1 above, semantic labels for three-dimensional LiDAR point cloud data of multiple target areas are generated by projecting a bird's-eye view based on the actual semantic labels; thereby obtaining multiple three-dimensional LiDAR point cloud data of the target areas.

[0050] In step S2 above, a kernel-point convolutional network is used to process each 3D LiDAR point cloud data to obtain the LiDAR point cloud data features of each level of the encoder in this branch.

[0051] In steps S3-S5 above, a bird's-eye view projection is performed on the features of the LiDAR point cloud data to obtain the LiDAR features (LiDAR features) of each level of each branch encoder; a two-dimensional optical image of the target area is acquired, and the Transformer module is used to extract multi-level features from the optical image to obtain RGB features of different granularities at multiple levels of the optical image.

[0052] Because the extracted lidar feature density is low and its importance is lower than that of RGB features in two-dimensional optical remote sensing images, a lidar-assisted attention module (L-AAM module) is provided in this embodiment of the invention. See details below. Figure 2 As shown, the obtained LiDAR features and RGB features are compressed and activated to obtain the importance of feature channels, and a self-attention mechanism is used to obtain better global contextual information. Therefore, the L-AAM module will only retain the fused RGB features to make the model focus more on important RGB features. The fusion process is as follows:

[0053] The LiDAR features and RGB features are concatenated along the channel dimension; the concatenated features are used as the input features of the L-AAM module; represented as:

[0054] x = concat(l, L) where x represents the input feature, x ∈ R H×W×C R represents a real number; H, W, and C all represent the dimensional values ​​of the input features; I represents RGB features. L represents the characteristics of a lidar system (LiDAR feature); connect.

[0055] In the L-AAM module:

[0056] First, the input features in the channel dimension are enhanced using LiDAR features; in short, the input requires compression, excitation, and reweighting operations; the specific details are as follows:

[0057] The input features are compressed along the channel dimension using global pooling to obtain the compressed result, which can be expressed by the formula:

[0058]

[0059] Where z represents the compression result, z∈R C i and j represent the indexes of the i-th and j-th positions in each channel of the feature, respectively; u represents any channel of the input feature x, u∈R H×W ;

[0060] The compression result is subjected to excitation processing through multiple fully connected layers (two fully connected layers in this embodiment of the invention) to obtain the excitation result, which is expressed as:

[0061] s=σ(W2σ(W1z))

[0062] Where s represents the excitation result, s∈R C σ represents the ReLU function.

[0063] The excitation results are processed using the Sigmoid function and pointwise multiplication. Specifically, the output size is adjusted, and the processed result is applied to the input features using pointwise multiplication to obtain the original features. The original features are then sliced ​​to half their original dimension, and the sliced ​​results are concatenated with the LiDAR features to obtain the enhanced input features. Represented as:

[0064]

[0065] in,

[0066] Secondly, the enhanced input features are processed through a self-attention mechanism to obtain an attention matrix; specifically, this includes the following:

[0067] A fine-grained linear transformation is performed on the enhanced input features based on a self-attention mechanism; the self-attention operation can be represented as:

[0068]

[0069] Where W1, W2, and W3 represent the learning weight matrices of the three linear layers Q, K, and V, respectively; Query(Q), Key(K), and Value(V) represent fine-grained linear transformations over all features.

[0070] Q, K, and V are passed to the following attention component, which calculates the attention matrix as follows:

[0071]

[0072] Where, d k Represents the enhanced input features The dimension;

[0073] Finally, based on the enhanced input features and attention matrix, the slice function is used to obtain the enhanced RGB features. and enhanced RGB features As the fused RGB feature, it is represented as:

[0074]

[0075] Among them, the features of self-attention module enhancement are:

[0076] Through the above steps, RGB features enhanced based on LiDAR point cloud data were obtained.

[0077] In step S6 above, a multilayer perceptron is used to upsample the densely fused RGB features at each level and then concatenates the sampling results of each level to obtain the semantic segmentation result.

[0078] This invention provides a remote sensing semantic segmentation method that integrates optical images and lidar point clouds. It involves a remote sensing semantic segmentation model based on two-dimensional optical images and three-dimensional lidar point cloud data, and a lidar-assisted attention module suitable for this task. To verify the technical effect of the method provided in this invention, a common remote sensing classification dataset, namely the GRSS18 dataset, is used for verification.

[0079] The sensors used to acquire this GRSS18 dataset include the Optech Titan MW (14SEN / CON340), a multispectral lidar sensor with an integrated camera that operates at three different laser wavelengths, the DiMAC ULTRALIGHT (very high resolution color imager), and the ITRES CASI 1500 (hyperspectral imager).

[0080] The GRSS18 dataset includes multispectral lidar point cloud data at 1550nm, 1064nm, and 532nm, first echo intensity gratings for each channel, and DSMs at a 50cm GSD (ground sample distance); hyperspectral data covering the 380-1050nm spectral range with 48 bands at a 1-meter GSD; and high-resolution RGB images at a 5-cm GSD. Experimental results from embodiments of this invention are as follows: Figure 3 As shown.

[0081] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A remote sensing semantic segmentation method that integrates optical images and lidar point clouds, characterized in that, Includes the following steps: S1. Acquire multiple 3D LiDAR point cloud data of the target area; S2. A kernel-point convolutional network is used to process each of the three-dimensional lidar point cloud data to obtain the lidar point cloud data features of each level of the encoder in this branch. S3. Project a bird's-eye view onto the features of the lidar point cloud data to obtain the lidar features of each level of each branch encoder. S4. Obtain the optical image of the target area, and perform multi-level feature extraction on the optical image to obtain the RGB features of the optical image; S5. The lidar features and the matching RGB features are fused using the L-AAM module to obtain the fused RGB features; S6. Upsample the RGB features after fusion at each level, and concatenate the sampling results of each level to obtain the semantic segmentation result; Step S5 specifically includes: S51. Connect the lidar features and the RGB features in the channel dimension; S52. Input the concatenated features as input features into the L-AAM module; S53. In the L-AAM module, the input features in the laser radar feature enhancement channel dimension are used; S54. The enhanced input features are processed through a self-attention mechanism to obtain an attention matrix; S55. Based on the enhanced input features and the attention matrix, slice the data to obtain enhanced RGB features, and use the enhanced RGB features as the fused RGB features.

2. The remote sensing semantic segmentation method fusing optical images and lidar point clouds as described in claim 1, characterized in that, In step S4, the Transformer module is used to extract multi-level features from the optical image.

3. The remote sensing semantic segmentation method fusing optical images and lidar point clouds as described in claim 1, characterized in that, Step S53 specifically includes: S531. The input features are compressed using global pooling to obtain a compression result; S532. The compression result is processed through multiple fully connected layers to obtain an excitation result; S533. The excitation result is processed by the Sigmoid function, and the processing result of the excitation result is applied to the input feature by point-by-point multiplication to obtain the original feature; S534. Slice the original features to half the original feature dimension, and concatenate the slicing result with the lidar features to obtain the enhanced input features.

4. The remote sensing semantic segmentation method fusing optical images and lidar point clouds as described in claim 1, characterized in that, Step S54 specifically includes: Fine-grained linear transformation is performed on the enhanced input features based on a self-attention mechanism; Based on the transformation results, the attention matrix is ​​obtained through the softmax function.

5. The remote sensing semantic segmentation method fusing optical images and lidar point clouds as described in claim 1, characterized in that, In step S6, a multilayer perceptron is used to upsample the RGB features fused at each level to obtain semantic labels.

Images