A method and system for fusing multi-view aerial oblique image LiDAR point cloud semantic segmentation

Abstract

The application discloses a kind of LiDAR point cloud semantic segmentation method and system of fusion multi-view aerial oblique image.First, construct parallel U-shaped structure backbone feature extraction network, extract multi-scale texture features and spatial features from multi-view aerial oblique image and LiDAR point cloud;On this basis, construct multi-modal data fusion module, use aggregated multi-view aerial oblique image to obtain strongly related texture features to guide attention to LiDAR point cloud features, highlight the key features in LiDAR point cloud;Global joint loss function is designed to optimize model parameters as a whole, the similarity loss of the predicted value of the strongly related pixel label in multiple images and the predicted value of point cloud label is considered, the interference of error matching between different modal features on model training is reduced, and the segmentation accuracy and reliability of ground object point cloud are guaranteed.

Description

Technical Field

[0001] This invention relates to the field of airborne LiDAR point cloud data processing technology, and in particular to an airborne LiDAR point cloud semantic segmentation method and system that fuses multi-view aerial oblique images. Background Technology

[0002] Airborne LiDAR point cloud data is a dense collection of high-precision 3D point coordinates on the ground. Semantic segmentation of it can assign a corresponding land feature category to any point in the irregular 3D point cloud data. LiDAR point cloud semantic segmentation is crucial for understanding large-scale scenes.

[0003] Currently, deep learning methods are widely used in LiDAR point cloud semantic segmentation tasks. Common point cloud semantic segmentation models can be divided into single-modal point cloud segmentation models and point cloud segmentation models that fuse multimodal data. Single-modal point cloud segmentation models rely on the spatial features of point clouds to complete the segmentation task, but they have difficulty distinguishing point cloud regions with similar distributions, and there are still many land cover category confusion phenomena in the segmentation results. Some studies tend to fuse multimodal data to optimize the land cover segmentation results, but due to problems such as the difficulty in focusing key information and incomplete feature matching between modal data feature fusion, the land cover misclassification of LiDAR point clouds is difficult to improve. Summary of the Invention

[0004] This invention mainly addresses the technical problems existing in the prior art by providing a semantic segmentation method for LiDAR point clouds that integrates multi-view aerial oblique imagery, thereby improving the ground feature confusion phenomenon in LiDAR point cloud classification results.

[0005] The above-mentioned technical problems of the present invention are mainly solved by the following technical solution: a LiDAR point cloud semantic segmentation method that fuses multi-view aerial oblique images, comprising the following steps: Step 1: Sample and geometrically register the two modal data, LiDAR point cloud and multi-view aerial oblique image, to obtain LiDAR point cloud and multi-view aerial oblique image samples. Step 2: Construct a LiDAR point cloud semantic segmentation model that integrates multi-view aerial oblique imagery, including a parallel LiDAR point cloud semantic segmentation branch and a multi-view aerial oblique imagery semantic segmentation branch, comprising the following sub-steps: Step 2.1, the semantic segmentation branch of multi-view aerial oblique image specifically includes: constructing a multi-view aerial oblique image feature extraction backbone network to obtain multi-level and multi-scale features of multi-view aerial oblique images, and then obtaining the semantic segmentation result of multi-view aerial oblique images through a convolutional output layer; the input of the multi-view aerial oblique image feature extraction backbone network is multi-view aerial oblique image samples. Step 2.2, the LiDAR point cloud semantic segmentation branch specifically includes: constructing a LiDAR point cloud feature extraction backbone network to obtain multi-level and multi-scale features of the LiDAR point cloud, and fusing the multi-scale features extracted by the multi-view aerial oblique image feature extraction backbone network with the multi-scale features of the LiDAR point cloud to enhance the LiDAR point cloud features. Then, the LiDAR point cloud semantic segmentation result is obtained through a convolutional output layer; the input of the LiDAR point cloud feature extraction backbone network is the LiDAR point cloud sample. Step 3: Calculate the global joint loss function to update the semantic segmentation model parameters; Step 4: Use the trained semantic segmentation model parameters to perform semantic segmentation on the LiDAR point cloud and multi-view aerial oblique image data to be segmented, and map the semantic segmentation results of the LiDAR point cloud samples onto the original LiDAR point cloud to obtain the final global segmentation result.

[0006] Furthermore, the specific implementation of step one includes the following sub-steps: Step 1.1: Randomly select a point from the original LiDAR point cloud, and randomly select a certain number of points within a certain rectangular area centered on that point as LiDAR point cloud samples. Step 1.2: Based on the center projection relationship, the selected LiDAR point cloud samples are used to find the corresponding coordinate set on the original multi-view aerial oblique image. The original multi-view aerial oblique image is then downsampled and cropped to obtain multi-view aerial oblique image samples.

[0007] Furthermore, in step 1.2, the calculation method for the central projection relationship is as follows: Equation (1) Equation (2) Equation (3) Equation (4) Equation (5) in, Represents the rotation matrix. This represents the three-dimensional coordinates of any point in the selected LiDAR point cloud sample. Indicates the camera's position coordinates. It is the focal length of the aerial oblique image in the x and y directions. These are the principal pixel coordinates of the image. This represents the projected pixel coordinates of the LiDAR point cloud calculated on the image; based on the camera parameters provided by each aerial oblique image, the pixel coordinates of each point in the LiDAR point cloud on the aerial oblique image are determined. If the set of pixel coordinates exceeds the range of the current aerial oblique image, skip the aerial oblique image and find other aerial oblique images for calculation to obtain the set of pixel coordinates of the image block; crop the original multi-view aerial oblique image to the same size according to the center of the coordinate set, and translate the original coordinates to obtain the coordinates of the LiDAR point cloud on the cropped image, as shown in Equations (6) and (7): Equation (6) Equation (7) in, and This represents the maximum and minimum pixel coordinates of the point cloud projection pixels in the current image, from which the image point coordinates corresponding to the center of the LiDAR point cloud data block can be calculated. New coordinates of LiDAR point cloud data blocks on the cropped image From the original coordinates The calculation yielded, where and This represents the number of pixels in the silhouette image in both the x and y directions.

[0008] Furthermore, the multi-view aerial oblique image feature backbone network adopts a U-shaped structure and uses 2×2 pooling operation to downsample aerial oblique image samples layer by layer, preserving multi-scale image features. It is also fused with the upsampled image features through skip connections to effectively transmit multi-level texture information. The Swin-Transformer structure is added to both upsampling and downsampling, and a 5×5 convolutional layer is added to the upsampling.

[0009] Furthermore, the LiDAR point cloud feature extraction backbone network adopts a U-shaped structure. In the downsampling stage, the random sampling method of RandLANet is used to improve the efficiency of point cloud feature extraction. At the same time, the point cloud features are enhanced based on the encoding of the point cloud neighborhood spatial relationship and the self-attention feature encoding. In the upsampling stage, the LiDAR point cloud features are enhanced by fusing multi-level image features, including view-based multi-view aerial oblique image feature aggregation and image-guided multimodal feature fusion.

[0010] Furthermore, the specific implementation of view-based multi-view aerial image feature aggregation is as follows: First, based on the aerial oblique image normal... With LiDAR point cloud normal Calculate the cosine of the angle between the two. Using this as the realism weight for the aerial oblique image texture, and then the pixel features of the multiview aerial oblique image... The global feature is obtained by weighting and summing the values ​​with the given weights, and then... Subtracting this global feature yields the optimized pixel features. The calculation is as follows: Equation (8) Optimized K-pixel features Connected along the channel dimension Next, channel attention structures and spatial attention structures are constructed to reduce redundant information in multi-view image features, where K represents the number of multi-view aerial oblique images, N represents the number of points in the point cloud samples corresponding to the pixels in the image, and C is the number of channels for the corresponding pixel features in the image; the calculation of the channel attention structure is shown in the following formula: Equation (9) Among them, salient features in image features With global features Depend on The MLP layer is obtained by performing max pooling and average pooling in the spatial dimension, and it serves to re-encode the features. Represents the sigmoid function; Next, the features optimized for channel attention are... Process it to calculate the maximum value along its channel dimension. Compared with the average After concatenation along the channel dimension, a convolutional layer is used for encoding, followed by a sigmoid activation function to obtain a spatial attention score, which is then compared with the features. Multiplication yields optimized features This process highlights salient regions in the input features and reduces the influence of irrelevant regions, as shown in the following formula: Equation (10) in, Indicates a connection operation. The convolutional layer is represented; finally, a one-dimensional convolutional layer is used to reduce the image features optimized by the two attention mechanisms. The channel dimension outputs aggregated multi-view image features. .

[0011] Furthermore, the specific implementation of image-guided multimodal feature fusion is as follows: For aggregated pixel features The number of feature channels is Calculate its maximum value along the channel dimension. and average This is used to represent the salient and global information of the pixels corresponding to N points in an image patch; then, the maximum and average values ​​are concatenated, a convolutional layer is used to re-encode the pixel features, and a sigmoid function is constructed to obtain the texture weight distribution map corresponding to the N points. As shown in the following formula: Equation (11) in, Indicates a connection operation. Represents a convolutional layer; texture weight distribution map Next, the number of channels extracted from the LiDAR point cloud feature extraction backbone network is... LiDAR point cloud spatial features Weighted calculations are performed to output optimized point cloud features. To preserve the original LiDAR point cloud spatial features, they are concatenated with the weighted LiDAR features to obtain preliminary fused LiDAR point cloud features. As shown in the following formula: Equation (12) The bottleneck structure is further utilized to aggregate features, which are then added to the initially fused LiDAR point cloud features to obtain the output features. As shown in the following formula: Equation (13) Here, F() and G() are convolutional blocks in the bottleneck structure. F() consists of a 1×1 convolutional layer, a batch normal layer, and a sigmoid layer, while G(x) consists of a 3×1 convolutional layer, a batch normal layer, and a sigmoid layer.

[0012] Furthermore, the global joint loss function is as follows:

[0013] Equation (14) in , This represents the input in two modes. A probability distribution diagram representing the categories. and These represent the weights used to calculate the loss for the image segmentation task. These represent cross-entropy loss and dice loss, respectively. This represents the KL divergence loss.

[0014] Furthermore, the KL divergence loss is defined as shown in equation (17): Equation (17) KL divergence loss is used to calculate the difference between the point cloud class probability distribution and the image probability distribution, where A probability distribution map representing the categories of a point cloud. A probability distribution map of the categories of pixels in the image corresponding to the point cloud; When using multi-view tilted images to extract features, class probability distribution maps of different viewpoints can be obtained. Since KL divergence can only be calculated using prediction probability distribution maps of the same size, the most likely prediction probability distribution is selected to calculate KL divergence, as shown in Equation (18): Equation (18) in, This represents the probability distribution of the pixel corresponding to point cloud i. This refers to the probability distribution of that point on a multi-view aerial oblique image. This indicates the predicted category for that point on any image in a multiview tilted image. This represents the count of predicted values. The `argmax` function is used to obtain the probability distribution of the pixel that appears most frequently in the predicted values. By selecting the most likely predicted category, the image index with the highest relevance is obtained, and the category probability distribution of the corresponding pixels in that image is used in the loss calculation. Ultimately, the result is... In the substitution (17) Participate in loss calculation.

[0015] This invention also provides a LiDAR point cloud semantic segmentation system that fuses multi-view aerial oblique imagery, comprising the following modules: The sample acquisition module is used to sample and geometrically register two modal data, LiDAR point cloud and multi-view aerial oblique image, to obtain LiDAR point cloud and multi-view aerial oblique image samples. The semantic segmentation model construction module is used to build a LiDAR point cloud semantic segmentation model that fuses multi-view aerial oblique imagery. It includes parallel LiDAR point cloud semantic segmentation branches and multi-view aerial oblique imagery semantic segmentation branches, specifically: The multi-view aerial oblique image semantic segmentation submodule is used to construct the multi-view aerial oblique image feature extraction backbone network to obtain multi-level and multi-scale features of multi-view aerial oblique images, and then obtain the multi-view aerial oblique image semantic segmentation result through the convolutional output layer; the input of the multi-view aerial oblique image feature extraction backbone network is multi-view aerial oblique image samples. The LiDAR point cloud semantic segmentation submodule is used to construct a LiDAR point cloud feature extraction backbone network, obtain multi-level and multi-scale features of LiDAR point clouds, and fuse the multi-scale features extracted by the multi-view aerial oblique image feature extraction backbone network with the multi-scale features of LiDAR point clouds to enhance the LiDAR point cloud features. Then, the LiDAR point cloud semantic segmentation result is obtained through a convolutional output layer. The input of the LiDAR point cloud feature extraction backbone network is LiDAR point cloud samples. The model training module is used to calculate the global joint loss function and update the semantic segmentation model parameters; The segmentation result output module is used to perform semantic segmentation on the LiDAR point cloud and multi-view aerial oblique image data to be segmented using the trained semantic segmentation model parameters, and to map the semantic segmentation results of the LiDAR point cloud samples onto the original LiDAR point cloud to obtain the final global segmentation result.

[0016] This invention proposes a semantic segmentation method for LiDAR point clouds that integrates multi-view aerial oblique imagery. It extracts multi-scale texture and spatial features from multi-view aerial oblique imagery and LiDAR point clouds, and then constructs a multimodal data fusion module. By aggregating strongly correlated texture features from multi-view aerial oblique imagery, it guides the attention of LiDAR point cloud features, highlighting key features and effectively improving the semantic segmentation accuracy of LiDAR point clouds. This meets the needs of downstream tasks such as large-scene understanding and 3D reconstruction. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the semantic segmentation model structure in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the semantic segmentation branch structure of multi-view aerial oblique images in an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the LiDAR point cloud semantic segmentation branch structure in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram showing the correspondence between LiDAR point cloud and multi-view aerial oblique image in an embodiment of the present invention.

[0021] Figure 5 This is a schematic diagram of multi-view aerial oblique image feature aggregation based on perspective in an embodiment of the present invention.

[0022] Figure 6 This is a schematic diagram of multimodal feature fusion guided by the aggregated multi-view aerial oblique image features in an embodiment of the present invention.

[0023] Figure 7 This is a comparative schematic diagram of LiDAR point cloud segmentation results of various models in the embodiments of the present invention. Different colors represent different land cover categories. Detailed Implementation

[0024] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0025] This invention proposes a semantic segmentation method for LiDAR point clouds based on multi-view aerial oblique imagery. Feature extraction from multi-view aerial oblique imagery and LiDAR point cloud data is performed at multiple levels. During the feature fusion stage, the aggregated multi-view imagery features are used to guide the attention of point cloud features, highlighting key features. Furthermore, the designed global joint loss function considers the similarity loss between strongly correlated pixel label predictions and point cloud label predictions from multiple images, reducing the interference of mismatches between different modal features on model training and ensuring the segmentation accuracy and reliability of ground feature point clouds. The overall model structure is described in [reference needed]. Figure 1 .

[0026] The data in this embodiment was acquired by CityMapper. The image sensor unit of this device consists of six 150M pixel measurable aerial cameras, capable of acquiring oblique aerial images from five perspectives (front, left, right, rear, and downward). The resolution of the downward-looking image can reach 5cm. Each acquired aerial image consists of 14192*10640 pixels, with a pixel size of 3.76μm. In addition to the image sensor, the device is also equipped with an all-terrain lidar, which can acquire a large amount of point cloud data, with a point cloud sampling density of approximately 39 points / square meter.

[0027] An embodiment provides a LiDAR point cloud semantic segmentation method fused with aerial imagery, comprising the following steps: Step 1 involves sampling and geometrically registering the LiDAR point cloud and multi-view aerial oblique imagery to obtain the required LiDAR point cloud and multi-view aerial oblique imagery samples for the model. This includes random sampling of the LiDAR point cloud based on samples from regions of a set size, downsampling of the original multi-view aerial oblique imagery, and pixel registration of the LiDAR point cloud and multi-view aerial oblique imagery according to camera parameters.

[0028] Step 1, which involves sampling and geometrically registering the LiDAR point cloud with the multi-view aerial oblique image, includes the following steps: Step 1.1: Randomly select a point from the original LiDAR point cloud, and randomly select a certain number of points within a certain rectangular area centered on that point as LiDAR point cloud samples. Step 1.2: Based on the center projection relationship, the selected LiDAR point cloud samples are used to find the corresponding coordinate set on the original multi-view aerial oblique image. The original multi-view aerial oblique image is then downsampled and cropped to obtain multi-view aerial oblique image samples.

[0029] In this embodiment, step 1 is implemented as follows: After randomly selecting a point from the original LiDAR point cloud, a search is performed within a rectangular area with a radius of 10m. The sample capacity of the LiDAR point cloud is 4096. If the number of points in the current area is less than 4096, the points in the area are sampled repeatedly.

[0030] After LiDAR point cloud sampling is completed, the set of corresponding pixel coordinates of the point cloud on each original multi-view aerial oblique image is calculated according to the central projection relationship, as shown below: Equation (1) Equation (2) Equation (3) Equation (4) Equation (5) in, Represents the rotation matrix. This represents the three-dimensional coordinates of any point in the selected LiDAR point cloud sample. Indicates the camera's position coordinates. It is the focal length of the aerial oblique image in the x and y directions. These are the principal pixel coordinates of the image. This represents the projected pixel coordinates of the LiDAR point cloud calculated on this image. Based on the camera parameters provided for each aerial oblique image, the pixel coordinates of each point in the LiDAR point cloud on this aerial oblique image can be determined.

[0031] The registration process between multiview aerial oblique imagery and LiDAR point cloud data is as follows: First, the LiDAR point cloud data is sampled into several data blocks with the same number of points. According to equations (1) to (5), the pixel coordinate set of the corresponding image block is found on the multiview aerial oblique imagery. If the image coordinate set exceeds the range of the current aerial oblique imagery, the aerial oblique imagery is skipped, and other aerial oblique imagery is searched for calculation to obtain the pixel coordinate set of the image block. Since the input image of the image feature extraction network is required to be of the same size, the original multiview aerial oblique imagery needs to be cropped to the same size according to the center of the coordinate set, and the original coordinates are translated to obtain the coordinates of the LiDAR point cloud on the cropped imagery. The calculation is shown in equations (6) and (7).

[0032] Equation (6) Equation (7) in, and This represents the maximum and minimum pixel coordinates of the point cloud projection pixels in the current image. The image point coordinates corresponding to the center of the LiDAR point cloud data block can be calculated from these coordinates. New coordinates of LiDAR point cloud data blocks on the cropped image It can be derived from the original coordinates The calculation yielded, where and This represents the number of pixels in the silhouette image in the x and y directions. For the correspondence between LiDAR point clouds and multi-view aerial oblique imagery, see [link to relevant documentation]. Figure 4 .

[0033] Step 2: Construct a parallel backbone network for feature extraction of LiDAR point cloud and multi-view aerial oblique imagery to obtain multi-level and multi-scale features of the two modalities and extract unique information of each modality. like Figure 2 and Figure 3 As shown, the construction of the parallel LiDAR point cloud and multi-view aerial oblique image feature extraction backbone network in step 2 includes the following steps: To fully utilize texture information at different scales, the multi-view aerial oblique image feature backbone network adopts a U-shaped structure. It uses 2×2 pooling to downsample aerial oblique image samples layer by layer, preserving multi-scale image features. These features are then fused with the upsampled image features through skip connections, effectively conveying multi-level texture information. Furthermore, a Swin-Transformer structure is incorporated into the upsampling and downsampling processes of the multi-view aerial oblique image feature extraction backbone network. This strengthens the internal communication of image features through window self-attention and moving window self-attention structures, enhancing the extraction of local and contextual information. Additionally, 5×5 convolutional layers are added to each upsampling layer of the image features to smooth the block-processed image features (obtained by the Swin-Transformer structure), outputting the pixel features of the multi-view aerial oblique image. In this embodiment, four upsampling and downsampling processes are performed.

[0034] To fully utilize the spatial information embodied in LiDAR point clouds at different scales, the LiDAR point cloud feature extraction backbone network also adopts a U-shaped structure. The input is LiDAR point cloud samples. In the downsampling stage, RandLANet's random sampling method is used to improve the efficiency of point cloud feature extraction. Simultaneously, point cloud features are enhanced based on point cloud neighborhood spatial relationship encoding and self-attention feature encoding. In the upsampling stage, LiDAR point cloud features are enhanced by fusing multi-level image features. This embodiment involves four rounds of upsampling and downsampling processing.

[0035] In this embodiment, the batch size is 4, the maximum number of iterations is 32, the LiDAR point cloud backbone network optimizer is Adam with a learning rate of 0.01, the multi-view aerial oblique image backbone network optimizer is SGD with a learning rate of 0.05, weight_decay is 0.0001, and momentum is set to 0.9. Both modalities contain three identical land cover category labels: buildings, impervious surfaces, and land vegetation. The downsampling ratio in the LiDAR point cloud feature extraction backbone network is set to [4, 4, 2, 2, 2]. The input image size is 224 × 224 pixels, the Swin-Transformer window segmentation size is set to 7 pixels, and the multi-head attention dimension is set to [3, 6, 12, 24].

[0036] Step 3 involves multi-scale feature fusion of multi-view aerial oblique imagery and LiDAR point cloud, including viewpoint-based multi-view aerial oblique imagery feature aggregation and image-guided multimodal feature fusion, to highlight key information in the point cloud features.

[0037] Step 3, which involves multi-scale feature fusion of multi-view aerial oblique imagery and LiDAR point cloud, includes the following steps: Step 3.1: Construct a view-based multi-view aerial oblique image feature aggregation module, and calculate the aggregated multi-view aerial oblique image features. See [link to full process] for details. Figure 5 When a ground object is photographed from the front, the normal of the ground object is almost in the same direction as the normal of the image. Therefore, the smaller the angle between the normal of the aerial oblique image and the normal of the target point, the more comprehensive the object texture it contains, and the greater the correlation between the aerial oblique image and the LiDAR point cloud modality. First, based on the aerial oblique image normal... With LiDAR point cloud normal Calculate the cosine of the angle between the two. Using this as the realism weight for the aerial oblique image texture, and then the pixel features of the multiview aerial oblique image... The global feature is obtained by weighting and summing the values ​​with the given weights, and then... Subtracting this global feature yields the optimized pixel features. The calculation is as follows: Equation (8) Then, the optimized K pixel features Connect along the channel dimension to obtain Next, channel attention structure and spatial attention structure are constructed to reduce redundant information in multi-view image features, where K represents the number of multi-view aerial oblique images, N represents the number of points in the point cloud sample corresponding to the image pixels, and C is the number of channels of the image corresponding to the pixel features. The calculation of the channel attention structure is shown in equation (9): Equation (9) Among them, salient features in image features With global features Depend on Max pooling and average pooling are performed in the spatial dimension to obtain the feature re-encoding layer of the multilayer perceptron (MLP). This represents the sigmoid function, used to calculate weighted scores.

[0038] Features optimized for channel attention Process it to calculate the maximum value along its channel dimension. Compared with the average After concatenation along the channel dimension, a convolutional layer is used for encoding, followed by a sigmoid activation function to obtain a spatial attention score, which is then compared with the features. Multiplication yields optimized features The process of highlighting significant regions in the input features and reducing the influence of irrelevant regions is shown in Equation (10).

[0039] Equation (10) in, Indicates a connection operation. This represents a convolutional layer. Finally, a one-dimensional convolutional layer is used to reduce the image features optimized by the two attention mechanisms. The channel dimension outputs aggregated multi-view image features. .

[0040] Step 3.2 involves encoding the aggregated image features and using them as weights to perform a weighted calculation with the corresponding LiDAR point cloud features. This highlights key information in the LiDAR point cloud features, achieving the fusion of image and point cloud features. (See [link to previous step] for details.) Figure 6 .

[0041] For aggregated pixel features That is, the multi-view image features after output aggregation. The number of feature channels is Calculate its maximum value along the channel dimension. and average This is used to represent the salient and global information of the pixels corresponding to N points in the image patch. Then, the maximum and average values ​​are concatenated, and a convolutional layer is used to re-encode the pixel features. A sigmoid function is constructed, and through a series of operations, the texture weight distribution map corresponding to the N points is obtained. As shown in equation (11): Equation (11) in, Indicates a connection operation. Texture weight distribution map. Next, the number of channels extracted from the point cloud backbone network is... LiDAR point cloud spatial features Weighted calculations are performed, and the optimized point cloud features are output. To preserve the original LiDAR point cloud spatial features, they are concatenated with the weighted LiDAR features to obtain preliminary fused LiDAR point cloud features. As shown in equation (12): Equation (12) Since concatenating the original LiDAR point cloud features with the weighted LiDAR point cloud features generates redundant information, this module further utilizes a bottleneck structure to aggregate the features. The bottleneck module consists of two consecutive convolutional blocks, F(x) and G(x). F(x) consists of a 1×1 convolutional layer, a batchnorm layer, and a sigmoid layer, while G(x) has a similar structure but uses a 3×1 convolutional kernel. The aggregated features are then added to the initially fused LiDAR point cloud features to obtain the output features. As shown in equation (13). Equation (13) This feature is compressed in dimensionality through convolutional layers in the upsampling layer of the LiDAR point cloud feature extraction backbone network. The model achieves the transmission of key information in multimodal data through layer-by-layer upsampling.

[0042] Step 4: Finally, after extracting the backbone network of LiDAR point cloud and multi-view aerial oblique image features, a convolutional output layer is added to obtain the semantic segmentation results of LiDAR point cloud and multi-view aerial oblique image. The semantic segmentation result of multi-view aerial oblique imagery is a predicted value for the semantic category of each pixel in the imagery.

[0043] The semantic segmentation result corresponding to a single LiDAR point cloud sample is the predicted value of the semantic category of each point.

[0044] Step 5: Calculate the similarity between the strongly correlated classification results of LiDAR point cloud and multi-view aerial oblique imagery as the loss function of the model, which is used to update the model parameters and strengthen the constraint between the two models.

[0045] The overall loss function of the LiDAR point cloud semantic segmentation model fused with multi-view aerial oblique imagery proposed in this invention (i.e., the overall model constituted by steps 2-4) is shown in Equation (14):

[0046] Equation (14) in , This represents the input in two modes. A probability distribution diagram representing the categories. and The weights used to calculate the loss of the image segmentation task are 0.6 and 0.4, respectively. These represent cross-entropy loss and dice loss, respectively. This represents the KL divergence loss.

[0047] For single-modal segmentation results, cross-entropy loss is mainly used to optimize the parameters. Cross-entropy loss is a measure of the difference between the predicted probability distribution and the target, and its definition is shown in equation (15): Equation (15) in Indicates the number of point clouds or pixels. Indicates the number of categories to be segmented. This represents the predicted probability of the segmentation result. For images, the cross-entropy in... It is a sign function; for each pixel, the true label is 1 if it equals the current category, and 0 otherwise. For point clouds, the true label of each point cloud is equal to the current category. Select 1 if the value is 1, otherwise select 0.

[0048] Furthermore, to better optimize the image model, the model applies DiceLoss to the loss metric of the multi-view aerial oblique image segmentation branch. Its loss calculation for a single class is defined as shown in Equation (16): Equation (16) in Indicates the number of pixels. This represents the segmentation probability of the current category. This is a sign function that indicates whether the label of the current pixel is equal to the true label. This is a constant used to avoid extreme values ​​during calculation.

[0049] In addition to cross-entropy loss in single-modal tasks With dice loss In addition, the joint loss function further considers the similarity between the segmentation results of the two modalities, and uses Kullback-Leibler divergence for further optimization, defined as in equation (17): Equation (17) KL divergence loss is used to calculate the difference between the point cloud class probability distribution and the image probability distribution, where This represents the probability distribution of point cloud categories. This represents the probability distribution map of pixel categories corresponding to point clouds in the image. In the calculation of the loss function, this model considers the probability distribution map of pixel segmentation categories corresponding to point clouds as a soft target. By calculating the difference in probability distribution between the image and LiDAR point cloud object segmentation results at the same location, KL divergence can more strictly constrain the segmentation results to achieve error correction.

[0050] When using multi-view tilted images to extract features, class probability distribution maps from different viewpoints can be obtained. Since KL divergence can only be calculated using prediction probability distribution maps of the same size, the most likely prediction probability distribution is selected to calculate KL divergence, as shown in equation (18): Equation (18) in, This represents the probability distribution of the pixel corresponding to point cloud i. This refers to the probability distribution of that point on the multi-view tilt image. This indicates the predicted category for that point on any image in a multiview tilted image. This represents the count of predicted values. The `argmax` function is used to obtain the probability distribution of the pixel that appears most frequently in the predicted values. By selecting the most likely predicted category, the image index with the highest relevance is obtained, and the category probability distribution of the corresponding pixels in that image is used in the loss calculation. Ultimately, the result is... In the substitution (17) Participate in loss calculation.

[0051] In practice, extreme values ​​may appear in the probability distribution of the segmentation results for the two modalities. In this example, the probability distribution results need to be normalized using the sigmoid function to ensure that the KL divergence can be calculated correctly.

[0052] Step 6: Use the trained model parameters to perform semantic segmentation on the input LiDAR point cloud and multi-view aerial oblique image data, and map the segmentation results of the LiDAR point cloud samples onto the original LiDAR point cloud to obtain the final global segmentation result.

[0053] Similar to the training process, the LiDAR point cloud samples to be segmented and the multi-view aerial oblique image samples are processed, and semantic segmentation is performed using the trained model parameters. In this process, the batch size can be set to 1. The size of the LiDAR point cloud samples and the corresponding multi-view aerial oblique image sample sizes are consistent with those in the training process: 4096 points and 224×224 pixels, respectively. The segmentation result of each LiDAR point cloud sample is mapped to the original LiDAR point cloud according to an index relationship. The final segmentation result of each point is determined by the maximum number of times that point appears in the segmentation results across all samples.

[0054] This invention also provides a LiDAR point cloud semantic segmentation system that fuses multi-view aerial oblique imagery, comprising the following modules: The sample acquisition module is used to sample and geometrically register two modal data, LiDAR point cloud and multi-view aerial oblique image, to obtain LiDAR point cloud and multi-view aerial oblique image samples. The semantic segmentation model construction module is used to build a LiDAR point cloud semantic segmentation model that fuses multi-view aerial oblique imagery. It includes parallel LiDAR point cloud semantic segmentation branches and multi-view aerial oblique imagery semantic segmentation branches, specifically: The multi-view aerial oblique image semantic segmentation submodule is used to construct the multi-view aerial oblique image feature extraction backbone network to obtain multi-level and multi-scale features of multi-view aerial oblique images, and then obtain the multi-view aerial oblique image semantic segmentation result through the convolutional output layer; the input of the multi-view aerial oblique image feature extraction backbone network is multi-view aerial oblique image samples. The LiDAR point cloud semantic segmentation submodule is used to construct a LiDAR point cloud feature extraction backbone network, obtain multi-level and multi-scale features of LiDAR point clouds, and fuse the multi-scale features extracted by the multi-view aerial oblique image feature extraction backbone network with the multi-scale features of LiDAR point clouds to enhance the LiDAR point cloud features. Then, the LiDAR point cloud semantic segmentation result is obtained through a convolutional output layer. The input of the LiDAR point cloud feature extraction backbone network is LiDAR point cloud samples. The model training module is used to calculate the global joint loss function and update the semantic segmentation model parameters; The segmentation result output module is used to perform semantic segmentation on the LiDAR point cloud and multi-view aerial oblique image data to be segmented using the trained semantic segmentation model parameters, and to map the semantic segmentation results of the LiDAR point cloud samples onto the original LiDAR point cloud to obtain the final global segmentation result.

[0055] To better illustrate the semantic segmentation effect of this invention, its performance was compared with that of existing point cloud semantic segmentation models for multimodal data. The results are shown in Table 1. The comparison shows that the semantic results of the method of this invention are relatively better. A comparative diagram of the segmentation results of various models for LiDAR point clouds can be found in [reference needed]. Figure 7 .

[0056] Table 1. Comparison of LiDAR point cloud semantic segmentation accuracy of various models (unit: %)

[0057] Therefore, the present invention has the following advantages: in the semantic segmentation task of LiDAR point cloud, not only the spatial features of point cloud are considered, but also the optimization effect of the strongly correlated texture features provided by multi-view aerial oblique images on point cloud features is taken into account, providing rich contextual information for LiDAR point cloud, effectively improving the confusion phenomenon in the semantic segmentation results of LiDAR point cloud, and improving the semantic segmentation accuracy of LiDAR point cloud.

[0058] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0059] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A semantic segmentation method for LiDAR point clouds fused with multi-view aerial oblique imagery, characterized in that, Includes the following steps: Step 1: Sample and geometrically register the two modal data, LiDAR point cloud and multi-view aerial oblique image, to obtain LiDAR point cloud and multi-view aerial oblique image samples. Step 2: Construct a LiDAR point cloud semantic segmentation model that integrates multi-view aerial oblique imagery, including a parallel LiDAR point cloud semantic segmentation branch and a multi-view aerial oblique imagery semantic segmentation branch. This includes the following sub-steps: Step 2.1, the semantic segmentation branch of multi-view aerial oblique image specifically includes: constructing a multi-view aerial oblique image feature extraction backbone network to obtain multi-level and multi-scale features of multi-view aerial oblique images, and then obtaining the semantic segmentation result of multi-view aerial oblique images through a convolutional output layer; the input of the multi-view aerial oblique image feature extraction backbone network is multi-view aerial oblique image samples. The backbone network for multi-view aerial oblique image feature extraction adopts a U-shaped structure and uses 2×2 pooling operation to downsample aerial oblique image samples layer by layer, preserving multi-scale image features. It also fuses with the upsampled image features through skip connections, effectively conveying multi-level texture information. The Swin-Transformer structure is added to both upsampling and downsampling, and a 5×5 convolutional layer is added to the upsampling. Step 2.2, the LiDAR point cloud semantic segmentation branch specifically includes: constructing a LiDAR point cloud feature extraction backbone network to obtain multi-level and multi-scale features of the LiDAR point cloud, and fusing the multi-scale features extracted by the multi-view aerial oblique image feature extraction backbone network with the multi-scale features of the LiDAR point cloud to enhance the LiDAR point cloud features. Then, the LiDAR point cloud semantic segmentation result is obtained through a convolutional output layer; the input of the LiDAR point cloud feature extraction backbone network is the LiDAR point cloud sample. The LiDAR point cloud feature extraction backbone network adopts a U-shaped structure. In the downsampling stage, RandLANet's random sampling method is used to improve the efficiency of point cloud feature extraction. At the same time, the point cloud features are enhanced based on the encoding of the point cloud neighborhood spatial relationship and the encoding of self-attention features. In the upsampling stage, the LiDAR point cloud features are enhanced by fusing multi-level image features, including view-based multi-view aerial oblique image feature aggregation and image-guided multimodal feature fusion. Step 3: Calculate the global joint loss function to update the semantic segmentation model parameters; Step 4: Use the trained semantic segmentation model parameters to perform semantic segmentation on the LiDAR point cloud and multi-view aerial oblique image data to be segmented, and map the semantic segmentation results of the LiDAR point cloud samples onto the original LiDAR point cloud to obtain the final global segmentation result.

2. The LiDAR point cloud semantic segmentation method based on multi-view aerial oblique imagery according to claim 1, characterized in that: Step one includes the following sub-steps: Step 1.1: Randomly select a point from the original LiDAR point cloud, and randomly select a certain number of points within a certain rectangular area centered on that point as LiDAR point cloud samples. Step 1.2: Based on the center projection relationship, the selected LiDAR point cloud samples are used to find the corresponding coordinate set on the original multi-view aerial oblique image. The original multi-view aerial oblique image is then downsampled and cropped to obtain multi-view aerial oblique image samples.

3. The LiDAR point cloud semantic segmentation method based on multi-view aerial oblique imagery according to claim 2, characterized in that: In step 1.2, the calculation method for the central projection relationship is as follows: Equation (1) Equation (2) Equation (3) Equation (4) Equation (5) in, Represents the rotation matrix. This represents the three-dimensional coordinates of any point in the selected LiDAR point cloud sample. Indicates the camera's position coordinates. It is the focal length of the aerial oblique image in the x and y directions. These are the principal pixel coordinates of the image. This represents the projected pixel coordinates of the LiDAR point cloud calculated on the image; based on the camera parameters provided by each aerial oblique image, the pixel coordinates of each point in the LiDAR point cloud on the aerial oblique image are determined. If the set of pixel coordinates exceeds the range of the current aerial oblique image, skip the aerial oblique image and find other aerial oblique images for calculation to obtain the set of pixel coordinates of the image block; crop the original multi-view aerial oblique image to the same size according to the center of the coordinate set, and translate the original coordinates to obtain the coordinates of the LiDAR point cloud on the cropped image, as shown in Equations (6) and (7): Equation (6) Equation (7) in, and This represents the maximum and minimum pixel coordinates of the point cloud projection pixels in the current image, from which the image point coordinates corresponding to the center of the LiDAR point cloud data block can be calculated. New coordinates of LiDAR point cloud data blocks on the cropped image From the original coordinates The calculation yielded, where and This represents the number of pixels in the silhouette image in both the x and y directions.

4. The LiDAR point cloud semantic segmentation method based on multi-view aerial oblique imagery according to claim 1, characterized in that: The specific implementation of view-based multi-view aerial image feature aggregation is as follows: First, based on the aerial oblique image normal... With LiDAR point cloud normal Calculate the cosine of the angle between the two. Using this as the realism weight for the aerial oblique image texture, and then the pixel features of the multiview aerial oblique image... The global feature is obtained by weighting and summing the values ​​with the given weights, and then... Subtracting this global feature yields the optimized pixel features. The calculation is as follows: Equation (8) Optimized K-pixel features Connected along the channel dimension Next, channel attention structures and spatial attention structures are constructed to reduce redundant information in multi-view image features, where K represents the number of multi-view aerial oblique images, N represents the number of points in the point cloud samples corresponding to the pixels in the image, and C is the number of channels for the corresponding pixel features in the image; the calculation of the channel attention structure is shown in the following formula: Equation (9) Among them, salient features in image features With global features Depend on The MLP layer is obtained by performing max pooling and average pooling in the spatial dimension, and it serves to re-encode the features. Represents the sigmoid function; Next, the features optimized for channel attention are... Process it to calculate the maximum value along its channel dimension. Compared with the average After concatenation along the channel dimension, a convolutional layer is used for encoding, followed by a sigmoid activation function to obtain a spatial attention score, which is then compared with the features. Multiplication yields optimized features This process highlights salient regions in the input features and reduces the influence of irrelevant regions, as shown in the following formula: Equation (10) in, Indicates a connection operation. The convolutional layer is represented; finally, a one-dimensional convolutional layer is used to reduce the image features optimized by the two attention mechanisms. The channel dimension outputs aggregated multi-view image features. .

5. The LiDAR point cloud semantic segmentation method based on multi-view aerial oblique imagery according to claim 1, characterized in that: The specific implementation of image-guided multimodal feature fusion is as follows: For aggregated pixel features The number of feature channels is Calculate its maximum value along the channel dimension. and average This is used to represent the salient and global information of the pixels corresponding to N points in an image patch; then, the maximum and average values ​​are concatenated, a convolutional layer is used to re-encode the pixel features, and a sigmoid function is constructed to obtain the texture weight distribution map corresponding to the N points. As shown in the following formula: Equation (11) in, Indicates a connection operation. Represents a convolutional layer; texture weight distribution map Next, the number of channels extracted from the LiDAR point cloud feature extraction backbone network is... LiDAR point cloud spatial features Weighted calculations are performed to output optimized point cloud features. To preserve the original LiDAR point cloud spatial features, they are concatenated with the weighted LiDAR features to obtain preliminary fused LiDAR point cloud features. As shown in the following formula: Equation (12) The bottleneck structure is further utilized to aggregate features, which are then added to the initially fused LiDAR point cloud features to obtain the output features. As shown in the following formula: Equation (13) Here, F() and G() are convolutional blocks in the bottleneck structure. F() consists of a 1×1 convolutional layer, a batch normal layer, and a sigmoid layer, while G(x) consists of a 3×1 convolutional layer, a batch normal layer, and a sigmoid layer.

6. The LiDAR point cloud semantic segmentation method based on multi-view aerial oblique imagery according to claim 1, characterized in that: The global joint loss function is shown in the following equation: Equation (14) in , This represents the input in two modes. A probability distribution diagram representing the categories. and These represent the weights used to calculate the loss for the image segmentation task. These represent cross-entropy loss and dice loss, respectively. This represents the KL divergence loss.

7. The LiDAR point cloud semantic segmentation method based on multi-view aerial oblique imagery according to claim 6, characterized in that: The KL divergence loss is defined as shown in equation (17): Equation (17) KL divergence loss is used to calculate the difference between the point cloud class probability distribution and the image probability distribution, where A probability distribution map representing the categories of a point cloud. A probability distribution map of the categories of pixels in the image corresponding to the point cloud; When using multi-view tilted images to extract features, class probability distribution maps of different viewpoints can be obtained. Since KL divergence can only be calculated using prediction probability distribution maps of the same size, the most likely prediction probability distribution is selected to calculate KL divergence, as shown in Equation (18): Equation (18) in, This represents the probability distribution of the pixel corresponding to point cloud i. This refers to the probability distribution of that point on a multi-view aerial oblique image. This indicates the predicted category for that point on any image in a multiview tilted image. This represents the count of predicted values. The `argmax` function is used to obtain the probability distribution of the pixel that appears most frequently in the predicted values. By selecting the most likely predicted category, the image index with the highest relevance is obtained, and the category probability distribution of the corresponding pixels in that image is used in the loss calculation. Ultimately, the result is... In the substitution (17) Participate in loss calculation.

8. A LiDAR point cloud semantic segmentation system integrating multi-view aerial oblique imagery, characterized in that, Includes the following modules: The sample acquisition module is used to sample and geometrically register two modal data, LiDAR point cloud and multi-view aerial oblique image, to obtain LiDAR point cloud and multi-view aerial oblique image samples. The semantic segmentation model construction module is used to build a LiDAR point cloud semantic segmentation model that fuses multi-view aerial oblique imagery. It includes parallel LiDAR point cloud semantic segmentation branches and multi-view aerial oblique imagery semantic segmentation branches, specifically: The multi-view aerial oblique image semantic segmentation submodule is used to construct the multi-view aerial oblique image feature extraction backbone network to obtain multi-level and multi-scale features of multi-view aerial oblique images, and then obtain the multi-view aerial oblique image semantic segmentation result through the convolutional output layer; the input of the multi-view aerial oblique image feature extraction backbone network is multi-view aerial oblique image samples. The backbone network for multi-view aerial oblique image feature extraction adopts a U-shaped structure and uses 2×2 pooling operation to downsample aerial oblique image samples layer by layer, preserving multi-scale image features. It also fuses with the upsampled image features through skip connections, effectively conveying multi-level texture information. The Swin-Transformer structure is added to both upsampling and downsampling, and a 5×5 convolutional layer is added to the upsampling. The LiDAR point cloud semantic segmentation submodule is used to construct a LiDAR point cloud feature extraction backbone network, obtain multi-level and multi-scale features of LiDAR point clouds, and fuse the multi-scale features extracted by the multi-view aerial oblique image feature extraction backbone network with the multi-scale features of LiDAR point clouds to enhance the LiDAR point cloud features. Then, the LiDAR point cloud semantic segmentation result is obtained through a convolutional output layer. The input of the LiDAR point cloud feature extraction backbone network is LiDAR point cloud samples. The LiDAR point cloud feature extraction backbone network adopts a U-shaped structure. In the downsampling stage, RandLANet's random sampling method is used to improve the efficiency of point cloud feature extraction. At the same time, the point cloud features are enhanced based on the encoding of the point cloud neighborhood spatial relationship and the encoding of self-attention features. In the upsampling stage, the LiDAR point cloud features are enhanced by fusing multi-level image features, including view-based multi-view aerial oblique image feature aggregation and image-guided multimodal feature fusion. The model training module is used to calculate the global joint loss function and update the semantic segmentation model parameters; The segmentation result output module is used to perform semantic segmentation on the LiDAR point cloud and multi-view aerial oblique image data to be segmented using the trained semantic segmentation model parameters, and to map the semantic segmentation results of the LiDAR point cloud samples onto the original LiDAR point cloud to obtain the final global segmentation result.

Images