Airborne multispectral lidar point cloud semantic segmentation method, storage medium and equipment

Abstract

The invention relates to a method, storage medium and device for semantic segmentation of airborne multispectral laser radar point cloud, which belongs to the technical field of airborne laser radar point cloud data processing. In order to solve the problem of poor point cloud segmentation caused by the uneven size and distribution of objects in the existing airborne multispectral point cloud. The present invention synthesizes a multispectral point cloud based on the airborne multispectral original point cloud, selects samples according to a uniform grid, and samples the density center spherical neighborhood of each object by a k-clustering method. The two sampling results are then spliced ​​to complete a comprehensive dense network sampling, and then separated according to the bands. The semantic segmentation network is used to perform semantic segmentation on the single-band point cloud band by band, and then a splicing matrix is ​​obtained and fused using a fusion network to obtain the final semantic segmentation result. The present invention also uses a joint training strategy for training to improve the overall fitting ability of the network, thereby improving the segmentation effect of the network.

Description

Technical Field

[0001] The present invention belongs to the technical field of airborne laser radar point cloud data processing, and specifically relates to a multi-band laser radar point cloud semantic segmentation method, storage medium and device. Background Art

[0002] With the continuous development of aerial remote sensing and lidar technologies, airborne lidar is evolving from single-band to multi-band. The core advantage of airborne multi-band lidar lies in acquiring three-dimensional spatial information and multi-band spectral information (multispectral point clouds) of the observed scene / target. This capability better reflects the geometric and spectral characteristics of ground objects, and holds great potential for application in urban planning, land use, resources, and the environment.

[0003] The task of semantic segmentation of multispectral point clouds aims to assign a category label to each point in the point cloud and is a prerequisite for the practical application of airborne multi-band lidar. Current point cloud semantic segmentation algorithms are typically applied to color point cloud data acquired by ground-based depth cameras, which contain both color and depth information. The basic idea is to use random sampling to obtain training and validation samples and then extract integrated spatial-spectral features using a deep learning model. Due to differences in detection angles, detection methods, and observation scenes between airborne multispectral lidar and ground-based depth cameras, existing point cloud semantic segmentation algorithms are not suitable for large-scale airborne multispectral point cloud data. On the one hand, the large scale variation of objects in airborne scenes and the uneven number and distribution of various objects result in non-uniform density in multispectral point clouds. Random sampling methods suffer from problems such as difficulty in selecting balanced samples and low robustness. On the other hand, integrated feature extraction fails to fully utilize the spatial-spectral information of multispectral point clouds. Therefore, the balanced extraction of high-order discriminative semantic segmentation features for various objects from multispectral point clouds remains a challenge. Summary of the Invention

[0004] The present invention aims to solve the problem of poor point cloud segmentation caused by uneven size and distribution of objects in existing airborne multispectral point clouds.

[0005] A method for semantic segmentation of airborne multispectral lidar point clouds, comprising the steps of establishing and training a segmentation network model, and performing semantic segmentation of the airborne multispectral point clouds using the trained segmentation network model.

[0006] The establishment and training process of the segmentation network model includes the following steps:

[0007] S1. For airborne multispectral point clouds, a comprehensive dense network sampling strategy is used for sampling:

[0008] The independent point cloud containing spatial position information and single-band echo intensity is called a single-band point cloud. The original airborne multispectral point cloud is a number of single-band point clouds. If there are M bands, the original airborne multispectral point cloud is M single-band point clouds.

[0009] First, a multispectral point cloud is synthesized based on the airborne multispectral original point cloud;

[0010] Then, in the synthesized multispectral point cloud, samples are selected according to a uniform grid. Where N is the number of points contained in each sample, and L1 is the number of samples selected using a uniform grid;

[0011] Then, the density center spherical neighborhood of each ground feature is sampled in the synthesized multispectral point cloud by using the k-clustering method. Where N is the number of points contained in each sample, and L2 is the number of samples selected using the k-clustering method;

[0012] Finally, the result vectors collected by the two sampling methods are directly spliced ​​along the dimension of the number of samples using the end-to-end method to complete the comprehensive dense network sampling and output the samples after comprehensive dense network sampling.

[0013] S2. Samples after comprehensive and dense network sampling It is separated into M bands to form M single-band point clouds. Then, the semantic segmentation network is used to perform semantic segmentation on the single-band point cloud band by band, and the single-band semantic segmentation result is output. The single-band semantic segmentation result is the category classification probability matrix of each point in the single-band point cloud.

[0014] S3. For all the point-by-point classification probability matrices obtained by single-band segmentation, first calculate the mean of the classification probability matrix of each band bit by bit to obtain the mean probability matrix;

[0015] Then take the minimum value of the classification probability matrix of each band bit by bit to obtain the minimum probability matrix;

[0016] Finally, the mean probability matrix and the minimum probability matrix are directly concatenated with the classification probability of each band, and then the concatenated features are activated by the activation function to obtain the feature matrix F Multiply , then F Multiply Input the fusion network for fusion, add the attention mechanism to the fusion network, and output the final semantic segmentation result;

[0017] S4. Training is performed using a joint training strategy, wherein the joint training strategy includes setting the total loss of the auxiliary loss function and multiple optimizers:

[0018] Set the total loss of the auxiliary loss function: Define the auxiliary loss function And the loss function of the fusion layer where x i are single original data, i=1,2,……,M, Θ i is the parameter learned by gradient descent in a single semantic segmentation network, is the loss function of semantic segmentation of single-band point cloud in each band; f i is the point-by-point category probability matrix obtained by semantic segmentation of single-band point cloud, Θ m is the parameter learned in the gradient descent of the fusion layer, is the loss function of the fusion network;

[0019] Set the total loss function

[0020] In the forward propagation process of the whole network, the band-by-band semantic segmentation network will first perform semantic segmentation on the single-band point cloud of each band, thereby obtaining a category probability matrix, which is then input into the fusion network and the final semantic segmentation result is obtained after fusion. In the backward propagation process, the gradient descent algorithm is used to perform semantic segmentation on the single-band point cloud of each band. Parameters are learned in the fusion layer, and Auxiliary loss function, completing the purpose of semantic segmentation module for fitting single-band point cloud;

[0021] Parallel optimizer: Add independent optimizers to the semantic segmentation network and fusion network of each band.

[0022] Furthermore, the process of synthesizing a multispectral point cloud based on the airborne multispectral original point cloud includes the following steps;

[0023] In the airborne multispectral original point cloud, for a single-band point cloud a, the nearest neighbor algorithm is used to find the point corresponding to the echo intensity information of the nearest neighbor point in the other single-band point clouds, and the found point is assigned to the single-band point cloud a, and a single-band point cloud is converted into a point cloud containing all bands; then the same processing is taken for each single-band point cloud, and finally several independent point clouds containing all bands are obtained; then several independent point clouds containing all bands are converted into a point cloud containing all bands, which is recorded as a multispectral point cloud.

[0024] Preferably, the nearest neighbor algorithm is a k-nearest neighbor algorithm, that is, for a single-band point cloud a, the k-nearest neighbor algorithm is used to find the point corresponding to the echo intensity information of the nearest neighbor point in the remaining single-band point clouds.

[0025] Furthermore, the mean probability matrix described in S3 Among them, F1, F2, ..., F M They represent the point-by-point category probability matrices output by the single-band point cloud semantic segmentation network for M bands.

[0026] Furthermore, the minimum probability matrix F described in S3 min =min(F1, F2, ..., F M ).

[0027] Furthermore, in S3, the feature matrix F after the spliced ​​features are activated by the activation function is Multiply as follows:

[0028] F Multiply =σ(F1||F2||……||F M ||F min ||F avg )

[0029] Among them, σ is the activation function, and || represents direct splicing.

[0030] Furthermore, F Multiply The process of inputting the fusion network for fusion includes the following steps:

[0031] Feature matrix before splicing The concatenated feature matrix Among them, N is the total number of point cloud points, label is the number of ground object categories, represents the field of real numbers;

[0032] Before using the fusion network for fusion, first perform the concatenation of the matrix F Multiply Add attention weight to the label dimension of F Multiply After adding the attention weights, the attention mechanism and fusion network are used to fuse features and output the point-by-point classification probability matrix. Finally, the category probability matrix is ​​converted into a point-by-point semantic segmentation result through the maximum pooling layer.

[0033] Furthermore, the fusion network is a multi-layer perceptron.

[0034] A computer storage medium, characterized in that at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the airborne multispectral lidar point cloud semantic segmentation method.

[0035] An airborne multispectral laser radar point cloud semantic segmentation device, the device comprising a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the airborne multispectral laser radar point cloud semantic segmentation method.

[0036] Beneficial effects:

[0037] The present invention can effectively improve the impact of the distribution and size imbalance of objects in airborne large-scene multispectral point cloud data on the accuracy of semantic segmentation; the present invention can take into account the independence of each band of airborne multispectral lidar point cloud, and utilize the spectral information of multiple bands to improve the accuracy of semantic segmentation results.

[0038] The present invention also proposes a joint training strategy so that the gradient descent algorithm can not only The parameters are learned in the fusion layer to complete the above fusion module so that the final fusion result is closer to the true value. The auxiliary loss function completes the purpose of the semantic segmentation module for fitting the single-band point cloud, thereby further improving the final semantic segmentation accuracy. At the same time, by adding independent optimizers to the semantic segmentation network and fusion network of each band, the hyperparameters of each module are iteratively optimized separately without interfering with each other, thereby improving the overall fitting ability of the network and further improving the segmentation effect of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the airborne multispectral point cloud semantic segmentation process based on comprehensive dense network sampling and decision fusion.

[0040] Figure 2 Schematic diagram of comprehensive dense network sampling.

[0041] Figure 3 Schematic diagram of the attention fusion process.

[0042] Figure 4 This is a false color composite image of the port's three-band lidar point cloud data.

[0043] Figure 5 This is the ground truth map of the port's three-band lidar point cloud data.

[0044] Figure 6 Comparison chart of semantic segmentation between full dense network sampling and original network. DETAILED DESCRIPTION

[0045] Based on the problems existing in the background technology, the present invention proposes a multispectral point cloud semantic segmentation method based on comprehensive dense network sampling and decision fusion, which has important research significance and application value in the fields of disaster warning, geological monitoring, vegetation detection, etc.

[0046] Specific implementation method 1: Combined with 1 to illustrate this implementation method,

[0047] This embodiment is a method for semantic segmentation of airborne multispectral lidar point clouds. More specifically, it is a method for semantic segmentation of airborne multispectral point clouds based on comprehensive dense network sampling and decision fusion, including the following steps:

[0048] S1. For airborne multispectral point clouds, a comprehensive dense network sampling strategy is used for sampling:

[0049] An independent point cloud containing spatial position information and single-band echo intensity is called a "single-band point cloud". The airborne multispectral original point cloud is a number of single-band point clouds. If there are M bands, then the airborne multispectral original point cloud is M single-band point clouds. Since the points contained in different band point clouds are not exactly the same, first, in the airborne multispectral original point cloud, for a single-band point cloud a, the k-nearest neighbor algorithm is used to find the points corresponding to the echo intensity information of the nearest neighbor points in the other single-band point clouds, and the found points are assigned to the single-band point cloud a, resulting in a single-band point cloud being converted into a point cloud containing all bands; then the same processing is applied to each single-band point cloud, and finally several independent point clouds containing all bands are obtained; then the several independent point clouds containing all bands are converted into a point cloud containing all bands, denoted as a multispectral point cloud, where each point can contain multi-band echo intensity. The purpose of this is to unify the spatial distribution of all point clouds and to increase the density of point clouds to a certain extent, laying the foundation for subsequent high-precision semantic segmentation.

[0050] In the synthesized multispectral point cloud, samples are selected according to a uniform grid Where N is the number of points contained in each sample, which is set to 4096 points in some embodiments, and L1 is the number of samples selected using a uniform grid.

[0051] Then, the density center spherical neighborhood of each ground feature is sampled in the synthesized multispectral point cloud by using the k-clustering method. Where N is the number of points contained in each sample, which is set to 4096 points in some embodiments, and L2 is the number of samples selected using the k-clustering method.

[0052] Finally, the result vectors collected by the two sampling methods are directly spliced ​​along the dimension of the number of samples using the end-to-end method (that is, the two sampling results are put together) to complete the comprehensive dense network sampling and output the samples after comprehensive dense network sampling.

[0053] Most of the point cloud segmentation training samples are obtained by random multiple sampling. However, for large remote sensing scenes, the classification and size of objects are not balanced. Random sampling can easily result in some small and sparsely distributed categories not being fully collected (such as cars, sailboats, power lines, etc.), thereby reducing the overall segmentation accuracy of the network. Therefore, this paper proposes a clustering-based full-category uniform sampling method, that is, a comprehensive dense network sampling strategy, such as Figure 2 shown.

[0054] The uniform grid sampling process allows for a flexible selection of sample sizes. After selecting sample centers at equal intervals, the centers are grouped with the remaining points by distance range. This method is highly efficient and ensures that all training point clouds are fully integrated into the training samples, thus ensuring segmentation progress at the input level.

[0055] The k-clustering method is based on the processing of sampling the spherical neighborhood of the density center of each feature to group the point cloud according to the feature category. A clustering algorithm is performed in each category of features. The cluster center can be approximately considered as the density center, which is called the centroid in the present invention. The clustering method is a type of unsupervised learning method. This method compares the similarity between samples through a similarity measurement function, and groups and merges similar samples, and finally iterates through a loss function to achieve higher clustering accuracy. The present invention adopts the k-clustering method to find the category center. First, the required number of categories k is defined, and the Euclidean distance is used as the similarity measurement. Finally, the clustering effect is evaluated through the loss function.

[0056] This comprehensive dense network sampling strategy has the advantages of including all training set points at the same time and sampling more evenly and with high coverage of ground objects.

[0057] S2. Samples after comprehensive and dense network sampling Separate it into M bands to form M single-band point clouds;

[0058] Then, the semantic segmentation network is used to perform semantic segmentation on the single-band point cloud band by band, and the single-band semantic segmentation result is output;

[0059] After a full, dense network sampling process, a balanced set of training samples containing more information about feature distribution is obtained. This allows for semantic segmentation of single-band point clouds using existing algorithms. By analyzing the characteristics of different bands, a more appropriate network is selected for processing based on feature distribution and spectral signatures. This processing yields a probability matrix for each point in the single-band point cloud, which is then fed into the next step.

[0060] S3. For all the point-by-point classification probability matrices obtained from the single-band segmentation, first calculate the mean of the classification probability matrix of each band bit by bit, then take the minimum value of the classification probability matrix of each band bit by bit, and finally directly concatenate the above mean probability matrix and minimum probability matrix with the classification probability of each band in an end-to-end manner. Then, the feature matrix obtained by splicing is activated by the activation function and input into the fusion network. The attention mechanism is added to the fusion network so that the network can learn the importance of different category probability matrices and the importance of different categories in the category probability matrix, and finally output the final semantic segmentation result.

[0061] The current field of multi-band point cloud semantic segmentation has the following problems:

[0062] (1) It is impossible to integrate the characteristics of each band. The single-band point cloud segmentation result cannot fully utilize the multi-band spectral information, resulting in poor segmentation effect for objects with similar spatial location characteristics (such as ground, grass, and road). Therefore, a multi-band point cloud fusion method is urgently needed to integrate the segmentation effects of different bands on different objects to obtain a more accurate point cloud.

[0063] (2) The correlation between spectra is not fully utilized. Simple splicing and other methods cannot utilize the correlation between spectra during the fusion process, and the results are not accurate.

[0064] (3) There is too much redundant information between spectra. Although there are differences between different spectra, if the features are not screened, there will be too much redundant information, which will interfere with the final segmentation accuracy.

[0065] In order to solve the above three problems, Figure 3 As shown, the present invention proposes the following solutions:

[0066] (1) In order to utilize the spectral correlation between the three-band point clouds, an explicit encoding method is proposed (actually, the emphasis here is on the encoding method). As shown in the following formula, the correlation between the spectra is artificially added to the input layer of the fusion layer to fully utilize the correlation between the spectra. Taking the Titan three-band point cloud data as an example (including 532nm, 1064nm, and 1550nm bands), this method can process point cloud data with more bands in actual application. The specific method is as follows:

[0067] F Multiply =σ(F 532nm ||F 1064nm ||F 1550nm ||F min ||F avg )

[0068] F min =min(F 532nm,F 1064nm ,F 1550nm )

[0069]

[0070] Among them, σ is the activation function, F 532nm 、F 1064nm 、F 1550nm They represent the point-by-point category probability matrix output by the single-band point cloud semantic segmentation network for the three bands of 532nm, 1064nm and 1550nm, respectively. min is the minimum value of the three-band eigenvectors after bit-by-bit comparison, F avg is the result of adding and averaging the three-band feature vectors bit by bit; the “||” symbol in the above formula indicates direct concatenation, where N is the total number of points in the point cloud, and label is the number of object categories.

[0071] (2) After the above-mentioned display encoding process, the three-band feature vectors and the two encoded feature vectors are directly spliced ​​together. In the next step, the spliced ​​vectors are input into the fusion network. The fusion network can use any existing fusion network, such as a multi-layer perceptron, and output a point-by-point classification probability matrix Finally, the category probability matrix is ​​converted into a point-by-point semantic segmentation result through the maximum pooling layer.

[0072] During the training process of the fusion network, the parameters in the hidden layer will not only fit the eigenvector of each band, but also take into account the fitting of the two explicit codes. min Indicates the value with the minimum probability of being classified into this category among the three band segmentation results. min The fitting of F can reduce the network’s mis-segmentation rate to a certain extent, thereby improving the overall segmentation accuracy of the network. avg Represents the average classification probability of the three band segmentation results, by F avg The fitting can simultaneously take into account the segmentation results of the three bands, thereby positively improving the segmentation accuracy of the network. In addition, by directly fitting each band, this encoding method can not only take into account the segmentation results of each band while also covering the overall results, but also reduce the overall missegmentation rate, and ultimately achieve an optimized fusion result.

[0073] (3) In order to solve the problem of feature redundancy after fusion, the present invention also proposes a feature selection method based on the attention mechanism to complete the feature screening. The concatenated vector F Multiply It contains two dimensions, one is the number dimension of probability matrices of different categories, and the other is the feature dimension of probability matrix of each category. Before using the fusion network for fusion, firstly perform the concatenation of the matrix F. MultiplyThe label dimension of F is added with attention weight, so that the feature dimension that is more conducive to semantic segmentation can be trained. Multiply The 5N dimension, that is, the five vectors F 532nm ,F 1064nm ,F 1550nm ,F min ,F avg By adding weights separately, training yields a category probability matrix that is more conducive to semantic segmentation. Hybridizing the two attention mechanisms can improve fusion accuracy to a certain extent. After adding attention weights, feature fusion is performed using the attention mechanism and fusion network. During the attention mechanism training process, the attention mechanism and fusion network can automatically evaluate which feature dimensions are more important.

[0074] S4. Training using joint training strategy:

[0075] The joint training strategy includes multiple loss functions and multiple optimizers:

[0076] In a fusion network that uses the attention mechanism, if only the loss function of the fusion layer is added, the network will lose the ability to fit a single band, resulting in a decrease in the accuracy of the fusion result.

[0077] This paper proposes a fusion network optimization module that ensures that both the feature extraction layer and the fusion layer can achieve high fitting capabilities by changing the parameter optimization strategy. The specific implementation method includes the following two parts:

[0078] (1) Setting the total loss of the auxiliary loss function: In the back propagation process of deep learning, in order to ensure that the feature extraction part has a good fitting effect on the single-band point cloud and the fusion part can take into account the characteristics of multiple bands, four loss functions are defined, namely Among them, x1, x2, x3 are the original data of the three band point clouds, Θ1, Θ2, Θ3 are the parameters learned by gradient descent in a single semantic segmentation network, are the loss functions (auxiliary loss functions) of semantic segmentation of single-band point cloud for each band. In this implementation, three bands are used, so And the corresponding Θ1, Θ2, Θ3, if it is M bands, then f1,f2,f3 are the point-by-point category probability matrices obtained by semantic segmentation of single-band point cloud, Θ m is the parameter learned in the gradient descent of the fusion layer, is the loss function of the fusion layer;

[0079] Total loss function as follows:

[0080]

[0081] During the forward propagation process of the entire network, the band-by-band semantic segmentation network will first perform semantic segmentation on the single-band point cloud of each band to obtain a category probability matrix, and input the above category probability matrix into the fusion network through a certain fusion method, and obtain the final semantic segmentation result after fusion. The above process involves two modules with large differences. The first module is the semantic segmentation module, whose task is to learn the characteristics of each band point cloud so that the trained model can better fit each single-band point cloud; the second module is the fusion module, and its purpose is to integrate the results obtained by the semantic segmentation module so that the final fusion result is closer to the true value. Therefore, in the process of back propagation, if a single loss function is used, it is impossible to take into account the different purposes of the two modules at the same time, and to a certain extent, it will cause gradient confusion in the gradient descent process of the two. By adding an auxiliary loss function, in the process of back propagation, the gradient descent algorithm can not only The parameters are learned in the fusion layer to complete the above fusion module so that the final fusion result is closer to the true value. The auxiliary loss function completes the purpose of the semantic segmentation module for fitting single-band point clouds, thereby further improving the final semantic segmentation accuracy.

[0082] (2) Parallel optimizer: In general deep learning tasks, only one optimizer is used to complete the iterative optimization of hyperparameters. However, since the model proposed in this invention contains many modules with different functions, such as the single-band semantic segmentation module and the fusion module, if a single optimizer is used, the iterative optimization of the hyperparameters of different modules will interfere with each other, reducing the overall fit of the network. This invention proposes a parallel optimizer training strategy. By adding independent optimizers to the semantic segmentation network and the fusion network of each band, the hyperparameters of each module are iteratively optimized separately without interfering with each other, thereby improving the overall fit of the network.

[0083] By adding two training strategies, auxiliary loss function and parallel optimizer, the final semantic segmentation accuracy of the network is improved.

[0084] S5. Use the model trained with the joint training strategy to perform semantic segmentation of airborne multispectral point clouds. Specific implementation method 2:

[0086] This embodiment is a computer storage medium, which stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the airborne multispectral lidar point cloud semantic segmentation method.

[0087] It should be understood that the instructions include computer program products, software, or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system or other electronic device. Computer storage media may include readable media on which instructions are stored, and may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions. Specific implementation method three:

[0089] This embodiment is an airborne multispectral lidar point cloud semantic segmentation device, which includes a processor and a memory. It should be understood that the device includes any device including a processor and a memory described in the present invention, and may also include other units and modules that perform display, interaction, processing, control, and other functions through signals or instructions;

[0090] At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the airborne multispectral lidar point cloud semantic segmentation method.

[0091] Example

[0092] The experiment uses Titan three-band LiDAR point cloud data. The point cloud is manually labeled and classified into nine types of features. The PointNet++ semantic segmentation network is used for band-by-band segmentation. First, follow the steps below to perform multi-band fusion semantic segmentation:

[0093] (1) Obtain multispectral point cloud based on the original point cloud;

[0094] The multispectral point cloud is grouped and sampled according to the grid, and the range includes all point clouds to make training set 1.

[0095] The multispectral point cloud is classified according to the ground object category, and the density center point of each category is calculated by the k-clustering algorithm. With the center point as the center of the circle, the points in the spherical neighborhood are grouped to form the training set 2.

[0096] The overall training set is created by directly concatenating the two training sets end to end.

[0097] (2) Input the training set of each band into the PointNet++ network for semantic segmentation, and obtain the probability matrix of the ground object category in each band.

[0098] (3) Calculate the bitwise minimum and bitwise average of the category probability matrix and concatenate them with the category probability matrix of all bands to form the input of the attention fusion network. Then, add a channel attention mechanism to the feature probability dimension and a spatial attention mechanism to the matrix directions of the five categories, and simultaneously complete feature dimensionality reduction and feature screening. Finally, the above input is trained through a multi-layer perceptron to output a fused category probability matrix, and finally, the category probability matrix is ​​converted into a predicted category label through maximum pooling.

[0099] (4) During the back-propagation process, an auxiliary loss function is added to each band-by-band segmentation network result, and different optimizers are added to the band-by-band segmentation network and the fusion network respectively, so that the final fitting effect is improved.

[0100] The comparison of the segmentation results of uniform sampling and random sampling of all samples and the comparison of the segmentation results of different methods are as follows:

[0101] Table 1 Comparison of full sample uniform sampling and random sampling segmentation results

[0102]

[0103]

[0104] Table 2 Comparison of segmentation results of different methods

[0105]

[0106] Using the present invention Figure 5 The ground feature map of the three-band lidar point cloud data of the port shown ( Figure 5 for Figure 4 The result of segmentation is shown in the figure (the true value image corresponding to the false color composite image of the port three-band lidar point cloud data). Figure 6 As shown, Figure 6 (a) is the true value map, (b) is the segmentation effect of the original network, and (c) is the segmentation effect of the comprehensive dense network sampling.

[0107] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A method for semantic segmentation of airborne multispectral lidar point clouds, characterized in that: It includes the establishment and training process of the segmentation network model, and the process of semantic segmentation of airborne multispectral point clouds using the trained segmentation network model; The establishment and training process of the segmentation network model includes the following steps: S1. For airborne multispectral point clouds, a comprehensive dense network sampling strategy is used for sampling: The independent point cloud containing spatial position information and single-band echo intensity is called a single-band point cloud. The original airborne multispectral point cloud is a number of single-band point clouds. If there are M bands, the original airborne multispectral point cloud is M single-band point clouds. First, a multispectral point cloud is synthesized based on the airborne multispectral original point cloud; Then, in the synthesized multispectral point cloud, samples are selected according to a uniform grid. Where N is the number of points contained in each sample, and L1 is the number of samples selected using a uniform grid; Then, the density center spherical neighborhood of each ground feature is sampled in the synthesized multispectral point cloud by using the k-clustering method. Where N is the number of points contained in each sample, and L2 is the number of samples selected using the k-clustering method; Finally, the result vectors collected by the two sampling methods are directly spliced ​​along the dimension of the number of samples using the end-to-end method to complete the comprehensive dense network sampling and output the samples after comprehensive dense network sampling. S2. Samples after comprehensive and dense network sampling It is separated into M bands to form M single-band point clouds. Then, the semantic segmentation network is used to perform semantic segmentation on the single-band point cloud band by band, and the single-band semantic segmentation result is output. The single-band semantic segmentation result is the category classification probability matrix of each point in the single-band point cloud. S3. For all the point-by-point classification probability matrices obtained by single-band segmentation, first calculate the mean of the classification probability matrix of each band bit by bit to obtain the mean probability matrix; Then take the minimum value of the classification probability matrix of each band bit by bit to obtain the minimum probability matrix; Finally, the mean probability matrix and the minimum probability matrix are directly concatenated with the classification probability of each band, and then the concatenated features are activated by the activation function to obtain the feature matrix F Multiply , then F Multiply Input the fusion network for fusion, add the attention mechanism to the fusion network, and output the final semantic segmentation result; S4. Training is performed using a joint training strategy, wherein the joint training strategy includes setting the total loss of the auxiliary loss function and multiple optimizers: Set the total loss of the auxiliary loss function: Define the auxiliary loss function And the loss function of the fusion layer where x i are single original data, i=1,2,……,M, Θ i is the parameter learned by gradient descent in a single semantic segmentation network, is the loss function of semantic segmentation of single-band point cloud in each band; f i is the point-by-point category probability matrix obtained by semantic segmentation of single-band point cloud, Θ m is the parameter learned in the gradient descent of the fusion layer, is the loss function of the fusion network; Set the total loss function In the forward propagation process of the whole network, the band-by-band semantic segmentation network will first perform semantic segmentation on the single-band point cloud of each band, thereby obtaining a category probability matrix, which is then input into the fusion network and the final semantic segmentation result is obtained after fusion. In the backward propagation process, the gradient descent algorithm is used to perform semantic segmentation on the single-band point cloud of each band. Parameters are learned in the fusion layer, and Auxiliary loss function, completing the purpose of semantic segmentation module for fitting single-band point cloud; Parallel optimizer: Add independent optimizers to the semantic segmentation network and fusion network of each band.

2. The method for semantic segmentation of airborne multispectral lidar point clouds according to claim 1, characterized in that: The process of synthesizing a multispectral point cloud based on an airborne multispectral raw point cloud includes the following steps: In the airborne multispectral original point cloud, for a single-band point cloud a, the nearest neighbor algorithm is used to find the point corresponding to the echo intensity information of the nearest neighbor point in the other single-band point clouds, and the found point is assigned to the single-band point cloud a, and a single-band point cloud is converted into a point cloud containing all bands; then the same processing is taken for each single-band point cloud, and finally several independent point clouds containing all bands are obtained; then several independent point clouds containing all bands are converted into a point cloud containing all bands, which is recorded as a multispectral point cloud.

3. The method for semantic segmentation of airborne multispectral lidar point clouds according to claim 2, characterized in that: The nearest neighbor algorithm is a k-nearest neighbor algorithm, that is, for a single-band point cloud a, the k-nearest neighbor algorithm is used to find the point corresponding to the echo intensity information of the nearest neighbor point in the remaining single-band point clouds.

4. The method for semantic segmentation of airborne multispectral lidar point clouds according to claim 3, characterized in that: Mean probability matrix described in S3 Among them, F1, F2, ..., F M They represent the point-by-point category probability matrices output by the single-band point cloud semantic segmentation network for M bands.

5. The method for semantic segmentation of airborne multispectral lidar point clouds according to claim 4, characterized in that: The minimum probability matrix F described in S3 min =min(F1, F2, ..., F M ).

6. The method for semantic segmentation of airborne multispectral lidar point clouds according to claim 5, characterized in that: In S3, the concatenated features are activated by the activation function to form the feature matrix F Multiply as follows: F Multiply =σ(F1||F2||……||F M ||F min ||F avg ) Among them, σ is the activation function, and || represents direct splicing.

7. The method for semantic segmentation of airborne multispectral lidar point clouds according to any one of claims 1 to 6, characterized in that: F Multiply The process of inputting the fusion network for fusion includes the following steps: Feature matrix before splicing The concatenated feature matrix Among them, N is the total number of point cloud points, label is the number of ground object categories, represents the field of real numbers; Before using the fusion network for fusion, first perform the concatenation of the matrix F Multiply Add attention weight to the label dimension of F Multiply After adding the attention weights, the attention mechanism and fusion network are used to fuse features and output the point-by-point classification probability matrix. Finally, the category probability matrix is ​​converted into a point-by-point semantic segmentation result through the maximum pooling layer.

8. The method for semantic segmentation of airborne multispectral lidar point clouds according to claim 7, characterized in that: The fusion network is a multi-layer perceptron.

9. A computer storage medium, characterized in that The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement an airborne multispectral lidar point cloud semantic segmentation method as described in any one of claims 1 to 8.

10. An airborne multispectral lidar point cloud semantic segmentation device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement an airborne multispectral lidar point cloud semantic segmentation method as described in any one of claims 1 to 8.

Images