A multispectral point cloud classification method for remote sensing scenarios

Abstract

This invention relates to a multispectral point cloud classification method for remote sensing scenarios. The purpose of this invention is to address the problems of poor classification performance in existing machine learning-based multispectral point cloud classification methods and the need for extensive manual annotation of ground truth values ​​for training in deep learning-based methods. The process is as follows: Multispectral point clouds are processed using a grid-balanced sampling method to obtain training and test samples; a feature aggregation network is constructed; a multi-classifier head and adaptive hybrid loss module are constructed; the trained feature aggregation network and the multi-classifier head and adaptive hybrid loss module are obtained; the multispectral point cloud to be tested is input into the trained feature aggregation network, the output features of the trained feature aggregation network are input into the trained multi-classifier head and adaptive hybrid loss module, and the trained multi-classifier head and adaptive hybrid loss module outputs the classification prediction result. This invention belongs to the field of multispectral point cloud classification.

Description

Technical Field

[0001] This invention relates to a multispectral point cloud classification method. Background Technology

[0002] Multispectral point clouds contain both three-dimensional spatial and spectral information of ground features, enabling the interpretation of their composition and distribution using stereoscopic observation methods. Compared to traditional observation techniques, multispectral point cloud data offers several advantages: multi-hyperspectral images are highly susceptible to illumination conditions and lack three-dimensional spatial information; lidar point clouds only provide three-dimensional spatial information of ground features but lack spectral information. Multispectral point clouds can simultaneously acquire both three-dimensional spatial and spectral information of scene features, demonstrating significant potential for various downstream tasks.

[0003] Currently, there are two main types of multispectral point cloud classification methods: machine learning-based methods and deep learning-based methods. Machine learning-based methods require less computation and less training data, but their classification performance is relatively poor. Deep learning-based methods have higher overall classification accuracy, but require a large amount of manually labeled ground truth for training. In addition, most existing deep learning methods have been extensively tested on indoor datasets, with few optimized designs for the characteristics of remote sensing scenarios. Summary of the Invention

[0004] The purpose of this invention is to address the problems of poor classification performance of existing machine learning-based multispectral point cloud classification methods and the need for a large number of manually labeled ground truth values ​​to assist training in deep learning-based multispectral point cloud classification methods. Therefore, this invention proposes a multispectral point cloud classification method for remote sensing scenarios.

[0005] The specific process of a multispectral point cloud classification method for remote sensing scenes is as follows:

[0006] Step 1: Process the multispectral point cloud using the grid balanced sampling method to obtain training and test samples;

[0007] Step 2: Construct a feature aggregation network;

[0008] Step 3: Construct a multi-classification head and adaptive hybrid loss module;

[0009] The multi-classification head includes two parts: a multi-scale classification head and a long-tailed distribution classification head.

[0010] The adaptive hybrid loss module includes multi-scale loss and long-tailed distribution loss;

[0011] Step 4: Input the training samples into the constructed feature aggregation network, and the feature aggregation network outputs features;

[0012] The feature aggregation network outputs the feature input to construct a multi-classification head and an adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module output the classification prediction result.

[0013] Train the feature aggregation network and the multi-classification head and adaptive hybrid loss module until convergence, and obtain the trained feature aggregation network and the multi-classification head and adaptive hybrid loss module.

[0014] Step 5: Input the test samples into the trained feature aggregation network. The trained feature aggregation network outputs features. Input the trained multi-classification head and adaptive hybrid loss module into the trained multi-classification head and adaptive hybrid loss module. The trained multi-classification head and adaptive hybrid loss module outputs the classification prediction results.

[0015] The beneficial effects of this invention are as follows:

[0016] This invention addresses the aforementioned problems by tackling the multi-scale differences and long-tail distribution of ground objects in remote sensing scenes. (Currently, there are two main types of multispectral point cloud classification methods: machine learning-based and deep learning-based. Machine learning-based methods require less computation and less training data, but their classification performance is poor. Deep learning-based methods have higher overall classification accuracy, but require a large amount of manually labeled ground values ​​for training. Furthermore, most existing deep learning methods are extensively tested on indoor datasets, with few optimized designs for the characteristics of remote sensing scenes.) First, to address the difficulty of obtaining manually labeled data for remote sensing scenes, a grid-balanced sampling method is used to reliably extract training samples from sparsely labeled multispectral point clouds. Then, to address the scale and quantity differences (long-tail distribution) of ground objects in remote sensing scenes, a multi-scale feature aggregation skip connection module is designed to optimize the feature extraction performance of the feature aggregation network. An adaptive hybrid loss module is designed to optimize the learning capability of the multi-classification head.

[0017] The method of this invention reliably extracts training regions from sparsely labeled multispectral point clouds through a grid balanced sampling method. It also optimizes the feature aggregation network and classifier through a multi-scale feature aggregation module and an adaptive hybrid loss module to address the problems of low small-class learning efficiency and poor classification accuracy in remote sensing multispectral point clouds with scale differences and long-tail distribution.

[0018] To verify the performance of this invention, it was tested on a set of real multispectral point cloud data. Experimental results show that, compared with current representative methods, this method can achieve higher overall accuracy, and even higher accuracy in classifying small categories of land features. The experimental results verify the effectiveness of the multispectral point cloud classification method for complex remote sensing scenes proposed in this invention. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the implementation process of the present invention;

[0020] Figure 2 This is a schematic diagram illustrating the implementation process of the grid-balanced sampling strategy in this invention;

[0021] Figure 3 This is a schematic diagram illustrating the implementation process of the multi-scale feature aggregation module in this invention;

[0022] Figure 4 This is a schematic diagram illustrating the implementation process of the adaptive hybrid loss module in this invention. and F5, same;

[0023] Figure 5a It is a false-color composite image of multispectral point clouds;

[0024] Figure 5b It is a sparsely labeled map of terrain features, with the black areas representing unlabeled features;

[0025] Figure 6 This is a comparison chart of the classification effects of different methods. Detailed Implementation

[0026] Specific Implementation Method 1: The specific process of this implementation method for multispectral point cloud classification in remote sensing scenarios is as follows:

[0027] Step 1: Process the multispectral point cloud using the grid balanced sampling method to obtain training and test samples;

[0028] Step 2: Construct a feature aggregation network;

[0029] Step 3: Construct a multi-classification head and adaptive hybrid loss module;

[0030] The multi-classification head includes two parts: a multi-scale classification head and a long-tailed distribution classification head.

[0031] The adaptive hybrid loss module includes multi-scale loss and long-tailed distribution loss;

[0032] Step 4: Input the training samples into the constructed feature aggregation network, and the feature aggregation network outputs features;

[0033] The feature aggregation network outputs the feature input to construct a multi-classification head and an adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module output the classification prediction result.

[0034] Train the feature aggregation network and the multi-classification head and adaptive hybrid loss module until convergence, and obtain the trained feature aggregation network and the multi-classification head and adaptive hybrid loss module.

[0035] Step 5: Input the test samples into the trained feature aggregation network. The trained feature aggregation network outputs features. Input the trained multi-classification head and adaptive hybrid loss module into the trained multi-classification head and adaptive hybrid loss module. The trained multi-classification head and adaptive hybrid loss module outputs the classification prediction results.

[0036] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that, in step one, the grid balanced sampling method is used to process the multispectral point cloud to obtain training samples and test samples.

[0037] The specific process is as follows:

[0038] Step 11, Order The multispectral point cloud is the manual annotation ground truth with unknown annotation ratio, where N represents the total number of points in the multispectral point cloud, 3 is the spatial dimension of the multispectral point cloud, and d represents the spectral dimension of the multispectral point cloud.

[0039] The size of the grid is determined by sampling the multispectral point cloud X based on the sampling rate ε (for example, if there are 100 points and the sampling rate is 30%, then according to this sampling rate, the 100 points are sampled to obtain 30 grids), and the number of grids is obtained.

[0040] The center point of each grid is obtained, and the center points of each grid form a sparse multispectral point cloud.

[0041] Steps 1 and 2: Calculate sparse multispectral point clouds The number of points in each manually labeled category is M = {m1, m2, ..., m C};

[0042] Where m1 is the number of multispectral point clouds labeled in the first category of the manually labeled point count M, and m2 is the number of multispectral point clouds labeled in the second category of the manually labeled point count M. C The number of multispectral point clouds labeled in the Cth category among the manually labeled points M, where C is the total number of land cover categories;

[0043] A fixed proportion is randomly selected from the multispectral point cloud labeled for each category. As the point of generation of training samples

[0044]

[0045] in, The point P for generating training samples train The number of multispectral point clouds in the first category, The point P for generating training samples train The number of multispectral point clouds in the second category, The point P for generating training samples trainThe number of multispectral point clouds in category C;

[0046] Sparse multispectral point clouds The multispectral point cloud, excluding the points generated from the training samples, is used as the point P generated from the test samples. test ;

[0047] Step 13: Generate points P from the training samples in the multispectral point cloud X. train The sample set X′ is extracted using the k-neighborhood method with X′ as the center, and X′ is used as the training sample.

[0048] In the multispectral point cloud X, the points P generated from the training samples are used as the basis for the calculation. test The sample set X″ is extracted using the k-neighborhood method centered on X″, and X″ is used as the test sample.

[0049] The other steps and parameters are the same as in Specific Implementation Method 1.

[0050] Specific Implementation Method 3: This implementation method differs from Specific Implementation Method 1 or 2 in that a feature aggregation network is constructed in step 2;

[0051] The specific process is as follows:

[0052] The feature aggregation network includes a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first multi-scale feature aggregation MSFF, a second multi-scale feature aggregation MSFF, a third multi-scale feature aggregation MSFF, a first decoder, a second decoder, a third decoder, a fourth decoder, and a fifth decoder.

[0053] Other steps and parameters are the same as in specific implementation method one or two.

[0054] Specific Implementation Method Four: This implementation method differs from one of the specific implementation methods one to three in that step three involves constructing a multi-classification head and an adaptive hybrid loss module;

[0055] The multi-classification head includes two parts: a multi-scale classification head and a long-tailed distribution classification head.

[0056] The adaptive hybrid loss module includes multi-scale loss and long-tailed distribution loss;

[0057] The specific process is as follows:

[0058] The multi-scale classification head includes a first adaptive classifier, a second adaptive classifier, a third adaptive classifier, and a fourth adaptive classifier;

[0059] The long-tailed distribution classification head includes the fifth adaptive classifier and the sixth adaptive classifier;

[0060] The first adaptive classifier, the second adaptive classifier, the third adaptive classifier, the fourth adaptive classifier, the fifth adaptive classifier, and the sixth adaptive classifier are all classifiers composed of three fully connected layers cascaded together (fully connected layer, fully connected layer, fully connected layer connected in sequence).

[0061] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0062] Specific Implementation Method Five: This implementation method differs from one of the specific implementation methods one to four in that, in step four, the training samples are input into the constructed feature aggregation network, and the feature aggregation network outputs features;

[0063] The specific process is as follows:

[0064] Step 41: Input the multispectral point cloud into the first encoder, and the first encoder outputs feature A;

[0065] Step 4.2: Input feature A into the second encoder and the first multi-scale feature aggregation MSFF respectively, and the second encoder outputs feature B;

[0066] Step 4.3: The output feature B of the second encoder is input into the third encoder, the first multi-scale feature aggregation MSFF, and the second multi-scale feature aggregation MSFF respectively. The third encoder outputs feature C.

[0067] Step 4: The output feature C of the third encoder is input into the fourth encoder, the first multi-scale feature aggregation MSFF, the second multi-scale feature aggregation MSFF, and the third multi-scale feature aggregation MSFF, respectively. The fourth encoder outputs feature D.

[0068] Steps 4 and 5: The output feature D of the fourth encoder is input into the fifth encoder, the second multi-scale feature aggregation MSFF, and the third multi-scale feature aggregation MSFF, respectively. The fifth encoder outputs feature E.

[0069] Steps 4 and 6: The output feature E of the fifth encoder is input into the first decoder and the third multi-scale feature aggregation MSFF, respectively. The first decoder outputs feature F1.

[0070] Step 47

[0071] The first multi-scale feature aggregation MSFF outputs feature F;

[0072] The second multi-scale feature aggregation MSFF output feature G;

[0073] The third multi-scale feature aggregation MSFF output feature H;

[0074] Step 48: The first decoder output feature I and the third multi-scale feature aggregated MSFF output feature H are input into the second decoder, and the second decoder outputs feature F2;

[0075] Step 49: The second decoder outputs feature J and the second multi-scale feature aggregates MSFF output feature G, which is then input into the third decoder. The third decoder outputs feature F3.

[0076] Step 40: The third decoder output feature K and the first multi-scale feature are aggregated into the MSFF output feature F and input into the fourth decoder. The fourth decoder outputs feature F4.

[0077] Step 41: The fourth decoder outputs feature L, which is input into the fifth decoder. The fifth decoder outputs feature F5, which is then used as the output feature of the feature aggregation network.

[0078] The feature aggregation network is based on a typical five-layer encoder-decoder structure. In the encoder layer, the feature encoding method proposed in the RandLA-Net network is used to aggregate spatial and spectral features, and a random sampling method is used to downsample the input features to improve computational efficiency. In the decoder layer, a simple fully connected layer and an upsampling module are used to maintain processing speed. A multi-scale feature aggregation module is used as a skip connection between the encoder and decoder layers to improve the preservation of ground feature details. Since the encoder-decoder network structure has five layers, three layers of the multi-scale feature aggregation module are applied.

[0079] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0080] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One to Five in that the specific operation process of the first multi-scale feature aggregation MSFF is as follows:

[0081] 1) The first encoder serves as the upper-level encoder;

[0082] The output feature A of the first encoder is used as the output feature of the upper encoder (shallow feature).

[0083] The second encoder serves as the encoder for this layer;

[0084] The output feature B of the second encoder is used as the output feature of this layer encoder (this layer feature).

[0085] The third encoder serves as the lower-level encoder;

[0086] The output feature C of the third encoder is used as the output feature (deep feature) of the lower encoder.

[0087] Where n and c are F L The corresponding receptive field size and feature dimension; It is a real number;

[0088] 2) To facilitate the aggregation of fine features at different scales;

[0089] For feature F S Downsampling to obtain features With F L It is a new feature that has the same receptive field;

[0090] For feature F D Upsampling to obtain features With F L It is a new feature that has the same receptive field;

[0091] 3) Feature F S ′ and feature F L Fusion to obtain shallow feature weights e S The expression is:

[0092]

[0093] in, This represents the global average pooling operation, where σ is the Sigmoid function.

[0094] Global average pooling is used to extract overall features, and these features are then fused through element-wise summation. The final shallow feature weights are e. S Calculated by the Sigmoid function;

[0095] 4) Based on shallow feature weight e S and feature F S ′, to obtain shallow fusion features The expression is:

[0096]

[0097] in, This is an element-wise multiplication operation;

[0098] 5) Transfer feature F′ D With feature F L Fusion to obtain deep feature weights e D The expression is:

[0099]

[0100] Global average pooling is used to extract overall features, and these features are then fused through element-wise multiplication to obtain the final deep feature weight e. D Calculated by the Sigmoid function;

[0101] 6) Based on deep feature weight e D By combining features F′D, deep fusion features are obtained. The expression is:

[0102]

[0103] 7) Connecting shallow fusion features Feature F L With deep fusion features And obtain the output features through the MLP layer. The expression is:

[0104]

[0105] in, It is an MLP layer.

[0106] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0107] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the specific operation process of the second multi-scale feature aggregation MSFF is as follows:

[0108] 1) The second encoder serves as the upper-level encoder;

[0109] The output feature B of the second encoder is used as the output feature of the upper encoder (shallow feature).

[0110] The third encoder serves as the encoder for this layer;

[0111] The output feature C of the third encoder is used as the output feature of the encoder at this layer (this layer feature).

[0112] The fourth encoder serves as the lower-level encoder;

[0113] The output feature D of the fourth encoder is used as the output feature (deep feature) of the lower encoder.

[0114] Where n and c are F L The corresponding receptive field size and feature dimension; It is a real number;

[0115] 2) To facilitate the aggregation of fine features at different scales;

[0116] For feature F S Downsampling to obtain features With F L It is a new feature that has the same receptive field;

[0117] For feature F D Upsampling to obtain features With F LIt is a new feature that has the same receptive field;

[0118] 3) Feature F S ′ and feature F L Fusion to obtain shallow feature weights e S The expression is:

[0119]

[0120] in, This represents the global average pooling operation, where σ is the Sigmoid function.

[0121] Global average pooling is used to extract overall features, and these features are then fused through element-wise summation. The final shallow feature weights are e. S Calculated by the Sigmoid function;

[0122] 4) Based on shallow feature weight e S and feature F S ′, to obtain shallow fusion features The expression is:

[0123]

[0124] in, This is an element-wise multiplication operation;

[0125] 5) Transfer feature F′ D With feature F L Fusion to obtain deep feature weights e D The expression is:

[0126]

[0127] Global average pooling is used to extract overall features, and these features are then fused through element-wise multiplication to obtain the final deep feature weight e. D Calculated by the Sigmoid function;

[0128] 6) Based on deep feature weight e D and feature F′ D To obtain deep fusion features The expression is:

[0129]

[0130] 7) Connecting shallow fusion features Feature F L With deep fusion features And obtain the output features through the MLP layer. The expression is:

[0131]

[0132] in, It is an MLP layer.

[0133] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0134] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One to Seven in that the specific operation process of the third multi-scale feature aggregation MSFF is as follows:

[0135] 1) The third encoder acts as the upper-level encoder;

[0136] The output feature C of the third encoder is used as the output feature of the upper encoder (shallow feature).

[0137] The fourth encoder serves as the encoder for this layer;

[0138] The output feature D of the fourth encoder is used as the output feature of the encoder in this layer (this layer feature).

[0139] The fifth encoder serves as the lower-level encoder;

[0140] The output feature E of the fifth encoder is used as the output feature (deep feature) of the lower encoder.

[0141] Where n and c are F L The corresponding receptive field size and feature dimension; It is a real number;

[0142] 2) To facilitate the aggregation of fine features at different scales;

[0143] For feature F S Downsampling to obtain features With F L It is a new feature that has the same receptive field;

[0144] For feature F D Upsampling to obtain features With F L It is a new feature that has the same receptive field;

[0145] 3) Feature F S ′ and feature F L Fusion to obtain shallow feature weights e S The expression is:

[0146]

[0147] in, This represents the global average pooling operation, where σ is the Sigmoid function.

[0148] Global average pooling is used to extract overall features, and these features are then fused through element-wise summation. The final shallow feature weights are e. S Calculated by the Sigmoid function;

[0149] 4) Based on shallow feature weight e S and feature F S ′, to obtain shallow fusion features The expression is:

[0150]

[0151] in, This is an element-wise multiplication operation;

[0152] 5) Transfer feature F′ D With feature F L Fusion to obtain deep feature weights e D The expression is:

[0153]

[0154] Global average pooling is used to extract overall features, and these features are then fused through element-wise multiplication to obtain the final deep feature weight e. D Calculated by the Sigmoid function;

[0155] 6) Based on deep feature weight e D and feature F′ D To obtain deep fusion features The expression is:

[0156]

[0157] 7) Connecting shallow fusion features Feature F L With deep fusion features And obtain the output features through the MLP layer. The expression is:

[0158]

[0159] in, It is an MLP layer.

[0160] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0161] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One to Eight in that, in step four, the feature aggregation network outputs features to construct a multi-classification head and an adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module outputs classification prediction results.

[0162] The specific process is as follows:

[0163] Step 42: Input the feature F1 output from the first decoder of the feature aggregation network into the first adaptive classifier. The first adaptive classifier outputs the class prediction result Z1.

[0164] Where L represents the number of categories, and N represents the total number of points in the multispectral point cloud. z represents the total number of points output by the first adaptive classifier. 11 z represents the class prediction result of the first point output by the first adaptive classifier. 12 This represents the class prediction result of the second point output by the first adaptive classifier. The output of the first adaptive classifier represents the first... Individual category prediction results;

[0165] The expression is:

[0166]

[0167] in w1 is the first adaptive classifier; w1 is the adaptive weight. This is an element-wise multiplication operation; based on the obtained category prediction result Z1 and the ground truth value of the corresponding ground feature at the output point of the first adaptive classifier. Calculate the loss using the cross-entropy loss function:

[0168]

[0169] Where y 1j z represents the ground truth value of the ground features at the j-th point corresponding to the first adaptive classifier after one-hot encoding. 1j The class prediction result of the j-th point output by the first adaptive classifier;

[0170] Step 43: Input the feature F2 output from the second decoder of the feature aggregation network into the second adaptive classifier. The first adaptive classifier outputs the class prediction result Z2.

[0171] Where L represents the number of categories, and N represents the total number of points in the multispectral point cloud. z represents the total number of points output by the second adaptive classifier. 21z represents the class prediction result of the first point output by the second adaptive classifier. 22 This represents the class prediction result of the second point output by the second adaptive classifier. The output of the second adaptive classifier represents the first... Individual category prediction results;

[0172] The expression is:

[0173]

[0174] in w2 is the second adaptive classifier; w2 is the adaptive weight. This is an element-wise multiplication operation;

[0175] Based on the obtained category prediction result Z2 and the ground cover true value corresponding to the output point of the second adaptive classifier Calculate the loss using the cross-entropy loss function:

[0176]

[0177] Where y 2j z represents the ground truth value of the ground features at the j-th point corresponding to the second adaptive classifier after one-hot encoding. 2j The class prediction result for the j-th point output by the second adaptive classifier;

[0178] Step 44: Input the feature F3 output from the third decoder of the feature aggregation network into the third adaptive classifier. The third adaptive classifier outputs the class prediction result Z3.

[0179] Where L represents the number of categories, and N represents the total number of points in the multispectral point cloud. z represents the total number of points output by the third adaptive classifier. 31 z represents the class prediction result of the first point output by the third adaptive classifier. 32 This represents the class prediction result of the second point output by the third adaptive classifier. The output of the third adaptive classifier represents the... Individual category prediction results;

[0180] The expression is:

[0181]

[0182] Where C(·) is the third adaptive classifier; w3 is the adaptive weight. This is an element-wise multiplication operation;

[0183] Based on the obtained category prediction result Z3 and the ground cover true value corresponding to the output point of the third adaptive classifier Calculate the loss using the cross-entropy loss function:

[0184]

[0185] Where y 3j z represents the ground truth value of the ground features at the j-th point corresponding to the third adaptive classifier after one-hot encoding. 3j The class prediction result for the j-th point output by the third adaptive classifier;

[0186] Step 45: Input the feature F4 output from the fourth decoder of the feature aggregation network into the fourth adaptive classifier. The fourth adaptive classifier outputs the class prediction result Z4.

[0187] Where L represents the number of categories, and N represents the total number of points in the multispectral point cloud. z represents the total number of points output by the fourth adaptive classifier. 41 z represents the class prediction result of the first point output by the fourth adaptive classifier. 42 This represents the class prediction result of the second point output by the fourth adaptive classifier. The output of the fourth adaptive classifier represents the... Individual category prediction results;

[0188] The expression is:

[0189]

[0190] in This is the fourth adaptive classifier; w4 represents the adaptive weights. This is an element-wise multiplication operation;

[0191] Based on the obtained category prediction result Z4 and the ground cover true value corresponding to the output point of the fourth adaptive classifier Calculate the loss using the cross-entropy loss function:

[0192]

[0193] Where y 4j Let z represent the ground truth value of the ground features at the j-th point corresponding to the fourth adaptive classifier after one-hot encoding. 4j The class prediction result for the j-th point output by the fourth adaptive classifier;

[0194] Step 46: Calculate the multi-scale loss based on steps 42 to 45:

[0195]

[0196] Step 47

[0197] The feature F5 output from the fifth decoder of the feature aggregation network is input into the fifth adaptive classifier, and the fifth adaptive classifier outputs the class prediction result. This represents the class prediction result of the first point output by the fifth adaptive classifier. This represents the class prediction result of the second point output by the fifth adaptive classifier. This represents the class prediction result of the Nth point output by the fourth adaptive classifier;

[0198] The expression is:

[0199]

[0200] in The fifth adaptive classifier; For adaptive weights, This is an element-wise multiplication operation;

[0201] The fifth adaptive classifier learns the classification features of the head and tail categories (all categories) in a long-tailed distribution scenario;

[0202] Based on the obtained category prediction results True values ​​of all categories of land features Calculate the loss using the root mean square loss function:

[0203]

[0204] in This represents the ground feature truth value after one-hot encoding of the ground feature truth value at point j. The class prediction result for the j-th point output by the fifth adaptive classifier;

[0205] The feature F5 output from the fifth decoder of the feature aggregation network is input into the sixth adaptive classifier, which outputs the class prediction result. This represents the class prediction result of the first point output by the sixth adaptive classifier. This represents the class prediction result of the second point output by the sixth adaptive classifier. This represents the class prediction result of the Nth point output by the sixth adaptive classifier;

[0206] The expression is:

[0207]

[0208] Where C(·) is the sixth adaptive classifier; For adaptive weights, This is an element-wise multiplication operation;

[0209] The sixth adaptive classifier learns the classification features of the tail category in the long-tail distribution scenario. To avoid affecting the results of the fifth adaptive classifier, the predicted probability of the head category output in the sixth adaptive classifier should be as small as possible.

[0210] Based on the obtained category prediction results True values ​​of land features relative to tail categories Calculate the loss using the root mean square loss function:

[0211]

[0212] in This represents the ground truth value of the j-th point after one-hot encoding, where the j-th point belongs to the head category. The class prediction result for the j-th point output by the sixth adaptive classifier;

[0213] During training, the ground truth values ​​of all head category objects are set to zero. This requires the tail category classifier to output a value as close to 0 as possible when encountering head categories, so as not to affect the classification results of the fifth adaptive classifier and thus distinguish the fifth and sixth adaptive classifiers.

[0214] If the proportion of multispectral points in a certain category to the total number of multispectral points in all categories of the entire scene is less than 5%, then that category is considered a tail category, and the other categories are considered head categories.

[0215] Step 48: Calculate the long-tailed distribution loss based on Step 47:

[0216]

[0217] Step 49: Calculate the adaptive hybrid loss based on multi-scale loss and long-tailed distribution loss.

[0218] Due to the long-tailed distribution and scale differences of ground features in remote sensing scenes, an adaptive hybrid loss module consisting of multi-scale loss and long-tail loss is proposed. This adaptive hybrid loss module can simultaneously utilize multiple classifiers to learn effective classification features for different categories, and finally uses adaptive weights to merge the outputs of different classifiers.

[0219] The multi-scale loss module is as follows: For the i-th decoder layer, the output feature can be denoted as... Therefore, the fifth decoder layer, i.e., the final layer, outputs the following features: Where N and C are the receptive field size and feature dimension corresponding to F5, respectively.

[0220] For the low-resolution features output by the first four decoder layers, a simple classifier consisting of three fully connected layers is used to obtain the class prediction at the corresponding resolution. Where L represents the number of categories.

[0221] Since different land cover types are classified differently at different scales, an adaptive weight is applied to each classifier. Automatically extract the features with the best classification accuracy at the current resolution:

[0222]

[0223] in For classifiers;

[0224] Then the obtained category predictions and the corresponding downsampled ground feature ground values ​​are compared. Calculate the loss using the cross-entropy loss function:

[0225]

[0226] The final multi-scale loss is a weighted sum of the losses at different resolutions:

[0227]

[0228] The long-tailed distribution loss module works as follows: the final feature F5 is input into two classifiers constructed from fully connected layers to obtain the class prediction. Similar to multi-scale loss, an adaptive weight is applied to each classifier.

[0229]

[0230] For the loss function, the goal is for the first classifier to learn all land cover categories, while the second classifier only learns the classification features of the tail categories. Therefore, the key to designing the loss function is how to make the two classifiers work together.

[0231] Compared to the common cross-entropy loss function, the root mean square loss function can better suppress the output probabilities of non-target classes and reduce the mutual influence of results between different classifiers.

[0232]

[0233] in Represents the true values ​​of all categories of land features. The ground truth value of the feature representing the tail category (set to 0 for the head category).

[0234] The final long-tail loss is a weighted sum of the losses from different classifiers:

[0235]

[0236] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0237] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One through Nine in that step forty-nine calculates the adaptive hybrid loss based on multi-scale loss and long-tail loss.

[0238] L = L scale +L tail .

[0239] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0240] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A multispectral point cloud classification method for remote sensing scenes, characterized in that: The specific process of the method is as follows: Step 1: Process the multispectral point cloud using the grid balanced sampling method to obtain training and test samples; Step 2: Construct a feature aggregation network; Step 3: Construct a multi-classification head and adaptive hybrid loss module; The multi-classification head includes two parts: a multi-scale classification head and a long-tailed distribution classification head. The adaptive hybrid loss module includes multi-scale loss and long-tailed distribution loss; Step 4: Input the training samples into the constructed feature aggregation network, and the feature aggregation network outputs features; The feature aggregation network outputs the feature input to construct a multi-classification head and an adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module output the classification prediction result. Train the feature aggregation network and the multi-classification head and adaptive hybrid loss module until convergence, and obtain the trained feature aggregation network and the multi-classification head and adaptive hybrid loss module. Step 5: Input the test samples into the trained feature aggregation network. The trained feature aggregation network outputs features into the trained multi-classification head and adaptive hybrid loss module. The trained multi-classification head and adaptive hybrid loss module outputs the classification prediction results. In step one, the multispectral point cloud is processed using the grid balanced sampling method to obtain training samples and test samples; The specific process is as follows: Step 11, Order Multispectral point clouds containing manually labeled ground truth values ​​for unknown label percentages, where This represents the total number of points in the multispectral point cloud, where 3 represents the spatial dimension of the multispectral point cloud. Represents the spectral dimension of a multispectral point cloud; Based on sampling rate Multispectral point clouds Sampling is performed to determine the grid size and obtain the number of grid cells; The center point of each grid is obtained, and the center points of each grid form a sparse multispectral point cloud. ; Steps 1 and 2: Calculate sparse multispectral point clouds Points for each manually labeled category ; in, Points for manually labeled categories The number of multispectral point clouds labeled in the first category, Points for manually labeled categories The number of multispectral point clouds labeled in the second category. Points for manually labeled categories The number of multispectral point clouds labeled in category C. This represents the total number of land cover categories. A fixed proportion is randomly selected from the multispectral point cloud labeled for each category. As the point of generation of training samples , , , ; in, The point of generation of training samples The number of multispectral point clouds in the first category, The point of generation of training samples The number of multispectral point clouds in the second category, The point of generation of training samples The number of multispectral point clouds in category C; Sparse multispectral point clouds The multispectral point cloud, excluding the points generated from the training samples, is used as the points generated from the test samples. ; Step 13: In multispectral point clouds The generation points of training samples in China and Israel Extracting the sample set using the k-neighborhood method centered on the sample set , As training samples; In multispectral point clouds The generation points of training samples in China and Israel Extracting the sample set using the k-neighborhood method centered on the sample set , As a test sample; In step two, a feature aggregation network is constructed; The specific process is as follows: The feature aggregation network includes a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first multi-scale feature aggregation MSFF, a second multi-scale feature aggregation MSFF, a third multi-scale feature aggregation MSFF, a first decoder, a second decoder, a third decoder, a fourth decoder, and a fifth decoder. In step three, a multi-classification head and an adaptive hybrid loss module are constructed. The multi-classification head includes two parts: a multi-scale classification head and a long-tailed distribution classification head. The adaptive hybrid loss module includes multi-scale loss and long-tailed distribution loss; The specific process is as follows: The multi-scale classification head includes a first adaptive classifier, a second adaptive classifier, a third adaptive classifier, and a fourth adaptive classifier; The long-tailed distribution classification head includes the fifth adaptive classifier and the sixth adaptive classifier; The first adaptive classifier, the second adaptive classifier, the third adaptive classifier, the fourth adaptive classifier, the fifth adaptive classifier, and the sixth adaptive classifier are all classifiers composed of three fully connected layers cascaded together.

2. The multispectral point cloud classification method for remote sensing scenes according to claim 1, characterized in that: In step four, the training samples are input into the constructed feature aggregation network, and the feature aggregation network outputs features. The specific process is as follows: Step 41: Input the multispectral point cloud into the first encoder, and the first encoder outputs feature A; Step 4.2: Input feature A into the second encoder and the first multi-scale feature aggregation MSFF respectively, and the second encoder outputs feature B; Step 4.3: The output feature B of the second encoder is input into the third encoder, the first multi-scale feature aggregation MSFF, and the second multi-scale feature aggregation MSFF respectively. The third encoder outputs feature C. Step 4: The output feature C of the third encoder is input into the fourth encoder, the first multi-scale feature aggregation MSFF, the second multi-scale feature aggregation MSFF, and the third multi-scale feature aggregation MSFF, respectively. The fourth encoder outputs feature D. Steps 4 and 5: The output feature D of the fourth encoder is input into the fifth encoder, the second multi-scale feature aggregation MSFF, and the third multi-scale feature aggregation MSFF, respectively. The fifth encoder outputs feature E. Steps four and six: The output features E of the fifth encoder are input into the first decoder and the third multi-scale feature aggregation (MSFF) respectively. The output features of the first decoder are... ; Step 47 The first multi-scale feature aggregation MSFF outputs feature F; The second multi-scale feature aggregation MSFF output feature G; The third multi-scale feature aggregation MSFF output feature H; Step 48: The first decoder output feature I and the third multi-scale feature are aggregated into the MSFF output feature H, which is then input into the second decoder. The second decoder outputs the feature... ; Step 49: The second decoder outputs feature J, and the second multi-scale feature aggregates with the MSFF output feature G, which is then input into the third decoder. The third decoder outputs feature G. ; Step 40: The third decoder output feature K and the first multi-scale feature are aggregated into the MSFF output feature F, which is then input into the fourth decoder. The fourth decoder outputs the feature... ; Step 41: The fourth decoder outputs feature L, which is input to the fifth decoder; the fifth decoder outputs feature L. ,feature As output features of the feature aggregation network.

3. The multispectral point cloud classification method for remote sensing scenes according to claim 2, characterized in that: The specific operation process of the first multi-scale feature aggregation MSFF is as follows: 1) The first encoder serves as the upper-level encoder; The output feature A of the first encoder is used as the output feature of the upper encoder. ; The second encoder serves as the encoder for this layer; The output feature B of the second encoder is used as the output feature of the encoder in this layer. ; The third encoder serves as the lower-level encoder; The output feature C of the third encoder is used as the output feature of the lower encoder. ; in, and They are respectively The corresponding receptive field size and feature dimension; It is a real number; 2)、 Features Downsampling to obtain features ; Features Upsampling to obtain features ; 3) Features With features Fusion to obtain shallow feature weights The expression is: in, This represents the global average pooling operation. For the Sigmoid function; 4) Based on shallow feature weights and characteristics To obtain shallow fusion features The expression is: in, This is an element-wise multiplication operation; 5) Features With features Fusion to obtain deep feature weights The expression is: 6) Based on deep feature weights and characteristics To obtain deep fusion features The expression is: 7) Connecting shallow fusion features ,feature With deep fusion features Output features are obtained through an MLP layer. The expression is: in, It is an MLP layer.

4. The multispectral point cloud classification method for remote sensing scenes according to claim 3, characterized in that: The specific operation process of the second multi-scale feature aggregation MSFF is as follows: 1) The second encoder serves as the upper-level encoder; The output feature B of the second encoder is used as the output feature of the upper encoder. ; The third encoder serves as the encoder for this layer; The output feature C of the third encoder is used as the output feature of the encoder at this layer. ; The fourth encoder serves as the lower-level encoder; The fourth encoder output feature D is used as the lower encoder output feature. ; in, and They are respectively The corresponding receptive field size and feature dimension; It is a real number; 2)、 Features Downsampling to obtain features ; Features Upsampling to obtain features ; 3) Features With features Fusion to obtain shallow feature weights The expression is: in, This represents the global average pooling operation. For the Sigmoid function; 4) Based on shallow feature weights and characteristics To obtain shallow fusion features The expression is: in, This is an element-wise multiplication operation; 5) Features With features Fusion to obtain deep feature weights The expression is: 6) Based on deep feature weights and characteristics To obtain deep fusion features The expression is: 7) Connecting shallow fusion features ,feature With deep fusion features Output features are obtained through an MLP layer. The expression is: in, It is an MLP layer.

5. A multispectral point cloud classification method for remote sensing scenarios according to claim 4, characterized in that: The specific operation process of the third multi-scale feature aggregation MSFF is as follows: 1) The third encoder acts as the upper-level encoder; The output feature C of the third encoder is used as the output feature of the upper encoder. ; The fourth encoder serves as the encoder for this layer; The fourth encoder output feature D is used as the output feature of this layer encoder. ; The fifth encoder serves as the lower-level encoder; The output feature E of the fifth encoder is used as the output feature of the lower encoder. ; in, and They are respectively The corresponding receptive field size and feature dimension; It is a real number; 2)、 Features Downsampling to obtain features ; Features Upsampling to obtain features ; 3) Features With features Fusion to obtain shallow feature weights The expression is: in, This represents the global average pooling operation. For the Sigmoid function; 4) Based on shallow feature weights and characteristics To obtain shallow fusion features The expression is: in, This is an element-wise multiplication operation; 5) Features With features Fusion to obtain deep feature weights The expression is: 6) Based on deep feature weights and characteristics To obtain deep fusion features The expression is: 7) Connecting shallow fusion features ,feature With deep fusion features Output features are obtained through an MLP layer. The expression is: in, It is an MLP layer.

6. The multispectral point cloud classification method for remote sensing scenarios according to claim 5, characterized in that: In step four, the feature aggregation network outputs the feature input to construct a multi-classification head and an adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module outputs the classification prediction result. The specific process is as follows: Step 42: Output features from the first decoder of the feature aggregation network. Input the first adaptive classifier, and the first adaptive classifier outputs the class prediction result. , ; in For the number of categories, This represents the total number of points in the multispectral point cloud. This represents the total number of points output by the first adaptive classifier. This represents the class prediction result of the first point output by the first adaptive classifier. This represents the class prediction result of the second point output by the first adaptive classifier. The output of the first adaptive classifier represents the first... Individual category prediction results; The expression is: in This is the first adaptive classifier; For adaptive weights, ; This is an element-wise multiplication operation; Based on the obtained category prediction results Ground truth value corresponding to the output point of the first adaptive classifier Calculate the loss using the cross-entropy loss function: in This indicates the first adaptive classifier corresponding to the first classifier. The ground feature ground feature true value after one-hot encoding of the ground feature true value of each point. The output of the first adaptive classifier Individual category prediction results; Step 43: Output features from the second decoder of the feature aggregation network. The first adaptive classifier outputs the class prediction result as input to the second adaptive classifier. , ; in For the number of categories, This represents the total number of points in the multispectral point cloud. This represents the total number of points output by the second adaptive classifier. This represents the class prediction result of the first point output by the second adaptive classifier. This represents the class prediction result of the second point output by the second adaptive classifier. The output of the second adaptive classifier represents the first... Individual category prediction results; The expression is: in This is the second adaptive classifier; For adaptive weights, ; This is an element-wise multiplication operation; Based on the obtained category prediction results Ground truth value corresponding to the output point of the second adaptive classifier Calculate the loss using the cross-entropy loss function: in This indicates the first corresponding to the second adaptive classifier. The ground feature ground feature true value after one-hot encoding of the ground feature true value of each point. The output of the second adaptive classifier Individual category prediction results; Step 44: Output features from the third decoder of the feature aggregation network. The input is a third adaptive classifier, and the output of the third adaptive classifier is the class prediction result. , ; in For the number of categories, This represents the total number of points in the multispectral point cloud. This represents the total number of points output by the third adaptive classifier. This represents the class prediction result of the first point output by the third adaptive classifier. This represents the class prediction result of the second point output by the third adaptive classifier. The output of the third adaptive classifier represents the... Individual category prediction results; The expression is: in For the third adaptive classifier; For adaptive weights, ; This is an element-wise multiplication operation; Based on the obtained category prediction results Ground truth value corresponding to the output point of the third adaptive classifier Calculate the loss using the cross-entropy loss function: in This indicates the first corresponding to the third adaptive classifier. The ground feature ground feature true value after one-hot encoding of the ground feature true value of each point. The output of the third adaptive classifier Individual category prediction results; Step 45: Output features from the fourth decoder of the feature aggregation network. Input the fourth adaptive classifier, and the fourth adaptive classifier outputs the class prediction result. , ; in For the number of categories, This represents the total number of points in the multispectral point cloud. This represents the total number of points output by the fourth adaptive classifier. This represents the class prediction result of the first point output by the fourth adaptive classifier. This represents the class prediction result of the second point output by the fourth adaptive classifier. The output of the fourth adaptive classifier represents the... Individual category prediction results; The expression is: in It is the fourth adaptive classifier; For adaptive weights, ; This is an element-wise multiplication operation; Based on the obtained category prediction results Ground truth value corresponding to the output point of the fourth adaptive classifier Calculate the loss using the cross-entropy loss function: in To represent the fourth adaptive classifier corresponding to the first The ground feature ground feature true value after one-hot encoding of the ground feature true value of each point. The output of the fourth adaptive classifier Individual category prediction results; Step 46: Calculate the multi-scale loss based on steps 42 to 45: Step 47 The fifth decoder of the feature aggregation network outputs features. Input the fifth adaptive classifier, and the fifth adaptive classifier outputs the class prediction result. , , This represents the class prediction result of the first point output by the fifth adaptive classifier. This represents the class prediction result of the second point output by the fifth adaptive classifier. The output of the fourth adaptive classifier represents the... Individual category prediction results; The expression is: in The fifth adaptive classifier; For adaptive weights, ; This is an element-wise multiplication operation; Based on the obtained category prediction results True values ​​of all categories of land features The loss is calculated using the root mean square loss function: in Indicates the first The ground feature ground feature true value after one-hot encoding of the ground feature true value of each point. The output of the fifth adaptive classifier Individual category prediction results; The fifth decoder of the feature aggregation network outputs features. Input the sixth adaptive classifier, and the sixth adaptive classifier outputs the class prediction result. , , This represents the class prediction result of the first point output by the sixth adaptive classifier. This represents the class prediction result of the second point output by the sixth adaptive classifier. This represents the output of the sixth adaptive classifier. Individual category prediction results; The expression is: in The sixth adaptive classifier; For adaptive weights, ; This is an element-wise multiplication operation; Based on the obtained category prediction results True values ​​of land features relative to tail categories The loss is calculated using the root mean square loss function: in Indicates the first The ground feature ground feature ground feature after uniquely hot encoding of the ground feature ... When a point belongs to the head category , The output of the sixth adaptive classifier Individual category prediction results; If the number of multispectral points in a certain category accounts for less than 5% of the total number of multispectral points in all categories of the entire scene, it is considered a tail category, and the other categories are considered head categories. Step 48: Calculate the long-tailed distribution loss based on Step 47: Step 49: Calculate the adaptive hybrid loss based on multi-scale loss and long-tailed distribution loss.

7. A multispectral point cloud classification method for remote sensing scenarios according to claim 6, characterized in that: In step 49, an adaptive hybrid loss is calculated based on multi-scale loss and long-tail loss: 。

Images