Multispectral point cloud classification method for remote sensing scene
Through grid balanced sampling and multi-scale feature aggregation network, combined with the adaptive hybrid loss module, the problems of poor classification effect and difficult to obtain manual labeling data in the multi-spectral point cloud classification method are solved, and the multi-spectral point cloud in remote sensing scenarios are improved Classification accuracy, especially the recognition effect of small categories of land objects.
Patent Information
- Application Number
- CN202411056528.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-02
AI Technical Summary
Among the existing multispectral point cloud classification methods, machine learning-based methods have poor classification effects, while deep learning-based methods require a large number of manual annotation truth-assisted training, and existing deep learning methods are mostly tested on indoor data sets. Lack of optimized design for remote sensing scenarios.
The grid balanced sampling method is used to extract training samples from sparsely annotated multispectral point clouds, and a multi-scale feature aggregation network and an adaptive hybrid loss module are designed to optimize the learning ability of feature extraction and classifiers, and target the land objects in remote sensing scenarios. Scale difference and long-tail distribution phenomenon.
The classification accuracy of remote sensing multispectral point clouds is improved, especially the classification accuracy of small-category land objects. The verification results show that higher overall accuracy and classification effect of small-category land objects are achieved on real data.
Smart Images

Figure CN118941870A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a multispectral point cloud classification method. Background Art
[0002] Multispectral point clouds contain the three-dimensional spatial and spectral information of objects, and can be used to interpret the composition and distribution of objects using stereoscopic observation methods. Compared with traditional observation techniques, multispectral point cloud data has many advantages: the imaging effect of multi-hyperspectral images is greatly affected by lighting conditions and lacks three-dimensional spatial information; LiDAR point clouds only obtain three-dimensional spatial information of objects but lack spectral information. Multispectral point clouds can simultaneously obtain three-dimensional spatial and spectral information of scene objects, and have great potential in a variety of downstream tasks.
[0003] At present, there are two main types of multispectral point cloud classification methods, one is based on machine learning, and the other is based on deep learning. The machine learning-based method has a small amount of calculation and requires less training data, but the classification effect is poor; the deep learning-based method has a higher overall classification accuracy, but requires a large amount of manually annotated truth-value auxiliary training; in addition, most of the existing deep learning methods are widely tested on indoor data sets, and few are optimized for the characteristics of remote sensing scenes. Summary of the invention
[0004] The purpose of the present invention is to solve the problem that the existing multispectral point cloud classification method based on machine learning has poor classification effect and the multispectral point cloud classification method based on deep learning requires a large amount of manually annotated truth-value auxiliary training, and propose a multispectral point cloud classification method for remote sensing scenes.
[0005] A multispectral point cloud classification method for remote sensing scenes. The specific process is as follows:
[0006] Step 1: Use the grid balanced sampling method to process the multispectral point cloud to obtain training and test samples;
[0007] Step 2: Build a feature aggregation network;
[0008] Step 3: Construct a multi-classification head and an adaptive hybrid loss module;
[0009] The multi-classification head includes a multi-scale classification head and a long-tail distribution classification head;
[0010] The adaptive hybrid loss module includes multi-scale loss and long-tail distribution loss;
[0011] Step 4: Input the training samples into the constructed feature aggregation network, and the feature aggregation network outputs features;
[0012] The output features of the feature aggregation network are input into the constructed multi-classification head and adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module output the classification prediction results;
[0013] The feature aggregation network, the multi-classification head, and the adaptive mixed loss module are trained until convergence, and the trained feature aggregation network, the multi-classification head, and the adaptive mixed loss module are obtained;
[0014] Step 5: Input the test sample into the trained feature aggregation network. The output features of the trained feature aggregation network are input into the trained multi-classification head and adaptive mixed loss module. The trained multi-classification head and adaptive mixed loss module output the classification prediction results.
[0015] The beneficial effects of the present invention are:
[0016] The present invention can solve the above problems starting from the multi-scale differences and long-tail distribution of objects in remote sensing scenes (currently, there are mainly two types of multispectral point cloud classification methods, one is a method based on machine learning, and the other is a method based on deep learning. The method based on machine learning has a small amount of calculation and requires less training data, but the classification effect is poor; the method based on deep learning has a higher overall classification accuracy, but requires a large amount of manually labeled true value auxiliary training; in addition, most of the existing deep learning methods are widely tested on indoor data sets, and few are optimized for the characteristics of remote sensing scenes.); first, in order to solve the problem that manually labeled data of remote sensing scenes is difficult to obtain, a grid balanced sampling method is used to reliably extract training samples from sparsely labeled multispectral point clouds; then, in view of the scale difference and quantity difference (long-tail distribution) of objects in remote sensing scenes, a multi-scale feature aggregation skip connection module is designed to optimize the feature extraction effect of the feature aggregation network; an adaptive mixed loss module is designed to optimize the learning ability of multiple classification heads.
[0017] The method of the present invention reliably extracts training areas from sparsely annotated multispectral point clouds through a grid balanced sampling method, and optimizes the feature aggregation network and classifier through a multi-scale feature aggregation module and an adaptive mixed loss module to solve the problems of low learning efficiency and poor classification accuracy of small categories when facing remote sensing multispectral point clouds with scale differences and long-tail distribution phenomena.
[0018] In order to verify the performance of the present invention, a set of real multispectral point cloud data was used for verification. The experimental results show that compared with the current representative methods, the present method can achieve higher overall accuracy and higher classification accuracy for small categories of objects. The experimental results verify the effectiveness of the multispectral point cloud classification method proposed in the present invention for complex remote sensing scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a schematic diagram of the implementation process of the present invention;
[0020] Figure 2 It is a schematic diagram of the implementation process of the grid balance sampling strategy in the present invention;
[0021] Figure 3 It is a schematic diagram of the implementation process of the multi-scale feature aggregation module in the present invention;
[0022] Figure 4 is a schematic diagram of the implementation flow of the adaptive mixed loss module in the present invention, and For F5, same;
[0023] Figure 5a It is a false color composite of a multispectral point cloud;
[0024] Figure 5b It is a manually annotated sparse feature label map, where the black area is the unlabeled feature;
[0025] Figure 6 This is a comparison chart of the classification effects of different methods. DETAILED DESCRIPTION
[0026] Specific implementation method 1: This implementation method is a multispectral point cloud classification method for remote sensing scenes. The specific process is as follows:
[0027] Step 1: Use the grid balanced sampling method to process the multispectral point cloud to obtain training and test samples;
[0028] Step 2: Build a feature aggregation network;
[0029] Step 3: Construct a multi-classification head and an adaptive hybrid loss module;
[0030] The multi-classification head includes a multi-scale classification head and a long-tail distribution classification head;
[0031] The adaptive hybrid loss module includes multi-scale loss and long-tail distribution loss;
[0032] Step 4: Input the training samples into the constructed feature aggregation network, and the feature aggregation network outputs features;
[0033] The output features of the feature aggregation network are input into the constructed multi-classification head and adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module output the classification prediction results;
[0034] The feature aggregation network, the multi-classification head, and the adaptive mixed loss module are trained until convergence, and the trained feature aggregation network, the multi-classification head, and the adaptive mixed loss module are obtained;
[0035] Step 5: Input the test sample into the trained feature aggregation network. The output features of the trained feature aggregation network are input into the trained multi-classification head and adaptive mixed loss module. The trained multi-classification head and adaptive mixed loss module output the classification prediction results.
[0036] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that in step 1, a grid-balanced sampling method is used to process the multispectral point cloud to obtain training samples and test samples;
[0037] The specific process is:
[0038] Step 1: Order is the multispectral point cloud of manually annotated true values with unknown annotation ratio, where N represents the total number of points in the multispectral point cloud, 3 represents the spatial dimension of the multispectral point cloud, and d represents the spectral dimension of the multispectral point cloud;
[0039] The multispectral point cloud X is sampled based on the sampling rate ε to determine the size of the grid (for example, if there are 100 points and the sampling rate is 30%, then according to this sampling rate, 100 points are sampled to obtain 30 grids), and the number of grids is obtained;
[0040] Get the center point of each grid, and the center point of each grid forms a sparse multispectral point cloud
[0041] Step 1 and 2: Calculate sparse multispectral point cloud The number of points in each manually labeled category is M = {m1,m2,…,m C};
[0042] Among them, m1 is the number of multispectral point clouds marked in the first category of the manually labeled points M, m2 is the number of multispectral point clouds marked in the second category of the manually labeled points M, and m C is the number of multispectral point clouds annotated in the Cth category among the number of points M of manually annotated categories, and C is the total number of ground object categories;
[0043] Randomly select a fixed proportion from the multispectral point cloud labeled for each category As the generation point of training samples
[0044]
[0045] in, is the generating point P of the training sample train The number of multispectral point clouds in the first category, is the generating point P of the training sample train The number of multispectral point clouds in the second category, is the generating point P of the training sample trainThe number of multispectral point clouds in the Cth category;
[0046] Sparse multispectral point cloud The multispectral point cloud except the generation point of the training sample is used as the generation point P of the test sample test ;
[0047] Step 13: Generate points P of training samples in multispectral point cloud X train The sample set X′ is extracted using the k-neighbor method as the center, and X′ is used as the training sample;
[0048] In the multispectral point cloud X, the generated points P of the training samples are test The k-neighbor method is used to extract the sample set X″, X″ as the test sample.
[0049] The other steps and parameters are the same as those in the first embodiment.
[0050] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that a feature aggregation network is constructed in step two;
[0051] The specific process is:
[0052] The feature aggregation network includes a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first multi-scale feature aggregation MSFF, a second multi-scale feature aggregation MSFF, a third multi-scale feature aggregation MSFF, a first decoder, a second decoder, a third decoder, a fourth decoder, and a fifth decoder.
[0053] The other steps and parameters are the same as those in the first or second embodiment.
[0054] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that a multi-classification head and an adaptive hybrid loss module are constructed in step 3;
[0055] The multi-classification head includes a multi-scale classification head and a long-tail distribution classification head;
[0056] The adaptive hybrid loss module includes multi-scale loss and long-tail distribution loss;
[0057] The specific process is:
[0058] The multi-scale classification head includes a first adaptive classifier, a second adaptive classifier, a third adaptive classifier, and a fourth adaptive classifier;
[0059] The long-tail distribution classification head includes a fifth adaptive classifier and a sixth adaptive classifier;
[0060] The first adaptive classifier, the second adaptive classifier, the third adaptive classifier, the fourth adaptive classifier, the fifth adaptive classifier, and the sixth adaptive classifier are all classifiers composed of three fully connected layers in cascade (fully connected layer, fully connected layer, and fully connected layer are connected in sequence).
[0061] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.
[0062] Specific implementation method five: This implementation method is different from any one of specific implementation methods one to four in that in step four, the training sample is input into the constructed feature aggregation network, and the feature aggregation network outputs features;
[0063] The specific process is:
[0064] Step 41: Input the multispectral point cloud into the first encoder, and the first encoder outputs feature A;
[0065] Step 42: Feature A is input into the second encoder and the first multi-scale feature aggregation MSFF respectively, and the second encoder outputs feature B;
[0066] Step 4: The second encoder outputs feature B which is input into the third encoder, the first multi-scale feature aggregation MSFF, and the second multi-scale feature aggregation MSFF respectively, and the third encoder outputs feature C;
[0067] Step 44: The third encoder outputs feature C which is respectively input into the fourth encoder, the first multi-scale feature aggregation MSFF, the second multi-scale feature aggregation MSFF, and the third multi-scale feature aggregation MSFF, and the fourth encoder outputs feature D;
[0068] Step 4 and 5, the fourth encoder outputs feature D which is respectively input into the fifth encoder, the second multi-scale feature aggregation MSFF, and the third multi-scale feature aggregation MSFF, and the fifth encoder outputs feature E;
[0069] Step 46: The fifth encoder outputs feature E which is input into the first decoder and the third multi-scale feature aggregation MSFF respectively, and the first decoder outputs feature F1;
[0070] Step 47
[0071] The first multi-scale feature aggregation MSFF outputs feature F;
[0072] The second multi-scale feature aggregation MSFF output feature G;
[0073] The third multi-scale feature aggregation MSFF output feature H;
[0074] Step 48: The first decoder outputs feature I and the third multi-scale feature aggregation MSFF output feature H, which are input into the second decoder, and the second decoder outputs feature F2;
[0075] Step 49: The second decoder output feature J and the second multi-scale feature aggregation MSFF output feature G are input into the third decoder, and the third decoder outputs feature F3;
[0076] Step 40: The third decoder output feature K and the first multi-scale feature aggregation MSFF output feature F are input into the fourth decoder, and the fourth decoder outputs feature F4;
[0077] Step 41: The fourth decoder outputs feature L which is input into the fifth decoder. The fifth decoder outputs feature F5, which is used as the output feature of the feature aggregation network.
[0078] The feature aggregation network is based on a typical five-layer encoder-decoder structure. In the encoder layer, the feature encoding method proposed in the RandLA-Net network is used to aggregate spatial and spectral features, and the input features are downsampled using a random sampling method to improve computational efficiency; in the decoder layer, a simple fully connected layer and upsampling module are used to maintain processing speed; between the encoder layer and the decoder layer, a multi-scale feature aggregation module is used as a skip connection to improve the ability to retain detailed features of the ground objects; since the encoder-decoder network structure has five layers, three layers of the multi-scale feature aggregation module are applied.
[0079] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0080] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the specific operation process of the first multi-scale feature aggregation MSFF is as follows:
[0081] 1) The first encoder is used as the upper encoder;
[0082] The first encoder outputs feature A as the output feature of the upper encoder (shallow feature)
[0083] The second encoder serves as the encoder of this layer;
[0084] The second encoder outputs feature B as the output feature of the encoder at this layer (feature of this layer)
[0085] The third encoder acts as the lower encoder;
[0086] The third encoder outputs feature C as the output feature of the lower encoder (deep feature)
[0087] Where n and c are F L The corresponding receptive field size and feature dimension; is a real number;
[0088] 2) To facilitate the aggregation of fine features at different scales;
[0089] For feature F S Downsampling to obtain features With F L is a new feature with the same receptive field;
[0090] For feature F D Upsampling to obtain features With F L is a new feature with the same receptive field;
[0091] 3) The feature F S ′ and feature F L Fusion, obtain shallow feature weight e S ; The expression is:
[0092]
[0093] in, represents the global average pooling operation, σ is the Sigmoid function;
[0094] The global average pooling operation is used to extract the overall features, and the above features are fused through the element-by-element summation operation. The final shallow feature weight e S Calculated by Sigmoid function;
[0095] 4) Based on shallow feature weight e S and feature F S ′, obtain shallow fusion features The expression is:
[0096]
[0097] in, It is an element-by-element multiplication operation;
[0098] 5) The feature F′ D With feature F L Fusion, obtain deep feature weight e D ; The expression is:
[0099]
[0100] The global average pooling operation is used to extract the overall features, and the above features are fused through element-by-element multiplication operations. The final deep feature weight e D Calculated by Sigmoid function;
[0101] 6) Based on deep feature weight e D And feature F′D, obtain deep fusion features The expression is:
[0102]
[0103] 7) Connect shallow fusion features Feature F L Fusion features with deep And get the output features through the MLP layer The expression is:
[0104]
[0105] in, is the MLP layer.
[0106] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.
[0107] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the specific operation process of the second multi-scale feature aggregation MSFF is as follows:
[0108] 1) The second encoder is used as the upper encoder;
[0109] The second encoder outputs feature B as the output feature of the upper encoder (shallow feature)
[0110] The third encoder serves as the encoder of this layer;
[0111] The third encoder outputs feature C as the output feature of the encoder at this layer (feature of this layer)
[0112] The fourth encoder serves as a lower encoder;
[0113] The fourth encoder outputs feature D as the output feature of the lower encoder (deep feature)
[0114] Where n and c are F L The corresponding receptive field size and feature dimension; is a real number;
[0115] 2) To facilitate the aggregation of fine features at different scales;
[0116] For feature F S Downsampling to obtain features With F L is a new feature with the same receptive field;
[0117] For feature F D Upsampling to obtain features With F Lis a new feature with the same receptive field;
[0118] 3) The feature F S ′ and feature F L Fusion, obtain shallow feature weight e S ; The expression is:
[0119]
[0120] in, represents the global average pooling operation, σ is the Sigmoid function;
[0121] The global average pooling operation is used to extract the overall features, and the above features are fused through the element-by-element summation operation. The final shallow feature weight e S Calculated by Sigmoid function;
[0122] 4) Based on shallow feature weight e S and feature F S ′, obtain shallow fusion features The expression is:
[0123]
[0124] in, It is an element-by-element multiplication operation;
[0125] 5) The feature F′ D With feature F L Fusion, obtain deep feature weight e D ; The expression is:
[0126]
[0127] The global average pooling operation is used to extract the overall features, and the above features are fused through element-by-element multiplication operations. The final deep feature weight e D Calculated by Sigmoid function;
[0128] 6) Based on deep feature weight e D and feature F′ D , obtain deep fusion features The expression is:
[0129]
[0130] 7) Connect shallow fusion features Feature F L Fusion features with deep And get the output features through the MLP layer The expression is:
[0131]
[0132] in, is the MLP layer.
[0133] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0134] Specific implementation eight: This implementation is different from specific implementations one to seven in that the specific operation process of the third multi-scale feature aggregation MSFF is as follows:
[0135] 1) The third encoder is used as the upper encoder;
[0136] The third encoder outputs feature C as the output feature of the upper encoder (shallow feature)
[0137] The fourth encoder is used as the encoder of this layer;
[0138] The fourth encoder output feature D is used as the output feature of the encoder at this layer (the feature at this layer)
[0139] The fifth encoder serves as the lower encoder;
[0140] The fifth encoder output feature E is used as the output feature of the lower encoder (deep feature)
[0141] Where n and c are F L The corresponding receptive field size and feature dimension; is a real number;
[0142] 2) To facilitate the aggregation of fine features at different scales;
[0143] For feature F S Downsampling to obtain features With F L is a new feature with the same receptive field;
[0144] For feature F D Upsampling to obtain features With F L is a new feature with the same receptive field;
[0145] 3) The feature F S ′ and feature F L Fusion, obtain shallow feature weight e S ; The expression is:
[0146]
[0147] in, represents the global average pooling operation, σ is the Sigmoid function;
[0148] The global average pooling operation is used to extract the overall features, and the above features are fused through the element-by-element summation operation. The final shallow feature weight e S Calculated by Sigmoid function;
[0149] 4) Based on shallow feature weight e S and feature F S ′, obtain shallow fusion features The expression is:
[0150]
[0151] in, It is an element-by-element multiplication operation;
[0152] 5) The feature F′ D With feature F L Fusion, obtain deep feature weight e D ; The expression is:
[0153]
[0154] The global average pooling operation is used to extract the overall features, and the above features are fused through element-by-element multiplication operations. The final deep feature weight e D Calculated by Sigmoid function;
[0155] 6) Based on deep feature weight e D and feature F′ D , obtain deep fusion features The expression is:
[0156]
[0157] 7) Connect shallow fusion features Feature F L Fusion features with deep And get the output features through the MLP layer The expression is:
[0158]
[0159] in, is the MLP layer.
[0160] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.
[0161] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that in step 4, the output features of the feature aggregation network are input into the constructed multi-classification head and adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module output classification prediction results;
[0162] The specific process is:
[0163] Step 42: Input the first decoder output feature F1 of the feature aggregation network into the first adaptive classifier, and the first adaptive classifier outputs the category prediction result Z1.
[0164] Where L is the number of categories, N is the total number of points in the multispectral point cloud, represents the total number of points output by the first adaptive classifier, z 11 Represents the first point category prediction result output by the first adaptive classifier, z 12 Represents the second point category prediction result output by the first adaptive classifier, represents the output of the first adaptive classifier Point category prediction results;
[0165] The expression is:
[0166]
[0167] in is the first adaptive classifier; w1 is the adaptive weight, is an element-by-element multiplication operation; based on the obtained category prediction result Z1 and the ground object true value corresponding to the output point of the first adaptive classifier The loss is calculated using the cross entropy loss function:
[0168]
[0169] where y 1j represents the ground object truth value after one-hot encoding of the ground object truth value of the j-th point corresponding to the first adaptive classifier, z 1j The j-th point category prediction result output by the first adaptive classifier;
[0170] Step 43: The second decoder output feature F2 of the feature aggregation network is input into the second adaptive classifier, and the first adaptive classifier outputs the category prediction result Z2.
[0171] Where L is the number of categories, N is the total number of points in the multispectral point cloud, represents the total number of points output by the second adaptive classifier, z 21Represents the first point category prediction result output by the second adaptive classifier, z 22 Represents the second point category prediction result output by the second adaptive classifier. represents the output of the second adaptive classifier Point category prediction results;
[0172] The expression is:
[0173]
[0174] in is the second adaptive classifier; w2 is the adaptive weight, It is an element-by-element multiplication operation;
[0175] Based on the obtained category prediction result Z2 and the ground object truth value corresponding to the output point of the second adaptive classifier The loss is calculated using the cross entropy loss function:
[0176]
[0177] where y 2j represents the ground object truth value after one-hot encoding of the ground object truth value of the j-th point corresponding to the second adaptive classifier, z 2j The j-th point category prediction result output by the second adaptive classifier;
[0178] Step 44: input the third decoder output feature F3 of the feature aggregation network into the third adaptive classifier, and the third adaptive classifier outputs the category prediction result Z3.
[0179] Where L is the number of categories, N is the total number of points in the multispectral point cloud, represents the total number of points output by the third adaptive classifier, z 31 Represents the first point category prediction result output by the third adaptive classifier, z 32 Represents the second point category prediction result output by the third adaptive classifier. represents the output of the third adaptive classifier Point category prediction results;
[0180] The expression is:
[0181]
[0182] Where C(·) is the third adaptive classifier; w3 is the adaptive weight, It is an element-by-element multiplication operation;
[0183] Based on the obtained category prediction result Z3 and the ground object truth value corresponding to the output point of the third adaptive classifier The loss is calculated using the cross entropy loss function:
[0184]
[0185] where y 3j represents the ground feature truth value after one-hot encoding of the ground feature truth value of the j-th point corresponding to the third adaptive classifier, z 3j The j-th point category prediction result output by the third adaptive classifier;
[0186] Step 45: input the fourth decoder output feature F4 of the feature aggregation network into the fourth adaptive classifier, and the fourth adaptive classifier outputs the category prediction result Z4.
[0187] Where L is the number of categories, N is the total number of points in the multispectral point cloud, represents the total number of points output by the fourth adaptive classifier, z 41 Represents the first point category prediction result output by the fourth adaptive classifier, z 42 Represents the second point category prediction result output by the fourth adaptive classifier, represents the output of the fourth adaptive classifier Point category prediction results;
[0188] The expression is:
[0189]
[0190] in is the fourth adaptive classifier; w4 is the adaptive weight, It is an element-by-element multiplication operation;
[0191] Based on the obtained category prediction result Z4 and the ground object true value corresponding to the output point of the fourth adaptive classifier The loss is calculated using the cross entropy loss function:
[0192]
[0193] where y 4j is the ground feature truth value after one-hot encoding of the ground feature truth value of the j-th point corresponding to the fourth adaptive classifier, z 4j The j-th point category prediction result output by the fourth adaptive classifier;
[0194] Step 46: Calculate the multi-scale loss based on steps 42 to 45:
[0195]
[0196] Step 47
[0197] The fifth decoder output feature F5 of the feature aggregation network is input into the fifth adaptive classifier, and the fifth adaptive classifier outputs the category prediction result Represents the first point category prediction result output by the fifth adaptive classifier, Represents the second point category prediction result output by the fifth adaptive classifier, Represents the Nth point category prediction result output by the fourth adaptive classifier;
[0198] The expression is:
[0199]
[0200] in is the fifth adaptive classifier; is the adaptive weight, It is an element-by-element multiplication operation;
[0201] The fifth adaptive classifier learns the classification features of the head and tail categories (all categories) in the long-tail distribution scenario;
[0202] Based on the obtained category prediction results And the ground truth of all categories The loss is calculated using the root mean square loss function:
[0203]
[0204] in It represents the ground feature truth value after one-hot encoding of the ground feature truth value of the j-th point. The j-th point category prediction result output by the fifth adaptive classifier;
[0205] The fifth decoder output feature F5 of the feature aggregation network is input into the sixth adaptive classifier, and the sixth adaptive classifier outputs the category prediction result Represents the first point category prediction result output by the sixth adaptive classifier, Represents the second point category prediction result output by the sixth adaptive classifier, Represents the Nth point category prediction result output by the sixth adaptive classifier;
[0206] The expression is:
[0207]
[0208] Where C(·) is the sixth adaptive classifier; is the adaptive weight, It is an element-by-element multiplication operation;
[0209] The sixth adaptive classifier learns the classification features of the tail category in the long-tail distribution scenario. To avoid affecting the results of the fifth adaptive classifier, the prediction probability of the head category output in the sixth adaptive classifier should be as small as possible;
[0210] Based on the obtained category prediction results The ground truth value of the tail category The loss is calculated using the root mean square loss function:
[0211]
[0212] in represents the ground feature truth value after the ground feature truth value of the j-th point is one-hot encoded, where when the j-th point belongs to the head category, The j-th point category prediction result output by the sixth adaptive classifier;
[0213] When training the tail classification head, the true values of all head category objects are set to zero. In this way, during the training process, the tail category classifier is required to output a value as close to 0 as possible when encountering the head category. This will not affect the classification results of the fifth adaptive classifier, and the fifth adaptive classifier is distinguished from the sixth adaptive classifier.
[0214] If the proportion of multispectral points in a category to the multispectral points in all categories of the entire scene is less than 5%, then the category is considered to be the tail category, and the remaining categories are considered to be the head categories;
[0215] Step 48: Calculate the long-tail distribution loss based on step 47:
[0216]
[0217] Step 49: Calculate the adaptive hybrid loss based on the multi-scale loss and the long-tail distribution loss.
[0218] Since the objects in remote sensing scenes have long-tail distribution and scale differences, an adaptive hybrid loss module consisting of multi-scale loss and long-tail loss is proposed. The adaptive hybrid loss module can use multiple classifiers to learn effective classification features of different categories at the same time, and finally use adaptive weights to merge the output results of different classifiers;
[0219] The multi-scale loss module is as follows: For the i-th decoder layer, the output feature can be recorded as Therefore, the output feature of the fifth decoder layer, that is, the final layer, is Where N and C are the receptive field size and feature dimension corresponding to F5 respectively.
[0220] For the low-resolution features output by the first four decoder layers, a simple classifier consisting of three fully connected layers is used to obtain the category prediction of the corresponding resolution Where L is the number of categories.
[0221] Since the classification of different objects at different scales is different, an adaptive weight is applied to each classifier. To automatically extract the features with the best classification accuracy at the current resolution:
[0222]
[0223] in is a classifier;
[0224] Then the obtained category prediction and the corresponding downsampled ground object value The loss is calculated using the cross entropy loss function:
[0225]
[0226] The final multi-scale loss is the weighted sum of losses at different resolutions:
[0227]
[0228] The long-tail distribution loss module is as follows: The final feature F5 is input into two classifiers constructed by the fully connected layer to obtain the category prediction Similar to the multi-scale loss, an adaptive weight is applied to each classifier
[0229]
[0230] For the loss function, we hope that the first classifier learns all the object categories, while the second classifier only learns the classification features of the tail categories. Therefore, how to make the two classifiers work together is the key to the loss function design.
[0231] Compared with the common cross entropy loss function, the root mean square loss function can better suppress the output probability of non-target categories and reduce the mutual influence of the results between different classifiers:
[0232]
[0233] in Represents the ground truth values of all categories, Represents the ground truth value of the tail category (the head category is set to 0).
[0234] The final long-tail loss is the weighted sum of the losses of different classifiers:
[0235]
[0236] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.
[0237] Specific implementation method ten: This implementation method is different from any one of specific implementation methods one to nine in that the adaptive hybrid loss is calculated based on the multi-scale loss and the long-tail loss in step forty-nine:
[0238] L=L scale +L tail .
[0239] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.
[0240] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.
Claims
1. A multispectral point cloud classification method for remote sensing scenes, characterized by: The specific process of the method is: Step 1: Use the grid balanced sampling method to process the multispectral point cloud to obtain training and test samples; Step 2: Build a feature aggregation network; Step 3: Construct a multi-classification head and an adaptive hybrid loss module; The multi-classification head includes a multi-scale classification head and a long-tail distribution classification head; The adaptive hybrid loss module includes multi-scale loss and long-tail distribution loss; Step 4: Input the training samples into the constructed feature aggregation network, and the feature aggregation network outputs features; The output features of the feature aggregation network are input into the constructed multi-classification head and adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module output the classification prediction results; The feature aggregation network, the multi-classification head, and the adaptive mixed loss module are trained until convergence, and the trained feature aggregation network, the multi-classification head, and the adaptive mixed loss module are obtained; Step 5: Input the test sample into the trained feature aggregation network. The output features of the trained feature aggregation network are input into the trained multi-classification head and adaptive mixed loss module. The trained multi-classification head and adaptive mixed loss module output the classification prediction results.
2. The multispectral point cloud classification method for remote sensing scenes according to claim 1, characterized in that: In the step 1, the multispectral point cloud is processed using a grid balanced sampling method to obtain training samples and test samples; The specific process is: Step 1: Order is the multispectral point cloud of manually annotated true values with unknown annotation ratio, where N represents the total number of points in the multispectral point cloud, 3 represents the spatial dimension of the multispectral point cloud, and d represents the spectral dimension of the multispectral point cloud; The multispectral point cloud X is sampled based on the sampling rate ε to determine the size of the grid and obtain the number of grids; Get the center point of each grid, and the center point of each grid forms a sparse multispectral point cloud Step 1 and 2: Calculate sparse multispectral point cloud The number of points in each manually labeled category is M = {m1,m2,…,m C }; Among them, m1 is the number of multispectral point clouds marked in the first category of the manually labeled points M, m2 is the number of multispectral point clouds marked in the second category of the manually labeled points M, and m C is the number of multispectral point clouds annotated in the Cth category among the number of points M of manually annotated categories, and C is the total number of ground object categories; Randomly select a fixed proportion from the multispectral point cloud labeled for each category As the generation point of training samples in, is the generating point P of the training sample train The number of multispectral point clouds in the first category, is the generating point P of the training sample train The number of multispectral point clouds in the second category, is the generating point P of the training sample train The number of multispectral point clouds in the Cth category; Sparse multispectral point cloud The multispectral point cloud except the generation point of the training sample is used as the generation point P of the test sample test ; Step 13: Generate points P of training samples in multispectral point cloud X train The sample set X′ is extracted using the k-neighbor method as the center, and X′ is used as the training sample; In the multispectral point cloud X, the generated points P of the training samples are test The k-neighbor method is used to extract the sample set X″, X″ as the test sample.
3. The multispectral point cloud classification method for remote sensing scenes according to claim 2, characterized in that: In the step 2, a feature aggregation network is constructed; The specific process is: The feature aggregation network includes a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first multi-scale feature aggregation MSFF, a second multi-scale feature aggregation MSFF, a third multi-scale feature aggregation MSFF, a first decoder, a second decoder, a third decoder, a fourth decoder, and a fifth decoder.
4. The multispectral point cloud classification method for remote sensing scenes according to claim 3, characterized in that: In the step 3, a multi-classification head and an adaptive hybrid loss module are constructed; The multi-classification head includes a multi-scale classification head and a long-tail distribution classification head; The adaptive hybrid loss module includes multi-scale loss and long-tail distribution loss; The specific process is: The multi-scale classification head includes a first adaptive classifier, a second adaptive classifier, a third adaptive classifier, and a fourth adaptive classifier; The long-tail distribution classification head includes a fifth adaptive classifier and a sixth adaptive classifier; The first adaptive classifier, the second adaptive classifier, the third adaptive classifier, the fourth adaptive classifier, the fifth adaptive classifier and the sixth adaptive classifier are all classifiers composed of three cascaded fully connected layers.
5. The multispectral point cloud classification method for remote sensing scenes according to claim 4, characterized in that: In the step 4, the training samples are input into the constructed feature aggregation network, and the feature aggregation network outputs features; The specific process is: Step 41: Input the multispectral point cloud into the first encoder, and the first encoder outputs feature A; Step 42: Feature A is input into the second encoder and the first multi-scale feature aggregation MSFF respectively, and the second encoder outputs feature B; Step 4: The second encoder outputs feature B which is input into the third encoder, the first multi-scale feature aggregation MSFF, and the second multi-scale feature aggregation MSFF respectively, and the third encoder outputs feature C; Step 44: The third encoder outputs feature C which is respectively input into the fourth encoder, the first multi-scale feature aggregation MSFF, the second multi-scale feature aggregation MSFF, and the third multi-scale feature aggregation MSFF, and the fourth encoder outputs feature D; Step 4 and 5, the fourth encoder outputs feature D which is respectively input into the fifth encoder, the second multi-scale feature aggregation MSFF, and the third multi-scale feature aggregation MSFF, and the fifth encoder outputs feature E; Step 46: The fifth encoder outputs feature E which is input into the first decoder and the third multi-scale feature aggregation MSFF respectively, and the first decoder outputs feature F1; Step 47 The first multi-scale feature aggregation MSFF outputs feature F; The second multi-scale feature aggregation MSFF output feature G; The third multi-scale feature aggregation MSFF output feature H; Step 48: The first decoder outputs feature I and the third multi-scale feature aggregation MSFF output feature H, which are input into the second decoder, and the second decoder outputs feature F2; Step 49: The second decoder output feature J and the second multi-scale feature aggregation MSFF output feature G are input into the third decoder, and the third decoder outputs feature F3; Step 40: The third decoder output feature K and the first multi-scale feature aggregation MSFF output feature F are input into the fourth decoder, and the fourth decoder outputs feature F4; Step 41: The fourth decoder outputs feature L which is input into the fifth decoder. The fifth decoder outputs feature F5, which is used as the output feature of the feature aggregation network.
6. The multispectral point cloud classification method for remote sensing scenes according to claim 5, characterized in that: The specific operation process of the first multi-scale feature aggregation MSFF is: 1) The first encoder is used as the upper encoder; The first encoder outputs feature A as the output feature of the upper encoder The second encoder serves as the encoder of this layer; The second encoder outputs feature B as the output feature of the encoder at this layer The third encoder acts as the lower encoder; The third encoder outputs feature C as the output feature of the lower encoder Where n and c are F L The corresponding receptive field size and feature dimension; is a real number; 2)、 For feature F S Downsampling to obtain features For feature F D Upsampling to obtain features 3) The feature F S ′ and feature F L Fusion, obtain shallow feature weight e S ; The expression is: in, represents the global average pooling operation, σ is the Sigmoid function; 4) Based on shallow feature weight e S and feature F S ′, obtain shallow fusion features The expression is: in, It is an element-by-element multiplication operation; 5) The feature F′ D With feature F L Fusion, obtain deep feature weight e D ; The expression is: 6) Based on deep feature weight e D and feature F′ D , obtain deep fusion features The expression is: 7) Connect shallow fusion features Feature F L Fusion features with deep And get the output features through the MLP layer The expression is: in, is the MLP layer.
7. The multispectral point cloud classification method for remote sensing scenes according to claim 6, characterized in that: The specific operation process of the second multi-scale feature aggregation MSFF is: 1) The second encoder is used as the upper encoder; The second encoder outputs feature B as the upper encoder output feature The third encoder serves as the encoder of this layer; The third encoder outputs feature C as the output feature of the encoder at this layer The fourth encoder serves as a lower encoder; The fourth encoder output feature D is used as the output feature of the lower encoder Where n and c are F L The corresponding receptive field size and feature dimension; is a real number; 2)、 For feature F S Downsampling to obtain features For feature F D Upsampling to obtain features 3) The feature F S ′ and feature F L Fusion, obtain shallow feature weight e S ; The expression is: in, represents the global average pooling operation, σ is the Sigmoid function; 4) Based on shallow feature weight e S and feature F S ′, obtain shallow fusion features The expression is: in, It is an element-by-element multiplication operation; 5) The feature F′ D With feature F L Fusion, obtain deep feature weight e D ; The expression is: 6) Based on deep feature weight e D and feature F′ D , obtain deep fusion features The expression is: 7) Connect shallow fusion features Feature F L Fusion features with deep And get the output features through the MLP layer The expression is: in, is the MLP layer.
8. The multispectral point cloud classification method for remote sensing scenes according to claim 7, characterized in that: The specific operation process of the third multi-scale feature aggregation MSFF is: 1) The third encoder is used as the upper encoder; The third encoder output feature C is used as the upper encoder output feature The fourth encoder is used as the encoder of this layer; The fourth encoder output feature D is used as the output feature of the encoder at this layer The fifth encoder serves as the lower encoder; The fifth encoder output feature E is used as the output feature of the lower encoder Where n and c are F L The corresponding receptive field size and feature dimension; is a real number; 2)、 For feature F S Downsampling to obtain features For feature F D Upsampling to obtain features 3) The feature F S ′ and feature F L Fusion, obtain shallow feature weight e S ; The expression is: in, represents the global average pooling operation, σ is the Sigmoid function; 4) Based on shallow feature weight e S and feature F S ′, obtain shallow fusion features The expression is: in, It is an element-by-element multiplication operation; 5) The feature F′ D With feature F L Fusion, obtain deep feature weight e D ; The expression is: 6) Based on deep feature weight e D and feature F′ D , obtain deep fusion features The expression is: 7) Connect shallow fusion features Feature F L Fusion features with deep And get the output features through the MLP layer The expression is: in, is the MLP layer.
9. The multispectral point cloud classification method for remote sensing scenes according to claim 8, characterized in that: In the step 4, the output features of the feature aggregation network are input into the constructed multi-classification head and adaptive hybrid loss module, and the multi-classification head and adaptive hybrid loss module output the classification prediction result; The specific process is: Step 42: Input the first decoder output feature F1 of the feature aggregation network into the first adaptive classifier, and the first adaptive classifier outputs the category prediction result Z1. Where L is the number of categories, N is the total number of points in the multispectral point cloud, represents the total number of points output by the first adaptive classifier, z 11 Represents the first point category prediction result output by the first adaptive classifier, z 12 Represents the second point category prediction result output by the first adaptive classifier, represents the output of the first adaptive classifier Point category prediction results; The expression is: in is the first adaptive classifier; w1 is the adaptive weight, It is an element-by-element multiplication operation; Based on the obtained category prediction result Z1 and the ground object truth value corresponding to the output point of the first adaptive classifier The loss is calculated using the cross entropy loss function: where y 1j represents the ground object truth value after one-hot encoding of the ground object truth value of the j-th point corresponding to the first adaptive classifier, z 1j is the j-th point category prediction result output by the first adaptive classifier; Step 43: The second decoder output feature F2 of the feature aggregation network is input into the second adaptive classifier, and the first adaptive classifier outputs the category prediction result Z2. Where L is the number of categories, N is the total number of points in the multispectral point cloud, represents the total number of points output by the second adaptive classifier, z 21 Represents the first point category prediction result output by the second adaptive classifier, z 22 Represents the second point category prediction result output by the second adaptive classifier. represents the output of the second adaptive classifier Point category prediction results; The expression is: in is the second adaptive classifier; w2 is the adaptive weight, It is an element-by-element multiplication operation; Based on the obtained category prediction result Z2 and the ground object truth value corresponding to the output point of the second adaptive classifier The loss is calculated using the cross entropy loss function: where y 2j represents the ground object true value after one-hot encoding of the ground object true value of the j-th point corresponding to the second adaptive classifier, z 2j The j-th point category prediction result output by the second adaptive classifier; Step 44: input the third decoder output feature F3 of the feature aggregation network into the third adaptive classifier, and the third adaptive classifier outputs the category prediction result Z3. Where L is the number of categories, N is the total number of points in the multispectral point cloud, represents the total number of points output by the third adaptive classifier, z 31 Represents the first point category prediction result output by the third adaptive classifier, z 32 Represents the second point category prediction result output by the third adaptive classifier. represents the output of the third adaptive classifier Point category prediction results; The expression is: in is the third adaptive classifier; w3 is the adaptive weight, It is an element-by-element multiplication operation; Based on the obtained category prediction result Z3 and the ground object truth value corresponding to the output point of the third adaptive classifier The loss is calculated using the cross entropy loss function: where y 3j represents the ground feature truth value after one-hot encoding of the ground feature truth value of the j-th point corresponding to the third adaptive classifier, z 3j The j-th point category prediction result output by the third adaptive classifier; Step 45: input the fourth decoder output feature F4 of the feature aggregation network into the fourth adaptive classifier, and the fourth adaptive classifier outputs the category prediction result Z4. Where L is the number of categories, N is the total number of points in the multispectral point cloud, represents the total number of points output by the fourth adaptive classifier, z 41 Represents the first point category prediction result output by the fourth adaptive classifier, z 42 Represents the second point category prediction result output by the fourth adaptive classifier, represents the output of the fourth adaptive classifier Point category prediction results; The expression is: in is the fourth adaptive classifier; w4 is the adaptive weight, It is an element-by-element multiplication operation; Based on the obtained category prediction result Z4 and the ground object true value corresponding to the output point of the fourth adaptive classifier The loss is calculated using the cross entropy loss function: where y 4j is the ground feature truth value after one-hot encoding of the ground feature truth value of the j-th point corresponding to the fourth adaptive classifier, z 4j The j-th point category prediction result output by the fourth adaptive classifier; Step 46: Calculate the multi-scale loss based on steps 42 to 45: Step 47 The fifth decoder output feature F5 of the feature aggregation network is input into the fifth adaptive classifier, and the fifth adaptive classifier outputs the category prediction result Represents the first point category prediction result output by the fifth adaptive classifier, Represents the second point category prediction result output by the fifth adaptive classifier, Represents the Nth point category prediction result output by the fourth adaptive classifier; The expression is: in is the fifth adaptive classifier; is the adaptive weight, It is an element-by-element multiplication operation; Based on the obtained category prediction results And the ground truth of all categories The loss is calculated using the root mean square loss function: in It represents the ground feature truth value after one-hot encoding of the ground feature truth value of the j-th point. The j-th point category prediction result output by the fifth adaptive classifier; The fifth decoder output feature F5 of the feature aggregation network is input into the sixth adaptive classifier, and the sixth adaptive classifier outputs the category prediction result Represents the first point category prediction result output by the sixth adaptive classifier, Represents the second point category prediction result output by the sixth adaptive classifier, Represents the Nth point category prediction result output by the sixth adaptive classifier; The expression is: in is the sixth adaptive classifier; is the adaptive weight, It is an element-by-element multiplication operation; Based on the obtained category prediction results The ground truth value of the tail category The loss is calculated using the root mean square loss function: in represents the ground feature truth value after the ground feature truth value of the j-th point is one-hot encoded, where when the j-th point belongs to the head category, The j-th point category prediction result output by the sixth adaptive classifier; If the proportion of multispectral points in a category to the multispectral points in all categories of the entire scene is less than 5%, it is considered to be a tail category, and the remaining categories are considered to be head categories; Step 48: Calculate the long-tail distribution loss based on step 47: Step 49: Calculate the adaptive hybrid loss based on the multi-scale loss and the long-tail distribution loss.
10. The multispectral point cloud classification method for remote sensing scenes according to claim 9, characterized in that: In step 49, the adaptive hybrid loss is calculated based on the multi-scale loss and the long-tail loss: L=L scale +L tail .
Citation Information
Patent Citations
Multi-spectral laser radar point cloud classification method based on multi-core graph learning
CN116933141A
Performing semantic segmentation of 3D data using deep learning
US11694333B1
Hyperspectral remote sensing image classification method based on self-attention context network
WO2022073452A1
Method for classifying hyperspectral images on basis of adaptive multi-scale feature extraction model
WO2022160771A1
