A meta-learning based multi-modal remote sensing data ground object classification method

Abstract

The present application relates to a kind of based on meta learning multi-modal remote sensing data ground feature classification method, belong to remote sensing data processing technical field.The present application constructs multi-modal fusion relationship network can only utilize a small amount of sample to realize the fusion classification of multi-modal remote sensing image, more internal complementary information is extracted from multi-modal data by cross-modal reconstruction;Learning ability is obtained by training on source data set using three-step meta learning process to train and learn the network constructed, and fine-tuning can quickly adapt to different target scenarios under the target data set only with a small amount of labeled samples, with good generalization performance.The present application effectively alleviates the mixed misclassification phenomenon existing in other methods in complex scene by more attention of each class in meta learning, more accurately divides the detail target, and the boundary information of ground feature is well preserved, and the precision of classification is improved.

Description

Technical Field

[0001] This invention relates to a method for classifying ground features in multimodal remote sensing data based on meta-learning, belonging to the field of remote sensing data processing technology. Background Technology

[0002] Hyperspectral images (HSIs) possess the unique advantage of rich spectral and spatial information, while LiDAR (Light Detection and Ranging) data can effectively supplement spatial structure information in two-dimensional images. Currently, deep learning-based multimodal fusion methods have demonstrated that, with supervised training using a sufficient number of labeled samples, multimodal models outperform single-modal models in land cover classification. However, multimodal remote sensing datasets are very limited, and the categories included in existing datasets cannot fully cover other regions. Furthermore, in practical work, labeling large-scale multimodal remote sensing data is extremely difficult and labor-intensive, severely limiting the application of multimodal data in real-world scenarios.

[0003] Recently, inspired by the ability of humans to learn, few-shot learning (FSL) has emerged as a powerful tool for rapidly acquiring knowledge from a limited number of labeled samples, transferable to other data—a capability known as meta-learning. The core of meta-learning is using a small number of labeled samples to identify previously unseen scenes and categories, possessing both FSL learning capabilities and transferability across different scenarios. Researchers have successfully applied it to land cover classification tasks in remote sensing imagery. Studies show that when faced with complex remote sensing imagery, meta-learning-based networks can learn discriminative features across different scenes, effectively mitigating the overfitting problem caused by insufficient labeled samples. However, some land cover categories may exhibit significant variations across different scenes, making it impossible for single-modal data to capture these changes.

[0004] For multimodal remote sensing data, the elevation features from LiDAR data and the spectral features from HSIs are typically different, but the spatial features of the two datasets are highly similar. There are not only numerous common features between modalities but also unique features specific to each modality. However, most existing multimodal remote sensing data fusion methods learn features for each modality separately and then fuse all features to obtain a fused feature representation. This leads to significant redundancy in common features after fusion, while unique features related to land cover attributes are ignored. Furthermore, in few-shot learning, each network training session relies on only a few samples, resulting in limited and significantly changing supervisory information during each training session. Existing supervised fusion methods struggle to learn robust fusion methods under this training mode. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal remote sensing data land cover classification method based on meta-learning, so as to solve the problems of low classification accuracy and low efficiency caused by the redundancy of common features and the lack of private features when fusing multimodal remote sensing data.

[0006] To address the aforementioned technical problems, this invention provides a multimodal remote sensing data feature classification method based on meta-learning. This classification method includes the following steps:

[0007] 1) Construct a multimodal fusion relationship network, which is used to extract features, reconstruct cross-modal features, and fuse features from input multimodal remote sensing images. It also assigns categories to samples based on learning the relationships between features. Cross-modal reconstruction loss is introduced as unsupervised information during cross-modal reconstruction, which makes the fused features more adaptable to the inherent information.

[0008] 2) Meta-training of the multimodal fusion relation network is performed using labeled source datasets;

[0009] 3) Fine-tune the meta-trained multimodal fusion relation network using labeled samples from the target dataset. In the fine-tuning learning, features related to the previous meta-training task are retained, and the focus is on learning the differences between different tasks in the target dataset.

[0010] 4) Classify unlabeled multimodal remote sensing data using a fine-tuned multimodal fusion relation network.

[0011] The multimodal fusion relation network constructed in this invention can learn more discriminative spectral, spatial, and elevation features from multimodal remote sensing data. Simultaneously, by introducing cross-modal reconstruction loss as unsupervised information, the fused features exhibit stronger data adaptability to inherent information. Meta-learning capabilities and transferable knowledge are acquired through meta-training on the source dataset. With a target dataset containing only a small number of multimodal labeled samples, the model rapidly adapts to different target scenarios through fine-tuning. During fine-tuning, features related to the previous meta-training task are preserved, and the network also focuses on learning the differences between different tasks on the target dataset, possessing meta-learning capabilities for the target dataset. Therefore, this invention can classify new categories of land features in different scenarios, exhibiting good generalization performance. Compared to supervised deep learning methods, it reduces dependence on labeled data, and compared to single-modal methods, it improves overall classification accuracy.

[0012] Furthermore, the multimodal fusion relationship network includes a cross-modal feature learning sub-network and a relationship learning sub-network. The cross-modal feature learning sub-network is used to encode the original data of each modality through an encoder to obtain the corresponding compact representation, reconstruct the input data of another modality through a decoder, and cascade the compact features of the two modalities through fusion to achieve the fusion of modal features.

[0013] The relation learning subnetwork is used to compare the relationships between features using a learnable depth metric function to assign categories to query samples based on the learned similarity.

[0014] This invention employs a bidirectional encoder-decoder structure to construct a multimodal fusion relationship network, enabling the network's multimodal feature learning process to rely more on the data itself and less on supervisory information. Considering the distributional differences between fused features of different categories, a learnable depth metric function is used to compare the relationships between features, thereby assigning categories to query samples based on the learned similarity.

[0015] Furthermore, during meta-training, multiple subtasks are sampled non-repeatedly from the source dataset for meta-training, with each subtask set up for small samples.

[0016] Furthermore, the source dataset includes a support set and a query set. Both the support set and the query set are simultaneously input into the cross-modal feature learning sub-network to obtain multimodal fusion features. For the multimodal fusion features of each class in the support set, the representative feature of each class is selected as the prototype feature by taking the mean to represent the feature distribution of that class. The prototype features of each class are then connected with all query features of the training task to obtain relation pairs. The relation learning sub-network calculates the similarity between each query sample and each class in the obtained relation pairs, and calculates the loss based on the similarity obtained from the real class relation pairs of the query set and the prototype. This loss, together with the loss of the cross-modal feature learning sub-network, is used to train the entire multimodal fusion relation network in a supervised manner.

[0017] Furthermore, during fine-tuning, multiple subtasks are sampled from a small set of labeled samples for fine-tuning the multimodal fusion relation network on the target dataset.

[0018] Furthermore, the loss function used in the multimodal fusion relationship network is:

[0019]

[0020] in It is weighted Hyperparameters of value, It is the loss of the cross-modal feature learning sub-network. and These are the parameters of the relation learning subnetwork and the cross-modal feature learning subnetwork, respectively. The similarity is obtained by comparing the true category relationships between the query set and the prototype. It is the similarity between the query set and the prototype learned by the multimodal fusion relation network.

[0021] Furthermore, the loss function used in the cross-modal feature learning sub-network is:

[0022]

[0023] It is the loss of the cross-modal feature learning sub-network. , These represent the hsi modal features before and after reconstruction, respectively. , These represent the lidar modal features before and after reconstruction, respectively. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the multimodal remote sensing data land cover classification method based on meta-learning of the present invention.

[0025] Figure 2 This is a schematic diagram of the cross-modal feature fusion network structure used in this invention;

[0026] Figure 3a These are hyperspectral false-color images from the Augsburg multimodal remote sensing dataset used in this embodiment of the invention.

[0027] Figure 3b These are DSM images from the Augsburg multimodal remote sensing dataset used in this embodiment of the invention;

[0028] Figure 3c These are the ground truth class-annotated images from the Augsburg multimodal remote sensing dataset used in this embodiment of the invention;

[0029] Figure 4a These are hyperspectral false-color images from the Trento multimodal remote sensing dataset used in this embodiment of the invention;

[0030] Figure 4b These are DSM images from the Trento multimodal remote sensing dataset used in this embodiment of the invention;

[0031] Figure 4c These are the real-class labeled images from the Trento multimodal remote sensing dataset used in this embodiment of the invention;

[0032] Figure 5aThese are hyperspectral false-color images from the MUUFL Gulfport multimodal remote sensing dataset used in this embodiment of the invention.

[0033] Figure 5b These are DSM images from the MUUFL Gulfport multimodal remote sensing dataset used in this embodiment of the invention;

[0034] Figure 5c These are the ground truth class-annotated images from the MUUFL Gulfport multimodal remote sensing dataset used in this embodiment of the invention;

[0035] Figure 6a These are hyperspectral false-color images from the Houston 2013 multimodal remote sensing dataset used in this embodiment of the invention;

[0036] Figure 6b These are DSM images from the Houston 2013 multimodal remote sensing dataset used in this embodiment of the invention;

[0037] Figure 6c These are the true category-annotated images from the Houston 2013 multimodal remote sensing dataset used in this embodiment of the invention;

[0038] Figure 7 This is a schematic diagram showing the comparison results between the present invention and six existing classification methods on the Trento dataset;

[0039] Figure 8 This is a schematic diagram showing the comparison results between the present invention and six existing classification methods in the Houston 2013 dataset;

[0040] Figure 9 This is a schematic diagram showing the comparison results between the present invention and six existing classification methods on the MUUFL Gulfport dataset. Detailed Implementation

[0041] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0042] Examples of multimodal remote sensing data land cover classification methods based on meta-learning:

[0043] This invention first constructs a multimodal fusion relation network, which is used for feature extraction, cross-modal reconstruction, and feature fusion of input multimodal remote sensing images, and assigns categories to samples based on learning the relationships between features. Then, the multimodal fusion relation network is meta-trained using labeled source datasets; and fine-tuned using labeled samples from the target dataset. Finally, the fine-tuned multimodal fusion relation network is used to classify unlabeled multimodal remote sensing data. The implementation flow of this method is as follows: Figure 1 As shown below, a detailed explanation will be provided with specific examples.

[0044] This embodiment uses hyperspectral and LiDAR remote sensing data as examples. The source dataset used for meta-training is the Augsburg multimodal dataset, containing 7 land cover classes and 78,294 markers. The target data include the Trento dataset, containing 6 land cover classes and 30,214 markers; the MUUFL Gulfport dataset, containing 11 land cover classes and 53,687 markers; and the Houston 2013 dataset, containing 15 land cover classes and 15,029 markers. The hyperspectral false-color images from the Augsburg multimodal dataset, the DSM images obtained from the LiDAR data, and the ground truth class annotations are shown below. Figure 3a , Figure 3b and Figure 3c As shown; hyperspectral false-color images from the Trento multimodal dataset, DSM images obtained from LiDAR data, and ground truth class annotations are as follows. Figure 4a , Figure 4b and Figure 4c As shown; hyperspectral false-color images from the MUUFL Gulfport multimodal dataset, DSM images obtained from LiDAR data, and ground truth class annotations are as follows. Figure 5a , Figure 5b and Figure 5c As shown; hyperspectral false-color images from the Houston 2013 multimodal dataset, DSM images obtained from LiDAR data, and ground truth class annotations are shown below. Figure 6a , Figure 6b and Figure 6c As shown.

[0045] In the few-shot learning process, 10,000 subtasks are first sampled non-repeatingly from the source dataset for meta-training. Each subtask is a 5-way, 5-shot few-shot setup, with 20 labeled samples per class used as the few-shot annotation set for the target dataset, thus each subtask contains 15 query samples. The Adam optimization algorithm is employed, with a batch size of 64 and a learning rate of 0.0001. Next, 1,000 subtasks are sampled from the small set of labeled samples for fine-tuning the model on the target dataset. Finally, for meta-testing, the fine-tuned learning parameters are used to classify all unlabeled multimodal data.

[0046] 1. Construct a multimodal fusion relationship network.

[0047] This embodiment constructs a multimodal fusion relationship network, which includes a cross-modal feature learning sub-network and a relationship learning sub-network, such as... Figure 1 As shown, the cross-modal feature learning sub-network is used to encode the original data of each modality through the encoder to obtain the corresponding compact representation. The compact representation is then reconstructed into the input data of another modality through the decoder. Finally, the compact features of the two modalities are concatenated through fusion to achieve the fusion of modal features. The relation learning sub-network is used to compare the relationships between features using a learnable depth metric function to assign categories to query samples based on the learned similarity.

[0048] Specifically, such as Figure 2 As shown, the cross-modal feature learning sub-network adopts a bidirectional encoder-decoder structure, aiming to make the network's multimodal feature learning process more dependent on the data itself and less dependent on supervisory information. The specific process is as follows: First, the encoder of each modality encodes its respective original data block ( Encoding 7-pixel tiles into a compact representation:

[0049]

[0050]

[0051] In the formula, and These represent the input hyperspectral and LiDAR-derived DSM, respectively; and It is a cross-modal feature encoder with two modes; and These are the features encoded by each individual.

[0052] Secondly, the decoder is used to reconstruct the compact representation to the input data of another modality, thereby realizing the interaction of modality learning. The reconstructed cross-modal features can be represented as:

[0053]

[0054]

[0055] In the formula, and It is a cross-modal feature decoder that combines two modal features; and This represents the features after reconstruction of each modality.

[0056] Since the reconstructed image has the same size as its original input, the unsupervised cross-modal reconstruction loss... The following can be calculated:

[0057]

[0058] It can be seen that the loss of cross-modal feature learning is not related to the real class supervision information of the input. However, through modal interaction, the feature learning of the feature encoder is related to another modality. The learned encoder not only embeds the deep features of the original modality, but also embeds the features related to another modality, thereby realizing the optimization of multimodal features in the learning process.

[0059] In the feature fusion stage, compact features from the two modalities are concatenated and embedded into convolutional blocks in the latent space to obtain fused features.

[0060]

[0061] In the formula, and represents the fusion feature of support set S and query set Q, respectively; C( , ) represents the cascade operator; This represents a convolutional block that includes convolutional layers, batch normalization (BN) layers, and ReLU activation functions. and It refers to the block size and channel size of the fusion feature.

[0062] The relation learning subnetwork aims to enable the model to adaptively learn from data with different distributions, thereby learning the categories of query set samples. Considering the distributional differences between fused features of different categories, directly calculating the similarity distance between query features and features of each category would confuse their semantic relationships. Therefore, this invention does not use distance-based similarity calculation methods, but instead uses a learnable depth metric function to compare the relationships between features, thereby assigning categories to query samples based on the learned similarity. Specifically, the features of the query sample and the features of each category are concatenated one-to-one into relation pairs:

[0063]

[0064] In the formula, and These represent the fused features of the obtained prototype and query set, respectively. This indicates the feature relationship pairs after they are spliced ​​together;

[0065] They are then fed into a two-layer neural network to learn their similarity scores. :

[0066]

[0067] In the formula, The relational network consists of two convolutional blocks, two fully connected blocks, and a random dropout layer.

[0068] Compared to manually selecting similarity calculation methods, the learning process is more data-adaptive. This invention establishes the principle that pairs with the same category combination score closer to 1, while pairs with different category combinations score closer to 0. In practice, features from the prototype and query set are combined with concatenation operators, and then the relation network generates a relation score for each query point. Two fully convolutional layers with loss operations are implemented in the relation learning module, fully connected layers, ReLU function, and batch normalization. The combined pairwise features are ultimately transformed into a distribution with a sigmoid activation function, and a mean squared error loss function is selected to regress the relation score of the label. The entire loss is calculated together with the cross-reconstruction loss during the feature learning process. Through multi-task learning, the network can learn to learn.

[0069]

[0070] in, It is weighted Hyperparameters of value, and These are the parameters of the relational network and the cross-modal feature embedding network, respectively. The similarity is obtained by comparing the true category relationships between the query set and the prototype. It is the similarity between the query set and the prototype learned by the multimodal fusion relation network.

[0071] 2. Perform meta-training on the source dataset.

[0072] On the source data Augsburg dataset, the MFRN network (Multimodal Fusion Relation Network) is first meta-trained using labeled samples in a supervised learning manner. For each training process, each subtask is a 5-way 5-shot small sample setting. A total of 20 labeled samples are used for each class as the small sample annotation set of the target dataset. Therefore, each subtask contains 15 query samples. A total of 10,000 subtasks are sampled without repetition for meta-training to ensure the richness of tasks and improve the robustness of the model. At the same time, there is no overlap between tasks.

[0073] In each subtask, the support set and query set are simultaneously input into the MFRN network. The multimodal data first undergoes feature learning through the cross-modal feature learning sub-network within the multimodal fusion relationship network. The deep features obtained from the two encoders are fused to obtain fused features, and the reconstructed features obtained from the decoder are used to calculate the cross-modal reconstruction loss with the input. Secondly, for each class's multimodal fused features in the support set, the representative feature of each class is selected as the prototype feature by taking the average, representing the feature distribution of that class. Finally, the prototype features of each class are concatenated with all query features of the task to obtain a relationship. The relationship learning sub-network calculates the similarity between each query sample and each class, and calculates the loss based on the true class relationship between the query set and the prototype. This loss, along with the cross-modal loss, is used to train the entire multimodal fusion relationship network in a supervised manner.

[0074] 3. Fine-tune on a small target dataset.

[0075] The MFRN network already possesses the ability to extract features and learn through multi-task meta-training on the source dataset. When the target domain has a small number of labeled samples, fine-tuning the model further adapts this meta-learning ability to different target domains. Specifically, multiple different sub-tasks are generated from a small sample dataset of target point clouds using meta-training. The training process also follows the same multi-task supervised training to update the network parameters. The only difference is that the initial training parameters of the model are selected from the meta-training parameters, making the fine-tuning process more efficient than the meta-training process.

[0076] 4. Perform meta-tests on the unlabeled target dataset.

[0077] After fine-tuning, the MFRN network can now classify multimodal remote sensing images. Meta-testing is then performed on unlabeled samples in the target dataset to address new cross-domain scenarios. Unlike the sampling methods used during training and fine-tuning, samples are no longer randomly selected from the target data. Instead, a small subset of labeled samples from the target domain are used as support samples to provide a reference for the meta-testing process. This ensures that all unlabeled points can obtain their class labels through the network. Finally, the class label for each point in the test set is obtained through a voting method.

[0078] To better illustrate the effectiveness of this invention, a comparative experiment was conducted between the multimodal fusion relation network of this invention's meta-learning method and six existing methods. The comparison methods specifically included two supervised learning networks, two semi-supervised learning methods, and two meta-learning methods. The supervised learning models were Contextual Deep CNN (CDCNN) and Spectral Spatial Residual Network (SSRN); the semi-supervised learning models were Transductive SVM (TSVM) and Semi-supervised Deep Graph Convolutional Network (DCGCN-SEMI); and the meta-learning methods were Deep Few-Shot Learning Network (DFSL) and Relation Network for Hyperspectral Few-Shot Classification (RN-FSC).

[0079] Among them, Tables 1-3 and Figure 7-9 The results of quantitative and qualitative experiments on three test data are as follows. Figure 7 , Figure 8 and Figure 9 From left to right, the images show the true class labels of the corresponding datasets, as well as the experimental results of CDCNN, SSRN, TSVM, DCGCN-SEMI, DFSL, DCGCN-SEMI, and the present invention (MFRN-ML) for segmenting the target dataset.

[0080] Table 1

[0081]

[0082] Table 2

[0083]

[0084] Table 3

[0085]

[0086] Experimental results show that for supervised methods (such as CDCNN and SSRN), sufficient annotated samples are essential for extracting discriminative features. A small number of labeled target samples can easily lead to overfitting of the source dataset by supervised learning methods. Although SSRN has a higher classification accuracy than CDCNN due to the presence of spectral information, its classification performance on target datasets is severely limited. Semi-supervised methods (i.e., TSVM and DCGCN-SEMI) can achieve better performance by utilizing both labeled and unlabeled information. Compared with TSVM, the deep learning classifier (DCGCN-SEMI) has a higher accuracy. Although it adopts a relatively simple structure, it introduces unlabeled samples during training, resulting in overall performance superior to supervised methods. Meta-learning, as a new few-shot learning paradigm, enables DFSL and RN-FSC methods to extract more robust features than supervised and semi-supervised methods, while learning transferable meta-knowledge, effectively improving classification accuracy.

[0087] Compared to all the methods mentioned above, the MFRN-ML designed in this invention achieves the highest classification accuracy. Compared to the two comparative meta-learning classification methods, this invention can extract more internal complementary information from multimodal data. The unsupervised learning process makes the fused features more adaptable to the data, unaffected by domain shifts. Compared to the methods mentioned above, this invention relies on meta-learning to pay more attention to each category, effectively mitigating the profanity and misclassification phenomena present in other methods in complex scenes. It more finely and accurately classifies detailed targets and well preserves the boundary information of ground features. Furthermore, the learning ability can be further learned as meta-knowledge and quickly transferred to different small sample target datasets.

Claims

1. A method for classifying ground features in multimodal remote sensing data based on meta-learning, characterized in that, This classification method includes the following steps: 1) Construct a multimodal fusion relationship network, including a cross-modal feature learning sub-network and a relationship learning sub-network. The cross-modal feature learning sub-network encodes the original data of each modality through an encoder to obtain the corresponding compact representation. The compact representation is reconstructed to the input data of another modality through a decoder. The compact features of the two modalities are concatenated through fusion. The relationship learning sub-network uses a learnable depth metric function to compare the relationship between features and assigns a category to the query sample based on the learned similarity. 2) Meta-training of the multimodal fusion relation network is performed using a labeled source dataset, which includes a support set and a query set. For the multimodal fusion features of each class in the support set, the representative feature of each class is selected as the prototype feature by taking the mean. The prototype feature of each class is concatenated with all query features of the training task to obtain relation pairs. The loss function used is: , It is weighted Hyperparameters of value, It is the loss of the cross-modal feature learning sub-network. and These are the parameters of the relation learning subnetwork and the cross-modal feature learning subnetwork, respectively. The similarity is obtained by comparing the true category relationships between the query set and the prototype. It is the similarity between the query set and the prototype learned by the multimodal fusion relationship network; 3) Fine-tune the meta-trained multimodal fusion relation network using labeled samples from the target dataset; 4) Classify unlabeled multimodal remote sensing data using a fine-tuned multimodal fusion relation network.

2. The method for classifying land cover in multimodal remote sensing data based on meta-learning according to claim 1, characterized in that, The cross-modal feature learning subnetwork adopts a bidirectional encoder-decoder structure.

3. The multimodal remote sensing data land cover classification method based on meta-learning according to claim 1, characterized in that, During meta-training, multiple subtasks are sampled non-repeatedly from the source dataset for meta-training, with each subtask set up for small samples.

4. The method for classifying land cover in multimodal remote sensing data based on meta-learning according to claim 1 or 3, characterized in that, The support set and query set are simultaneously input into the cross-modal feature learning sub-network to obtain multimodal fusion features; The relation learning subnetwork calculates the similarity between each query sample and each class based on the obtained relation pairs, and calculates the loss based on the real class relation pairs obtained from the query set and the prototype. Together with the loss of the cross-modal feature learning subnetwork, the entire multimodal fusion relation network is trained in a supervised manner.

5. The multimodal remote sensing data land cover classification method based on meta-learning according to claim 1, characterized in that, During fine-tuning, multiple subtasks are sampled from a small set of labeled samples for fine-tuning the multimodal fusion relation network on the target dataset.

6. The method for classifying ground features in multimodal remote sensing data based on meta-learning according to claim 4, characterized in that, During meta-training, for each training process, each subtask is set up with a 5-way 5-shot small sample.

7. The multimodal remote sensing data land cover classification method based on meta-learning according to claim 6, characterized in that, The loss function used in the cross-modal feature learning sub-network is: , It is the loss of the cross-modal feature learning sub-network. , These represent the hsi modal features before and after reconstruction, respectively. , These represent the lidar modal features before and after reconstruction, respectively.

Images