Multi-modal remote sensing image classification method under any modal missing condition

Abstract

The invention discloses a multi-modal remote sensing image classification method under any modal missing condition, which comprises the following steps of: acquiring multi-modal remote sensing image data which comprises a hyperspectral image and other modal images, and a ground object type label; constructing a hyperspectral image encoder for the hyperspectral image, constructing other modal image encoders for other modal images, and extracting real specific features of each modal image; constructing a modal sharing feature encoder, and extracting sharing features of each modal image; real specific feature loss and shared feature alignment loss are constructed; generating a missing mode and generating a specific feature; the total loss of first-stage pre-training is constructed, and first-stage pre-training is carried out; performing two-stage fine tuning, and constructing total classification features; obtaining ground feature classification of the multi-modal remote sensing image through a classifier; the method is high in flexibility, can adapt to the deficiency of any number of modes, and has higher ground feature classification accuracy under the condition of mode deficiency.

Description

Technical Field

[0001] This invention relates to the field of satellite remote sensing imagery, and more specifically to a multimodal remote sensing image classification method under conditions of arbitrary modality loss. Background Technology

[0002] In the process of acquiring remote sensing data, we often face the challenges of weather and sensor instability. Natural disasters such as earthquakes, floods, and typhoons are frequently accompanied by severe rainy weather, making it difficult for traditional optical satellites (such as the Gaofen series and Landsat) to visualize the damage on the ground. Accurate identification of ground features is crucial for responding to emergency disasters, agricultural monitoring, and military reconnaissance. Most existing multimodal remote sensing data fusion models assume they can acquire complete, unbiased raw images. However, these models struggle to cope with modal gaps caused by extreme weather (cloud and fog obstruction) or specific time periods (nighttime imaging), significantly impacting model performance.

[0003] With the development of deep learning, numerous AI models for remote sensing image land cover classification have emerged, but those suitable for modality missing problems are relatively scarce. Currently, there are many methods to solve the modality missing problem, such as early retrieval models that retrieve similar samples from a pre-prepared database to replace missing samples; generative models such as GANs and diffusion models have provided means to recover the original image, but most methods have some drawbacks. Retrieval methods require searching and comparing in a huge database for each sample, resulting in high computational cost, slow inference speed, and the need for additional memory to store the database; image recovery methods require complex model design and extremely large training loads. Typical diffusion models require training for four to five hundred rounds over thousands of diffusion time steps, which is not advantageous in terms of time and parameter scale; furthermore, many models are designed based on bimodal missing features, and extending to trimodal or higher requires significant changes to the model structure and adjustments for each combination of missing features, which is particularly evident in distillation and illusion-based networks. Therefore, how to design a method that can solve the problem of modality loss while taking into account the training cost and flexibility of the model to adapt to arbitrary modality combinations has become an urgent problem to be solved in the field of remote sensing image fusion. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal remote sensing image classification method under arbitrary modality loss conditions. This method can solve the problems of existing methods that rely heavily on complete modality data and are difficult to deal with modality loss caused by special weather (cloud and fog obstruction) or specific time periods (night imaging).

[0005] To achieve the above functions, this invention designs a multimodal remote sensing image classification method under arbitrary modality loss conditions. For the target area, the following steps S1-S8 are performed to construct and train a remote sensing image land cover classification model, thereby completing the land cover classification of the multimodal remote sensing image:

[0006] Step S1: Acquire multimodal remote sensing images of the target area, which include hyperspectral images, synthetic aperture radar images, and digital surface model images; label the corresponding land cover types in each image;

[0007] Step S2: For hyperspectral images, construct a hyperspectral image encoder to extract real-specific features of the hyperspectral images; for images of other modalities, construct image encoders for other modalities to extract real-specific features of the other modalities; and construct a modal shared feature encoder to extract shared features of hyperspectral images, shared features of synthetic aperture radar images, and shared features of digital surface model images.

[0008] Step S3: Based on the real-specific features of hyperspectral images, synthetic aperture radar images, and digital surface model images, construct the real-specific feature loss. Based on the shared features of hyperspectral images, synthetic aperture radar images, and digital surface model images, construct the shared feature alignment loss in the shared feature alignment task based on KL divergence loss.

[0009] Step S4: Construct a missing modality generation module, randomly discard each modality to generate missing modalities, and generate generation-specific features of missing modalities based on an attention mechanism; based on the generation-specific features of missing modalities and the corresponding real-specific features of missing modalities in step S2, construct the total loss for one-stage pre-training, and perform one-stage pre-training.

[0010] Step S5: After the first stage of pre-training is completed, each modality is randomly discarded to generate training samples for the second stage of fine-tuning.

[0011] Step S6: Perform two-stage fine-tuning, construct a logic-guided gating fusion module, input the real specific features and generated specific features of the missing modalities in the training samples, combine the CNN gating network and the cue learning method to obtain the overall specific features, and construct the gating weight loss.

[0012] Step S7: Based on the fusion of shared features of each modality, obtain the overall shared features; add the overall shared features and the overall specific features to obtain the overall classification features; pass through a classifier to obtain the land cover classification of the multimodal remote sensing image;

[0013] Step S8: After the second-stage fine-tuning is completed, the training of the remote sensing image land cover classification model is completed. The trained remote sensing image land cover classification model is applied to complete the land cover classification of the multimodal remote sensing images of the target area.

[0014] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0015] 1. This invention can adapt to the absence of any number of modes without requiring a redesign of the model structure for each missing combination, and has good scalability and versatility;

[0016] 2. In small sample categories (such as commercial areas, water bodies, etc.), the classification performance of this invention is better than the comparison method, demonstrating its robustness and learning ability to imbalanced data;

[0017] 3. On datasets with different regions and different modal combinations, the present invention demonstrates stable high performance, the classification results are closer to the true labels, and the classification of edge regions and dense regions is more refined;

[0018] 4. By introducing a language-guided missing modality generation module (LPG) and a logic-guided gated fusion module (LGF), and combining them with a multi-task loss function, the model's ability to reconstruct missing modalities and the rationality of feature fusion are effectively improved. Attached Figure Description

[0019] Figure 1 This is a flowchart of a multimodal remote sensing image classification method under arbitrary modality missing conditions provided by an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of a one-stage pre-training provided according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a missing mode generation module provided according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of two-stage fine-tuning provided according to an embodiment of the present invention;

[0023] Figure 5 This is a flowchart of a two-stage fine-tuning training method provided according to an embodiment of the present invention;

[0024] Figure 6 This is a category diagram of the Augsburg dataset provided according to an embodiment of the present invention;

[0025] Figure 7 This is a category diagram of the Berlin dataset provided according to an embodiment of the present invention;

[0026] Figure 8 This is a classification visualization diagram of the Augsburg dataset with missing modalities provided in an embodiment of the present invention;

[0027] Figure 9This is a visualization of the classification performance of the Berlin dataset provided in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0029] This invention provides a multimodal remote sensing image classification method under arbitrary modality loss conditions, which, for a target area, refers to... Figure 1 Perform the following steps S1-S8 to construct and train the remote sensing image land cover classification (LGWG-Net) model to complete the land cover classification of multimodal remote sensing images:

[0030] Step S1: Acquire multimodal remote sensing images of the target area, which include hyperspectral images, synthetic aperture radar images, and digital surface model images; label the corresponding land cover types in each image;

[0031] The specific steps of step S1 are as follows:

[0032] Step S1.1: Obtain multimodal remote sensing images of the target area and corresponding ground truth labels for land cover classification, where the ground truth labels are the land cover types;

[0033] In this embodiment, satellite data of hyperspectral (HSI), synthetic aperture radar (SAR), and digital surface model (DSM) for the Augsburg region are acquired, along with a training set TrainImage and a test set TestImage.

[0034] Acquire satellite data for the Berlin region, including hyperspectral (HSI), multispectral (MS), and synthetic aperture radar (SAR) data, as well as the groundtruth file for the entire map.

[0035] Step S1.2: Divide the multimodal remote sensing images and corresponding ground object classification ground truth labels into training set, validation set, and test set according to a preset ratio;

[0036] In this example, the Augsburg dataset was created using a recognized partitioning method, with 761 training samples and 77,533 test samples.

[0037] The Berlin dataset is divided into training and testing sets in a 1:9 ratio. Each class is sampled in this ratio to ensure class fairness. After the division, there are 161,426 training samples and 1,452,889 test samples.

[0038] For the training samples of each dataset, 10% of each class is set as a validation set to preserve the best training model.

[0039] Step S2: For hyperspectral images, construct a hyperspectral image encoder to extract real-specific features of the hyperspectral images; for images of other modalities, construct image encoders for other modalities to extract real-specific features of the other modalities; and construct a modal shared feature encoder to extract shared features of hyperspectral images, shared features of synthetic aperture radar images, and shared features of digital surface model images.

[0040] Reference Figure 2 The specific steps of step S2 are as follows:

[0041] Step S2.1: Input the images in the training set in the form of image patches. For hyperspectral images... A hyperspectral image encoder containing a 3D convolution module and a 2D convolution module is constructed, wherein the 3D convolution module includes three 3D convolution layers;

[0042] The first 3D convolutional layer includes a convolutional kernel of size (7, 3, 3) with a stride of 1 and padding of (3, 1, 1), keeping the size of the input and output images unchanged, and increasing the number of channels from 1 to 8.

[0043] The second 3D convolutional layer uses a convolutional kernel with size (7, 1, 1) and stride (3, 1, 1), and no longer performs spatial convolution. It performs spectral dimensionality reduction on the image, increasing the number of channels from 8 to 16.

[0044] The structure of the third 3D convolutional layer is exactly the same as that of the second 3D convolutional layer. For images with more than a preset threshold of spectral quantity, further spectral dimensionality reduction and channel number increase are performed.

[0045] Each 3D convolutional layer is followed by a 3D batch normalization layer and a corrected linear unit, which flattens the 3D output image into a 2D image format.

[0046] The 2D convolutional module consists of three residual layers with 64, 128, and 256 channels respectively. Finally, after passing through a global average pooling layer and a fully connected layer, the hyperspectral image encoder outputs the true specific features of the hyperspectral image. ;

[0047] Step S2.2: For the synthetic aperture radar image and the digital surface model image respectively, construct synthetic aperture radar image encoder and digital surface model image encoder with identical structures, as follows:

[0048] First, the image passes through an initial 2D convolutional layer with a kernel size of 3×3, a stride of 1, and a uniform channel number of 64. Then, it passes through three standard ResNet residual layers, each containing two 3×3 convolutional layers and a residual connection step with a stride of 2, performing downsampling. The channel numbers of the three residual layers are 64, 128, and 256 respectively. Finally, through a global average pooling layer and a fully connected layer, the synthetic aperture radar image encoder and the digital surface model image encoder output the true specific features of the synthetic aperture radar image. Real-world specific features of digital surface model images ;

[0049] Step S2.3: Construct a modality-shared feature encoder that receives images of any modality and maps them to the same shared feature space. The modality-shared feature encoder consists of three parts: the first part is a modality adaptation layer that unifies the input image into a 64-channel feature map; for hyperspectral images... The first part uses 1×1 2D convolutional layers to compress the number of spectra, while images of other modalities are extracted and their dimensions increased using 3×3 convolutional layers. The second part is a shared backbone layer, where images of all modalities are adapted to a shape of (B, 64, H, W), where the first dimension B represents the batch size, the second dimension represents the number of channels, the third dimension H represents the image height, and the fourth dimension W represents the image width. Deep features are extracted through a set of parameter-shared ResNet layers. The third part is the output head, which consists of standard pooling layers and fully connected layers to obtain the shared features of the hyperspectral images. Shared features of synthetic aperture radar images Shared features of digital surface model images .

[0050] Step S3: Based on the real-specific features of hyperspectral images, synthetic aperture radar images, and digital surface model images, construct the real-specific feature loss. Based on the shared features of hyperspectral images, synthetic aperture radar images, and digital surface model images, construct the shared feature alignment loss in the shared feature alignment task based on KL divergence loss.

[0051] The specific steps of step S3 are as follows:

[0052] Step S3.1: Based on the true specific features of the hyperspectral image True specific features of synthetic aperture radar images Real-world specific features of digital surface model images After pairwise similarity calculation, the loss of the true specific feature is calculated using the following formula. :

[0053] ;

[0054] in, Representing modes i The true specific characteristics, Representing modes j The true specific characteristics, This indicates the calculation of cosine similarity. Indicates the number of modal longitudinals. This indicates the calculation of the average value; by minimizing the absolute value of the cosine similarity between feature vectors, they are made orthogonal to each other in the feature space, further ensuring that the features extracted by the encoder of a specific modality are unique to that modality.

[0055] Step S3.2: Targeting shared features of hyperspectral images Shared features of synthetic aperture radar images Shared features of digital surface model images Alignment is performed using KL divergence loss, assuming mode i The shared feature distribution follows a Gaussian distribution. ,in and For modality i The shared characteristics of the modalities are the mean and standard deviation, while the distribution of the average statistic of all modalities is considered as the central distribution. ,in , The KL divergence loss represents the average of the mean and standard deviation of the shared features across all modalities, as shown in the following formula:

[0056] ;

[0057] in, Indicates the KL divergence loss;

[0058] In the shared feature alignment task, the averaged shared feature distribution is used as the center. KL divergence loss is then used to pull the shared feature distributions of all single modalities towards the center, thereby achieving semantic alignment of different modalities in the shared feature space. The average loss between all modalities and the center is denoted as the shared feature alignment loss. .

[0059] Step S4: Construct a missing modality generation module, randomly discard each modality to generate missing modalities, and generate generation-specific features of missing modalities based on an attention mechanism; based on the generation-specific features of missing modalities and the corresponding real-specific features of missing modalities in step S2, construct the total loss for one-stage pre-training, and perform one-stage pre-training.

[0060] The specific steps of step S4 are as follows:

[0061] Step S4.1: Refer to Figure 3 A missing modality generation (LPG) module is constructed to receive real-specific features and shared features of all modalities. Then, images of random modalities are discarded, and the discarded modalities are taken as missing modalities, while the retained modalities are taken as available modalities. The shared features of the discarded modalities will not be used in the reconstruction, and the real-specific features are used for the final reconstruction loss.

[0062] Step S4.2: Construct a reconstruction module. Based on the attention mechanism, the real-specific features and shared features of all available modalities are fused and passed through a linear layer to become the key K and value V in the attention mechanism.

[0063] Step S4.3: Generate descriptive text for the inherent features of each modality, and input it into a pre-trained text encoder. In this embodiment, a clip-ViT-B / 32 text encoder is used to obtain the linguistic features of the missing modality. , serving as query Q in the attention mechanism;

[0064] Step S4.4: Perform multi-head attention operation on the obtained query Q, key K, and value V to obtain the interaction context vector. Missing modes are generated via multilayer perceptron. i Generate specific features ;

[0065] Step S4.5: Remove missing modes i Generate specific features The corresponding mode obtained in step S2.2 i True specific features Calculate the MSE loss as the reconstruction loss for generating specific features. ;

[0066] Reference Figure 2 A first-stage pre-training process is performed, and the total loss for the first-stage pre-training is constructed as follows:

[0067] ;

[0068] in, This represents the total loss during the first phase of training. Represents the loss of true specific features. This represents the reconstruction loss for generating specific features. This represents the shared feature alignment loss; This is an adjustable parameter.

[0069] Step S5: After the first stage of pre-training is completed, each modality is randomly discarded to generate training samples for the second stage of fine-tuning.

[0070] The specific steps of step S5 are as follows:

[0071] Step S5.1: After the first stage of pre-training is completed, save and freeze the parameters of each modality image encoder and missing modality generation module;

[0072] Step S5.2: For the M modes, first randomly discard them. Data from each modality is recorded as multimodal data. Subsequently, for multimodal data The data in the middle is then randomly discarded. Each modality is recorded as the few-modal data. ; let the multimodal data sequentially and few modal data The model is trained.

[0073] Step S6: Perform two-stage fine-tuning, construct a logic-guided gating fusion module, input the real specific features and generated specific features of the missing modalities in the training samples, combine the CNN gating network and the cue learning method to obtain the overall specific features, and construct the gating weight loss.

[0074] Reference Figure 4 , Figure 5 The specific steps of step S6 are as follows:

[0075] Step S6.1: Perform two-stage fine-tuning. The specific features for generating missing modalities are reconstructed and replaced by the missing modal generation module trained in the first-stage pre-training, while the shared features of missing modalities are directly discarded.

[0076] Step S6.2: After the missing modality generation module reconstructs the specific features, the specific features of all modalities will be divided into the original true specific features. and the generation of specific features of reconstruction Construct a logic-guided gated fusion module (LGF) and input real-specific features. and generating specific features The gated fusion module combines a CNN gated network and a cue learning method. For each modality, a specific input feature is concatenated with a learnable vector as a cue. The cue length is an adjustable parameter. All specific features with cue input are then fed into the CNN gated network to generate weight coefficients. Weighted fusion is then performed to obtain the overall specific features. ;

[0077] Step S6.3: Construct the gating weight loss calculation formula as follows:

[0078] ;

[0079] in, This indicates the gating weight loss. i Representing few modal data Index of missing modalities j Representing few modal data The modal index is available in the middle; Indicates the total number of modes. n For fewer modal data The total number of missing modes; Representing few modal data intermediate mode i The weights for generating specific features, Representing multimodal data intermediate mode i The weights of the true specific features; Representing multimodal data intermediate mode j The weights of the true specific features, Representing few modal data intermediate mode j The weights of the true specific features.

[0080] Taking the trimodal Augsburg dataset as an example, there are a total of three modes: HSI, SAR, and DSM; in a certain round of training, multimodal data... For full-modal Few modal data for The missing synthetic aperture radar modes are indicated by "miss". When assigning weights to the gating network, SAR features with fewer modes are used to generate specific features, and their weights are determined accordingly. It should be less than the weight corresponding to the true specific feature of SAR under multimodal data. Both HSI and DSM represent available modalities, but with limited modal data, the model should prioritize the available true modal information. Should be greater than .

[0081] Step S7: Based on the fusion of shared features of each modality, obtain the overall shared features; add the overall shared features and the overall specific features to obtain the overall classification features; pass through a classifier to obtain the land cover classification of the multimodal remote sensing image;

[0082] The specific steps of step S7 are as follows:

[0083] Step S7.1: The overall shared features are obtained by averaging the shared features of all available modalities. Shared characteristics of the whole and overall specific characteristics Add them together to obtain the total classification features. The data is then fed into a classifier to obtain the final logits value, which is then compared with the ground truth labels for land cover classification to calculate the classification cross-entropy loss. ;

[0084] Step S7.2: Generally, the more modalities available, the better the model performance and the lower the classification loss. Therefore, a loss function was designed to approximate multimodal data with fewer modalities; constructing fewer modalities... With multimodal data Distance loss:

[0085] ;

[0086] in, For fewer modal data With multimodal data Distance loss; Indicating in multimodal data The classification loss below, Indicates the presence of few modal data Classification loss;

[0087] The overall loss for the two-stage fine-tuning phase is calculated as follows:

[0088] ;

[0089] ;

[0090] in, For classification cross-entropy loss, This indicates the gating weight loss. This represents the distance loss between few-modal data and multimodal data. and This is an adjustable parameter.

[0091] Step S8: After the second-stage fine-tuning is completed, the training of the remote sensing image land cover classification model is completed. The trained remote sensing image land cover classification model is applied to complete the land cover classification of the multimodal remote sensing images of the target area.

[0092] The following is an application example of the present invention:

[0093] See Figure 6 , Figure 7 As shown, two remote sensing image datasets from different regions were collected. The Augsburg dataset is a small dataset with 7 land cover categories and a relatively serious class imbalance problem. The training samples for industrial areas, commercial areas, and water bodies are significantly scarce, which tests the model's ability to learn from small samples. The Berlin dataset is a large dataset with more than 160,000 training samples and more than 1.45 million test samples. It also has a certain degree of class bias, but the training samples for each class are relatively sufficient.

[0094] Referring to Tables 1 and 2, this embodiment provides test indicators for land cover classification in the Augsburg and Berlin regions to verify the experimental results, and conducts detailed comparative experiments with other models used in recent years to solve the modality missing problem. The following is a brief introduction to the other comparative models:

[0095] Vit_Prompt: From the 2023 CVPR conference (Y.-L. Lee, Y.-H. Tsai, W.-C. Chiu, andC.-Y. Lee, "Multimodal prompting with missing modalities for visualrecognition," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2023, pp. 14943-14952.), the authors addressed the lack of modality by adding cue vectors to the trained Transformer model.

[0096] Mmanet: From the 2023 CVPR conference (S. Wei, C. Luo, and Y. Luo, "Mmanet: Margin-aware distillation and modality-aware regularization for incomplete multimodal learning," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2023, pp. 20039-20049.), the authors employed a self-designed "margin-aware" distillation algorithm, using a full-modal teacher network to teach the student network to address modality loss.

[0097] LDS2AE: From the 2024 AAAI meeting (J. Qu, Y. Yang, W. Dong, and Y. Yang, "LDS2AE: Local diffusion shared-specific autoencoder for multimodal remotesensing image classification with arbitrary missing modalities," in Proceedings of the AAAI Conference on Artificial Intelligence ( , vol. 38, no.13, 2024, pp. 14731-14739.), mainly adopts the diffusion model combined with mask reconstruction to deal with the lack of modes.

[0098] SimMLM: From the 2025 ICCV conference (S. Li, C. Chen, and J. Han, "SimMLM: Asimple framework for multimodal learning with missing modality," in Proceedings of the IEEE / CVF International Conference on Computer Vision (2025, pp. 24068-24077.) mainly proposes a general network architecture that dynamically fuses each modality network after training them separately to address modality loss.

[0099] Overall accuracy (OA), average class accuracy (AA), and Kappa coefficient were used as evaluation metrics in the experiment. As can be clearly seen from the data in Tables 1 and 2, the method proposed in this invention achieved consistent and significant optimal performance across all modality combinations in both datasets.

[0100] On the Augsburg dataset, when the input includes all modalities (HSI + SAR + DSM), the method of this invention achieves an OA of 91.81% and a Kappa coefficient of 88.42%, representing improvements of 2.4% and 3.74% respectively compared to the suboptimal method SimMLM. Notably, the improvement in the AA metric is particularly significant (from 62.97% to 68.11%), indicating that the method not only performs well in overall classification but also excels in handling class imbalance and identifying small sample classes. Furthermore, under the condition of full-modal input (HSI + SAR + DSM) on the Berlin dataset, the method achieves a peak accuracy of 86.19%.

[0101] In the most challenging single-modal tests, many methods often suffer performance collapse due to insufficient information. For example, in the Augsburg dataset, using only the DSM modality, the OA of Vit_Prompt and Mmanet is only 52.91% and 60.80%, respectively, which is insufficient for practical needs. In contrast, this invention, through an effective feature extraction and reconstruction mechanism, remarkably improves the OA of the DSM single modality to 82.33%, an increase of over 20%. Similarly, in the Berlin dataset, using only SAR data, this invention also improves the OA from 53.94% for Mmanet to 74.43%. This fully demonstrates that this invention can maximize the extraction of information from single modalities and between modalities.

[0102] In the dual-modal case, the performance of the method of this invention steadily improves and is consistently superior to the comparative methods. For example, in the HSI + SAR combination of the Berlin dataset, the OA of the method of this invention reaches 84.05%, a result that even surpasses the performance of some comparative methods (such as Vit_Prompt, LDS2AE) under full-modal input.

[0103] Furthermore, the data in the comparison table reveals that different modalities contribute differently to the classification results (e.g., HSI is generally superior to SAR). However, regardless of the missing modality, the performance degradation of the method in this invention is less than that of other comparative methods. Taking the Augsburg dataset as an example, when degrading from full modality (OA 91.81%) to HSI only (OA 90.42%), the accuracy of this invention is only about 1.4%; while SimMLM experiences a degradation of about 2%. This further verifies the effectiveness of the model architecture of this invention in feature alignment and cross-modal interaction, reducing over-reliance on specific modalities and achieving more robust land cover classification.

[0104] Table 1. Comparison of test metrics for the Augsburg dataset

[0105]

[0106] Table 2. Comparison of test metrics for the Berlin dataset

[0107]

[0108] In Tables 1 and 2, √ indicates that the input dataset contains images of the corresponding modality.

[0109] See Figure 8 , Figure 9 As shown, a multimodal remote sensing image classification method designed under arbitrary modality missing conditions based on an embodiment of the present invention was used to visualize the test results of the dataset.

[0110] After training the model, the entire image data was used as the test set and input into the model to obtain the predicted values ​​for the entire image. The visualization was created using Matlab software, with appropriate colors selected for each class in the dataset to represent the instances.

[0111] Figure 8 This reflects the performance of the Augsburg dataset in land cover classification across all modalities (including full modality). Compared to the ground truth map (GT), the classification performance for bimodal and higher modalities is quite refined, correctly classifying even a few categories (such as commercial areas), which are represented by small red dots on the map. The performance for unimodal maps is slightly lower, but the overall bias is not significant. Figure 9This reflects the comparative effects of different models on the Berlin dataset under full modality. It is easy to see that the effect of the present invention is closer to the true value, especially in dense areas of multiple categories and at the edges of different categories.

[0112] The visualization results demonstrate that the invention has strong generalization ability and robustness in different datasets, and can achieve more accurate land cover classification even in the case of modality loss.

[0113] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A multimodal remote sensing image classification method under arbitrary modality loss conditions, characterized in that, For the target area, perform the following steps S1-S8 to build and train a remote sensing image land cover classification model to complete the land cover classification of multimodal remote sensing images: Step S1: Acquire multimodal remote sensing images of the target area, which include hyperspectral images, synthetic aperture radar images, and digital surface model images; label the corresponding land cover types in each image; Step S2: For hyperspectral images, construct a hyperspectral image encoder to extract the true specific features of the hyperspectral images; for images of other modalities, construct image encoders for those modalities respectively to extract the true specific features of those modalities. A modal shared feature encoder was constructed to extract shared features from hyperspectral images, synthetic aperture radar images, and digital surface model images. Step S3: Based on the real-specific features of hyperspectral images, synthetic aperture radar images, and digital surface model images, construct the real-specific feature loss. Based on the shared features of hyperspectral images, synthetic aperture radar images, and digital surface model images, construct the shared feature alignment loss in the shared feature alignment task based on KL divergence loss. Step S4: Construct a missing modality generation module, randomly discard each modality to generate missing modalities, and generate specific features for the generation of missing modalities based on an attention mechanism; Based on the generated specific features of the missing modalities and the corresponding real specific features of the missing modalities in step S2, the total loss of the one-stage pre-training is constructed, and one-stage pre-training is performed. Step S5: After the first stage of pre-training is completed, each modality is randomly discarded to generate training samples for the second stage of fine-tuning. Step S6: Perform two-stage fine-tuning, construct a logic-guided gating fusion module, input the real specific features and generated specific features of the missing modalities in the training samples, combine the CNN gating network and the cue learning method to obtain the overall specific features, and construct the gating weight loss. Step S7: Obtain the overall shared features by fusing the shared features of each modality; The overall shared features and overall specific features are added together to obtain the overall classification features; after passing through a classifier, the land cover classification of the multimodal remote sensing image is obtained. Step S8: After the second-stage fine-tuning is completed, the training of the remote sensing image land cover classification model is completed. The trained remote sensing image land cover classification model is applied to complete the land cover classification of the multimodal remote sensing images of the target area.

2. The multimodal remote sensing image classification method under arbitrary modality loss conditions according to claim 1, characterized in that, The specific steps of step S1 are as follows: Step S1.1: Obtain multimodal remote sensing images of the target area and corresponding ground truth labels for land cover classification, where the ground truth labels are the land cover types; Step S1.2: Divide the multimodal remote sensing images and corresponding ground object classification ground truth labels into training set, validation set, and test set according to a preset ratio.

3. The multimodal remote sensing image classification method under arbitrary modality loss conditions according to claim 1, characterized in that, The specific steps of step S2 are as follows: Step S2.1: For hyperspectral images A hyperspectral image encoder containing a 3D convolution module and a 2D convolution module is constructed, wherein the 3D convolution module includes three 3D convolution layers; The first 3D convolutional layer includes a convolutional kernel of size (7, 3, 3) with a stride of 1 and padding of (3, 1, 1), keeping the size of the input and output images unchanged, and increasing the number of channels from 1 to 8. The second 3D convolutional layer uses a convolutional kernel with size (7, 1, 1) and stride (3, 1, 1), and no longer performs spatial convolution. It performs spectral dimensionality reduction on the image, increasing the number of channels from 8 to 16. The structure of the third 3D convolutional layer is exactly the same as that of the second 3D convolutional layer. For images with more than a preset threshold of spectral quantity, further spectral dimensionality reduction and channel number increase are performed. Each 3D convolutional layer is followed by a 3D batch normalization layer and a corrected linear unit, which flattens the 3D output image into a 2D image format. The 2D convolutional module consists of three residual layers with 64, 128, and 256 channels respectively. Finally, after passing through a global average pooling layer and a fully connected layer, the hyperspectral image encoder outputs the true specific features of the hyperspectral image. ; Step S2.2: For the synthetic aperture radar image and the digital surface model image respectively, construct synthetic aperture radar image encoder and digital surface model image encoder with identical structures, as follows: First, the image passes through an initial 2D convolutional layer with a kernel size of 3×3, a stride of 1, and a uniform channel number of 64. Then, it passes through three standard ResNet residual layers, each containing two 3×3 convolutional layers and a residual connection step with a stride of 2, performing downsampling. The channel numbers of the three residual layers are 64, 128, and 256 respectively. Finally, through a global average pooling layer and a fully connected layer, the synthetic aperture radar image encoder and the digital surface model image encoder output the true specific features of the synthetic aperture radar image. Real-world specific features of digital surface model images ; Step S2.3: Construct a modality-shared feature encoder that receives images of any modality and maps them to the same shared feature space. The modality-shared feature encoder consists of three parts: the first part is a modality adaptation layer that unifies the input image into a 64-channel feature map; for hyperspectral images... The first part uses 1×1 2D convolutional layers to compress the number of spectra, while images of other modalities are extracted and their dimensions increased using 3×3 convolutional layers. The second part is a shared backbone layer, where images of all modalities are adapted to a shape of (B, 64, H, W), where the first dimension B represents the batch size, the second dimension represents the number of channels, the third dimension H represents the image height, and the fourth dimension W represents the image width. Deep features are extracted through a set of parameter-shared ResNet layers. The third part is the output head, which consists of standard pooling layers and fully connected layers to obtain the shared features of the hyperspectral images. Shared features of synthetic aperture radar images Shared features of digital surface model images .

4. The multimodal remote sensing image classification method under arbitrary modality loss conditions according to claim 3, characterized in that, The specific steps of step S3 are as follows: Step S3.1: Based on the true specific features of the hyperspectral image True specific features of synthetic aperture radar images Real-world specific features of digital surface model images After pairwise similarity calculation, the loss of the true specific feature is calculated using the following formula. : ； in, Representing modes i The true specific characteristics, Representing modes j The true specific characteristics, This indicates the calculation of cosine similarity. Indicates the number of modal longitudinals. This indicates the calculation of the average value; Step S3.2: Targeting shared features of hyperspectral images Shared features of synthetic aperture radar images Shared features of digital surface model images Alignment is performed using KL divergence loss, assuming mode i The shared feature distribution follows a Gaussian distribution. ,in and For modality i The shared characteristics of the modalities are the mean and standard deviation, while the distribution of the average statistic of all modalities is considered as the central distribution. ,in , The KL divergence loss represents the average of the mean and standard deviation of the shared features across all modalities, as shown in the following formula: ； in, Indicates the KL divergence loss; In the shared feature alignment task, the averaged shared feature distribution is used as the center. KL divergence loss is then used to pull the shared feature distributions of all single modalities towards the center. The average loss of all modalities relative to the center is denoted as the shared feature alignment loss. .

5. The multimodal remote sensing image classification method under arbitrary modality loss conditions according to claim 4, characterized in that, The specific steps of step S4 are as follows: Step S4.1: Construct a missing modality generation module, which receives the real specific features and shared features of all modalities, and then discards the images of random modalities. The discarded modalities are taken as missing modalities, and the retained modalities are taken as available modalities. Step S4.2: Construct a reconstruction module. Based on the attention mechanism, the real-specific features and shared features of all available modalities are fused and passed through a linear layer to become the key K and value V in the attention mechanism. Step S4.3: Generate descriptive text for the inherent features of each modality, input it into the pre-trained text encoder, and obtain the linguistic features of the missing modalities. , serving as query Q in the attention mechanism; Step S4.4: Perform multi-head attention operation on the obtained query Q, key K, and value V to obtain the interaction context vector. Missing modes are generated via multilayer perceptron. i Generate specific features ; Step S4.5: Remove missing modes i Generate specific features The corresponding mode obtained in step S2.2 i True specific features Calculate the MSE loss as the reconstruction loss for generating specific features. ; The total loss for the first-stage pre-training is as follows: ； in, This represents the total loss during the first phase of training. Represents the loss of true specific features. This represents the reconstruction loss for generating specific features. This represents the shared feature alignment loss. This is an adjustable parameter.

6. The multimodal remote sensing image classification method under arbitrary modality loss conditions according to claim 5, characterized in that, The specific steps of step S5 are as follows: Step S5.1: After the first stage of pre-training is completed, save and freeze the parameters of each modality image encoder and missing modality generation module; Step S5.2: For the M modes, first randomly discard them. Data from each modality is recorded as multimodal data. Subsequently, for multimodal data The data in the middle is then randomly discarded. Each modality is recorded as the few-modal data. .

7. The multimodal remote sensing image classification method under arbitrary modality loss conditions according to claim 6, characterized in that, The specific steps of step S6 are as follows: Step S6.1: Perform two-stage fine-tuning. The specific features of the missing modality are reconstructed and replaced by the missing modality generation module trained in the first-stage pre-training, while the shared features of the missing modality are directly discarded. Step S6.2: After the missing modality generation module reconstructs the specific features, the specific features of all modalities will be divided into the original true specific features. and the generation of specific features of reconstruction ; Next, a logic-guided gated fusion module is constructed. This module combines a CNN gated network and a cue learning method. Whether it's a real-world or generated-specific feature, a learnable vector is appended as a cue. The cue length is an adjustable parameter. All specific features with the cue are then input into the CNN gated network to generate weight coefficients, and weighted fusion is performed to obtain the overall specific features. ; Step S6.3: Construct the gating weight loss calculation formula as follows: ； in, This indicates the gating weight loss. i Representing few modal data Index of missing modalities j Representing few modal data The modal index is available in the middle; Indicates the total number of modes. n For fewer modal data The total number of missing modes; Representing few modal data Mid-mode i The weights for generating specific features, Representing multimodal data intermediate mode i The weights of the true specific features; Representing multimodal data Mid-mode j The weights of the true specific features, Representing few modal data intermediate mode j The weights of the true specific features.

8. The multimodal remote sensing image classification method under arbitrary modality loss conditions according to claim 7, characterized in that, The specific steps of step S7 are as follows: Step S7.1: The overall shared features are obtained by averaging the shared features of all available modalities. Shared characteristics and overall specific characteristics Add them together to obtain the total classification features. The data is then fed into a classifier to obtain the final logits value, which is then compared with the ground truth labels for land cover classification to calculate the classification cross-entropy loss. ; Step S7.2: Constructing few-modal data With multimodal data Distance loss: ； in, For fewer modal data With multimodal data Distance loss; Indicating in multimodal data The following classification cross-entropy loss, Indicates the presence of few modal data The following classification cross-entropy loss; Overall loss during the two-stage fine-tuning phase The calculation is as follows: ； ； in, For classification cross-entropy loss, This indicates the gating weight loss. This represents the distance loss between few-modal data and multimodal data. and This is an adjustable parameter.

Images