Multi-modal remote sensing image fusion classification method under modal missing condition

Abstract

The invention discloses a multi-modal remote sensing image fusion classification method under a modal missing condition, and the method comprises the steps: constructing a multi-modal remote sensing image model image fusion classification model, taking the data of three remote sensing image modals as input, carrying out the feature extraction of the data of each modal through a respective specific feature encoder and a shared feature encoder, and carrying out the feature extraction of the data of each modal; the method comprises the following steps: firstly, generating a specific feature, then fusing the specific feature and the shared feature, generating a single-mode fusion feature by adopting a jump link, and finally, fusing the fusion features of the three modes again, and generating a final fusion feature for model prediction. Two auxiliary tasks are introduced into the model, one is a domain classification task based on specific features, and the other is a comparative learning task for shared features. The model is not only suitable for full-modal training, but also effectively extracts and fuses the multi-modal features under the condition that partial modals are missing, and the classification performance of the model is improved.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image classification, specifically relating to a multimodal remote sensing image fusion classification method under modality missing conditions. Background Technology

[0002] Land cover classification using remote sensing image data is an important research field with broad application prospects, playing a significant role in urban planning, mineral exploration, and environmental monitoring. With the development of remote sensing technology, descriptions of potential land cover targets can be simultaneously collected using a series of sensors, such as hyperspectral (HS), lidar (LiDAR), synthetic aperture radar (SAR), and multispectral (MS). By analyzing remote sensing image data, different types of land cover can be identified and classified, such as buildings, roads, water bodies, and vegetation. Initially, researchers attempted to use the specific advantages of single-source sensor data for land cover classification; however, single-source sensor data provides limited feature dimensions and information, making it difficult to fully capture the complex attributes of land cover. This limitation has gradually prompted researchers to turn to the fusion of multi-source remote sensing data. Multi-source data fusion can integrate the advantages of various sensors, thereby more comprehensively characterizing land cover in terms of spatial, spectral, and textural features, effectively improving classification accuracy.

[0003] Although multimodal methods exhibit better performance than single-modal methods, they still face certain challenges in practical applications. One common problem is modality loss, especially in imaging specific regions. Low sampling frequencies of some sensors or factors such as weather or sensor malfunctions can make it difficult to acquire multimodal data simultaneously. This issue significantly increases the uncertainty of multimodal methods in real-world scenarios, necessitating effective strategies to address modality loss and ensure the robustness of classification models under missing modalities. Summary of the Invention

[0004] This invention addresses the problems existing in the prior art by providing a multimodal remote sensing image fusion and classification method under modality missing conditions. It can effectively extract and fuse multimodal features not only when the modalities are complete, but also when some modalities are missing, thereby improving the classification performance of the model.

[0005] To address the above technical problems, this invention provides the following technical solution: a multimodal remote sensing image fusion and classification method under modality missing conditions, which uses a specific encoder to extract specific features of each modality and a shared encoder to extract shared features between modalities, while simultaneously using two auxiliary tasks to learn the specific features and shared features respectively, including the following steps: S1. Obtain multimodal data covering the same area to construct a multimodal dataset and perform preprocessing. The multimodal dataset includes: hyperspectral image data, DSM image data and SAR image data, or a number of real pixels extracted from one or two of these image data, and the corresponding classification labels. S2. Using a number of real pixels extracted from three types of image data, namely hyperspectral image data, DSM image data, and SAR image data, or one or two of them, as inputs and corresponding specific features as outputs, construct a specific feature encoder. A shared feature encoder is constructed by concatenating three types of image data, namely hyperspectral image data, DSM image data, and SAR image data, or one or two of them, as input and obtaining the shared features corresponding to each mode as output. S3. Using specific features and shared features as inputs and the final features as outputs, construct a residual fusion module; S4. Construct a prediction module using the final features as input and the corresponding classification labels as output; S5. Based on the specific feature encoder, shared feature encoder, residual fusion module, and prediction module from steps S1 to S4, a multimodal remote sensing image fusion classification network is constructed. The multimodal remote sensing image fusion classification network is trained using the multimodal dataset from step S1 to obtain the multimodal remote sensing image fusion classification model. During the training of the multimodal remote sensing image fusion classification network, overall loss optimization is performed, including using a contrastive loss function to minimize the distance between similar features at the same location in different modalities and to maximize the difference between features at different locations; using a domain classification loss function to assist in the learning of specific features; and using a main classification task loss function to guide the prediction results.

[0006] S6. Input three or one or two of the collected hyperspectral image data, DSM image data, and SAR image data into the multimodal remote sensing image fusion classification model to obtain classification labels, i.e., the terrain classification results.

[0007] Furthermore, step S1 as described above includes the following sub-steps: S1.1 Acquire multimodal data covering the same area, including hyperspectral image data, DSM image data, and SAR image data, denoted as follows: , and Where W and H represent width and height, and C represents the number of spectral bands in the hyperspectral image; S1.2 Data Preprocessing: Principal component analysis (PCA) is performed on the hyperspectral image to reduce its dimensionality. The first d principal components are extracted, and the resulting dimensionality-reduced hyperspectral data is represented as follows: .

[0008] Furthermore, in step S2 above, the specific feature encoder extracts specific features from a number of real pixels extracted from the three types of image data—hyperspectral image data, DSM image data, and SAR image data—or one or two of them, as shown in the following formula:

[0009] in, X i Pixels representing hyperspectral image data, DSM image data, or SAR image data , Represents a specific feature encoder.

[0010] Furthermore, in step S2 above, the shared feature encoder extracts the shared features corresponding to each mode from the three types of image data—hyperspectral image data, DSM image data, and SAR image data—or one or two of them, as follows:

[0011] in, X i Represents pixels in hyperspectral image data, DSM image data, or SAR image data. , This represents a shared feature encoder.

[0012] Furthermore, step S3 mentioned above includes the following sub-steps: S3.1, Residual fusion module for specific features and shared features The data is concatenated along the channel dimension and then projected, with the output added as a residual to the shared features to form a more semantically rich modality embedding. The following expression:

[0013] in, This represents a projection operation. If a mode is missing, the embedding of the missing mode is obtained by averaging the shared features of the remaining existing modes. S3.2, embed the obtained modes into the modal data. The features are then combined to form the final feature representation f.

[0014] Furthermore, in step S4 above, the prediction module inputs the final feature f into the fully connected layer and uses the weights of the fully connected layer to generate the prediction result y, as shown in the following expression:

[0015] in, This represents a fully connected layer operation, where W is the connection weight corresponding to the fully connected layer operation.

[0016] Furthermore, in the aforementioned step S4, The loss function for the main classification task is as follows:

[0017] in, It is the cross-entropy loss function. y is the true label, and y is the predicted result.

[0018] Furthermore, in step S4 above, the contrast loss includes: the loss of the same position pair in different modes. Loss of different position pairs in the same mode , Loss between different position pairs across modes As shown in the following formula:

[0019]

[0020]

[0021]

[0022]

[0023] in, Cosine similarity between modes A and B at different positions. It is a threshold. The function ensures that only when the similarity is less than Only when this happens will there be a positive loss, and only then will optimization be possible; simultaneously, a masking method is used to reduce the interference of missing modal data. When modality A is missing, the mask... It is 0 when the mode exists, and 1 when the mode does not exist. The contrastive loss function is: .

[0024] Furthermore, in step S4 above, the domain classification loss function is:

[0025] in, Given a remote sensing image dataset with N modalities, , For the i-th modal data of the j-th sample, Classify and label them. yes The modal label is set to 1 at the i-th position and 0 at the other positions. ; This represents a specific feature of the i-th modality of the j-th sample; It is a mapping of specific features to A function of the probabilistic simplex, where S represents a specific feature space. It represents the probability simplex with N classes.

[0026] Furthermore, in step S4 above, the total loss function is expressed as:

[0027] in, , and The loss weights are assigned to the main classification task, the contrastive learning task, and the domain classification task, respectively. The model parameters are optimized through backpropagation to minimize the total loss function.

[0028] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows: This invention provides a multimodal data processing method that combines specific modality feature extraction capabilities with cross-modality shared feature extraction capabilities, enabling relatively accurate classification tasks in cases of complete modalities and random modality loss. Through contrastive learning and domain classification task-assisted optimization, the robustness and prediction accuracy of the model in multimodal data fusion analysis are effectively improved. Attached Figure Description

[0029] Figure 1 This is the complete modal structure diagram of the model.

[0030] Figure 2 This is the model's missing modal structure diagram.

[0031] Figure 3 These are the real label map and the color map; in the figure, (a) is the real label map and (b) is the color map.

[0032] Figure 4 The figures show a comparison of model prediction results under complete modal training and complete modal testing conditions. In the figures, (a) is the prediction graph of the MMANet network, (b) is the prediction graph of the mmFormer network, and (c) is the prediction graph of the network of this invention.

[0033] Figure 5 This is a comparison of model prediction results under random missing modal training and complete modal testing. In the figure, (a) is the prediction graph of the MMANet network, (b) is the prediction graph of the mmFormer network, and (c) is the prediction graph of the network of this invention.

[0034] Figure 6This is a diagram illustrating the prediction accuracy of different models under training in each complete modality.

[0035] Figure 7 This is a diagram illustrating the prediction accuracy of different models under training with random missing modalities. Detailed Implementation

[0036] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0037] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0038] This invention uses data from three remote sensing image modalities as input. Each modal data is first processed by its own specific modal encoder and a shared encoder for feature extraction. Then, the obtained specific features and shared features are fused, and a skip link is used to generate a single modal fusion feature. Finally, the fusion features of the three modalities are fused again to generate the final fusion feature for model prediction.

[0039] Furthermore, this invention introduces two auxiliary tasks into the model to facilitate the learning of specific and shared features. One is a domain classification task for learning specific features, and the other is a contrastive learning task for shared features. This model is not only suitable for full-modal training but also effectively extracts and fuses multimodal features even when some modalities are missing. This model is designed for the modality missing problem in trimodal remote sensing data, aiming to effectively extract and fuse multimodal features and improve the model's classification performance even when some modalities are missing.

[0040] refer to Figure 1 This invention provides a method for multimodal remote sensing image fusion and classification under modality missing conditions, comprising the following steps: S1. Obtain multimodal data covering the same area to construct a multimodal dataset and perform preprocessing. The multimodal dataset includes: hyperspectral image data, DSM image data and SAR image data, or a number of real pixels extracted from one or two of these image data, and the corresponding classification labels. S2. Using a number of real pixels extracted from three types of image data, namely hyperspectral image data, DSM image data, and SAR image data, or one or two of them, as inputs and the corresponding specific features as outputs, construct a specific feature encoder. A shared feature encoder is constructed by concatenating three types of image data, namely hyperspectral image data, DSM image data, and SAR image data, or one or two of them, as input and obtaining the shared features corresponding to each mode as output. S3. Using specific features and shared features as inputs and the final features as outputs, construct a residual fusion module; S4. Construct a prediction module using the final features as input and the corresponding classification labels as output; S5. Based on the specific feature encoder, shared feature encoder, residual fusion module, and prediction module from steps S1 to S4, construct a multimodal remote sensing image fusion classification network. Train the multimodal remote sensing image fusion classification network using the multimodal dataset from step S1 to obtain the multimodal remote sensing image fusion classification model. During the training of the multimodal remote sensing image fusion classification network, perform overall loss optimization, including using a contrastive loss function to minimize the distance between similar features of the same location pair in different modalities, maximizing the difference between features of different location pairs in different modalities and different location pairs in the same modality; using a domain classification loss function to assist in the learning of specific features; and using a main classification task loss function to guide the prediction results. S6. Input three or one or two of the collected hyperspectral image data, DSM image data, and SAR image data into the multimodal remote sensing image fusion classification model to obtain classification labels, i.e., the terrain classification results.

[0041] To validate the effectiveness of our research method, we conducted experiments using the Augsburg dataset. This dataset consists of three different data sources: hyperspectral images (HSI), synthetic aperture radar (SAR) images, and digital surface model (DSM) images. Specifically, the hyperspectral images contain 180 bands, the SAR images consist of four feature channels, and the DSM images are grayscale images. Each image has a spatial resolution of 332 × 485 pixels. Figure 3 The visualization dataset is divided into 7 categories and contains 78,294 ground truth pixels, of which 761 pixels were used for training and 77,533 pixels were used for testing. In the figure, (a) shows the ground truth labels and (b) shows the color mapping.

[0042] In a preferred embodiment of the present invention, step S1 involves acquiring multimodal data covering the same area, including hyperspectral data, DSM data, and SAR data, denoted as follows: , and , where and represent the width and height of the three, and represents the number of spectral bands in the hyperspectral image.

[0043] Principal component analysis (PCA) was performed on the hyperspectral image to reduce its dimensionality, and the first d principal components were extracted. The resulting dimensionality-reduced hyperspectral data is represented as follows: This reduces the high-dimensional redundancy of hyperspectral images.

[0044] In a preferred embodiment of the present invention, the preprocessed hyperspectral data, DSM data, and SAR data are respectively input into their respective specific encoders to extract specific features, which are represented as follows: , and Simultaneously, the shared features extracted by the shared encoder are represented as follows: , ,and Specifically, the construction of both the specific encoder and the shared encoder is identical, both beginning with the extraction of local features from three 3×3 convolutional blocks. Each convolution contains a convolutional layer, a batch normalization layer, and a ReLU activation function; these modules help extract low-level features specific to the data. Building upon this, various convolutional operations of different sizes (including 1×1 convolutions, 3×3 convolutions, and global average pooling) are used to further extract multi-scale features, capturing multi-level spatial information. Finally, dimensionality reduction of the features is achieved through 1×1 convolutions and pooling operations, and the final representational power is enhanced by concatenating features from different convolutional branches. The shared encoder ensures that cross-modal data can share the same set of convolutional weights, thereby extracting consistent shared features.

[0045] In step S2, the specific feature encoder extracts specific features from several real pixels extracted from three types of image data—hyperspectral image data, DSM image data, and SAR image data—or two of them, as shown in the following formula:

[0046] in, X i Pixels representing hyperspectral image data, DSM image data, or SAR image data , Represents a specific feature encoder.

[0047] The shared feature encoder extracts shared features corresponding to each mode from three types of image data—hyperspectral image data, DSM image data, and SAR image data—or two of them, as shown in the following formula:

[0048] in, X iRepresents pixels in hyperspectral image data, DSM image data, or SAR image data. , This represents a shared feature encoder.

[0049] In a preferred embodiment of the present invention, step S3 involves constructing a residual fusion module to fuse specific features with shared features. This involves fusing specific features of the same modality. and shared features They are all input into the residual fusion module.

[0050] Step S3 includes the following sub-steps: S3.1, Residual fusion module for specific features and shared features The data is concatenated along the channel dimension and then projected, with the output added as a residual to the shared features to form a more semantically rich modality embedding. The following expression:

[0051] in, This represents a projection operation. If a mode is missing, the embedding of the missing mode is obtained by averaging the shared features of the remaining existing modes. S3.2, embed the obtained modes into the modal data. The features are then combined to form the final feature representation f.

[0052] In a preferred embodiment of the present invention, in step S4, the prediction module inputs the final feature f into the fully connected layer and uses the weights of the fully connected layer to generate the prediction result y, as shown in the following expression:

[0053] in, This indicates a fully connected layer operation. These are the connection weights corresponding to the operations in the fully connected layer.

[0054] When training a multimodal remote sensing image fusion classification network, overall loss optimization is performed, including using a contrastive loss function to minimize the distance between similar features at the same location in different modalities and to maximize the difference between features at different locations; using a domain classification loss function to assist in the learning of specific features; and using a main classification task loss function to guide the prediction results, as detailed below: The loss function for the main classification task is:

[0055] in, It is the cross-entropy loss function. y is the true label, and y is the predicted result.

[0056] To assist in the learning of shared features across multiple modalities, this invention designs a contrastive loss function based on cosine similarity measurement, aiming to minimize the distance between similar features at the same location in different modalities and maximize the difference between features at different locations.

[0057] Specifically, the losses for positive and negative samples are obtained by calculating the cosine similarity between the same-modal and cross-modal feature pairs of modality A and modality B.

[0058] The loss function consists of two parts: first, the positive sample loss. The similarity of shared features across modalities is enhanced by minimizing the feature similarity of pairs of the same position across different modalities. Secondly, the negative sample loss maximizes the distance between different position pairs (including the loss of features from different position pairs within the same modality). , Loss of features at different locations across modalities This can reduce interference from irrelevant features.

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] in, Cosine similarity between modes A and B at different positions. It is a threshold. The function ensures that only when the similarity is less than Only when this happens will there be a positive loss, and only then will optimization be possible; simultaneously, a masking method is used to reduce the interference of missing modal data. When modality A is missing, the mask... It is 0 when the mode exists, and 1 when the mode does not exist. The contrastive loss function is: .

[0065] Similarly, domain classification is used to assist in the learning of specific features. In order to further explore the specific features of each modality, this invention uses domain classification technology to assist in the learning of specific features.

[0066] Specifically, domain classification tasks are used for all existing modal data. This can be understood as learning specific features to predict the classification accuracy of the modality to which a data point belongs, and using these features as the training loss. Its loss function is expressed as:

[0067] in, Given a remote sensing image dataset with N modalities, , For the i-th modal data of the j-th sample, Classify and label them. yes The modal label is set to 1 at the i-th position and 0 at the other positions. ; This represents a specific feature of the i-th modality of the j-th sample; It is a mapping of specific features to A function of the probabilistic simplex, where S represents a specific feature space. It represents the probability simplex with N classes.

[0068] Finally, the model undergoes total loss optimization, and the total model loss is... It consists of three parts: the loss from the main classification task, the loss from the contrastive learning task, and the loss from the domain classification task. Therefore, the total loss... The formula is expressed as:

[0069] in, , and The loss weights are assigned to the main classification task, the contrastive learning task, and the domain classification task, respectively. The model parameters are optimized through backpropagation to minimize the total loss function.

[0070] like Figure 4 As shown, (a) illustrates the training and testing of the MMANet network, (b) illustrates the training and testing of the mmFormer network, and (c) illustrates the training and testing of the network of this invention. The model prediction accuracy is shown in the case of full modality training, using only single-modality, bimodal, and trimodal testing, respectively. Figure 6 As shown, the prediction accuracy of single-modal training is significantly lower than that of bimodal training, while the prediction accuracy of trimodal training is higher than that of bimodal training. This indicates that each modality has unique data characteristics, and multimodal fusion can more comprehensively utilize these characteristics, thereby improving the model's prediction performance.

[0071] like Figure 5 As shown, (a) presents the training and testing results of the MMANet network, (b) presents the training and testing results of the mmFormer network, and (c) presents the training and testing results of the network of this invention. Figure 7 This demonstrates that the model makes accurate predictions using data from each modality even when random modalities are missing during training. Figure 7The comparison results show that although the model performance decreases somewhat in the case of missing random modes, the model of the present invention can still maintain a high classification accuracy. This indicates that the model of the present invention has good robustness in dealing with the problem of missing modes.

[0072] Furthermore, comparative experiments with existing methods show that the model of this invention exhibits significant advantages in multimodal remote sensing image classification tasks. The MMANet and mmFormer models only utilize specific features of each modality, failing to fully explore the correlations between shared features across modalities, thus resulting in inferior classification performance compared to our model. In contrast, the model of this invention improves classification performance by deeply mining both shared and specific features across modalities.

[0073] In summary, the model of this invention demonstrates extremely high classification accuracy and robustness, whether in the case of complete modalities or missing modalities.

[0074] In summary, the model of this invention demonstrates extremely high classification accuracy and robustness, whether in the case of complete modalities or missing modalities.

[0075] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A multimodal remote sensing image fusion classification method under modality missing conditions, characterized in that: The specific features of each modality are extracted using a specific encoder and the shared features between modalities are extracted using a shared encoder. At the same time, two auxiliary tasks are used to learn the specific features and shared features respectively, including the following steps: S1. Acquire multimodal data covering the same area to construct a multimodal dataset and perform preprocessing. The multimodal dataset includes: three types of image data, namely, hyperspectral image data, DSM image data, and SAR image data, or a number of real pixels extracted from one or two of the image data, and corresponding classification labels; S2, taking the three image data, the hyperspectral image data, the DSM image data and the SAR image data, or a number of real pixels extracted from one or two of the image data as input, and corresponding specific features as output, to construct a specific feature encoder; Taking the three image data of hyperspectral image data, DSM image data and SAR image data or connecting one or two of them as input, obtaining the shared features corresponding to each modality as output, and constructing a shared feature encoder; S3, using specific features and shared features as input and final features as output to construct a residual fusion module; S4, construct a prediction module with the final features as input and the corresponding classification labels as output; S5, constructing a multimodal remote sensing image fusion classification network based on the specific feature encoder, shared feature encoder, residual fusion module, and prediction module of step S1 to step S4, training the multimodal remote sensing image fusion classification network with the multimodal data set of step S1, and obtaining a multimodal remote sensing image fusion classification model; when training the multimodal remote sensing image fusion classification network, performing total loss optimization, including minimizing the distance between similar features at the same position of different modalities by using the contrast loss function, and maximizing the difference between features at different positions; using the domain classification loss function to assist the learning of specific features; and using the main classification task loss function to guide the prediction results; S6. Input three or one or two of the collected hyperspectral image data, DSM image data, and SAR image data into a multimodal remote sensing image fusion classification model to obtain a classification label, that is, a classification result of the terrain.

2. According to the multi-modal remote sensing image fusion classification method under modality loss conditions of claim 1, it is characterized in that: Step S1 includes the following sub-steps: S1.

1. Obtain multimodal data covering the same area, including hyperspectral image data, DSM image data and SAR image data, respectively denoted as , and , where W and H represent the width and height, and C represents the number of spectral bands of the hyperspectral image; S1.

2. Data preprocessing: Perform principal component analysis and dimensionality reduction on the hyperspectral image, extract the first d principal components, and the generated hyperspectral data after dimensionality reduction is expressed as .

3. The multimodal remote sensing image fusion classification method under modality missing conditions according to claim 1 is characterized in that: In step S2, the specific feature encoder extracts specific features from a number of real pixels extracted from three types of image data, or one or two of the image data, namely, the hyperspectral image data, the DSM image data, and the SAR image data, as shown in the following formula: ， in, X i represents pixels of hyperspectral image data, DSM image data or SAR image data, , Represents a specific feature encoder.

4. The multimodal remote sensing image fusion classification method under modality loss conditions according to claim 1 is characterized in that: In step S2, the shared feature encoder extracts the shared features corresponding to the respective modalities of the three image data, namely, the hyperspectral image data, the DSM image data and the SAR image data, or one or two of the image data, as shown in the following formula: ， in, X i Represents hyperspectral image data, DSM image data or SAR image data pixels, , Represents shared feature encoder.

5. The multimodal remote sensing image fusion classification method under modality missing conditions according to claim 1 is characterized in that: Step S3 includes the following sub-steps: S3.

1. Residual fusion module for specific features and shared features After concatenation in the channel dimension and projection operation, its output is added to the shared features as a residual to form a more semantically rich modality embedding The following expression: ， in, represents the projection operation. If one modality is missing, the embedding of the missing modality is obtained by averaging the shared features of the remaining existing modalities. S3.

2. Embed the obtained modes of each mode Concatenate to form the final feature representation f.

6. The multimodal remote sensing image fusion classification method under modality loss conditions according to claim 1 is characterized in that: In step S4, the prediction module inputs the final feature f into the fully connected layer and generates the prediction result y using the weight of the fully connected layer, as shown in the following expression: ， in, represents the fully connected layer operation, and W is the connection weight corresponding to the fully connected layer operation.

7. The multi-modal remote sensing image fusion classification method under modality loss conditions according to claim 1 is characterized in that: In step S4, The main classification task loss function is as follows: , in, is the cross entropy loss function, is the true label and y is the predicted result.

8. The multi-modal remote sensing image fusion classification method under modality missing conditions according to claim 1 is characterized in that: In step S4, the contrast loss includes: the loss of the same position pair of different modalities , the loss of different position pairs of the same modality , and the loss of different position pairs across modalities As follows: ， ， ， ， ， in, represents the cosine similarity between modality A and modality B at different position pairs, is the threshold value, The function ensures that only when the similarity is lower than When , there will be positive loss and it will be optimized; at the same time, the mask method is used to reduce the interference of missing modal data. When modality A is missing, the mask is 0 when the mode exists, and 1 when the mode does not exist; The contrast loss function is: .

9. The multi-modal remote sensing image fusion classification method under modality missing conditions according to claim 8 is characterized in that: In step S4, the domain classification loss function is: ， in, is a remote sensing image dataset of N modalities, , is the i-th modal data of the j-th sample, For its classification label, yes The modal tag is 1 in the i-th position and 0 in the other positions. ; Represents the specific features of the i-th mode of the j-th sample; is a function that maps a specific feature to dimensional probability simplex function, where S represents a specific feature space, represents a probability simplex with N classes.

10. The multi-modal remote sensing image fusion classification method under modality missing conditions according to claim 8, characterized in that: In step S4, the total loss function is expressed as: ， in, , and They are the loss weights for the main classification task, contrastive learning task, and domain classification task, respectively. The model parameters are optimized through back propagation to minimize the total loss function.

Images