Multi-mode-based unsupervised domain adaptive hyperspectral image classification method

Abstract

The invention provides a multi-mode-based unsupervised domain adaptive hyperspectral image classification method and system. The method comprises the following steps: extracting global features through an image encoder, classifying image samples by using a double classifier, aligning the same-kind distribution of a source domain and a target domain, and generating a pseudo label for the target domain; text features are extracted through a text encoder, the text features and image features are subjected to comparative learning, the distance between the text features and the image features of the same category is shortened, and the separability of data is improved. The system comprises a domain adaptation module and a vision-language alignment module, the domain adaptation module obtains image features through an image encoder, obtains prediction results through a double classifier, and selects one of the prediction results as a target domain pseudo tag; and the vision-language alignment module obtains corresponding text description by using the source domain label and the target domain pseudo label, and after a text encoder obtains text features, the text features are aligned with the image features obtained by the domain adaptation part.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image classification, and in particular to a multimodal unsupervised domain adaptation hyperspectral image classification method. Background Art

[0002] As a typical spatial-spectral multidimensional data, hyperspectral images can capture the diagnostic spectral information of the object, represent the intrinsic characteristics of the object, and are important parameters for distinguishing and identifying the object. Hyperspectral images have been widely used in many fields, such as geological rock and mineral exploration, land cover classification, agriculture, military, etc., because of their advantages such as high spectral resolution, strong material recognition ability, relatively high spatial resolution, detection of imperceptible information, data availability and historical data analysis. However, in many practical scenarios, labeling samples is time-consuming and costly. Therefore, most of the collected data has only a few labels or no labels at all. This also poses a great challenge to the accurate classification of hyperspectral images. In addition, under the influence of different acquisition conditions such as different time, different location, and different sensors, there will be large spectral drift between the obtained hyperspectral images. Therefore, even if the model is fully trained on a hyperspectral image with sufficient labels, if such a model is directly applied to another hyperspectral image with spectral shift phenomenon, it is often impossible to obtain satisfactory results. Therefore, it is very important to alleviate the problems caused by spectral drift and improve the performance of the model.

[0003] In recent years, domain adaptation has received increasing attention in the field of computer vision, and researchers have also found that domain adaptation can improve the performance of hyperspectral image classification to a certain extent. The purpose of domain adaptation is to learn the target domain with fewer or no labels by leveraging the knowledge in the source domain with rich labels, so as to extend the generalization ability of the model from the source domain to the target domain and alleviate the problem of spectral drift. Traditional domain adaptation methods can be divided into three categories: instance-based methods, feature-based methods, and classifier-based methods. Instance-based domain adaptation methods mainly adjust the marginal distribution of source samples or target samples to reduce the difference between domains and align the distributions of the two domains. Feature-based domain adaptation methods transform the source data and target data into the feature space so that the data distributions of the two domains in the feature space are similar. Classifier-based domain adaptation methods adapt the classifier trained on the source domain to the target domain by considering the unlabeled samples of the target domain. Although traditional methods have achieved certain results in image classification, they can usually only capture the surface features of the image, and they often rely heavily on well-trained neural networks and do not fully utilize the deep structure of the data and the complex local correlations. Compared with traditional methods, the features obtained by deep learning-based domain adaptation methods not only have stronger generalization ability but also better transferability. Deep learning-based domain adaptation methods automatically extract more expressive features directly from the original data. At present, most deep learning-based domain adaptation methods add an adaptation layer to the original deep network architecture to achieve source-to-target adaptation, or adopt adversarial learning strategies to minimize cross-domain differences. However, these methods still ignore the text information related to land cover categories, which is not enough to fully utilize the information of each modality of hyperspectral images.

[0004] With the success of large language models, the combination of images and language has become a research field that has attracted much attention. The success of large language models also inspires us that combining remote sensing images with language to complete image description in the process of domain adaptation can set domain-invariant features for images, thereby improving the robustness and generalization ability of the model. In hyperspectral image classification, many domain adaptation methods focus on extracting domain-invariant features at the visual level, but few attempts have been made to use text information to collaboratively assist in the extraction of visual-level features. This text-guided domain adaptation method can create a semantic space by aligning image features and text features, and adapt in the target domain. Therefore, in the hyperspectral image classification task, how to introduce text modality into the domain adaptation method deserves further study. Summary of the invention

[0005] According to the above, it is difficult to obtain labeled samples of hyperspectral images, there is spectral drift between the hyperspectral images of the source scene and the target scene, and the existing technology is still mostly studied from the visual level, and the technical problem of failing to fully utilize the information of hyperspectral images is provided. A multimodal unsupervised domain adaptation hyperspectral image classification method is provided. The present invention mainly utilizes (briefly describe the technical solution, usually write the distinguishing technical features), so as to achieve (the effect achieved).

[0006] The technical means adopted by the present invention are as follows:

[0007] A multimodal unsupervised domain adaptation hyperspectral image classification method, comprising:

[0008] S1. Extract global features through the image encoder, use the dual classifier to classify image samples, align the distribution of the same type of source domain and target domain, and generate pseudo labels for the target domain;

[0009] S2. Extract text features through the text encoder, compare and learn the text features and image features, narrow the distance between text and image features of the same category, and improve the separability of the data.

[0010] Furthermore, step S1 specifically includes:

[0011] S11, sending the image into the image encoder, and extracting the attention-based spatial features and the attention-based spectral features of the hyperspectral image through two branches respectively;

[0012] S12, fusing the attention-based spatial features and the attention-based spectral features of the extracted hyperspectral image to obtain fused image features;

[0013] S13. Send the fused image features to the dual classifier to obtain the prediction probability.

[0014] Furthermore, after step S13, step S14 is further included to calculate the source classification loss as follows:

[0015] The source classification loss is the cross entropy loss, and the calculation formula of the source classification loss is as follows:

[0016]

[0017] in, is the label, IE(·) is the output of the image encoder, is the image feature obtained by the image encoder, F(·) is the output of the two classifiers, is the predicted probability.

[0018] Further, after step S14, a step of minimizing class confusion in target prediction using MCC loss is included, as follows:

[0019] The correlation between any two categories is adjusted according to the contribution of different instances. When there are many categories, in order to avoid the problem of extremely unbalanced category distribution, the category normalization technology is used. The final MCC loss calculation formula is as follows:

[0020]

[0021] Among them, p ij represents the probability that the i-th sample belongs to the j-th category, W represents the weights of different categories, Indicates the number of categories.

[0022] Furthermore, step S1 also includes the step of updating the classifier using the adversarial loss function, as follows:

[0023] The outputs of the target domain samples obtained by the two classifiers use the absolute value indicator to measure the difference, and the adversarial loss function is as follows:

[0024]

[0025] Further, step S2 specifically includes:

[0026] S21. Calculate the alignment loss between image features and coarse-grained text features. The calculation formula is as follows:

[0027]

[0028] Among them, P x (i) and A x (i) are the positive sample set and negative sample set of image features, P t (i) and A t (i) are the positive sample set and negative sample set of text features, where |P(i)| = |P x (i)|=|P t (i)|;x i represents the image features, t i Represents text features; τ is a temperature parameter used to control the smoothness of the softmax distribution;

[0029] S22. Calculate the alignment loss between image features and fine-grained text features. The calculation formula is as follows:

[0030]

[0031] in, Represents the loss of the image feature corresponding to the jth text feature and the fine-grained text feature;

[0032] S23. Calculate the supervised contrastive learning loss of the source domain. The calculation formula is as follows:

[0033]

[0034] S24. Calculate the supervised contrastive learning loss of the target domain. The calculation formula is as follows:

[0035]

[0036] S25. Calculate the alignment loss of the total image features and text features. The calculation formula is as follows:

[0037]

[0038] in, represents the supervised contrastive learning loss of the source domain; represents the supervised contrastive learning loss for the target domain.

[0039] The present invention also provides a multimodal unsupervised domain adaptation hyperspectral image classification system implemented based on the multimodal unsupervised domain adaptation hyperspectral image classification method, comprising: a domain adaptation module and a visual-language alignment module, wherein:

[0040] The domain adaptation module obtains image features by the image encoder, obtains prediction results by the dual classifier, and selects one of the prediction results as the target domain pseudo label;

[0041] The visual-language alignment module obtains the corresponding text description using the source domain label and the target domain pseudo label. After the text features are obtained by the text encoder, the text features are aligned with the image features obtained by the domain adaptation part.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] 1. The present invention provides an unsupervised domain adaptation hyperspectral image classification method based on multimodality, which enhances the separability of data by aligning text modality and image modality. First, an image encoder is used to extract the spatial features and spectral features of the image respectively, and the two are fused. The fused image features are passed through a dual classifier to obtain a prediction probability. Then, the corresponding text description is obtained according to the label, and the source domain text description is obtained according to the source domain label. In the target domain, due to the lack of label information, the prediction result obtained by one of the classifiers is used as the pseudo label of the target domain sample, so as to obtain the text description of the target domain according to the pseudo label. The text description is passed through a text encoder to obtain text features, and the text features and image features are paired through a visual-language alignment module. The text features are used as domain invariant features to optimize the aggregation of text and image features of the same category, while increasing the separation between text and image features of different categories, and enhancing the model's recognition ability for features of different categories.

[0044] 2. The present invention provides a multi-modal unsupervised domain adaptation hyperspectral image classification method. In order to improve the distinguishability of target domain data, the text modality auxiliary network is used to align the feature distribution of the source domain and the target domain. Contrastive learning is used to deeply compare and measure the similarity between the hyperspectral image and the natural language description. By measuring and optimizing these similarities, the effective alignment of image and text features in the feature space is achieved.

[0045] Based on the above reasons, the present invention can be widely promoted in the fields of hyperspectral image classification and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 This is a framework diagram of the multimodal unsupervised domain adaptation hyperspectral image classification method of the present invention.

[0048] Figure 2 A framework diagram of an image encoder provided in an embodiment of the present invention.

[0049] Figure 3 A framework diagram of a vision-language alignment module provided in an embodiment of the present invention.

[0050] Figure 4 This is a diagram of the classification results of the Houston dataset provided by an embodiment of the present invention.

[0051] Figure 5A diagram of the classification results of the Pavia dataset provided in an embodiment of the present invention.

[0052] Figure 6 A diagram of the classification results of the HyRANK data set provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] like Figure 1 As shown, the present invention provides a multimodal unsupervised domain adaptation hyperspectral image classification method, comprising:

[0056] S1. Extract global features through the image encoder, use the dual classifier to classify image samples, align the distribution of the same type of source domain and target domain, and generate pseudo labels for the target domain;

[0057] S2. Extract text features through the text encoder, compare and learn the text features and image features, narrow the distance between text and image features of the same category, and improve the separability of the data.

[0058] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:

[0059] S11, sending the image into the image encoder, and extracting the attention-based spatial features and the attention-based spectral features of the hyperspectral image through two branches respectively;

[0060] S12, fusing the attention-based spatial features and the attention-based spectral features of the extracted hyperspectral image to obtain fused image features;

[0061] S13. Send the fused image features to the dual classifier to obtain the prediction probability.

[0062] In specific implementation, as a preferred embodiment of the present invention, after step S13, step S14 is further included to calculate the source classification loss as follows:

[0063] The source classification loss is the cross entropy loss, and the calculation formula of the source classification loss is as follows:

[0064]

[0065] in, is the label, IE(·) is the output of the image encoder, is the image feature obtained by the image encoder, F(·) is the output of the two classifiers, is the predicted probability.

[0066] In specific implementation, as a preferred embodiment of the present invention, after step S14, a step of minimizing class confusion in target prediction using MCC loss is further included, as follows:

[0067] The correlation between any two categories is adjusted according to the contribution of different instances. When there are many categories, in order to avoid the problem of extremely unbalanced category distribution, the category normalization technology is used. The final MCC loss calculation formula is as follows:

[0068]

[0069] Among them, p ij represents the probability that the i-th sample belongs to the j-th category, W represents the weights of different categories, Indicates the number of categories.

[0070] In specific implementation, as a preferred embodiment of the present invention, step S1 also includes a step of using an adversarial loss function to update the classifier, as follows:

[0071] The outputs of the target domain samples obtained by the two classifiers use the absolute value indicator to measure the difference, and the adversarial loss function is as follows:

[0072]

[0073] In specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes:

[0074] S21. Calculate the alignment loss between image features and coarse-grained text features, maximize the similarity between positive sample pairs, and minimize the distance between positive sample pairs. The calculation formula is as follows:

[0075]

[0076] Among them, P x (i) and A x (i) are the positive sample set and negative sample set of image features, P t (i) and A t (i) are the positive sample set and negative sample set of text features, where |P(i)| = |P x (i)|=|P t (i)|;x i represents the image features, t i Represents text features; τ is a temperature parameter used to control the smoothness of the softmax distribution;

[0077] In this embodiment, the text description consists of a coarse-grained text description and two fine-grained text descriptions, wherein the coarse-grained text description is designed to construct a coarse-grained text description for each class in a cloze manner using the HSI of the class name as a template, and the fine-grained text is manually written for each class based on the color, shape, distribution and adjacency of each class. After the text features are obtained through the text encoder, contrastive learning loss is used as the optimization target. After both the image features and the text features are projected into the multimodal embedding space through normalization, only by shortening the distance between the image features and the text features can the alignment between the same categories be better achieved. In other words, maximizing the similarity between positive sample pairs can minimize the distance between positive sample pairs.

[0078] S22. Calculate the alignment loss between image features and fine-grained text features. The calculation formula is as follows:

[0079]

[0080] in, Represents the loss of the image feature corresponding to the jth text feature and the fine-grained text feature;

[0081] S23. Calculate the supervised contrastive learning loss of the source domain. The calculation formula is as follows:

[0082]

[0083] S24. Calculate the supervised contrastive learning loss of the target domain. The calculation formula is as follows:

[0084]

[0085] S25. Calculate the alignment loss of the total image features and text features. The calculation formula is as follows:

[0086]

[0087] in, represents the supervised contrastive learning loss of the source domain; represents the supervised contrastive learning loss for the target domain.

[0088] The embodiment of the present invention further provides a multimodal unsupervised domain adaptation hyperspectral image classification system implemented based on the multimodal unsupervised domain adaptation hyperspectral image classification method, comprising: a domain adaptation module and a visual-language alignment module, wherein:

[0089] The domain adaptation module obtains image features by the image encoder, obtains prediction results by the dual classifier, and selects one of the prediction results as the target domain pseudo label;

[0090] In this embodiment, the domain adaptation module includes an attention-based image encoder and a dual classifier, wherein:

[0091] like Figure 2 As shown in Figure 1, the attention-based image encoder includes a spectral feature extraction branch and a spatial feature extraction branch. Taking the Houston dataset as an example, the input data block size is 7×7×48, where 7×7 is the spatial neighborhood size and 48 is the number of bands. Among them:

[0092] The spectral feature extraction branch consists of three Conv3d-BN-ReLUs, a series of Conv3d and SE attention modules; the output of the first Conv3d-BN-ReLU and the output of Conv3d are used for residual connection, and then passed through the third Conv3d-BN-ReLU, and the obtained data is input into the spectral SE attention module, and the attention weights obtained by learning are multiplied by the spectral features to obtain attention-based spectral features; the importance of each spectral band is taken into account, so that the model can pay more attention to the spectral features.

[0093] The spatial feature extraction branch is used to extract the spatial information of the image. It consists of two Conv3d-BN rectified linear units (ReLU), a series of Conv3d and SE attention modules. After the data passes through two Conv3d-BN-ReLU and residual operations, it is input into the spatial SE attention module. In this module, the model is able to pay attention to and utilize spatial features. Finally, the spectral and spatial attention features obtained by the spectral feature extraction branch and the spatial feature extraction branch are fused to obtain the final spatial-spectral features as image features. This processing enables the model to fully understand all aspects of the sample information, improving the classification accuracy and the generalization ability of the model.

[0094] The dual classifier consists of a multi-layer FC-BN-ReLU-Dropout structure, where FC, BN and ReLU represent full connection, batch normalization and activation function layers respectively; the two classifiers generate probability vectors from the image features obtained by the image encoder, and apply the SoftMax function to obtain a logical vector, which represents the prediction probability of the corresponding classifier for the image category; such a structure not only increases the nonlinear expression ability of the model, but also prevents overfitting and enhances the generalization ability of the model.

[0095] Finally, the entire network is optimized by alternately updating the image encoder and the dual classifier.

[0096] The visual-language alignment module obtains the corresponding text description using the source domain label and the target domain pseudo label. After the text features are obtained by the text encoder, the text features are aligned with the image features obtained by the domain adaptation part.

[0097] In this embodiment, if Figure 3 As shown, the text encoder uses a Transformer architecture modified from Radford et al. A 63M-parameter 12-layer 512-wide model with 8 attention heads is used. The transformer operates on the lowercase byte pair encoding (BPE) representation of the text. To improve computational efficiency, the maximum sequence length is capped at 76. The text features are then normalized layer by layer and then linearly projected into the multimodal embedding space. Masked self-attention is used in the text encoder to retain the ability to use a pre-trained language model for initialization or to add language modeling as an auxiliary objective. Among them, the text description consists of a coarse-grained text description and two fine-grained text descriptions, where the coarse-grained text description is designed to construct a coarse-grained text description for each class in a cloze manner using the HSI of the class name as a template, and the fine-grained text is artificially written for each class based on its color, shape, distribution, and adjacency.

[0098] Example

[0099] In this embodiment, in order to fully extract image features, the image encoder adopts a dual-branch network to extract the attention-based spatial features and attention-based spectral features of the hyperspectral image respectively, and fuses the two. In order to improve the distinguishability of the target domain data, this implementation scheme uses a text modality auxiliary network to realize the alignment of the feature distribution of the source domain and the target domain, and uses contrastive learning to deeply compare and measure the similarity between the hyperspectral image and the natural language description. By measuring and optimizing these similarities, effective alignment of image and text features in the feature space is achieved. In this process, image and text features belonging to the same category attract each other in the feature space, while features of different categories repel each other, thereby pulling apart. Since text features are regarded as domain-invariant features, image features will effectively improve the separability of data in the process of approaching text features. Since the labels of the source domain are not applicable to the target domain data, the predicted labels obtained from the target data are used as pseudo-labels of the target data, and text descriptions are obtained based on the pseudo-labels.

[0100] In addition, in this embodiment, the domain adaptation module and the visual-language alignment module are combined together through three stages A, B and C. In stage A, the image encoder and the dual classifier are updated using labeled source domain samples so that the image encoder can fully learn the discriminative features of the source domain, thereby training the dual classifier to accurately classify the source domain samples, and using the MCC loss to reduce category confusion in the target domain. The visual-language alignment module is also performed in this stage. In stage B, the image encoder trained in stage A is used, and the classifier is updated by increasing the output difference of the dual classifier, while also minimizing the MCC loss. In stage C, the image encoder is optimized by using the fixed dual classifier, that is, minimizing the output difference of the dual classifier on the target sample, so that the prediction of the target sample by the two classifiers is as consistent as possible, while also minimizing the MCC loss.

[0101] In addition, this paper selects three different hyperspectral datasets (Houston, Pavia, HyRANK) and compares them with seven related working algorithms (Domain Adaptation Network, DAN; Domain-Adversarial Neural Networks, DANN; Class-wise Distribution Adaptation, CDA; Confident Learning-based Domain Adaptation, CLDA; Two-branch Attention Adversarial DA, TAADA; Topological structure and Semantic information Transfer network, TSTnet; Supervised Contrastive Learning-based UDA, SCLUDA). The results are as follows: Figure 4-6 As shown, the effectiveness of the present invention is proved.

[0102] Table 1 shows the average accuracy results (%) obtained on the three data sets, and Table 2 shows the Kappa coefficient results (kappa×100) obtained on the three data sets.

[0103] Table 1

[0104]

[0105] Table 2

[0106]

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal unsupervised domain adaptation hyperspectral image classification method, characterized in that: include: S1. Extract global features through the image encoder, use the dual classifier to classify image samples, align the distribution of the same type of source domain and target domain, and generate pseudo labels for the target domain; S2. Extract text features through the text encoder, compare and learn the text features and image features, narrow the distance between text and image features of the same category, and improve the separability of the data.

2. The method for unsupervised domain adaptation hyperspectral image classification based on multimodality according to claim 1, characterized in that: Step S1 specifically includes: S11, sending the image into the image encoder, and extracting the attention-based spatial features and the attention-based spectral features of the hyperspectral image through two branches respectively; S12, fusing the attention-based spatial features and the attention-based spectral features of the extracted hyperspectral image to obtain fused image features; S13. Send the fused image features to the dual classifier to obtain the prediction probability.

3. The multimodal unsupervised domain adaptation hyperspectral image classification method according to claim 2, characterized in that: After step S13, step S14 is also included to calculate the source classification loss as follows: The source classification loss is the cross entropy loss, and the calculation formula of the source classification loss is as follows: in, is the label, IE(·) is the output of the image encoder, is the image feature obtained by the image encoder, F(·) is the output of the two classifiers, is the predicted probability.

4. The multimodal unsupervised domain adaptation hyperspectral image classification method according to claim 3, characterized in that: After step S14, a step of minimizing class confusion in target prediction using MCC loss is also included, as follows: The correlation between any two categories is adjusted according to the contribution of different instances. When there are many categories, in order to avoid the problem of extremely unbalanced category distribution, the category normalization technology is used. The final MCC loss calculation formula is as follows: Among them, p ij represents the probability that the i-th sample belongs to the j-th category, W represents the weights of different categories, and c represents the number of categories.

5. The multimodal unsupervised domain adaptation hyperspectral image classification method according to claim 3, characterized in that: Step S1 also includes the step of updating the classifier using the adversarial loss function, as follows: The outputs of the target domain samples obtained by the two classifiers use the absolute value indicator to measure the difference, and the adversarial loss function is as follows:

6. The multimodal unsupervised domain adaptation hyperspectral image classification method according to claim 1, characterized in that: Step S2 specifically includes: S21. Calculate the alignment loss between image features and coarse-grained text features. The calculation formula is as follows: Among them, P x (i) and A x (i) are the positive sample set and negative sample set of image features, P t (i) and A t (i) are the positive sample set and negative sample set of text features, where |P(i)| = |P x (i)|=|P t (i)|;x i represents the image features, t i Represents text features; τ is a temperature parameter used to control the smoothness of the softmax distribution; S22. Calculate the alignment loss between image features and fine-grained text features. The calculation formula is as follows: in, Represents the loss of the image feature corresponding to the jth text feature and the fine-grained text feature; S23. Calculate the supervised contrastive learning loss of the source domain. The calculation formula is as follows: S24. Calculate the supervised contrastive learning loss of the target domain. The calculation formula is as follows: S25. Calculate the alignment loss of the total image features and text features. The calculation formula is as follows: in, represents the supervised contrastive learning loss of the source domain; represents the supervised contrastive learning loss for the target domain.

7. A multimodal unsupervised domain adaptation hyperspectral image classification system implemented based on the multimodal unsupervised domain adaptation hyperspectral image classification method according to any one of claims 1 to 6, characterized in that: include: Domain adaptation module and vision-language alignment module, where: The domain adaptation module obtains image features by the image encoder, obtains prediction results by the dual classifier, and selects one of the prediction results as the target domain pseudo label; The visual-language alignment module obtains the corresponding text description using the source domain label and the target domain pseudo label. After the text features are obtained by the text encoder, the text features are aligned with the image features obtained by the domain adaptation part.

Images