Open set cross-domain hyperspectral classification method and system based on multiple modes and optimal transmission

Abstract

The invention provides an open set cross-domain hyperspectral classification method and system based on multi-mode and optimal transmission, and relates to the technical field of hyperspectral image classification. The method specifically comprises the following steps: acquiring and utilizing an original hyperspectral image to construct a labeled source domain data set and an unlabeled target domain data set; utilizing a spatial-spectral feature extractor to respectively obtain spatial-spectral features of the source domain data and spatial-spectral features of the target domain data; and determining a source domain prototype, calculating a similarity matrix between the target domain data and the source domain prototype, solving an optimal transmission matching matrix, distributing a pseudo tag for each target domain data, and performing text correction on all the source domain data and all the target domain data by using a contrast learning loss function to obtain the target domain data. And the source domain prototype and the spatial spectrum feature extractor are optimized by constructing the total loss of minimizing hyperspectral image classification, so that classification prediction can be carried out on the hyperspectral image to be classified, and target domain data can be classified more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image classification, and in particular to an open set cross-domain hyperspectral classification method and system based on multimodality and optimal transmission. Background Art

[0002] As a typical spatial-spectral multidimensional data, hyperspectral images can capture the diagnostic spectral information of objects, reflect the intrinsic characteristics of objects, and are an important basis for distinguishing and identifying objects. Due to its advantages such as high spectral resolution, strong material recognition ability, relatively high spatial resolution, ability to detect weak information, and can be used for historical data analysis and data availability, hyperspectral images are widely used in geological rock and mineral exploration, land cover classification, agriculture, military and other fields. However, in practical applications, the acquisition of hyperspectral image labeled samples is costly and time-consuming, resulting in most of the collected hyperspectral image data having only a few labels or no labels at all, which brings great challenges to the accurate classification of hyperspectral images.

[0003] In addition, due to the influence of different acquisition conditions, such as time, location, sensor and other factors, significant spectral drift may occur between the acquired hyperspectral images. Even if a model is fully trained on hyperspectral image data with sufficient labels, it is often difficult to obtain satisfactory classification results when the trained model is directly applied to other hyperspectral image datasets with spectral shifts. Therefore, alleviating the spectral drift problem and improving the generalization ability of the model in different scenarios have become important research directions for improving the classification accuracy of hyperspectral images, which are crucial to ensuring the robust performance of hyperspectral images in different environments.

[0004] In addition, in the case of actual cross-scene hyperspectral image classification, there may be categories in the target domain that do not appear in the source domain, which will also lead to a decrease in the performance of the model in the target domain. Summary of the invention

[0005] In view of the shortcomings of the above-mentioned prior art, the present invention proposes an open set cross-domain hyperspectral classification method and system based on multimodality and optimal transmission, so as to solve the problem that the existing open set cross-domain hyperspectral image classification method only considers transferring the knowledge of the source domain to the target domain, ignoring the categories in the target domain that are not in the source domain, resulting in negative transfer to the target domain, and thus failing to obtain better classification performance in the target domain.

[0006] The first aspect of the present invention proposes an open set cross-domain hyperspectral classification method based on multimodality and optimal transmission, the method comprising the following steps:

[0007] Step 1: Obtain the original hyperspectral image and use it to construct a labeled source domain dataset and an unlabeled target domain dataset;

[0008] Step 2: Construct a spatial-spectral feature extractor, and input the labeled source domain dataset and the unlabeled target domain dataset into the spatial-spectral feature extractor for feature extraction, so as to obtain the spatial-spectral features of all source domain data and the spatial-spectral features of all target domain data;

[0009] Step 3: Build a dual classifier and determine the source domain prototype. Input the spatial-spectral features of all source domain data into the dual classifier for classification prediction, obtain the prediction probability of all source domain data and calculate the source classification loss of the source domain prototype.

[0010] Step 4: Divide the target domain data into several batches according to the set batch size, calculate the similarity matrix between the spatial-spectral features of all target domain data in each batch and the source domain prototype, and then assign pseudo labels to each target domain data by solving the optimal transfer matching matrix, and calculate the class alignment loss of the optimal transfer class alignment;

[0011] Step 5: Use the Clip large model to perform feature mapping on all real labels in the labeled source domain data set, and then use the contrastive learning loss function to calculate the alignment loss of the source domain data and the alignment loss of the target domain data to perform text correction on all source domain data and all target domain data, and calculate the total loss in the text correction process;

[0012] Step 6: Calculate the total loss of hyperspectral image classification by adding the source classification loss of the source domain prototype, the class alignment loss of the optimal transfer class alignment, and the total loss in the text correction process, and optimize the source domain prototype and the spatial spectrum feature extractor by minimizing the total loss of hyperspectral image classification;

[0013] Step 7: Obtain the hyperspectral image to be classified as a sample, use the optimized spatial spectrum feature extractor to extract the spatial-spectral features of the sample, and calculate the similarity between the spatial-spectral features of the sample and the optimized source domain prototype. When the similarity exceeds the set similarity threshold, the sample is classified as the category of the source domain data with the highest similarity to the sample; otherwise, the sample is classified as an unknown class;

[0014] The specific contents of step 1 are as follows: obtaining an original hyperspectral image, cutting the original hyperspectral image into pieces to obtain a number of samples, taking the samples with true labels as source domain data and constructing a labeled source domain dataset, taking the samples without true labels as target domain data and constructing an unlabeled target domain dataset; wherein the true labels are text information used to indicate the category to which the source domain data belongs;

[0015] The spatial spectrum feature extractor comprises: a spectral feature extraction branch, a spatial feature extraction branch, a feature fusion layer and an average pooling layer;

[0016] The spectral feature extraction branch is used to extract spectral attention features of the input hyperspectral image;

[0017] The spatial feature extraction branch is used to extract the spatial attention features of the input hyperspectral image;

[0018] The feature fusion layer is used to add the spectral attention feature and the spatial attention feature to obtain a spatial-spectral feature map of the hyperspectral image;

[0019] The average pooling layer is used to reduce the dimension of the spatial-spectral feature map of the hyperspectral image to obtain the spatial-spectral features of the hyperspectral image;

[0020] The spectral feature extraction branch includes: a first Conv3d-BN-ReLU module, a second Conv3d-BN-ReLU module, a third Conv3d-BN-ReLU module, a 3D convolution layer Conv3d, a fourth Conv3d-BN-ReLU module and a spectral SE attention module connected in series in sequence; wherein the output of the first Conv3d-BN-ReLU module and the output of the 3D convolution layer Conv3d are residually connected; the first Conv3d-BN-ReLU module, the second Conv3d-BN-ReLU module, the third Conv3d-BN-ReLU module and the fourth Conv3d-BN-ReLU module are all Conv3d-BN-ReLU structures composed of a 3D convolution layer Conv3d, a batch normalization layer BN and a rectified linear unit ReLU;

[0021] The spectral SE attention module is used to learn the feature attention weights of the spectral features extracted in the fourth Conv3d-BN-ReLU module, and multiply the learned feature attention weights by the spectral features extracted by the fourth Conv3d-BN-ReLU module element by element to obtain spectral attention features;

[0022] The spatial feature extraction branch includes: a fifth Conv3d-BN-ReLU module, a sixth Conv3d-BN-ReLU module, a seventh Conv3d-BN-ReLU module, a 3D convolution layer Conv3d and a spatial SE attention module; wherein the output of the fifth Conv3d-BN-ReLU module and the output residual of the 3D convolution layer Conv3d are connected; the fifth Conv3d-BN-ReLU module, the sixth Conv3d-BN-ReLU module and the seventh Conv3d-BN-ReLU module are all Conv3d-BN-ReLU structures;

[0023] The spatial SE attention module is used to learn the feature attention weights of the spatial features extracted from the seventh Conv3d-BN-ReLU module, and multiply the learned feature attention weights by the spatial features extracted from the seventh Conv3d-BN-ReLU module element by element to obtain the spatial attention features;

[0024] The step 3 further comprises:

[0025] Step 3.1: For any source domain data, the spatial-spectral features of the source domain data are input into the dual classifier for classification prediction to generate the prediction probability of the source domain data, and then the prediction probability of all source domain data is obtained;

[0026] The dual classifier is composed of two single classifiers, and the probability vectors output by the two single classifiers are connected to the last fully connected layer FC for classification, and the logic vector is obtained by using the SoftMax function, and the obtained logic vector is used as the prediction probability of the source domain data corresponding to the spatial-spectral features input into the dual classifier; wherein the two single classifiers are both neural networks formed by stacking several layers of FC-BN-ReLU-Dropout structures; the FC-BN-ReLU-Dropout structure includes: a fully connected layer FC, a batch normalization layer BN, a ReLU activation function layer and a Dropout layer connected in series;

[0027] Step 3.2: The weight of the last fully connected layer FC in the dual classifier is used as the source domain prototype;

[0028] Step 3.3: According to the predicted probabilities and true labels of all source domain data, the source classification loss of the source domain prototype is calculated using cross entropy loss;

[0029] The source classification loss is expressed as:

[0030]

[0031] in is the source classification loss; X s Y is the set of all source domain data in the labeled source domain dataset; s is the set of true labels of all source domain data in the labeled source domain dataset; S represents the source domain; n s represents the total number of source domain data in the input source domain dataset; i represents the index of the source domain data; L ce (·) represents the cross entropy loss; Represents the i-th source domain data; represents the true label of the i-th source domain data; G(·) represents the processing process of the spatial spectrum feature extractor; F(·) is the processing process of the dual classifier; Represents source domain data The predicted probability of

[0032] The method for assigning pseudo labels to each target domain data described in step 4 is shown in the following formula:

[0033]

[0034] Among them, OT ε,κ (M, α, β) represents the solution of the optimal transfer matching matrix with entropy regularization; ε and κ are both hyperparameters; M represents the cost matrix; α represents the number of source domain prototypes; β represents the target domain data; argmax represents the maximization operation; Q represents the transfer matching matrix; represents a two-dimensional matrix; It means solving the optimal transmission matching matrix; represents the optimal transmission matching matrix; H(Q) represents the entropy regularization term; D KL represents KL divergence; 1 c The number of one-dimensional vectors representing source domain prototypes; 1 B Represents the number of one-dimensional vectors of target domain data in a batch; A one-dimensional vector representing the prototype of the source domain; represents the number of source domain prototypes; Q st represents the optimal transmission matching matrix obtained; B represents the size of a batch; M st is the similarity matrix;

[0035] The class alignment loss is expressed as:

[0036]

[0037] Where L CD represents the class alignment loss; j represents the index of the target domain data in the current batch; A is the confidence of the target domain data in the current batch. If A is greater than B / 1, the target domain data in the current batch is considered to be a known class, otherwise it is an unknown class target domain data; L CE is the cross entropy loss; Represents the jth target domain data in the current batch; To match the target domain data according to the optimal transmission matching matrix Assigned pseudo labels;

[0038] The step 5 further comprises:

[0039] Step 5.1: Use the Clip large model to perform feature mapping on all real labels in the labeled source domain dataset to generate text features of all source domain data;

[0040] Step 5.2: Calculate the alignment loss of the source domain data using the contrastive learning loss function based on the spatio-spectral features of all the source domain data and the text features of all the source domain data;

[0041] Step 5.3: Calculate the alignment loss of the target domain data using the contrastive learning loss function based on the spatio-spectral features of all the target domain data and the text features of all the source domain data;

[0042] Step 5.4: Calculate the total loss in the text correction process using the alignment loss of the source domain data and the alignment loss of the target domain data;

[0043] The total loss in the text correction process is expressed as:

[0044]

[0045] where L text represents the total loss in the text correction process; represents the alignment loss of the source domain data; represents the alignment loss of the target domain data;

[0046] The method for generating text features described in Step 5.1 is as follows: In the labeled source domain dataset, the true label of each source domain data represents the category of the source domain data. There are a total of K known classes and 1 unknown class in the source domain data. For the true label T of any source domain data k , where k represents the k-th category of the source domain data, and 0 < k ≤ K + 1, use the Clip large model to map the true label of the source domain data to the feature space to generate the text features of the source domain data;

[0047] The text features of the source domain data are expressed as:

[0048] t k = c(T k )

[0049] where t k is the text feature of the k-th category of the source domain data after feature mapping; c(·) represents using the text extractor in the Clip large model to extract the feature of the input text information so as to map the extracted text modality information in the feature space;

[0050] The contrastive learning loss function described in Step 5.2 is expressed as:

[0051]

[0052] where L c represents the alignment loss of the spatio-spectral features and the text features; P(i) represents the total number of positive sample pairs and negative sample pairs related to the i-th sample; p represents the current sample; Pt (i) represents the positive sample set of text features related to the i-th sample, including all text features that match the i-th sample features; x i represents the spatial-spectral characteristics of the i-th sample; represents the positive sample of the text feature; τ represents the temperature parameter; a represents the negative sample; A t (i) A set of negative samples representing text features related to the i-th sample; Represents negative samples of text features; P x (i) represents the positive sample set of image features related to the i-th sample, including all image features that match the i-th sample features; where |P(i)| = |P x (i)|=|P t (i)|;A x (i) represents the negative sample set of image features related to the i-th sample; t i Represents text features; Represents positive samples of image features; Negative samples of text features.

[0053] The second aspect of the present invention proposes an open set cross-domain hyperspectral classification system based on multimodality and optimal transmission, which is used to implement an unsupervised domain adaptation hyperspectral image classification method based on multimodality. The system includes: a data preprocessing module, an image feature extraction module, a classification prediction module, an optimal transmission class alignment module and a visual-language alignment module;

[0054] The data preprocessing module is used to input the original hyperspectral image and perform preprocessing, thereby constructing a labeled source domain data set and an unlabeled target domain data set and transmitting them to the image feature extraction module;

[0055] The image feature extraction module is used to extract features from all source domain data and target domain data using a spatial spectrum feature extractor, obtain the spatial-spectral features of all source domain data and the spatial-spectral features of all target domain data, and transmit them to the visual-language alignment module and the optimal transfer class alignment module respectively; it is used to extract features from the input hyperspectral image to be classified, and then transmit the spatial-spectral features of the hyperspectral image to be classified to the classification prediction module;

[0056] The classification prediction module is used to determine and optimize the source domain prototype and transmit it to the optimal transmission class alignment module; it is used to classify and predict the hyperspectral image to be classified using the optimized source domain prototype and the spatial-spectral characteristics of the hyperspectral image to be classified;

[0057] The optimal transmission class alignment module is used to calculate the similarity matrix between the spatial-spectral features of the target domain data and the source domain prototype, and assign pseudo labels to the target domain data by solving the optimal transmission matching matrix, and then apply the pseudo labels of the target domain data to the optimization of the source domain prototype and the spatial spectrum feature extractor;

[0058] The visual-language alignment module is used to use the Clip large model to perform feature mapping on all real labels in the labeled source domain data set, and then perform text correction on all source domain data and all target domain data by calculating the alignment loss of the source domain data and the alignment loss of the target domain data using the contrastive learning loss function, and apply the text-corrected source domain data and target domain data to the optimization of the source domain prototype and the empty spectrum feature extractor.

[0059] The beneficial effects of adopting the above technical solution are:

[0060] The method of the present invention uses an adaptation layer to unify the number of source domain and target domain channels, utilizes a spatial spectrum feature extractor, i.e., a spatial-spectral feature extractor, to extract features, and then performs cross-entropy loss learning on the source domain prototype to obtain a classifier that accurately classifies the source domain data, and takes the weight of the last fully connected layer of the classifier as the source domain prototype.

[0061] The method of the present invention uses the features of text mapping to perform comparative learning on the feature mappings of the source domain and the target domain before the last layer of the classifier is fully connected, so that the feature mappings between different categories of the source domain and the target domain can be more discriminative.

[0062] The method of the present invention uses source domain prototypes and target domain samples to calculate the optimal transport (OT) similarity matrix, solves the optimal matching matrix, assigns the target domain samples with the pseudo labels of the source domain prototypes that have the greatest matching degree, and then performs domain alignment to reduce domain differences, that is, cross entropy loss is performed on the target domain samples and their pseudo labels, which is beneficial to reducing domain differences while avoiding negative migration generated by unknown class samples, and can maintain classification boundaries for domain alignment.

[0063] The method of the present invention can accurately classify the source domain categories of the target domain and accurately find out the unknown categories to avoid misclassification due to the existence of unknown categories. It is more in line with the application of hyperspectral images in actual scenes and can classify the target domain more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Flow chart of an open set cross-domain hyperspectral classification method based on multimodality and optimal transmission in this embodiment;

[0065] Figure 2Schematic diagram of the overall framework of an open set cross-domain hyperspectral classification method based on multimodality and optimal transmission in this embodiment;

[0066] Figure 3 is a schematic diagram of a spatial-spectral feature extractor in this embodiment;

[0067] Figure 4 Result comparison diagrams of different classification methods on the Houston dataset in this embodiment; wherein Figure (a) is a true label value diagram of the Houston dataset; Figure (b) is a classification result diagram of the application method DACD; Figure (c) is a classification result diagram of the application method OSBP; Figure (d) is a classification result diagram of the application method DAMC; Figure (e) is a classification result diagram of the application method MST classification; Figure (f) is a classification result diagram of the application method STA classification; Figure (g) is a classification result diagram of the application method UADAL; Figure (h) is a classification result diagram of the method of this embodiment;

[0068] Figure 5 Result comparison diagrams of different classification methods on the Pavia dataset in this embodiment; wherein Figure (a) is a true label value diagram of the Pavia dataset; Figure (b) is a classification result diagram of the application method DACD; Figure (c) is a classification result diagram of the application method OSBP; Figure (d) is a classification result diagram of the application method DAMC; Figure (e) is a classification result diagram of the application method MST classification; Figure (f) is a classification result diagram of the application method STA classification; Figure (g) is a classification result diagram of the application method UADAL; Figure (h) is a classification result diagram of the method of this embodiment;

[0069] Figure 6 Result comparison diagrams of different classification methods on the HyRANK data set in this embodiment; wherein Figure (a) is a true label value diagram of the HyRANK data set; Figure (b) is a classification result diagram of the application method DACD; Figure (c) is a classification result diagram of the application method OSBP; Figure (d) is a classification result diagram of the application method DAMC; Figure (e) is a classification result diagram of the application method MST classification; Figure (f) is a classification result diagram of the application method STA classification; Figure (g) is a classification result diagram of the application method UADAL; Figure (h) is a classification result diagram of the method of this embodiment;

[0070] Figure 7 This is a structural diagram of an open set cross-domain hyperspectral classification system based on multimodality and optimal transmission in this implementation. DETAILED DESCRIPTION

[0071] In order to facilitate the understanding of the present application, the specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thoroughly understood.

[0072] In this embodiment, an open set cross-domain hyperspectral classification method based on multimodality and optimal transmission is proposed. Figure 1 and Figure 2 As shown, the method comprises the following steps:

[0073] Step 1: Obtain the original hyperspectral image and use it to construct a labeled source domain dataset and an unlabeled target domain dataset;

[0074] The specific content of step 1 is: obtain the original hyperspectral image, cut the original hyperspectral image into pieces to obtain several samples, use the samples with real labels as source domain data and construct a labeled source domain data set, use the samples without real labels as target domain data and construct an unlabeled target domain data set; wherein the real label is text information, which is used to indicate the category to which the source domain data belongs.

[0075] In this embodiment, a certain number of original hyperspectral images are obtained from existing hyperspectral datasets, such as the Houston dataset, the Pavia dataset, and the HyRANK dataset. The original hyperspectral images are sliced ​​and preprocessed into datasets of specified size and specification to obtain labeled source domain data and unlabeled target domain data.

[0076] Step 2: Construct a spatial-spectral feature extractor and input the labeled source domain dataset and the unlabeled target domain dataset into the spatial-spectral feature extractor for feature extraction to obtain the spatial-spectral features of all source domain data and the spatial-spectral features of all target domain data.

[0077] The spatial-spectral feature extractor comprises: a spectral feature extraction branch, a spatial feature extraction branch, a feature fusion layer and an average pooling layer; wherein the spectral feature extraction branch is used to extract the spectral attention feature of the input hyperspectral image; the spatial feature extraction branch is used to extract the spatial attention feature of the input hyperspectral image; the feature fusion layer is used to add the spectral attention feature and the spatial attention feature to obtain the spatial-spectral feature map of the hyperspectral image; the average pooling layer is used to reduce the dimension of the spatial-spectral feature map of the hyperspectral image to obtain the spatial-spectral feature of the hyperspectral image.

[0078] In this embodiment, the spectral attention features and spatial attention features obtained by the spectral feature extraction branch and the spatial feature extraction branch are fused and then average pooled to obtain the final spatial-spectral features and use them as image features of the input data; the above processing enables the spatial-spectral feature extractor to fully understand all aspects of the data sample, thereby improving the classification accuracy and the generalization ability of the model.

[0079] The spectral feature extraction branch includes: a first Conv3d-BN-ReLU module, a second Conv3d-BN-ReLU module, a third Conv3d-BN-ReLU module, a 3D convolution layer Conv3d, a fourth Conv3d-BN-ReLU module and a spectral SE attention module connected in series in sequence; wherein the output of the first Conv3d-BN-ReLU module and the output of the 3D convolution layer Conv3d are residually connected; the first Conv3d-BN-ReLU module, the second Conv3d-BN-ReLU module, the third Conv3d-BN-ReLU module and the fourth Conv3d-BN-ReLU module are all Conv3d-BN-ReLU structures composed of a 3D convolution layer Conv3d, a batch normalization layer (BatchNormalization, BN) and a rectified linear unit (Rectified Linear Unit, ReLU).

[0080] The spectral SE attention module is used to learn the feature attention weights of the spectral features extracted in the fourth Conv3d-BN-ReLU module, and multiply the learned feature attention weights by the spectral features extracted by the fourth Conv3d-BN-ReLU module element by element to obtain the spectral attention features.

[0081] In this embodiment, if Figure 3 As shown in the figure, the spectral feature extraction branch is used to extract the spectral information of the hyperspectral image, and the Conv3d-BN-ReLU structure is used to extract features from the input hyperspectral image data and learn the spatial and temporal features of the input data. The spectral SE attention module first compresses the spatial dimension of the feature map through the Global Average Pooling operation to obtain a channel descriptor. Then, the correlation between channels is learned through two fully connected layers, and the weight of each channel is generated through the Sigmoid activation function. The spectral feature extraction branch takes into account the importance of each spectral band, so that the constructed model can focus more on the spectral features. Taking the Houston dataset as an example, the input data block size is 7×7×48, 7×7 is the spatial neighborhood size, and 48 is the number of bands.

[0082] The spatial feature extraction branch includes: a fifth Conv3d-BN-ReLU module, a sixth Conv3d-BN-ReLU module, a seventh Conv3d-BN-ReLU module, a 3D convolution layer Conv3d and a spatial SE attention module; wherein the output of the fifth Conv3d-BN-ReLU module and the output of the 3D convolution layer Conv3d are residually connected; the fifth Conv3d-BN-ReLU module, the sixth Conv3d-BN-ReLU module and the seventh Conv3d-BN-ReLU module are all Conv3d-BN-ReLU structures.

[0083] The spatial SE attention module is used to learn the feature attention weights of the spatial features extracted in the seventh Conv3d-BN-ReLU module, and multiply the learned feature attention weights by the spatial features extracted by the seventh Conv3d-BN-ReLU module element by element to obtain the spatial attention features.

[0084] In this embodiment, if Figure 3 As shown in the figure, the spatial feature extraction branch is used to extract the spatial information of the hyperspectral image. After the hyperspectral image in the input spatial feature extraction branch passes through three Conv3d-BN-ReLU structures, 3D convolution layer Conv3d and residual operation, the data is input into the spatial SE attention module; in this module, the model is enabled to pay attention to and utilize spatial features. The structure of the spatial SE attention module is the same as that of the spectral SE attention module.

[0085] Step 3: Build a dual classifier and determine the source domain prototype. Input the spatial-spectral features of all source domain data into the dual classifier for classification prediction, obtain the prediction probability of all source domain data and calculate the source classification loss of the source domain prototype.

[0086] Step 3.1: For any source domain data, the spatial-spectral features of the source domain data are input into the dual classifier for classification prediction to generate the prediction probability of the source domain data, and then the prediction probability of all source domain data is obtained.

[0087] The dual classifier is composed of two single classifiers, and the probability vectors output by the two single classifiers are connected to the last fully connected layer FC for classification, and the logical vector is obtained by using the SoftMax function, and the obtained logical vector is used as the predicted probability of the source domain data corresponding to the spatial-spectral features input into the dual classifier; wherein the two single classifiers are both neural networks formed by stacking several layers of FC-BN-ReLU-Dropout structures; the FC-BN-ReLU-Dropout structure includes the following series: a fully connected layer (Fully Connected Layer, FC), a batch normalization layer (BatchNormalization, BN), a ReLU activation function layer and a Dropout layer.

[0088] In this embodiment, the fully connected layer converts the input spatial-spectral features into a one-dimensional feature vector; the batch normalization layer normalizes the output of the fully connected layer so that the mean of each feature is 0 and the variance is 1; the ReLU activation function layer introduces nonlinearity so that the model can learn more complex feature representations; the Dropout layer prevents the model from overfitting by randomly discarding the output of a part of the neurons during the training process. Each layer in the FC-BN-ReLU-Dropout structure processes the output of the previous layer and provides input for the next layer. In this way, each single classifier can gradually extract and combine features and classify them through the fully connected layer, which not only increases the nonlinear expression ability of the model, but also prevents overfitting and enhances the generalization ability of the model. The two single classifiers are converted into probability vectors based on the spatial-spectral features obtained by the spatial spectrum feature extractor, and the SoftMax function is applied to obtain a logical vector, which represents the prediction probability of the corresponding single classifier for the source domain data category.

[0089] Step 3.2: The weights of the last fully connected layer FC in the dual classifier are used as the source domain prototype.

[0090] In this embodiment, Y s and X s In supervised training, the dual classifier calculates the predicted probability of the source domain data through forward propagation, and then uses the loss function, namely the cross entropy loss, to measure the difference between the predicted probability of the source domain data and the true label. The weight of the last fully connected layer FC in the dual classifier is taken as the source domain prototype, and the accurate source domain prototype can be obtained through subsequent training optimization.

[0091] Step 3.3: According to the predicted probabilities and true labels of all source domain data, the source classification loss of the source domain prototype is calculated using the cross entropy loss.

[0092] The source classification loss is expressed as:

[0093]

[0094] in is the source classification loss; X s Y is the set of all source domain data in the labeled source domain dataset; s is the set of true labels of all source domain data in the labeled source domain dataset; S represents the source domain; n s represents the total number of source domain data in the source domain dataset; i represents the index of the source domain data; L ce (·) represents the cross entropy loss; Represents the i-th source domain data; represents the true label of the i-th source domain data; G(·) represents the processing process of the spatial spectrum feature extractor; F(·) is the processing process of the dual classifier; Represents source domain data The predicted probability of .

[0095] Step 4: Divide the target domain data into several batches according to the set batch size, and calculate the similarity matrix between the spatial-spectral features of all target domain data in each batch and the source domain prototype respectively. Then, assign a pseudo label to each target domain data by solving the optimal transfer matching matrix, and calculate the class alignment loss of the optimal transfer class alignment.

[0096] The method for assigning a pseudo label to each target domain data is shown in the following formula:

[0097]

[0098] Among them, OT ε,κ (M, α, β) represents the solution of the optimal transfer matching matrix with entropy regularization; ε and κ are both hyperparameters; M represents the cost matrix; α represents the number of source domain prototypes; β represents the target domain data; argmax represents the maximization operation; Q represents the transfer matching matrix; represents a two-dimensional matrix; It means solving the optimal transmission matching matrix; represents the optimal transmission matching matrix; H(Q) represents the entropy regularization term; D KL represents KL divergence; 1 c The number of one-dimensional vectors representing source domain prototypes; 1 B Represents the number of one-dimensional vectors of target domain data in a batch; A one-dimensional vector representing the prototype of the source domain; represents the number of source domain prototypes; Q st represents the optimal transmission matching matrix obtained; B represents the size of a batch; M st is the similarity matrix.

[0099] The class alignment loss is expressed as:

[0100]

[0101] Where L CD represents the class alignment loss; j represents the index of the target domain data in the current batch; A is the confidence of the target domain data in the current batch. If A is greater than B / 1, the target domain data in the current batch is considered to be a known class, otherwise it is an unknown class target domain data; L CE is the cross entropy loss; Represents the jth target domain data in the current batch; To match the target domain data according to the optimal transmission matching matrix Assigned pseudo labels.

[0102] In this embodiment, the similarity matrix of the optimal transmission is constructed using the features of the corrected target data and the source domain prototype, and then the optimal matching matrix is ​​solved to assign pseudo labels to the unlabeled samples of the target domain, and cross-entropy loss training is performed to achieve category alignment between the target domain and the source domain data, greatly reducing domain differences and improving the accuracy of cross-scene classification. Domain differences can be reduced by calculating the class alignment loss of the optimal transmission class alignment. When solving the optimal allocation matrix, samples that are similar to the source domain prototype are often given a large weight, which is the so-called high-confidence sample, and these high-confidence samples are assigned pseudo-labels. For samples that are not similar to the source domain prototype, the sample is considered to be an unknown class in the target domain, and such samples are discarded. Then, cross-entropy loss is used for training to achieve alignment between domains.

[0103] Step 5: Use the Clip large model to perform feature mapping on all real labels in the labeled source domain dataset, and then perform text correction on all source domain data and all target domain data by calculating the alignment loss of the source domain data and the alignment loss of the target domain data using the contrastive learning loss function, and calculate the total loss in the text correction process.

[0104] In this embodiment, the text modality is added to correct the mapping of samples in the feature space, which can make the various prototypes in the source domain more discriminable, and the various data in the target domain also have a certain degree of discriminability, so that the weights obtained when solving the optimal transmission optimal allocation matrix after the corrected feature mapping are more accurate, and the pseudo-labels obtained are more accurate, which can more effectively reduce the domain differences, thereby helping to improve the final classification accuracy.

[0105] Step 5.1: Use the Clip large model to perform feature mapping on all real labels in the labeled source domain dataset to generate text features of all source domain data.

[0106] The method for generating text features is as follows: In the labeled source domain dataset, the true label of each source domain data represents the category of the source domain data. There are a total of K known classes and 1 unknown class in the source domain data. For the true label T of any source domain data k , where k represents the k-th category of the source domain data, and 0 < k ≤ K + 1. The Clip large model is used to map the true label of the source domain data to the feature space to generate the text features of the source domain data.

[0107] The text features of the source domain data are expressed as:

[0108] t k = C(T k ) (5)

[0109] where t k is the text feature of the k-th category of the source domain data after feature mapping; c(·) represents using the text extractor in the Clip large model to extract features from the input text information, so as to map the extracted text modality information in the feature space.

[0110] In this embodiment, the labeled source domain class text information is input into the existing Clip large model, and the text features are obtained by mapping it in the feature space; the existing language large model Clip large model (Contrastive Language-Image Pretraining) is a multi-modal learning model that jointly learns images and texts through contrastive learning. The core idea of CLIP is to map images and texts to a shared embedding space and maximize the similarity between matching image-text pairs and minimize the similarity between non-matching image-text pairs through contrastive learning.

[0111] Step 5.2: According to the spatial-spectral features of all source domain data and the text features of all source domain data, use the contrastive learning loss function to calculate the alignment loss of the source domain data.

[0112] The contrastive learning loss function is expressed as:

[0113]

[0114] where L c represents the alignment loss between the spatial-spectral features and the text features; P(i) represents the total number of positive and negative sample pairs related to the i-th sample; p represents the current sample; P t (i) represents the positive sample set of the text features related to the i-th sample, including all text features that match the features of the i-th sample; x i represents the spatial-spectral feature of the i-th sample; represents the positive sample of text features; τ represents the temperature parameter, which is used to control the smoothness of softmax distribution; a represents the negative sample; A t (i) A set of negative samples representing text features related to the i-th sample; Represents negative samples of text features; P x (i) represents the positive sample set of image features related to the i-th sample, including all image features that match the i-th sample features; where |P(i)| = |P x (i)|=|P t (i)|;A x (i) represents the negative sample set of image features related to the i-th sample; t i Represents text features; Represents positive samples of image features; Negative samples of text features.

[0115] In this embodiment, each source domain data is regarded as a sample. For any sample i and the text features of all samples, if the true label of sample i matches a certain text feature, the true label of sample i and the text feature are taken as a positive sample pair of a text feature related to sample i, and the text feature is a positive sample of the text feature related to the i-th sample; if the true label of sample i does not match a certain text feature, the true label of sample i and the text feature are taken as a negative sample pair of a text feature related to sample i, and the text feature is a negative sample of the text feature related to the i-th sample. Similarly, for any sample i and the spatial-spectral features of all samples, if the true label of sample i matches a certain spatial-spectral feature, the true label of sample i and the spatial-spectral feature are taken as a positive sample pair of image features related to sample i, and the image feature is a positive sample of the image feature related to the i-th sample; if the true label of sample i does not match a certain spatial-spectral feature, the true label of sample i and the spatial-spectral feature are taken as a negative sample pair of image features related to sample i, and the image feature is a negative sample of the image feature related to the i-th sample.

[0116] In this embodiment, the true label of the source domain data is artificially written as a text description for each category of the source domain data based on the color, shape, distribution and adjacency relationship. After the text features are obtained through the text encoder, contrastive learning loss is used as the optimization target. After both the image features and the text features are projected into the multimodal embedding space through normalization, only by shortening the distance between the image features and the text features can the alignment between the same categories be better achieved. The alignment loss of the spatial-spectral features and the text features is calculated using the contrastive learning loss function in order to maximize the similarity between the positive sample pairs, thereby minimizing the distance between the positive sample pairs. In other words, maximizing the similarity between the positive sample pairs can minimize the distance between the positive sample pairs.

[0117] Step 5.3: Based on the spatial-spectral features of all target domain data and the text features of all source domain data, the alignment loss of the target domain data is calculated using the contrastive learning loss function.

[0118] Step 5.4: Calculate the total loss in the text correction process using the alignment loss of the source domain data and the alignment loss of the target domain data.

[0119] The total loss in the text correction process is expressed as:

[0120]

[0121] Where L text represents the total loss during text correction; represents the alignment loss of source domain data; represents the alignment loss of target domain data.

[0122] In this embodiment, text is used to guide the feature mapping of the source domain and the target domain, and the features of the source domain and the target domain data are made more discriminative by optimizing the contrastive learning loss, so that the pseudo-label can be solved more accurately by using the optimal transmission.

[0123] Step 6: The total loss of hyperspectral image classification is calculated by adding the source classification loss of the source domain prototype, the class alignment loss of the optimal transfer class alignment, and the total loss in the text correction process, and the source domain prototype and the empty spectrum feature extractor are optimized by minimizing the total loss of hyperspectral image classification.

[0124] The total loss of the hyperspectral image classification is expressed as:

[0125]

[0126] Where Loss is the total loss of hyperspectral image classification; is the source classification loss of the source domain prototype.

[0127] Step 7: Obtain the hyperspectral image to be classified as a sample, use the optimized spatial spectrum feature extractor to extract the spatial-spectral features of the sample, and calculate the similarity between the spatial-spectral features of the sample and the optimized source domain prototype. When the similarity exceeds the set similarity threshold, the sample is classified as the category of the source domain data with the highest similarity to the sample; otherwise, the sample is classified as an unknown class.

[0128] In this embodiment, the trained empty spectrum feature extractor and the source domain prototype are retained, and a similarity threshold is used to distinguish whether the target sample is a known class or an unknown class. If it is a known class, subsequent classification is performed.

[0129] In this embodiment, if Figure 4 , Figure 5 and Figure 6 As shown in the figure, the application effect of the open set cross-domain hyperspectral classification method based on multimodality and optimal transmission proposed in this embodiment is verified on the Houston dataset, Pavia dataset and HyRANK dataset respectively, and the method is compared with the existing methods, wherein the existing methods include: a domain adaptive classification method based on deep networks (Domain Adaptive Classification using Deep networks, DACD) aims to reduce the difference between domains by learning the feature mapping between the source domain and the target domain, thereby improving the classification performance of the target domain; a one-shot hyperspectral band selection and classification method (One-Shot Hyperspectral Band Selection and Classification, OSBP), which focuses on the single-shot band selection and classification of hyperspectral images, and realizes effective feature selection and classification under limited samples through deep learning technology; a deep learning framework (Deep Attention-based Multimodal Classification, using attention mechanism to process multimodal data; a multi-stage network based on Transformer (Multi-Stage Transformer (MST), which improves the performance of image classification through multiple stages of feature extraction and transformation; a spectral-temporal attention (STA) that combines spectral and temporal attention mechanisms to process hyperspectral data with time series characteristics; an unsupervised adversarial domain adaptation learning method (UADAL), which reduces the distribution difference between the source domain and the target domain through adversarial training. Figure 4 , Figure 5 and Figure 6The white area in the middle represents unknown categories. Figure 4 , Figure 5 and Figure 6 From the comparison of the classification results in , it can be clearly seen that the performance of the open set cross-domain hyperspectral classification method based on multimodality and optimal transmission proposed in this embodiment is better.

[0130] The present embodiment provides a multi-modal unsupervised domain adaptation hyperspectral image classification system, which is used to implement a multi-modal unsupervised domain adaptation hyperspectral image classification method, such as Figure 7 As shown, the system includes: a data preprocessing module, an image feature extraction module, a classification prediction module, an optimal transmission class alignment module and a vision-language alignment module.

[0131] The data preprocessing module is used to input the original hyperspectral image and perform preprocessing, thereby constructing a labeled source domain data set and an unlabeled target domain data set and transmitting them to the image feature extraction module.

[0132] The image feature extraction module is used to use a spatial-spectral feature extractor to extract features from all source domain data and target domain data, obtain the spatial-spectral features of all source domain data and the spatial-spectral features of all target domain data, and transmit them to the visual-language alignment module and the optimal transfer class alignment module respectively; it is used to extract features from the input hyperspectral image to be classified, and then transmit the spatial-spectral features of the hyperspectral image to be classified to the classification prediction module.

[0133] The classification prediction module is used to determine and optimize the source domain prototype and transmit it to the optimal transmission class alignment module; and is used to perform classification prediction on the hyperspectral image to be classified using the optimized source domain prototype and the spatial-spectral characteristics of the hyperspectral image to be classified.

[0134] The optimal transmission class alignment module is used to calculate the similarity matrix between the spatial-spectral features of the target domain data and the source domain prototype, and assign pseudo labels to the target domain data by solving the optimal transmission matching matrix, and then apply the pseudo labels of the target domain data to the optimization of the source domain prototype and the spatial spectrum feature extractor.

[0135] The visual-language alignment module is used to use the Clip large model to perform feature mapping on all real labels in the labeled source domain data set, and then perform text correction on all source domain data and all target domain data by calculating the alignment loss of the source domain data and the alignment loss of the target domain data using the contrastive learning loss function, and apply the text-corrected source domain data and target domain data to the optimization of the source domain prototype and the empty spectrum feature extractor.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. An open set cross-domain hyperspectral classification method based on multimodality and optimal transmission, characterized in that: The method comprises the following steps: Step 1: Obtain the original hyperspectral image and use it to construct a labeled source domain dataset and an unlabeled target domain dataset; Step 2: Construct a spatial-spectral feature extractor, and input the labeled source domain dataset and the unlabeled target domain dataset into the spatial-spectral feature extractor for feature extraction, so as to obtain the spatial-spectral features of all source domain data and the spatial-spectral features of all target domain data; Step 3: Build a dual classifier and determine the source domain prototype. Input the spatial-spectral features of all source domain data into the dual classifier for classification prediction, obtain the prediction probability of all source domain data and calculate the source classification loss of the source domain prototype. Step 4: Divide the target domain data into several batches according to the set batch size, calculate the similarity matrix between the spatial-spectral features of all target domain data in each batch and the source domain prototype, and then assign pseudo labels to each target domain data by solving the optimal transfer matching matrix, and calculate the class alignment loss of the optimal transfer class alignment; Step 5: Use the Clip large model to perform feature mapping on all real labels in the labeled source domain data set, and then use the contrastive learning loss function to calculate the alignment loss of the source domain data and the alignment loss of the target domain data to perform text correction on all source domain data and all target domain data, and calculate the total loss in the text correction process; Step 6: Calculate the total loss of hyperspectral image classification by adding the source classification loss of the source domain prototype, the class alignment loss of the optimal transfer class alignment, and the total loss in the text correction process, and optimize the source domain prototype and the spatial spectrum feature extractor by minimizing the total loss of hyperspectral image classification; Step 7: Obtain the hyperspectral image to be classified as a sample, use the optimized spatial spectrum feature extractor to extract the spatial-spectral features of the sample, and calculate the similarity between the spatial-spectral features of the sample and the optimized source domain prototype. When the similarity exceeds the set similarity threshold, the sample is classified as the category of the source domain data with the highest similarity to the sample; otherwise, the sample is classified as an unknown class.

2. The open set cross-domain hyperspectral classification method based on multimodality and optimal transmission according to claim 1 is characterized in that: The specific content of step 1 is: obtain the original hyperspectral image, cut the original hyperspectral image into pieces to obtain several samples, use the samples with real labels as source domain data and construct a labeled source domain dataset, use the samples without real labels as target domain data and construct an unlabeled target domain dataset; The true label is text information, which is used to indicate the category to which the source domain data belongs.

3. The open set cross-domain hyperspectral classification method based on multimodality and optimal transmission according to claim 2 is characterized in that: The spatial spectrum feature extractor comprises: a spectral feature extraction branch, a spatial feature extraction branch, a feature fusion layer and an average pooling layer; The spectral feature extraction branch is used to extract spectral attention features of the input hyperspectral image; The spatial feature extraction branch is used to extract the spatial attention features of the input hyperspectral image; The feature fusion layer is used to add the spectral attention feature and the spatial attention feature to obtain a spatial-spectral feature map of the hyperspectral image; The average pooling layer is used to reduce the dimension of the spatial-spectral feature map of the hyperspectral image to obtain the spatial-spectral features of the hyperspectral image.

4. The open set cross-domain hyperspectral classification method based on multimodality and optimal transmission according to claim 3 is characterized in that: The spectral feature extraction branch includes: a first Conv3d-BN-ReLU module, a second Conv3d-BN-ReLU module, a third Conv3d-BN-ReLU module, a 3D convolution layer Conv3d, a fourth Conv3d-BN-ReLU module and a spectral SE attention module connected in series in sequence; wherein the output of the first Conv3d-BN-ReLU module and the output of the 3D convolution layer Conv3d are residually connected; the first Conv3d-BN-ReLU module, the second Conv3d-BN-ReLU module, the third Conv3d-BN-ReLU module and the fourth Conv3d-BN-ReLU module are all Conv3d-BN-ReLU structures composed of a 3D convolution layer Conv3d, a batch normalization layer BN and a rectified linear unit ReLU; The spectral SE attention module is used to learn the feature attention weights of the spectral features extracted in the fourth Conv3d-BN-ReLU module, and multiply the learned feature attention weights by the spectral features extracted by the fourth Conv3d-BN-ReLU module element by element to obtain spectral attention features; The spatial feature extraction branch includes: a fifth Conv3d-BN-ReLU module, a sixth Conv3d-BN-ReLU module, a seventh Conv3d-BN-ReLU module, a 3D convolution layer Conv3d and a spatial SE attention module; wherein the output of the fifth Conv3d-BN-ReLU module and the output residual of the 3D convolution layer Conv3d are connected; the fifth Conv3d-BN-ReLU module, the sixth Conv3d-BN-ReLU module and the seventh Conv3d-BN-ReLU module are all Conv3d-BN-ReLU structures; The spatial SE attention module is used to learn the feature attention weights of the spatial features extracted in the seventh Conv3d-BN-ReLU module, and multiply the learned feature attention weights by the spatial features extracted by the seventh Conv3d-BN-ReLU module element by element to obtain the spatial attention features.

5. The open set cross-domain hyperspectral classification method based on multimodality and optimal transmission according to claim 4 is characterized in that: The step 3 further comprises: Step 3.1: For any source domain data, the spatial-spectral features of the source domain data are input into the dual classifier for classification prediction to generate the prediction probability of the source domain data, and then the prediction probability of all source domain data is obtained; The dual classifier is composed of two single classifiers, and the probability vectors output by the two single classifiers are connected to the last fully connected layer FC for classification, and the logic vector is obtained by using the SoftMax function, and the obtained logic vector is used as the prediction probability of the source domain data corresponding to the spatial-spectral features input into the dual classifier; wherein the two single classifiers are both neural networks formed by stacking several layers of FC-BN-ReLU-Dropout structures; the FC-BN-ReLU-Dropout structure includes: a fully connected layer FC, a batch normalization layer BN, a ReLU activation function layer and a Dropout layer connected in series; Step 3.2: The weight of the last fully connected layer FC in the dual classifier is used as the source domain prototype; Step 3.3: According to the predicted probabilities and true labels of all source domain data, the source classification loss of the source domain prototype is calculated using cross entropy loss; The source classification loss is expressed as: in is the source classification loss; X s Y is the set of all source domain data in the labeled source domain dataset; s is the set of true labels of all source domain data in the labeled source domain dataset; S represents the source domain; n s represents the total number of source domain data in the input source domain dataset; i represents the index of the source domain data; L ce (·) represents the cross entropy loss; Represents the i-th source domain data; represents the true label of the i-th source domain data; G(·) represents the processing process of the spatial spectrum feature extractor; F(·) is the processing process of the dual classifier; Represents source domain data The predicted probability of .

6. The open set cross-domain hyperspectral classification method based on multimodality and optimal transmission according to claim 5 is characterized in that: The method for assigning pseudo labels to each target domain data described in step 4 is shown in the following formula: Among them, OT ε ,κ (M, α, β) represents the solution of the optimal transfer matching matrix with entropy regularization; ε and κ are both hyperparameters; M represents the cost matrix; α represents the number of source domain prototypes; β represents the target domain data; argmax represents the maximization operation; Q represents the transfer matching matrix; represents a two-dimensional matrix; It means solving the optimal transmission matching matrix; represents the optimal transmission matching matrix; H(Q) represents the entropy regularization term; D KL represents KL divergence; 1 c The number of one-dimensional vectors representing source domain prototypes; 1 B Represents the number of one-dimensional vectors of target domain data in a batch; A one-dimensional vector representing the prototype of the source domain; represents the number of source domain prototypes; Q st represents the optimal transmission matching matrix obtained; B represents the size of a batch; M st is the similarity matrix; The class alignment loss is expressed as: Where L CD represents the class alignment loss; j represents the index of the target domain data in the current batch; A is the confidence of the target domain data in the current batch. If A is greater than B / 1, the target domain data in the current batch is considered to be a known class, otherwise it is an unknown class target domain data; L CE is the cross entropy loss; Represents the jth target domain data in the current batch; To match the target domain data according to the optimal transmission matching matrix Assigned pseudo labels.

7. The open set cross-domain hyperspectral classification method based on multimodality and optimal transmission according to claim 6 is characterized in that: The step 5 further comprises: Step 5.1: Use the Clip large model to perform feature mapping on all real labels in the labeled source domain dataset to generate text features of all source domain data; Step 5.2: Calculate the alignment loss of source domain data using contrastive learning loss function based on the spatial-spectral features of all source domain data and the text features of all source domain data; Step 5.3: Calculate the alignment loss of the target domain data using the contrastive learning loss function based on the spatial-spectral features of all target domain data and the text features of all source domain data; Step 5.4: Calculate the total loss in the text correction process using the alignment loss of the source domain data and the alignment loss of the target domain data; The total loss in the text correction process is expressed as: Where L text represents the total loss during text correction; represents the alignment loss of source domain data; represents the alignment loss of target domain data.

8. The open set cross-domain hyperspectral classification method based on multimodality and optimal transmission according to claim 7 is characterized in that: The method for generating text features described in Step 5.1 is as follows: In the labeled source domain dataset, the true label of each source domain data represents the category of the source domain data, where there are a total of K known classes and 1 unknown class in the source domain data. For the true label T of any source domain data k , where k represents the k-th category of the source domain data, and 0 < k ≤ K + 1. The Clip large model is used to map the true label of the source domain data to the feature space to generate the text features of the source domain data; The text features of the source domain data are expressed as: t k =c(T k ) where t k is the text feature of the kth category of the source domain data after feature mapping; c(·) represents the feature extraction of the input text information by the text extractor in the Clip large model, so as to map the extracted text modal information in the feature space.

9. The open set cross-domain hyperspectral classification method based on multimodality and optimal transmission according to claim 8, characterized in that: The contrastive learning loss function described in step 5.2 is expressed as: Where L c represents the alignment loss of spatial-spectral features and text features; P(i) represents the total number of positive sample pairs and negative sample pairs related to the i-th sample; p represents the current sample; P t (i) represents the positive sample set of text features related to the i-th sample, including all text features that match the i-th sample features; x i represents the spatial-spectral characteristics of the i-th sample; represents the positive sample of text features; τ represents the temperature parameter; a represents the negative sample; A t (i) A set of negative samples representing text features related to the i-th sample; Negative samples representing text features; P x (i) represents the positive sample set of image features related to the i-th sample, including all image features that match the i-th sample features; where |P(i)|=|P x (i)|=|P t (i)|;A x (i) represents the negative sample set of image features related to the i-th sample; t i Represents text features; Represents positive samples of image features; Negative samples of text features.

10. An open set cross-domain hyperspectral classification system based on multimodality and optimal transmission is used to implement an unsupervised domain adaptation hyperspectral image classification method based on multimodality, characterized in that: The system includes: a data preprocessing module, an image feature extraction module, a classification prediction module, an optimal transmission class alignment module and a vision-language alignment module; The data preprocessing module is used to input the original hyperspectral image and perform preprocessing, thereby constructing a labeled source domain data set and an unlabeled target domain data set and transmitting them to the image feature extraction module; The image feature extraction module is used to extract features from all source domain data and target domain data using a spatial spectrum feature extractor, obtain the spatial-spectral features of all source domain data and the spatial-spectral features of all target domain data, and transmit them to the visual-language alignment module and the optimal transfer class alignment module respectively; it is used to extract features from the input hyperspectral image to be classified, and then transmit the spatial-spectral features of the hyperspectral image to be classified to the classification prediction module; The classification prediction module is used to determine and optimize the source domain prototype and transmit it to the optimal transmission class alignment module; it is used to classify and predict the hyperspectral image to be classified using the optimized source domain prototype and the spatial-spectral characteristics of the hyperspectral image to be classified; The optimal transmission class alignment module is used to calculate the similarity matrix between the spatial-spectral features of the target domain data and the source domain prototype, and assign pseudo labels to the target domain data by solving the optimal transmission matching matrix, and then apply the pseudo labels of the target domain data to the optimization of the source domain prototype and the spatial spectrum feature extractor; The visual-language alignment module is used to use the Clip large model to perform feature mapping on all real labels in the labeled source domain data set, and then perform text correction on all source domain data and all target domain data by calculating the alignment loss of the source domain data and the alignment loss of the target domain data using the contrastive learning loss function, and apply the text-corrected source domain data and target domain data to the optimization of the source domain prototype and the empty spectrum feature extractor.

Images