Cross-scene hyperspectral image classification method based on multi-modal domain generalization

Abstract

The application discloses a cross-scene hyperspectral image classification method based on multi-modal domain generalization, comprising the following steps: step one, data preparation; step two, image feature extraction; step three, construction of an optimized semantic space; step four, loss calculation and model optimization; step five, model testing; the application uses a multi-modal fusion network to optimize the classification algorithm, the network cooperates and operates in parallel with two feature extraction modules, which can mine more rich and diverse image features; the prior text knowledge of each category is processed to construct an optimized semantic space, so that the image features can be accurately aligned by category, thereby significantly improving the accuracy of classification; the optimized semantic space realizes the deep interactive fusion of the two modalities of images and texts, promotes the learning of cross-domain invariant representation, and through targeted optimization of the supervised contrast loss, the features are aligned by category in the semantic space, further reducing the cross-domain difference and enhancing the adaptability and stability of the model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a cross-scene hyperspectral image classification method based on multimodal domain generalization. Background Technology

[0002] In the field of hyperspectral image classification, deep learning technology, especially convolutional neural networks, has made significant progress, enabling the extraction and application of implicit information from hyperspectral images in areas such as environmental monitoring, medical imaging, agriculture, and geological exploration. However, in cross-scene hyperspectral image classification, training and test samples often come from different scenes with different spectral and spatial characteristics, leading to domain shifts or distribution mismatches. This makes it difficult for models trained in a single scene to generalize to other scenes, severely impacting their performance.

[0003] In existing technologies, transfer learning is generally used to address the poor adaptability of deep learning in cross-scene hyperspectral image classification. Domain adaptation within transfer learning is the most widely used method. For example, feature adaptation aligns source and target domain samples to a unified feature space through mapping, thus ensuring alignment between different domains. Deep Adaptive Networks utilize maximum mean difference to mitigate domain discrepancies, while Deep Adversarial Neural Networks (DANNNs) pioneered adversarial techniques for solving the domain adaptation problem, focusing on enabling the trained network to extract similar features from the source and target domains. However, all of the above methods focus on feature alignment and do not fully explore and utilize other potential correlation information. Some transfer learning techniques employ domain generalization methods, which currently involve data augmentation, representation learning, and learning strategy development. Among these, representation learning is the most important method. Representation learning mainly considers how to train the model to reduce the representational discrepancies of one or more source domains, but it does not consider the facilitating effect of linguistic features on visual representation learning. Furthermore, for hyperspectral images, since they lack textual descriptive information that reflects feature categories, how to introduce text and achieve efficient image-text fusion has become a pressing problem to be explored. If this problem is not solved, it will hinder the improvement of the accuracy and comprehensiveness of hyperspectral image classification tasks in cross-scene applications. Summary of the Invention

[0004] The purpose of this invention is to provide a cross-scene hyperspectral image classification method based on multimodal domain generalization to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a cross-scene hyperspectral image classification method based on multimodal domain generalization, comprising the following steps: Step 1, data preparation; Step 2, image feature extraction; Step 3, constructing and optimizing the semantic space; Step 4, calculating the loss and optimizing the model; Step 5, model testing;

[0006] In step one above, the hyperspectral images in the dataset are cut into image patches and sent to the image encoder to build prior text knowledge for the dataset and form a knowledge base.

[0007] Step two above specifically includes the following steps:

[0008] 2.1 Extraction of spatial and spectral features: In the image encoder, SSEM based on three-dimensional convolution extracts the spatial and spectral features of hyperspectral image patches through three-dimensional spectral information;

[0009] 2.2 Extraction of multi-scale features: In the image encoder, the MSEM based on the two-dimensional residual pyramid structure extracts multi-scale features of hyperspectral image patches through the Conv2d-BN2d module, three pyramid residual modules and average pooling layer;

[0010] 2.3 Feature Concatenation: The features extracted by SSEM and MSEM are concatenated and sent to the classification head and semantic space optimization.

[0011] Step three above specifically includes the following steps:

[0012] 3.1 Text Processing: The semantic space construction module is optimized to obtain prior text knowledge from the knowledge base based on tags, and then encodes the prior text knowledge through a text encoder;

[0013] 3.2 Extracting Text Features: The text encoder uses the Transformer architecture to extract text features from the prior knowledge of the encoded text;

[0014] 3.3 Constructing the optimized semantic space and projecting image features: The optimized semantic space construction module uses the extracted text features to construct the optimized semantic space and projects the image features extracted by the image encoder into the optimized semantic space;

[0015] Step four above specifically includes the following steps:

[0016] 4.1 Calculate the classification loss: Input the image features extracted by the image encoder into the classification head, the classification head outputs the classification probability, and use the ground truth to calculate the cross-entropy loss of supervised learning, and obtain the classification loss of the network;

[0017] 4.2 Calculate the contrast loss: Calculate the contrast loss based on the similarity between image features and semantic space features;

[0018] 4.3 Calculate the overall loss: the overall loss during model training. Defined as:

[0019]

[0020] in, It is classification loss. It is a comparison of losses. It's a hyperparameter;

[0021] 4.4 Model Parameter Optimization: The gradient with respect to the model parameters is calculated based on the overall loss. Based on the gradient of the model parameters, the Adam optimizer is used to optimize the image encoder and text encoder.

[0022] In step five above, SSEM and MSEM are used to extract and classify image features, and the performance of the model optimized in step four is evaluated in the target domain based on OA and KC.

[0023] Preferably, in step one, the image patch size is ,in and These are all patch sizes of image blocks. Represents the number of spectral channels.

[0024] Preferably, in step one, for the knowledge base Prior knowledge for each text All related to tags One-to-one correspondence; prior textual knowledge describes the spatial relationships and image features of the categories corresponding to the labels. Given a label... It can be achieved through the following mapping relationship Find the corresponding text :

[0025] .

[0026] Preferably, in step 2.1, SSEM uses a serial 3D convolutional residual block, a max pooling layer, and 3D convolution to extract depth information. The 3D convolutional residual block consists of two Conv3d-BN3d-ReLU modules and one Conv3d module. The 3D convolution formula is:

[0027]

[0028] in, , , These represent the length, width, and number of channels of a 3D convolution, respectively. Indicates the first In the layer network Feature samples The value of the neuron at that location, Indicates in The weights of the (i-1)th feature map in the convolution kernel. This indicates network bias.

[0029] Preferably, in step 2.2, the three pyramid residual modules middle, It contains three residual modules with the same structure. Each residual module contains three BN2d-Conv2d blocks and one BN2d-ReLU block. The kernel size of the second convolutional layer is 7, and the kernel size of the rest is 3. and The first residual module in Its second convolutional layer has a kernel size of 8 and a stride of 2, and the rest of the structure is the same as... Similarly, MSEM employs the following rules to systematically increase the depth of feature maps at each layer:

[0030]

[0031] in, This is the initial number of channels. It is the first In the pyramid residual module, the first The depth of each unit, This is the total number of residual units in the entire network. In this case, the depth of each layer of the feature map depends on... and .

[0032] Preferably, in step 3.1, the encoding formula is as follows:

[0033]

[0034] in, It uses lowercase byte pair encoding, with a vocabulary of 49152. It is a tag embedding operation. It is a positional encoding.

[0035] Preferably, in step 3.2, the Transformer architecture employs a multi-head self-attention mechanism for feature extraction and text tokenization. Divided into Parts, each part is set as ,in, Indicates batch size. Indicates the maximum sentence length. Indicates the size of the embedding dimension. Indicates the number of heads. After linear transformation, we obtain and The specific calculation process is as follows:

[0036]

[0037]

[0038]

[0039] in , , All are learnable matrices;

[0040] Then, the attention weight matrix for each head is calculated, and the attention matrices of all heads are merged:

[0041]

[0042]

[0043] in, Used for each data sequence Operation, and Then the attention matrices of each data header are merged into a single attention matrix;

[0044] The equations for the entire feature extraction process can be simplified as follows:

[0045]

[0046]

[0047] in, and It is a layer normalization operation. It is a feedforward neural network. The above feature extraction process only involves one layer, and the number of repetitions is the same as the number of layers. The Transformer architecture has 3 layers and 8 heads, with a width of 512.

[0048] Preferably, in step 3.3, the specific formula is as follows:

[0049]

[0050]

[0051]

[0052] in, It is a learnable matrix. It is a bias term, and the linear layer is responsible for mapping image features to the semantic space.

[0053] Preferably, in step 4.1, the cross-entropy loss The formula is:

[0054]

[0055] Classification loss of the network The formula is:

[0056]

[0057] in, yes One-hot encoding of category information and It is a prediction probability. Indicates the number of samples.

[0058] Preferably, in step 4.2, the similarity measurement formula is:

[0059]

[0060]

[0061] To highlight the crucial role of similarity, After squaring, the optimized supervised contrast loss is:

[0062]

[0063]

[0064]

[0065] in, Corresponding image features, These do not correspond to image features, which are positive and negative features, respectively. and This represents one of the positive and negative samples. , As a contribution weight.

[0066] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention utilizes a multimodal fusion network to optimize the classification algorithm. This network coordinates and operates two feature extraction modules in parallel, enabling the mining of richer and more diverse image features. By processing prior textual knowledge for each category to construct an optimized semantic space, image features are accurately aligned by category, thereby significantly improving classification accuracy. The optimized semantic space achieves deep interactive fusion of image and text modalities, promotes the learning of cross-domain invariant representations, and through targeted optimization of supervised contrastive loss, promotes feature alignment by category within the semantic space, further reducing cross-domain differences and enhancing the model's adaptability and stability. Attached Figure Description

[0067] Figure 1 This is a diagram illustrating the overall training framework of the image classification method of the present invention.

[0068] Figure 2 A flowchart for the image encoder SSEM;

[0069] Figure 3 This is a schematic diagram of the bottleneck residual structure of the pyramid.

[0070] Figure 4 A flowchart for the image encoder MSEM;

[0071] Figure 5 A pseudo-color image of the Houston dataset;

[0072] Figure 6 Pseudocolor images for the Pavia dataset;

[0073] Figure 7 Pseudocolor images for the Indiana dataset;

[0074] Figure 8 Line graphs of OA for the proposed method using different base learning rates on three datasets;

[0075] Figure 9 Line graphs of OA for the proposed method using different regularization parameters on three datasets;

[0076] Figure 10 Line graphs of OA for the proposed method using different contribution weights on three datasets;

[0077] Figure 11 Line graphs of OA for the proposed method using different contribution patch sizes on three datasets;

[0078] Figure 12 The data are the ground truth plot and model prediction results for Houston2018, where (a) GT; (b) GroupDRO; (c) ANDMask; (d) VREx; (e) DIFEX; (f) SDEnet; (g) S2ECnet; (h) LLURnet; and (i) the method of this invention.

[0079] Figure 13 The data are the ground truth plot and model prediction results for PaviaCenter, where (a) GT; (b) GroupDRO; (c) ANDMask; (d) VREx; (e) DIFEX; (f) SDEnet; (g) S2ECnet; (h) LLURnet; and (i) the method of this invention.

[0080] Figure 14The data are the ground truth plot and model prediction results for IndianaTD, where (a) GT; (b) GroupDRO; (c) ANDMask; (d) VREx; (e) DIFEX; (f) SDEnet; (g) S2ECnet; (h) LLURnet; and (i) the method of this invention.

[0081] Figure 15 This is a flowchart of the method of the present invention. Detailed Implementation

[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] Please see the appendix Figure 1 -Appendix Figure 15 The present invention provides a technical solution:

[0084] Example:

[0085] A cross-scene hyperspectral image classification method based on multimodal domain generalization includes the following steps: Step 1, data preparation; Step 2, image feature extraction; Step 3, constructing and optimizing the semantic space; Step 4, calculating the loss and optimizing the model; Step 5, model testing.

[0086] In step one above, the hyperspectral images in the dataset are segmented into image patches and sent to the image encoder to construct prior textual knowledge for the dataset, forming a knowledge base; the image patch size is... ,in and These are all patch sizes of image blocks. Represents the number of spectral channels, for the knowledge base Prior knowledge for each text All related to tags One-to-one correspondence; prior textual knowledge describes the spatial relationships and image features of the categories corresponding to the labels. Given a label... It can be achieved through the following mapping relationship Find the corresponding text :

[0087] ;

[0088] Step two above specifically includes the following steps:

[0089] 2.1 Extraction of Spatial and Spectral Features: In the image encoder, SSEM based on 3D convolution extracts the spatial and spectral features of hyperspectral image patches through 3D spectral information. SSEM uses a series of 3D convolutional residual blocks, max pooling layers, and 3D convolution to extract depth information. The 3D convolutional residual block consists of two Conv3d-BN3d-ReLU modules and one Conv3d module. The outputs of the first and last Conv3d-BN3d-ReLU modules are used for residual merging. The 3D convolution formula is:

[0090]

[0091] in, , , These represent the length, width, and number of channels of a 3D convolution, respectively. Indicates the first In the layer network Feature samples The value of the neuron at that location, Indicates in The weights of the (i-1)th feature map in the convolution kernel. Indicates network bias;

[0092] 2.2 Multi-scale Feature Extraction: In the image encoder, the MSEM based on the two-dimensional residual pyramid structure extracts multi-scale features of hyperspectral image patches through the Conv2d-BN2d module, three pyramid residual modules, and an average pooling layer. middle, It contains three residual modules with the same structure. Each residual module contains three BN2d-Conv2d blocks and one BN2d-ReLU block. The kernel size of the second convolutional layer is 7, and the kernel size of the rest is 3. and The first residual module in Its second convolutional layer has a kernel size of 8 and a stride of 2, and the rest of the structure is the same as... Similarly, MSEM employs the following rules to systematically increase the depth of feature maps at each layer:

[0093]

[0094] in, This is the initial number of channels. It is the first In the pyramid residual module, the first The depth of each unit, This is the total number of residual units in the entire network. In this case, the depth of each layer of the feature map depends on... and ;

[0095] 2.3 Feature Concatenation: The features extracted by SSEM and MSEM are concatenated and sent to the classification head and semantic space optimization.

[0096] Step three above specifically includes the following steps:

[0097] 3.1 Text Processing: The semantic space construction module optimizes the acquisition of prior text knowledge from the knowledge base based on tags, and then encodes the prior text knowledge using a text encoder. The encoding formula is as follows:

[0098]

[0099] in, It uses lowercase byte pair encoding, with a vocabulary of 49152. It is a tag embedding operation. It is a positional encoding;

[0100] 3.2 Text Feature Extraction: The text encoder uses a Transformer architecture to extract text features from the prior knowledge of the encoded text. The Transformer architecture employs a multi-head self-attention mechanism for feature extraction, and the text is then labeled. Divided into Parts, each part is set as ,in, Indicates batch size. Indicates the maximum sentence length. Indicates the size of the embedding dimension. Indicates the number of heads. After linear transformation, we obtain and The specific calculation process is as follows:

[0101]

[0102]

[0103]

[0104] in , , All are learnable matrices;

[0105] Then, the attention weight matrix for each head is calculated, and the attention matrices of all heads are merged:

[0106]

[0107]

[0108] in, Used for each data sequence Operation, and Then the attention matrices of each data header are merged into a single attention matrix;

[0109] The equations for the entire feature extraction process can be simplified as follows:

[0110]

[0111]

[0112] in, and It is a layer normalization operation. It is a feedforward neural network. The above feature extraction process only involves one layer, and the number of repetitions is the same as the number of layers. The Transformer architecture has 3 layers and 8 heads, with a width of 512.

[0113] 3.3 Constructing the Optimized Semantic Space and Projecting Image Features: The optimized semantic space construction module uses the extracted text features to construct the optimized semantic space and projects the image features extracted by the image encoder into the optimized semantic space. The specific formula is as follows:

[0114]

[0115]

[0116]

[0117] in, It is a learnable matrix. It is a bias term; the linear layer is responsible for mapping image features to the semantic space.

[0118] Step four above specifically includes the following steps:

[0119] 4.1 Calculate the classification loss: Input the image features extracted by the image encoder into the classification head, the classification head outputs the classification probability, and calculate the supervised learning cross-entropy loss using the ground truth, thus obtaining the network's classification loss and cross-entropy loss. The formula is:

[0120]

[0121] Classification loss of the network The formula is:

[0122]

[0123] in, yes One-hot encoding of category information and It is a prediction probability. Indicates the number of samples;

[0124] 4.2 Calculate the contrast loss: The contrast loss is calculated based on the similarity between image features and semantic space features. The similarity measurement formula is as follows:

[0125]

[0126]

[0127] To highlight the crucial role of similarity, After squaring, the optimized supervised contrast loss is:

[0128]

[0129]

[0130]

[0131] in, Corresponding image features, These do not correspond to image features, which are positive and negative features, respectively. and This represents one of the positive and negative samples. Contribution weight Its function is to adjust the degree of influence of the image features of the two branches on the optimization of the semantic space;

[0132] 4.3 Calculate the overall loss: the overall loss during model training. Defined as:

[0133]

[0134] in, It is classification loss. It is a comparison of losses. It's a hyperparameter. Its function is to balance the impact of semantic space optimization on the overall model;

[0135] 4.4 Model Parameter Optimization: The gradient with respect to the model parameters is calculated based on the overall loss. Based on the gradient of the model parameters, the Adam optimizer is used to optimize the image encoder and text encoder.

[0136] In step five above, SSEM and MSEM are used to extract and classify image features, and the performance of the model optimized in step four is evaluated in the target domain based on OA and KC.

[0137] Experimental Example 1:

[0138] To verify the effectiveness of the proposed method in the embodiments, the following experiments were conducted: Three publicly available datasets were used: Houston, Pavia, and Indiana. The Houston dataset is shown in Table 1, the Pavia dataset in Table 2, and the Indiana dataset in Table 3. All experiments were run in the same environment using Python 3.6 and PyTorch 11.3 frameworks. The default weight decay of L2 regularization for all modules was set to 1e-4. The image encoder and text encoder were optimized using Adam, and the CPU was an AMD EPYC7642. The network was built using a 48-core processor and an RTX 3090 GPU with 24GB of RAM. Knowledge bases were created for three datasets, with each category corresponding to a prior knowledge set. The knowledge bases for Houston, Pavia, and Indiana are shown in Table 4, Table 5, and Table 6, respectively. Parameter sensitivity analysis was performed to evaluate the network's sensitivity to the three target domains. The four adjustable hyperparameters were the base learning rate η, regularization parameter λ, contribution weight α, and patch size. η was selected from 1e-1, 1e-2, 1e-3, 1e-4, and 1e-5; λ was selected from 1e+2, 1e+1, 1e+0, 1e-1, and 1e-2; α was selected from 0.1, 0.3, 0.5, 0.7, and 0.9; and the patch size was selected from 9, 11, 13, 15, and 17. Figure 8 The classification results are shown on three datasets with different base learning rates; Figure 9 The classification results for different regularization parameters are shown on three datasets; Figure 10 The classification results with different weighted contributions are shown on three datasets; Figure 11 The classification results are shown for different patch sizes on three datasets. A base learning rate of 1e-2, a contribution weight of 0.5, and a patch size of 13 were used on these three datasets. For the Houston and Pavia datasets, the regularization parameter was set to 1e-0, which effectively prevents overfitting and improves the stability of the model. For the Indiana dataset, the regularization parameter was set to 1e+1, which helps to address the problem of large feature differences in this dataset, thereby improving classification performance.

[0139] Experimental Example 2:

[0140] To verify the effectiveness of the proposed SSEM, MSEM, and optimized semantic space, ablation experiments were designed. In the experiments, the first two modules were SSEM and MSEM, and the latter two modules were SS and OSS. SS represents the use of the original supervised contrastive learning, while OSS introduces optimized supervised contrastive learning. By comparing the experimental results of SS and OSS, the effectiveness of the proposed contrastive learning loss optimization method was demonstrated. Table 7 shows that the experimental results indicate that the proposed method of this invention outperforms these variants. Specifically, designing SSEM and MSEM as two branches of a parallel network and training them together can fully leverage their respective advantages and achieve complementary strengths. The optimization of the semantic space and supervised contrastive loss further improves the generalization ability of the model.

[0141] Comparative example:

[0142] To demonstrate the superiority of the proposed method in cross-domain classification tasks, comparative experiments were conducted with other methods. Specifically, SDEnet, S2ECnet, and LLURnet were used as comparison methods. Due to the limited availability of existing cross-scene hyperspectral classification methods, four additional RGB image classification methods—GroupDRO, ANDMask, VREx, and DIFEX—were selected to further validate the method's effectiveness. All data samples and labels from the source domain were used as training instances, with 80% used for training and 20% for validation. Simultaneously, all data from the target domain were used for testing. To minimize the impact of random sampling, the average of ten runs was used to determine the result for each algorithm. To measure the accuracy and consistency between the true labels and predicted values, the overall accuracy (OA) and Kappa coefficient (KC) of the target domain were used as evaluation metrics. The results are shown in Tables 8, 9, and 10. On the Houston dataset, LLURNet performs best among all compared methods; the OA performance of our method is 4.22 higher than LLURNet, and the KC performance is 7.16 higher than LLURNet. On the Pavia dataset, LLURNet performs best among all compared methods; the OA performance of our method is 0.86 higher than LLURNet, and the KC performance is 0.96 higher than LLURNet. On the Indiana dataset, DIFEX performs best among all compared methods; the OA performance of our method is 2.08 higher than DIFEX, and the KC performance is 3.76 higher than DIFEX. In the Indiana dataset, the standard deviation of some classes is greater than the accuracy, indicating that the experimental results fluctuate greatly, reflecting the challenge of classifying these feature classes; our method effectively alleviates this problem. Overall, our method performs better than the compared methods in classification; the classification graphs for the three datasets are shown below. Figure 12 , Figure 13 and Figure 14As shown; each unclassified pixel is used as the background, while classified pixels are predicted as contrast; each image contains a ground truth image and predictions from all models; Figure 14 As can be clearly seen, the method of the present invention performs better in predicting the second and fifth categories.

[0143] Table 1 Houston Dataset

[0144]

[0145] Table 2 Pavia dataset

[0146]

[0147] Table 3 Indiana Dataset

[0148] Table 4 Knowledge Base of the Houston Dataset

[0149] Class Name Prior knowledge 1 Grass healthy The grass healthy is lush 2 Grass stressed The grass stressed by the road appears pale 3 Trees The trees grow steadily along the road 4 Water Water appears smooth with a dark blue or black color 5 Residential buildings Residential buildings arranged neatly 6 Non-residential buildings Non-residential buildings vary in shape 7 Road Roads divide buildings into blocks

[0150] Table 5 Knowledge Base of the Pavia Dataset

[0151] Class Name Prior knowledge 1 Tree The trees grow steadily along the road 2 Asphalt Asphalt is used to pave roads 3 Brick A brick is a type of construction material 4 Bitumen Bitumen is a material for building surfaces 5 Shadow Shadows will appear on the backlight of the building 6 Meadow Meadow is a land covered with grass 7 Bare soil No vegetation on the surface of bare soil

[0152] Table 6 Knowledge Base of the Indiana Dataset

[0153] Class Name Prior knowledge 1 Concrete / Asphalt No crops on the surfaces of Concrete or Asphalt 2 Corn-CleanTill Corn-CleanTill planted with corn 3 Corn-CleanTill-EW Corn-CleanTill-EW planted with early maturing maize 4 Orchard The orchard is full of fruit trees 5 Soybeans-CleanTill Soybeans-CleanTill planted with soybeans 6 Soybeans-CleanTill-EW Soybeans-CleanTill-EW grows early soybeans 7 Wheat Wheat is an important food crop

[0154] Table 7 Ablation Experiment Results

[0155] SSEM MSEM SS OSS OA onHouston KC onHouston OA onPavia KC onPavia OA onIndiana KC onIndiana √ √ 75.35 63.05 74.86 69.81 55.27 41.94 √ √ 78.32 63.05 72.83 66.93 54.99 41.28 √ √ 74.72 51.87 71.68 66.42 55.20 41.41 √ √ √ 79.81 63.21 78.58 74.33 55.37 41.67 √ √ √ 82.29 68.62 84.53 81.37 56.56 44.81

[0156] Table 8. Comparison of accuracy, OA (%), and KC (κ) of different methods for the target scene in the Houston 2018 class.

[0157] Class GroupDRO ANDMask VREx DIFEX SDEnet S2ECnet LLURnet Ours 1 30.36±10.64 35.02±10.54 33.75±10.98 23.62±8.55 62.94±18.32 80.34±22.14 33.06±15.05 46.27±16.87 2 71.18±2.30 70.76±4.13 69.87±2.43 68.28±7.37 78.13±7.89 65.09±11.96 69.01±8.00 67.96±8.20 3 63.98±2.55 64.16±6.13 64.30±5.23 66.41±6.42 57.91±17.52 46.71±4.89 64.90±5.63 68.51±11.32 4 81.82±7.19 81.82±3.21 78.18±8.74 91.82±11.32 100.00±0.00 100.00±0.00 100.00±0.00 100.00±0.00 5 50.73±9.14 53.30±10.07 55.58±5.51 54.34±10.63 62.02±5.42 73.62±7.46 67.80±5.60 74.10±5.45 6 80.92±9.60 78.11±9.03 78.06±6.76 81.87±6.04 84.54±4.40 83.42±2.60 74.25±36.97 94.53±3.26 7 59.83±9.53 65.46±6.12 62.86±8.69 60.13±9.42 50.05±2.75 47.02±5.96 46.63±5.22 51.33±8.79 OA 72.30±3.73 71.61±4.27 71.39±2.73 72.97±2.77 75.63±2.90 75.61±1.82 78.07±0.88 82.29±1.41 KC 55.33±3.19 55.34±4.29 54.65±2.50 55.89±3.10 59.59±3.51 59.96±2.38 61.46±2.31 68.62±3.04

[0158] Table 9. Comparison of accuracy, OA (%), and KC (κ) of different methods for the target scene Pavia Center class.

[0159] Class GroupDRO ANDMask VREx DIFEX SDEnet S2ECnet LLURnet Ours 1 79.55±18.56 81.85±6.94 81.51±19.60 82.53±11.00 92.79±2.99 95.36±2.89 83.29±7.54 9.44±2.04 2 82.50±5.50 76.45±4.36 80.82±3.58 81.55±8.12 84.73±3.08 81.68±9.26 84.48±1.66 87.73±4.13 3 19.11±16.56 23.07±18.75 21.42±14.04 25.20±30.33 78.39±9.08 58.89±36.36 74.67±5.96 71.23±9.14 4 73.18±17.14 68.40±23.06 78.13±11.05 74.71±13.95 81.05±2.54 78.44±9.34 84.96±1.58 82.76±4.25 5 80.00±4.31 80.44±10.80 81.15±6.79 90.78±5.99 83.89±7.92 84.48±6.83 86.16±5.50 88.95±4.38 6 77.86±6.86 80.82±7.91 76.30±7.43 76.99±5.31 71.42±9.00 61.49±10.32 75.91±6.82 81.49±5.80 7 74.65±14.07 82.76±9.91 70.79±11.50 69.85±23.01 71.21±5.44 79.58±9.24 87.62±5.25 81.28±11.10 OA 74.02±4.31 74.04±3.11 74.39±4.38 74.98±4.83 81.89±1.22 80.58±0.42 83.67±0.53 84.53±1.31 KC 68.64±4.94 68.78±3.56 69.17±5.01 69.98±5.62 78.27±1.43 75.88±2.00 80.41±0.64 81.37±1.67

[0160] Table 10 Comparison of accuracy, OA (%), and KC (κ) of different methods for the target scene IndianaTD class

[0161] Class GroupDRO ANDMask VREx DIFEX SDEnet S2ECnet LLURnet Ours 1 10.29±8.59 11.10±10.54 8.45±9.65 14.73±21.01 0.01±0.03 0.63±0.90 0.24±0.55 1.43±2.30 2 29.24±25.99 8.47±16.92 4.13±1.57 9.51±20.39 0.82±1.11 2.39±3.52 1.95±3.03 46.39±9.45 3 68.82±22.99 85.06±13.51 92.21±6.50 86.05±16.72 94.52±2.54 92.80±5.38 92.93±5.52 44.17±7.63 4 77.89±6.44 70.82±11.83 66.91±26.16 77.55±17.04 83.18±4.10 86.03±3.89 86.89±5.51 66.07±4.29 5 0.66±1.34 0.10±0.23 0.02±0.03 0.01±0.16 0.01±0.03 0.00±0.00 0.00±0.00 44.03±2.86 6 0.00±0.00 0.00±0.00 0.00±0.00 0.09±0.19 0.00±0.00 0.00±0.00 0.00±0.00 0.00±0.00 7 96.16±1.44 95.40±1.63 96.34±2.02 94.83±2.64 95.96±0.75 94.81±2.40 94.77±2.78 99.42±0.50 OA 53.65±0.95 53.21±1.74 53.96±1.07 54.48±1.40 54.00±0.36 53.53±0.69 53.46±0.60 56.56±0.65 KC 40.30±0.80 39.32±1.95 39.86±1.62 41.05±2.01 40.13±0.46 39.55±0.81 39.52±0.77 44.81±0.84

[0162] Based on the above, the advantages of this invention are that the proposed method is a multimodal fusion network. By introducing a language modality, it reduces the impact of data distribution differences, thereby assisting in learning cross-domain invariant representations of hyperspectral images. The method of this invention includes a parallel architecture of a spatial-spectral feature extraction module (SSEM) and a multi-scale feature extraction module (MSEM) to capture richer image information during the feature extraction stage. In addition to being used for classification, the extracted features are also mapped to an optimized semantic space (OSS). The optimized semantic space brings similar features closer together and separates different features, realizing deep interactive fusion of image and text modalities and promoting the learning of cross-domain invariant representations. Compared with existing methods, the method of this invention has superior performance in cross-scene hyperspectral image classification tasks.

[0163] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A cross-scene hyperspectral image classification method based on multimodal domain generalization, comprising the following steps: Step 1: Data preparation; Step 2: Image feature extraction; Step 3: Construct and optimize the semantic space; Step 4: Calculate the loss and optimize the model; Step 5: Test the model; Its features are: In step one above, the hyperspectral images in the dataset are cut into image patches and sent to the image encoder to build prior text knowledge for the dataset and form a knowledge base. Step two above specifically includes the following steps: Step 2.1 Extracting Spatial and Spectral Features: In the image encoder, the spatial-spectral feature extraction module SSEM based on three-dimensional convolution extracts the spatial and spectral features of hyperspectral image patches through three-dimensional spectral information; Step 2.2 Extracting Multi-Scale Features: In the image encoder, the multi-scale feature extraction module MSEM based on the two-dimensional residual pyramid structure extracts multi-scale features of hyperspectral image patches through the Conv2d-BN2d module, three pyramid residual modules and average pooling layer; Step 2.3 Feature Concatenation: Concatenate the features extracted by SSEM and MSEM and send them to the classification head and optimize the semantic space; Step three above specifically includes the following steps: Step 3.1 Text Processing: The semantic space construction module optimizes the text prior knowledge obtained from the knowledge base based on the tags, and then encodes the text prior knowledge through a text encoder; Step 3.2 Extracting Text Features: The text encoder uses the Transformer architecture to extract text features from the prior knowledge of the encoded text; Step 3.3 Constructing the optimized semantic space and projecting image features: The optimized semantic space construction module uses the extracted text features to construct the optimized semantic space and projects the image features extracted by the image encoder into the optimized semantic space; Step four above specifically includes the following steps: Step 4.1 Calculate the classification loss: Input the image features extracted by the image encoder into the classification head, the classification head outputs the classification probability, and use the ground truth to calculate the cross-entropy loss of supervised learning, and obtain the classification loss of the network; Step 4.2 Calculate contrast loss: Calculate contrast loss based on the similarity between image features and semantic space features; Step 4.3 Calculate the overall loss: the overall loss during model training. Defined as: ， in, It is classification loss. It is a comparison of losses. It's a hyperparameter; Step 4.4 Model parameter optimization: Calculate the gradient with respect to the model parameters based on the overall loss, and use the Adam optimizer to optimize the image encoder and text encoder based on the gradient of the model parameters; In step five above, SSEM and MSEM are used to extract and classify image features, and the performance of the model optimized in step four is evaluated in the target domain based on OA and KC.

2. The cross-scene hyperspectral image classification method based on multimodal domain generalization according to claim 1, characterized in that: In step one, the image patch size is ,in and These are all patch sizes of image blocks. Represents the number of spectral channels.

3. The cross-scene hyperspectral image classification method based on multimodal domain generalization according to claim 1, characterized in that: In step one, for the knowledge base Prior knowledge for each text All related to tags One-to-one correspondence; prior textual knowledge describes the spatial relationships and image features of the categories corresponding to the labels. Given a label... It can be achieved through the following mapping relationship Find the corresponding text : 。 4. The cross-scene hyperspectral image classification method based on multimodal domain generalization according to claim 1, characterized in that: In step 2.1, SSEM uses a series of 3D convolutional residual blocks, max pooling layers, and 3D convolution to extract depth information. The 3D convolutional residual block consists of two Conv3d-BN3d-ReLU modules and one Conv3d module. The 3D convolution formula is: ， in, , , These represent the length, width, and number of channels of a 3D convolution, respectively. Indicates the first In the layer network Feature samples The value of the neuron at that location, This indicates network bias.

5. The cross-scene hyperspectral image classification method based on multimodal domain generalization according to claim 1, characterized in that: In step 2.2, the three pyramid residual modules middle, Contains three residual modules with identical structures Each residual module contains three BN2d-Conv2d blocks and one BN2d-ReLU block. The kernel size of the second convolutional layer is 7, and the kernel size of the rest is 3. and The first residual module in Its second convolutional layer has a kernel size of 8 and a stride of 2, and the rest of the structure is the same as... Similarly, MSEM employs the following rules to systematically increase the depth of feature maps at each layer: ， in, This is the initial number of channels. It is the first In the pyramid residual module, the first The depth of each unit, This is the total number of residual units in the entire network. In this case, the depth of each layer of the feature map depends on... and .

6. The cross-scene hyperspectral image classification method based on multimodal domain generalization according to claim 1, characterized in that: In step 3.1, the encoding formula is as follows: ， in, It uses lowercase byte pair encoding, with a vocabulary of 49152. It is a tag embedding operation. It is a positional encoding.

7. The cross-scene hyperspectral image classification method based on multimodal domain generalization according to claim 1, characterized in that: In step 3.2, the Transformer architecture employs a multi-head self-attention mechanism for feature extraction and text tokenization. Divided into Parts, each part is set as ,in, Indicates batch size. Indicates the maximum sentence length. Indicates the size of the embedding dimension. Indicates the number of heads. After linear transformation, we obtain and The specific calculation process is as follows: ， ， ， in , , All are learnable matrices; Then, the attention weight matrix for each head is calculated, and the attention matrices of all heads are merged: ， ， in, Used for each data sequence Operation, and Then the attention matrices of each data header are merged into a single attention matrix; The equations for the entire feature extraction process can be simplified as follows: ， ， in, and It is a layer normalization operation. It is a feedforward neural network. The above feature extraction process only involves one layer, and the number of repetitions is the same as the number of layers. The Transformer architecture has 3 layers and 8 heads, with a width of 512.

8. The cross-scene hyperspectral image classification method based on multimodal domain generalization according to claim 1, characterized in that: In step 3.3, the specific formula is as follows: ， ， ， in, It is a learnable matrix. It is a bias term, and the linear layer is responsible for mapping image features to the semantic space.

9. The cross-scene hyperspectral image classification method based on multimodal domain generalization according to claim 1, characterized in that: In step 4.1, the cross-entropy loss The formula is: ， Classification loss of the network The formula is: ， in, yes One-hot encoding of category information and It is a prediction probability. Indicates the number of samples.

10. The cross-scene hyperspectral image classification method based on multimodal domain generalization according to claim 1, characterized in that: In step 4.2, the similarity measurement formula is: ， ， right After squaring, the optimized supervised contrast loss is: ， ， ， in, Corresponding image features, These do not correspond to image features, which are positive and negative features, respectively. and This represents one of the positive and negative samples. , As a contribution weight.

Images