Cross-domain small sample learning hyperspectral image classification method and system based on text perception

Abstract

The invention relates to an image processing technology, in particular to a cross-domain small sample learning hyperspectral image classification method and system based on text perception. The method comprises the following steps: designing a domain-agnostic prior semantic information description method, carrying out secondary classification on category names of a source domain and a target domain, and embedding secondary classification semantic information into a text template of each hyperspectral image pixel; the method comprises the following steps: constructing a hyperspectral image classification model based on a double-branch transformer, and extracting a cross-domain visual spatial spectrum feature and a cross-domain text spatial spectrum feature; designing a text perception spatial-spectral domain self-adaptive strategy; calculating the total loss in the training stage, performing joint training on the classification model and the condition domain discriminator, and defining the total loss of the classification model as the sum of the feature classification comparison loss of the classification model and the loss of the text perception spatial spectral domain adaptive strategy; and classifying the cross-domain hyperspectral image by using the trained classification model. According to the method, the model is generalized to the unseen category of the target domain, the domain offset is reduced, and the generalization performance of the model is improved.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a cross-domain small sample learning hyperspectral image classification method and system based on text perception. Background Art

[0002] Hyperspectral imagery (HSI) has a unique advantage because it can provide rich spectral information of object categories, making it possible to identify objects with high precision. HSI has been widely used in many fields, such as environmental monitoring, resource management, military reconnaissance and urban planning. HSI classification is a key technology in HSI processing, which aims to assign all pixels in HSI to specific categories. However, the complexity and risk of the acquisition environment bring small sample problems to HSI classification.

[0003] In recent years, studies have found that cross-domain information can help solve the problem of limited labeled samples, so the research and development of cross-domain HSI classification has become a hot topic. In cross-domain HSI classification, the source domain contains sufficient labeled samples, while the target domain has fewer labeled samples. In addition, the categories in the source domain are called seen categories, and the categories in the target domain that do not exist in the source domain are called unseen categories. The core idea of ​​cross-domain HSI classification is to train a model that can effectively generalize to small sample labeled data in the target domain by learning the transfer knowledge of the source domain.

[0004] Few-shot learning (FSL) is a typical cross-domain learning method that obtains a general model by constructing cross-domain tasks. Few-shot learning (FSL) is an effective solution for cross-domain hyperspectral image (HSI) classification. The principle behind it is that such a model trained by source domain knowledge can be easily generalized to the target domain with few labeled samples. FSL methods are mainly divided into two categories: meta-learning and metric learning. In terms of HSI processing, current research mainly uses metric learning methods to achieve cross-domain HSI classification. For example, deep few-shot learning (DFSL) and deep cross-domain few-shot learning (DCFSL) are both classic applications of FSL in cross-domain HSI classification, which learn a metric space in the source domain and then use the metric space to classify HSI pixels in the target domain.

[0005] The core goal of few-shot learning is to train a classifier in the source domain so that it can generalize to new categories, i.e., to unseen categories, based on a small number of labeled samples in the target domain. Since text information is considered independent of the data domain, many text-based few-shot learning methods have been proposed in the prior art to improve the generalization performance of the model. Most existing text-based few-shot learning methods first form a set of positive and negative sample pairs. Then, the goal of these methods is to align the features of positive sample pairs in the projection space while clearly distinguishing negative sample pairs. Under this guiding principle, researchers have actively developed a variety of text-based few-shot learning methods to improve the efficiency of cross-scene hyperspectral image (HSI) classification. Although many text-based few-shot learning methods have been proposed in the prior art for remote sensing image scene classification or cross-scene HSI classification, the research on text-based few-shot learning methods in cross-domain HSI classification is still in its early stages. Current methods have not yet considered the use of semantic information to assist the learning of visual features of unseen categories in the target domain.

[0006] Although FSL methods have made significant progress in cross-domain HSI classification, most FSL methods only focus on learning domain-invariant representations at the visual level and ignore the importance of textual information. Textual information can provide rich prior knowledge of objects and help achieve object recognition and classification. In addition, such information is considered to be independent of the data domain. With the rapid development of deep learning and natural language processing techniques, multimodal learning models, especially image-text models, have shown excellent performance in various image processing tasks, including small-sample transfer tasks. For example, the contrastive language-image pre-training model CLIP performs well in small-sample and zero-sample tasks. Recently, the language-aware domain generalization network (LDGnet) has also been successfully applied to HSI classification. However, the application of multimodal learning-based models in cross-domain HSI classification is still in its early stages. Existing methods do not consider unseen categories in the target domain and fail to exploit its language modality to learn domain-invariant visual representations.

[0007] Recently, multimodal learning models, especially image-text models, have been shown to have many advantages in remote sensing image processing. More importantly, text information contains various prior knowledge about ground objects, which is considered to be independent of the data domain. However, existing FSL methods for hyperspectral images ignore the language modality knowledge and fail to utilize it to improve the generalization performance of the model.

[0008] Hyperspectral image classification is a key technology in the field of hyperspectral image processing, and its goal is to classify all pixels in a hyperspectral image into specific categories. Cross-domain information can effectively solve the problem of limited labeled samples. Although cross-domain HSI classification algorithms have made some progress, existing methods still lack the exploration of prior semantic information and have not fully utilized textual knowledge to enhance the domain invariance of the model. Summary of the invention

[0009] In order to solve the technical problems existing in the prior art, the present invention proposes a text-aware cross-domain small sample learning hyperspectral image classification method and system, designs a domain-independent prior semantic information description method to effectively represent intra-class relations and cross-class relations; the image-text model adopted can extract domain-invariant spatial-spectral features to effectively extract cross-domain visual and textual spatial-spectral features, and use these features to achieve vision and language alignment; a text-aware spatial-spectral domain adaptation strategy is adopted to generalize the model to unseen categories in the target domain, reduce domain shift, and further improve the generalization performance of the model.

[0010] In an embodiment of the present invention, a text-aware cross-domain small sample learning hyperspectral image classification method specifically includes the following steps:

[0011] S1. Design a domain-agnostic prior semantic information description method. By performing secondary classification on the category names of the source domain and the target domain, the secondary classification semantic information of the category names is embedded into the text template of each hyperspectral image pixel, so that the text template combines the intra-class and inter-class relationships to adapt to different data distributions.

[0012] S2. Build a hyperspectral image classification model based on dual-branch transformer to achieve vision and language alignment, obtain domain-invariant visual representation, and extract cross-domain visual spatial-spectral features and textual spatial-spectral features;

[0013] S3. Design a text-aware spatial-spectral domain adaptation strategy to improve the generalization ability of the dual-branch transformer-based hyperspectral image classification model;

[0014] S4. Calculate the total loss in the training phase. The hyperspectral image classification model based on the dual-branch transformer and the conditional domain discriminator are jointly trained. The total loss L of the hyperspectral image classification model is total It is defined as the sum of the feature classification contrast loss of the hyperspectral image classification model and the loss of the text-aware spatial-spectral domain adaptation strategy;

[0015] S5. Apply the trained dual-branch transformer-based hyperspectral image classification model to classify cross-domain hyperspectral images.

[0016] The embodiment of the present invention also provides a text-aware cross-domain small sample learning hyperspectral image classification system, which is implemented based on the aforementioned hyperspectral image classification method; and includes the following modules:

[0017] The information description module is used to describe domain-agnostic prior semantic information. By performing secondary classification on the category names of the source domain and the target domain, the secondary classification semantic information of the category names is embedded into the text template of each hyperspectral image pixel, so that the text template combines the intra-class and inter-class relationships to adapt to different data distributions.

[0018] A dual-branch transformer-based hyperspectral image classification model is used to achieve vision and language alignment, obtain domain-invariant visual representations, and extract cross-domain visual spatial-spectral features and textual spatial-spectral features;

[0019] A text-aware spatial-spectral domain adaptation strategy module, which is used to improve the generalization ability of the dual-branch transformer-based hyperspectral image classification model;

[0020] The total loss calculation module is based on the joint training of the hyperspectral image classification model and the conditional domain discriminator based on the dual-branch transformer, and the total loss L of the hyperspectral image classification model is calculated. total It is defined as the sum of the feature classification contrast loss of the hyperspectral image classification model and the loss of the text-aware spatial-spectral domain adaptation strategy;

[0021] The classification module applies the trained dual-branch transformer-based hyperspectral image classification model to classify cross-domain hyperspectral images.

[0022] From the above technical solutions, it can be seen that the present invention proposes a novel text-aware small sample learning (TAFSL) method for cross-domain hyperspectral image classification. Compared with the prior art, the technical effects achieved by the present invention include the following:

[0023] 1. This paper is the first study to use image-text model to process unseen categories in target domain in cross-domain HSI classification, making full use of the advantages of language modality to assist visual representation learning; a domain-independent prior semantic information description method is designed to effectively represent intra-class and cross-class relationships.

[0024] 2. The present invention adopts a two-stream transformer-based model as the image-text model, which can extract domain-invariant spatial-spectral features, that is, spatial-spectral features for capturing vision and text across domains, and use these features to achieve alignment of vision and language to obtain domain-invariant visual representation.

[0025] 3. The present invention proposes a text-aware spatial-spectral domain adaptation strategy to generalize the model to unseen categories in the target domain. By combining adversarial learning, the domain shift is alleviated. This strategy enables the model to generalize to unseen categories in the target domain, further improving the generalization performance of the model.

[0026] 4. The experimental results of cross-domain HSI classification on four hyperspectral image datasets show that the TAFSL proposed in this paper is superior to the most advanced existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The present invention is a flowchart of a method for classifying hyperspectral images based on text-aware cross-domain small sample learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0029] Example

[0030] This embodiment provides a text-aware cross-domain small sample learning hyperspectral image classification method, which first gives the domain prior semantic information of each class to effectively describe the intra-class and inter-class relationships; then, a hyperspectral image classification model based on a dual-branch transformer is used to extract cross-domain visual and textual spatial-spectral features, and visual language alignment technology is used to enhance the domain invariance of visual representation to obtain a domain-invariant visual representation; a text-aware spatial-spectral domain adaptation strategy is designed to further improve the generalization ability of the hyperspectral image classification model based on a dual-branch transformer.

[0031] like Figure 1 As shown, the hyperspectral image classification method of this embodiment specifically includes the following steps:

[0032] S1. Design a domain-agnostic prior semantic information description method. By performing secondary classification on the category names of the source domain and the target domain, the secondary classification semantic information of the category names is embedded into the text template of each hyperspectral image pixel, so that the text template can combine intra-class and inter-class relationships to adapt to different data distributions.

[0033] In previous image-text models for hyperspectral images (HSI), the category name is mainly used as the semantic information keyword for each HSI pixel. Specifically, these methods first build a coarse-grained text template expressed as "hyperspectral image of <category name>", and then provide some fine-grained text information containing adjacency relations and other prior knowledge, such as "stressed grass is located next to the road". However, this fine-grained prior semantic information description method is challenging to adapt to the distribution of unseen categories in different target domains. In addition, using only coarse-grained prior semantic information cannot express the relationship between categories, nor can it extract domain invariance of text information from source and target domains.

[0034] To this end, this embodiment designs a domain-agnostic prior semantic information description method, which performs secondary classification on the category names of the source domain and the target domain, so that the text template can combine intra-class and inter-class (i.e., cross-class) relationships to adapt to different data distributions.

[0035] Specifically, this step mainly uses the text understanding and generation capabilities of the large language model (LLM) to enhance the association between seen categories and unseen categories. First, all the category names of the source domain and the target domain are input into the large language model LLM at one time; the large language model LLM divides the input category names into three categories based on prior geographical knowledge: "natural", "artificial" and "agricultural" features, among which artificial features are also called man-made features. Then, the secondary classification semantic information of the category name is embedded into the text template of each hyperspectral image HSI pixel, so that the text template effectively utilizes the intra-class and inter-class relationships. Taking "grassland" as an example, its corresponding text template can be expressed as "hyperspectral image of grassland [natural class]".

[0036] In the following sections, the text representation of each pixel (whether from the source or target domain) is defined as X test , which is the basis for text spatial spectral feature extraction and visual-linguistic domain discriminator.

[0037] S2. Build a hyperspectral image classification model based on a dual-branch transformer to achieve vision and language alignment, obtain domain-invariant visual representation, and extract cross-domain visual spatial-spectral features and textual spatial-spectral features.

[0038] In the field of computer vision, traditional deep learning methods mainly use CNN modules to extract visual features. CNN modules can provide local receptive fields and have the advantage of parameter sharing. In recent years, transformers have shown extraordinary capabilities in the field of text processing due to their ability to capture long-range dependencies and contextual relationships in data. Recent achievements in applying transformers to image processing include the development of visual transformers, which can provide better performance than CNN-based models on multiple benchmark datasets. These advances highlight the potential of transformers to revolutionize image processing by providing a more comprehensive and context-aware approach to understanding visual data. However, existing image-text models only use transformers for text feature extraction, ignoring the advantages of transformers in global image context understanding.

[0039] Therefore, in order to effectively capture the visual spatial-spectral features and textual spatial-spectral features of hyperspectral images, this embodiment constructs a novel hyperspectral image classification model based on dual-branch transformer, namely, a dual-stream transformer architecture model, which can be used to achieve the alignment of vision and language, thereby obtaining a domain-invariant visual representation. The hyperspectral image classification model based on dual-branch transformer mainly includes a visual branch transformer encoder and a text branch transformer encoder.

[0040] In the vision branch transformer encoder, a 3D local block is first segmented from the hyperspectral image to represent each hyperspectral image pixel; then, each 3D local block is further divided into a center sequence X cen and the neighborhood sequence X nbr Specifically, let X 3D is the three-dimensional local block of the hyperspectral image, then the three-dimensional local block X 3D The partition function It can be expressed as follows:

[0041]

[0042] Where ω represents the segmentation rate. Let the three-dimensional local block X 3D The size of is x1×y1×λ1, then the central sequence X cen and the neighborhood sequence X nbr The sizes are x2×λ1 and x3×λ1 respectively, where x2+x3=x1×y1.

[0043] Then, the obtained center sequence X cen and the neighborhood sequence X nbrAs input to the transformer encoder of the vision branch.

[0044] The visual branch transformer encoder mainly includes a multi-head attention module and a feed-forward network (FFN). As for the multi-head attention module, it includes three inputs: query (i.e. Q), key (i.e. K), and value (i.e. V), as shown below:

[0045]

[0046] Among them, d k represents the scaling factor, whose size is based on the dimension size of the query. As can be seen from formula (2), the attention mechanism of the multi-head attention module is realized by calculating the dot product between the query and the keyword, and the result of the dot product is further used as the weight of the value through the softmax function and the scaling factor. cen As the query, the neighborhood sequence X nbr As the key and value; by calculating the difference distribution of the local neighborhood, we can make full use of the neighborhood correlation and obtain the three inputs of the multi-head attention module, which are as follows:

[0047]

[0048] in, represents the weight matrix of the query (Q) for computing visual features, represents the weight matrix of the key (K) for computing visual features, Represents the weight matrix for calculating the value (V) of the visual feature.

[0049] Therefore, the single-head attention score with neighborhood correlation can be calculated as follows:

[0050] head i =Attention(Q′ i ,K′ i ,V′ i ) (6)

[0051] Multi-head attention is composed of single-head attention in series. The calculation of multi-head attention score is as follows:

[0052] MultiHead(Q′,K′,V′)=Concat(head1,head2,…,head h ) 0 (7)

[0053] Next, the multi-attention scores are combined with the center sequence X cenThe local visual spatial spectral features are input into the feedforward network FFN to calculate the local visual spatial spectral features (i.e., the local visual spatial spectral features of the three-dimensional local block of the hyperspectral image). The feedforward network FFN includes two fully connected layers, the activation function of the first fully connected layer is ReLU, and the second fully connected layer is a linear transformation layer. In this embodiment, the local visual spatial spectral features are calculated by the following feature extraction function:

[0054]

[0055] Where w1, b1 are the weights and biases of the first fully connected layer, w2, b2 are the weights and biases of the second fully connected layer; X nbr is the neighborhood sequence, MultiHead img Score for multi-head attention.

[0056] Finally, the local visual spatial spectral feature in the feature metric is one-dimensional, so the following flattening transformation is required to obtain the visual spatial spectral feature vector of the three-dimensional local block of the hyperspectral image:

[0057] f image (X 3D )=faltten(F image ) (9)

[0058] Among them, F image is the local visual spatial spectrum feature of the three-dimensional local block of the hyperspectral image calculated by formula (8), f image (X 3D ) is the visual spatial-spectral feature vector of the three-dimensional local block of the hyperspectral image after flattening transformation.

[0059] As far as the text branch transformer encoder is concerned, some components are similar to those of the vision branch transformer encoder, but there are still some differences, which are explained as follows:

[0060] First, the text is tokenized and then encoded using methods such as byte pair encoding (BPE) to convert into a vector for the text branch transformer encoder model, which is defined as follows:

[0061] X emb =Embedding(X text ) (10)

[0062] Among them, X emb is the encoded text vector. Then, add position encoding to the encoded text vector:

[0063] X e ' mb =X emb +position(Xemb ) (11)

[0064] For the text attention mechanism, their Q′ text ,K′ text and V′ text All are text vectors X after adding position encoding e ' mb The calculation method obtained by linear mapping is similar to formula (3), (4) and (5).

[0065]

[0066] in, represents the weight matrix of the query (Q) for calculating text features, represents the weight matrix of the key (K) for calculating text features, Represents the weight matrix for calculating the value (V) of the text feature.

[0067] In addition, their multi-head attention calculation method is the same as formula (6) and (7), which can be expressed as follows:

[0068] MultHead text =MultiHead(Q′ text ,K′ text ,V′ text ) (15)

[0069] Different from the visual branch transformer encoder, the text branch transformer encoder is mainly composed of multiple residual blocks (ResBlocks). The calculation method of a residual block is as follows:

[0070] resblock(X text )=Ln(FFN(Ln(Q′ text +MuliHead text ))) (16)

[0071] Where Ln() represents layer normalization. Then, the text space spectrum feature vector f is obtained from l stacked residual blocks. text (X text ):

[0072] f text (X text )=Stack(resblock1,resblock2,…,resblock l ) (17)

[0074] In addition, the hyperspectral image classification model of the dual-branch transformer in this embodiment can be updated in two stages: FSL and visual language alignment.

[0075] In the FSL stage, the data of each batch will be divided into a support set and a query set:

[0076]

[0077] Among them, x i Indicates the support sample, y i represents the label of the supporting sample, S s represents the support set of K samples labeled with category C; x j represents the query sample, y j represents the label of the query sample, Q s represents a query set of K samples labeled with category C.

[0078] In the visual-language alignment phase, each batch of data will be displayed as the sum of the support set and the query set:

[0079]

[0080] Among them, Img s represents all hyperspectral images of C categories in the current batch, each category contains K+N samples, x u represents the image sample of the u-th category, y u Represents the label corresponding to the u-th category image sample.

[0081] In the training phase, this embodiment will calculate the distance metric between the text set and the image set, and the distance metric between the support set and the query set, respectively, to obtain the category probability distribution. According to the calculation results of the distance metric, the parameters of each network will be updated through the softmax function. The category probability distribution based on the distance metric between the support set and the query set can be defined as follows:

[0082]

[0083] Similarly, the category probability distribution based on the distance metric between the text set and the image set can be expressed as follows:

[0084]

[0085] where d(·) represents the Euclidean distance metric function, C K represents the kth th categories, text k is the kth th The feature encoding of each category, represents the visual branch transformer encoder with parameter λ, represents the text branch transformer encoder with parameter β, xj is from the sample query set Q s Or hyperspectral image Img s The query sample, y j is the query sample x j .

[0086] The category probability distribution based on the distance metric between the support set and the query set, and the category probability distribution based on the distance metric between the text set and the image set are used to further calculate the loss.

[0087] According to the negative log probability that the query sample has the true label, the support-query sample classification loss (also called FSL loss) of each episode can be expressed as:

[0088]

[0089] Similarly, the image-text contrast loss calculated by visual-language alignment can be calculated as follows:

[0090]

[0091] Finally, the feature classification contrast loss L of the hyperspectral image classification model based on the dual-branch transformer is feat It is the sum of the support-query sample classification loss and the image-text contrast loss:

[0092]

[0093] In summary, in the process of vision and language registration and alignment, text features are considered as a prototype, and the resulting cross-entropy loss is combined with the FSL loss to represent the multiple alignment of spatial-spectral features in the semantic space.

[0094] S3. Design a text-aware spatial-spectral domain adaptation strategy to improve the generalization ability of the hyperspectral image classification model based on the dual-branch transformer.

[0095] This embodiment defines the visual domain and the language domain, the source domain and the target domain. The designed text-aware spatial-spectral domain adaptation strategy includes a confrontation strategy between the source domain and the target domain, and a confrontation strategy between the visual domain and the language domain.

[0096] In order to enhance the perception of prior semantic information, the existing data distribution is adjusted through the spatial-spectral domain adaptation strategy, which helps to reduce the domain shift between the source and target domains, as well as the domain shift between the visual and language domains. The adversarial strategy between the source and target domains is also used to improve the generalization ability of the target domain, while the adversarial strategy between the visual and language domains adjusts the distribution of visual spatial-spectral features and the distribution of textual spatial-spectral features, using the domain independence of text to help the visual branch transformer encoder improve its generalization ability.

[0097] S31, through the conditional domain discriminator D on the source distribution and the target distribution s-t , reducing the offset of the source and target domain feature extraction by the dual-branch transformer-based hyperspectral image classification model to set up an adversarial strategy between the source and target domains.

[0098] This embodiment proposes a method for source distribution P s (x) and the target distribution p t The conditional domain discriminator D of (x) s-t , which is used for adversarial training with a dual-branch transformer-based hyperspectral image classification model (also called a dual-branch transformer feature extraction model) to reduce the offset of the dual-branch transformer feature extraction model in the source domain and target domain features.

[0099] In this step, the data of the source domain and the target domain are first taken as input, and the empty spectrum feature vector from the source domain dataset is obtained through the dual-branch transformer feature extraction model. And the prediction results of the classification labels of the input source domain dataset At the same time, the empty spectrum feature vector from the target domain dataset is also obtained And the prediction results of the classification labels of the input target domain dataset Among them, the prediction results of the classification labels By the empty spectrum feature vector The input classifier G (a learnable linear mapping layer) is obtained. Then, the prediction results of the empty spectrum feature vector and the classification label are used as the conditional domain discriminator D s-t Input.

[0100] The prediction result g of the classification label contains discriminative information, which can condition the adversarial adaptability of the feature representation. The empty spectrum feature vector f (specifically including the empty spectrum feature vector from the source domain dataset Empty spectral feature vector from target domain dataset ) and the prediction results of the classification labels g (specifically including the prediction results of the classification labels of the source domain dataset Prediction results of the classification labels of the target domain dataset ) can be simultaneously in the conditional domain discriminator D s-t On modeling.

[0101] In this step, the conditional domain discriminator D s-t The input of is the data of the source domain and the target domain, so it is also called the source domain target domain discriminator. The domain adversarial loss of the source domain target domain discriminator is calculated as follows:

[0102]

[0103] in Represents a random variable According to the source domain data distribution P s (x) Expected value calculation during sampling, Represents a random variable According to the target domain data distribution P t (x) Expected calculation during sampling; Receive the empty spectrum feature vector of the source domain data And the prediction results of the classification labels As input, the computation is done on samples from the source domain The probability of Receive the empty spectrum feature vector of the target domain data And the classification results As input, a sample from the target domain During the training process, the conditional domain discriminator D s-t Minimize the above domain adversarial function, that is, the training goal of the conditional domain discriminator is to judge the correct result as much as possible (correctly judge whether the input data comes from the source domain or the target domain); and the two-branch transformer feature extraction model and classifier G maximize it, that is, let the conditional domain discriminator D s-t Try to judge the wrong results as much as possible. Through this adversarial method, the visual branch transformer encoder can be trained to extract approximate representations in different scenes.

[0104] When the empty spectrum feature vector f and the prediction result g of the classification label are simultaneously modeled as the domain judgment information model h(f,g), multilinear mapping is selected for modeling. However, multilinear mapping requires the outer product of f and g, which will lead to dimensionality explosion. Therefore, the following adjustment strategy is adopted to avoid dimensionality explosion:

[0105]

[0106] in, represents the element-wise product of f and g, d f and d g are the dimensions of f and g respectively, d represents the configurable dimension size, which is set to 1024 in this example, ⊙ represents the dot product, and represents two random matrices, which are sampled only once and remain fixed during the training phase, and d<<d f ×d g If the dimension of the linear mapping is larger than the maximum number of units in the deep network, 1024, then use The domain-invariant encoding feature loss from the source domain to the target domain is recorded as:

[0107]

[0108] in The domain decision information model representing the source domain, A domain decision information model representing the target domain, The source domain determination information model is represented by a model that prevents dimensionality explosion. The target domain determination information model is derived from formula (29) through a model that prevents dimensionality explosion.

[0109] At this point, the adversarial strategy setting between the source domain and the target domain is completed.

[0110] S32, through the conditional domain discriminator D on the visual domain data and the text domain data img-text , reducing the offset of visual feature and text feature extraction in the dual-branch transformer-based hyperspectral image classification model to set up an adversarial strategy between the visual domain and the language domain.

[0111] This embodiment proposes a conditional domain discriminator D for visual domain data and text domain data. img-text , in order to better utilize language information to improve the generalization ability of the transformer branch encoder.

[0112] In this step, the visual domain data and text domain data are first taken as input, and the spatial spectrum feature vector f from the visual domain data is obtained through the dual-branch transformer feature extraction model. img And the prediction result g of the classification label of the input visual domain dataset img At the same time, the empty spectrum feature vector f from the text domain data is also obtained text And the prediction result g of the classification label of the input text domain data text ; Among them, the prediction result of the classification label g img By the empty spectrum characteristic vector f text The input classifier G (a learnable linear mapping layer) is obtained. Then, the prediction results of the empty spectrum feature vector and the classification label are used as the conditional domain discriminator D img-text Input.

[0113] In this step, the input of the conditional domain discriminator is the data of the visual domain and the text domain, so it is also called the visual language domain discriminator. The loss of the visual language domain discriminator is calculated as follows:

[0114]

[0115] in Represents the random variable Img i According to the image data distribution P img(x) Mean calculation during sampling, Represents the random variable text j According to the text data distribution P text (x) Mean calculation during sampling; D img-text (f img ,g img ) receives the spatial spectral feature vector f of the visual domain data img And the prediction result g of the classification label img , calculates the probability from the visual domain; while 1-D img-text (f text ,g text ) Receive the empty spectrum feature vector f of the text domain data text And the prediction result g of the classification label text , computing the probability from the language domain.

[0116] Similarly, the spatial spectral feature vector f from the visual domain data img And the prediction result g of the classification label img Modeled to be resistant to dimensionality explosion The empty spectrum feature vector f from the text domain data text And the prediction result g of the classification label text Modeled to be resistant to dimensionality explosion The loss of the adversarial strategy from the visual domain to the language domain is expressed as follows:

[0117]

[0118] At this point, the adversarial strategy between the visual domain and the language domain is set up. Through this adversarial strategy, a dual-branch transformer encoder can be trained to extract domain-invariant feature vector representations under different modalities.

[0119] S33. Couple the adversarial strategy between the source domain and the target domain and the adversarial strategy between the visual domain and the language domain to obtain a text-aware spatial-spectral domain adaptation strategy.

[0120] The loss L for text-aware spatial-spectral domain adaptation adapt is the conditional domain discriminator D s-t Domain adversarial loss and conditional domain discriminator D img-text The weighted sum of the adversarial losses is expressed as follows:

[0121] L adapt =L s-t +λL img-text (33)

[0122] The above conditional domain discriminator D s-t , Conditional Domain Discriminator D img-textThe structure of is the same, but the input data is different. In the network setting of the corresponding conditional domain discriminator of the above strategy, after stacking multiple perceptron layers, multi-layer linear mapping is applied. The first four operation layers each consist of a fully connected layer, a ReLU activation layer, and a dropout layer; the last layer is a fully connected layer with a softmax activation function to determine the domain dependency of the input.

[0123] S4. Calculate the total loss during the training phase.

[0124] In TAFSL, the dual-branch transformer-based hyperspectral image classification model and the conditional domain discriminator are jointly trained. Combined with the above loss function, the total loss L of the dual-branch transformer-based hyperspectral image classification model is total Defined as:

[0125] L total =L feat +L adapt (34)

[0126] Among them, L feat is the feature classification contrast loss of the dual-branch transformer-based hyperspectral image classification model calculated in step S2.

[0127] S5. Apply the trained dual-branch transformer-based hyperspectral image classification model to classify cross-domain hyperspectral images.

[0128] In this embodiment, the hyperspectral image classification model is first switched to the evaluation mode, in which the behaviors specific to the training phase are disabled to ensure the stability and consistency of the performance measurement.

[0129] The 3D local block of the hyperspectral image is input into the visual branch transformer encoder to obtain the spatial spectrum feature vector. The minimum and maximum values ​​of the spatial spectrum feature vector are calculated, and then these extreme values ​​are used for linear normalization to map the spatial spectrum feature vector to the range of [0,1].

[0130] Next, a KNN (K-Nearest Neighbors) classifier is initialized and trained using the normalized spatial spectral feature vector and the corresponding training labels. The classifier provides category predictions for the input test data using the nearest neighbor strategy. For the test data, the test features are first generated by the visual branch transformer encoder and normalized based on the extreme values ​​of the KNN classifier training phase; the KNN classifier is used to predict the category of the test data, and the predicted results are compared with the true labels to generate a reward value sequence.

[0131] Finally, the reward values ​​of the correct samples are accumulated and the average accuracy is calculated based on the total number of test samples.

[0132] Based on the same inventive concept, this embodiment also provides a text-aware cross-domain small sample learning hyperspectral image classification system, which is implemented based on the aforementioned hyperspectral image classification method; specifically includes the following modules:

[0133] The information description module is used to describe domain-agnostic prior semantic information. By performing secondary classification on the category names of the source domain and the target domain, the secondary classification semantic information of the category names is embedded into the text template of each hyperspectral image pixel, so that the text template combines the intra-class and inter-class relationships to adapt to different data distributions.

[0134] A dual-branch transformer-based hyperspectral image classification model is used to achieve vision and language alignment, obtain domain-invariant visual representations, and extract cross-domain visual spatial-spectral features and textual spatial-spectral features;

[0135] A text-aware spatial-spectral domain adaptation strategy module, which is used to improve the generalization ability of the dual-branch transformer-based hyperspectral image classification model;

[0136] The total loss calculation module is based on the joint training of the hyperspectral image classification model and the conditional domain discriminator based on the dual-branch transformer, and the total loss L of the hyperspectral image classification model is calculated. total It is defined as the sum of the feature classification contrast loss of the hyperspectral image classification model and the loss of the text-aware spatial-spectral domain adaptation strategy;

[0137] The classification module applies the trained dual-branch transformer-based hyperspectral image classification model to classify cross-domain hyperspectral images.

[0138] The above modules are respectively used to implement the various steps of the aforementioned hyperspectral image classification method. The detailed implementation process refers to steps S1-S5 and will not be repeated here.

[0139] The experiments were conducted on a computer equipped with a 2.1 GHz Intel Xeon Silver 4130 processor, 32 GB DDR4 memory, and an NVIDIA GeForce RTX 4090 graphics processing unit (GPU). The open source PyTorch framework was used for training and testing experiments.

[0140] In order to verify the effectiveness of the method of this embodiment, the TAFSL method is compared with four supervised methods (support vector machine (SVM), spectral-spatial residual network (SSRN), etc.), and six FSL methods (DFSL+SVM, DCFSL, CMFSL, Gia-FSL, MRLF and HyMuT). As for the FSL method, a cross-domain diachronic strategy is adopted in the training phase. Specifically, 200 labeled samples are selected from the source domain for each category to obtain transferable knowledge, and a limited number of labeled samples in the target domain are used for training. The remaining samples of the target domain dataset are reserved for testing.

[0141] In the experiment, the spatial size of the three-dimensional local tensor of the star instrument is defined as 9×9, and the spectrum size e of S0 is set to 128. The enlightened initialization method is used for parameter initialization. In addition, when extracting text features, the text encoder is initialized as a 33 million parameter, three-layer, 512-wide model with eight attention heads. It uses a transformer architecture similar to CLIP, uses lowercase byte pair encoding (BPE) to represent text, and has a vocabulary of 49,152. To maintain computational efficiency, the maximum sequence length is capped at 76. This example uses the overall accuracy, average accuracy (Average Accuracy), and Kappa coefficient (Kappa coefficient) to evaluate the classification performance of all methods.

[0142] Hyperparameter experiment: This embodiment performs parameter sensitivity analysis to evaluate the sensitivity of TAFSL to changes in the regularization parameter λ of the visual language domain adaptive discriminator. The basic regularization parameter λ is regarded as an adjustable hyperparameter and is selected from the set of {0.9, 0.95, 1.0, 1.05, 1.1}. After the weight λ is adjusted, the gradient of the loss function is used to estimate the model weight hyperparameters during the descent process. Table 1 lists the classification results corresponding to different basic learning rates in the four datasets. For these four datasets, a weight λ of 1.0 is the most ideal.

[0143] Table 1 Overall accuracy of the classification parameters of the four datasets under 5-SHOT

[0144]

[0145] Ablation experiment: The domain discriminator is a key component of TAFSL. An important strategy to enhance the domain generalization ability of the model is to use a combination of enhanced text and visual language domain discriminators. The elimination analysis evaluates the important contribution of TAFSL components by eliminating each component from the entire framework. There are three variants of the elimination analysis:

[0146] (1) "TAFSL (text removed)": removes the visual-language alignment and visual-language domain adaptation discriminators from FAFSL;

[0147] (2) "TAFSL (Removing Image-Text Modality Discriminator)": remove the visual-language domain adaptation discriminator from FAFSL;

[0148] (3) "TAFSL (remove prior knowledge text)": remove the prior text information from FAFSL used to enhance the domain discrimination ability of the domain discriminator.

[0149] Table 2

[0150]

[0151] As can be seen from Table 2, the TAFSL method of this embodiment significantly outperforms existing variants, marking a significant improvement. Specifically, the integration of the visual language alignment architecture improves TAFSL (removing the image-text modality discriminator) by about 3% compared to the TAFSL (removing text) baseline. This highlights the positive impact of language patterns in enhancing visual representation learning. In addition, compared with TAFSL (removing the image-text modality discriminator), TAFSL is improved by more than 0.6%, which shows that the visual language domain discrimination strategy effectively exploits the domain agnosticism of text, thereby enhancing the cross-domain generalization ability of the visual branch transformer model. In addition, compared with TAFSL (removing prior knowledge text), TAFSL is improved by nearly 0.5%, which shows that the land cover type classification effectively establishes intra-class and inter-class relationships, thereby improving the cross-domain generalization ability of the dual-branch transformer model.

[0152] The TAFSL proposed in this embodiment combines visual and language modalities together, and does so with a small amount of learning; a dual-branch transformer model is used to simultaneously extract visual and language features; the text enhanced by secondary classification of category names is regarded as domain-independent shared information, aiming to reduce the domain shift between the source and target domains. Specifically, the text features obtained by the text encoder are regarded as prototypes, and the distance between the query and the text is calculated. By minimizing the distance metric, the gap between visual and language features is narrowed. In addition, this embodiment also designs the adaptation of the source-target domain and the visual-language domain, and uses adversarial learning to obtain cross-domain invariant visual feature representations. Finally, comprehensive experiments on four datasets verify the effectiveness of the proposed TAFSL in domain generalization.

[0153] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A text-aware cross-domain small sample learning hyperspectral image classification method, characterized in that: The following steps are involved: S1. Design a domain-agnostic prior semantic information description method. By performing secondary classification on the category names of the source domain and the target domain, the secondary classification semantic information of the category names is embedded into the text template of each hyperspectral image pixel, so that the text template combines the intra-class and inter-class relationships to adapt to different data distributions. S2. Build a hyperspectral image classification model based on dual-branch transformer to achieve vision and language alignment, obtain domain-invariant visual representation, and extract cross-domain visual spatial-spectral features and textual spatial-spectral features; S3. Design a text-aware spatial-spectral domain adaptation strategy to improve the generalization ability of the dual-branch transformer-based hyperspectral image classification model; S4. Calculate the total loss in the training phase. Perform joint training on the hyperspectral image classification model based on the dual-branch transformer and the conditional domain discriminator. total It is defined as the sum of the feature classification contrast loss of the hyperspectral image classification model and the loss of the text-aware spatial-spectral domain adaptation strategy; S5. Apply the trained dual-branch transformer-based hyperspectral image classification model to classify cross-domain hyperspectral images.

2. The hyperspectral image classification method according to claim 1, characterized in that: Step S1 utilizes the text understanding and generation capabilities of the large language model (LLM) to enhance the association between seen and unseen categories.

3. The hyperspectral image classification method according to claim 2, characterized in that: Step S1 includes: All category names of the source domain and the target domain are input into the large language model LLM at one time; The large language model (LLM) classifies the input category names into "natural", "artificial" and "agricultural" features based on prior geographical knowledge; The secondary classification semantic information of the category name is embedded into the text template of each hyperspectral image pixel, so that the text template can effectively utilize the intra-class and inter-class relations.

4. The hyperspectral image classification method according to claim 1, characterized in that: The dual-branch transformer-based hyperspectral image classification model constructed in step S2 includes a visual branch transformer encoder and a text branch transformer encoder; The visual branch transformer encoder segments a 3D local block from the hyperspectral image to represent each hyperspectral image pixel; each 3D local block is further divided into a center sequence X cen and the neighborhood sequence X nbr , as the input of the visual branch transformer encoder; The text branch transformer encoder tokenizes the text, encodes the text to convert it into a vector; and adds positional encoding to the encoded text vector.

5. The hyperspectral image classification method according to claim 4, characterized in that: The visual branch transformer encoder includes a multi-head attention module and a feed-forward network FFN; Select the center sequence X cen For query, the neighborhood sequence X nbr are keys and values; by calculating the difference distribution of the local neighborhood and using the neighborhood correlation, the three inputs of the multi-head attention module are obtained; Combine the multi-attention scores with the center sequence X cen The feed-forward network FFN is input to calculate the local visual spatial-spectral features of the 3D local patches of the hyperspectral image.

6. The hyperspectral image classification method according to claim 5, characterized in that: The feed-forward network FFN consists of two fully connected layers, the activation function of the first fully connected layer is ReLU, and the second fully connected layer is a linear transformation layer; The local visual spatial spectrum features are calculated by the following feature extraction function: F(X 3D )=FFN(X nbr +MultiHead img ) =max{0,(((X nbr +MultiHead img )w1+b1)w2+b2)} Where w1, b1 are the weights and biases of the first fully connected layer, w2, b2 are the weights and biases of the second fully connected layer; X nbr is the neighborhood sequence, MultiHead img Score for multi-head attention.

7. The hyperspectral image classification method according to claim 5, characterized in that: Flatten the local visual spatial-spectral features to obtain the visual spatial-spectral feature vector f of the three-dimensional local block of the hyperspectral image image (X 3D ): f image (X 3D )=faltten(F image ) Among them, F image It is the local visual spatial spectrum feature.

8. The hyperspectral image classification method according to claim 1, characterized in that: The dual-branch transformer hyperspectral image classification model is updated in two stages: FSL and visual language alignment. In the FSL stage, the data of each batch will be divided into a support set and a query set; Among them, x i Indicates the support sample, y i represents the label of the supporting sample, S s represents the support set of K samples labeled with category C; x j represents the query sample, y j represents the label of the query sample, Q s represents a query set of K samples labeled with category C; In the visual-language alignment phase, each batch of data will be displayed as the sum of the support set and the query set: Among them, Img s represents all hyperspectral images of C categories in the current batch, each category contains K+N samples, x u represents the image sample of the u-th category, y u Represents the label corresponding to the u-th category image sample.

9. The hyperspectral image classification method according to claim 1, characterized in that: The text-aware spatial-spectral domain adaptation strategy designed in step S3 includes a confrontation strategy between the source domain and the target domain, and a confrontation strategy between the visual domain and the language domain; Step S3 includes: S31, through the conditional domain discriminator D on the source distribution and the target distribution s-t , reduce the offset of the source and target domain feature extraction of the dual-branch transformer-based hyperspectral image classification model to set up an adversarial strategy between the source and target domains; The data of the source domain and the target domain are taken as input, and the empty spectrum feature vector from the source domain dataset is obtained through the dual-branch transformer feature extraction model. And the prediction results of the classification labels of the input source domain dataset At the same time, the empty spectrum feature vector from the target domain dataset is also obtained And the prediction results of the classification labels of the input target domain dataset Among them, the prediction results of the classification labels By the empty spectrum feature vector Input classifier G to obtain; then, the prediction results of the empty spectrum feature vector and the classification label are used as the conditional domain discriminator D s-t Input; S32, through the conditional domain discriminator D on the visual domain data and the text domain data img-text , reduce the offset of visual and text feature extraction in the dual-branch transformer-based hyperspectral image classification model to set up an adversarial strategy between the visual domain and the language domain; Taking the visual domain data and text domain data as input, the double-branch transformer feature extraction model is used to obtain the spatial spectrum feature vector f from the visual domain data. img And the prediction result g of the classification label of the input visual domain dataset img At the same time, the empty spectrum feature vector f from the text domain data is also obtained text And the prediction result g of the classification label of the input text domain data text ; Among them, the prediction result of the classification label g img By the empty spectrum characteristic vector f text Input classifier G to obtain; then, the prediction results of the empty spectrum feature vector and the classification label are used as the conditional domain discriminator D img-text Input; S33, coupling the adversarial strategy between the source domain and the target domain and the adversarial strategy between the visual domain and the language domain, to obtain a text-aware spatial-spectral domain adaptation strategy; s-t Domain adversarial loss and conditional domain discriminator D img-text The weighted sum of the adversarial losses is used as the loss for text-aware spatial-spectral domain adaptation.

10. A text-aware cross-domain small sample learning hyperspectral image classification system, characterized in that: This is achieved based on the hyperspectral image classification method described in any one of claims 1 to 9; Includes the following modules: The information description module is used to describe domain-agnostic prior semantic information. By performing secondary classification on the category names of the source domain and the target domain, the secondary classification semantic information of the category names is embedded into the text template of each hyperspectral image pixel, so that the text template combines the intra-class and inter-class relationships to adapt to different data distributions. A dual-branch transformer-based hyperspectral image classification model is used to achieve vision and language alignment, obtain domain-invariant visual representations, and extract cross-domain visual spatial-spectral features and textual spatial-spectral features; A text-aware spatial-spectral domain adaptation strategy module, which is used to improve the generalization ability of the dual-branch transformer-based hyperspectral image classification model; The total loss calculation module is based on the joint training of the hyperspectral image classification model and the conditional domain discriminator based on the dual-branch transformer, and the total loss L of the hyperspectral image classification model is calculated. total It is defined as the sum of the feature classification contrast loss of the hyperspectral image classification model and the loss of the text-aware spatial-spectral domain adaptation strategy; The classification module applies the trained dual-branch transformer-based hyperspectral image classification model to classify cross-domain hyperspectral images.

Images