Small sample domain generalization hyperspectral image classification method based on cycle prompt and related device

Abstract

The invention belongs to the technical field of image classification, and discloses a small sample domain generalization hyperspectral image classification method based on cycle prompt and a related device. The small sample domain generalization hyperspectral image classification method based on the loop prompt comprises the steps of constructing and training a network composed of a mapping layer and based on a Mangban converter to form a hyperspectral image classification network, constructing a composite source domain loss function and training the hyperspectral image classification network to obtain a trained hyperspectral image classification network, constructing a target domain classification loss function, freezing a network based on a Mangban converter in the trained hyperspectral image classification network, training a mapping layer in the trained hyperspectral image classification network, and obtaining the hyperspectral image classification network after the mapping layer is trained; a hyperspectral image classification result and a hyperspectral image classification graph are obtained; according to the method, basic spectral space information can be effectively captured, and the generalization ability of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a small sample domain generalized hyperspectral image classification method based on cyclic prompts and a related device. Background Art

[0002] With the rapid development of sensors and image processing technology, hyperspectral images have received widespread attention in various application fields. Compared with traditional chemical analysis methods, hyperspectral imaging systems have the advantages of being fast, accurate, non-contact, and non-destructive to the object being measured. They are irreplaceable and are currently widely used in national defense security, public security criminal investigation, mineral exploration, ecological monitoring, environmental monitoring, mineralogy, food, agriculture and forestry, life sciences, medical diagnosis, smart agriculture, and medical diagnosis. Hyperspectral image classification is one of the important means of hyperspectral image interpretation, and has been most widely and deeply studied in the field of hyperspectral image processing research. It can provide strong support for higher-level applications such as ground object monitoring and change detection. Under the objective environment where the country attaches great importance to life safety, environmental protection, and economic sustainable development, the hyperspectral imaging system market has ushered in a golden period of development.

[0003] In the actual process, although deep neural networks with more hidden layers have strong feature learning and feature representation capabilities, their feature representation capabilities tend to decline when the training samples are limited. Cross-domain few-shot learning helps transfer knowledge from the source domain to the target domain through a limited number of annotated samples. This strategy has been proven to be very effective in hyperspectral image classification. Most existing methods focus on iterative training between the source domain and the target to achieve domain adaptation in the case of few samples. Ye et al. proposed a cross-domain few-shot learning based on graph convolution contrast in their article "Cross-Domain Few-Shot Learning Based on Graph Convolution Contrast for Hyperspectral Image Classification" (IEEE Transactions on Geoscience and Remote Sensing, 2024). This method converts the similarity and connectivity relationships of samples into graph structures to extract domain-invariant features, and uses cross-domain learning to transfer knowledge from the source domain to classify samples in the target domain.

[0004] However, most of the works described in the above papers focus on alternating training in the source and target domains to transfer general knowledge from the source domain to the target domain with limited samples, and the domain gap still affects the overall performance of the model on the target domain. Summary of the invention

[0005] The object of the present invention is to provide a few-shot domain generalization hyperspectral image classification method and related device based on cyclic prompting to overcome the problems existing in the prior art. The present invention can introduce Mamba into the feature extraction network, effectively extract general knowledge of different domains by quickly switching between different domains, obtain a network with generalization ability, and enable it to obtain better performance on the target domain; in order to reduce the computational requirements of ordinary transformers, the network integrates a bidirectional Mamba and a transformer architecture to effectively capture basic spectral-spatial information; through a domain-specific promptor, it aims to prevent the model from overfitting to the current domain and mitigate catastrophic forgetting of previous domain knowledge, thereby enhancing the generalization ability of the model.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] In the first aspect, the present invention provides a few-shot domain generalization hyperspectral image classification method based on cyclic prompting, including the following steps:

[0008] Obtain hyperspectral image data of a number of composite source domains and a number of target domains respectively;

[0009] Construct a network consisting of a mapping layer and a Mamba-based transformer, and alternately iteratively train the network consisting of the mapping layer and the Mamba-based transformer through the hyperspectral image data of a number of composite source domains to form a hyperspectral image classification network;

[0010] Construct a composite source domain loss function, and train the hyperspectral image classification network through the composite source domain loss function and the hyperspectral image data of the composite source domain to obtain a trained hyperspectral image classification network;

[0011] Construct a target domain classification loss function, freeze the network based on the Mamba transformer in the trained hyperspectral image classification network, and train the mapping layer in the trained hyperspectral image classification network through the target domain classification loss function and the hyperspectral image data of the target domain to obtain a hyperspectral image classification network after training the mapping layer;

[0012] Input the hyperspectral image data of the target domain that has not participated in the training into the hyperspectral image classification network after training the mapping layer to obtain a hyperspectral image classification result and a hyperspectral image classification map;

[0013] Further, the obtaining of a number of composite source domains specifically includes:

[0014] Use M few-shot source domain datasets D, D = {D1, D2,..., D M} and a base source domain dataset D0 to construct a hyperspectral image source domain training set and train it. During the training process, the combination of the few-shot source domain datasets D and the base source domain dataset D0 is regarded as a single composite source domain Through a number of individual composite source domains constitute a number of composite source domains;

[0015] The step of separately obtaining the hyperspectral image data of a number of composite source domains and a number of target domains further includes:

[0016] Construct the support set data and query set data of the composite source domain through the hyperspectral image data of a number of composite source domains respectively, and construct the support set data and query set data of the target domain through the hyperspectral image data of a number of target domains respectively;

[0017] Furthermore, the step of constructing a network composed of a mapping layer and a Mamba Transformer, and alternately iteratively training the network composed of the mapping layer and the Mamba Transformer through the hyperspectral image data of a number of composite source domains to form a hyperspectral image classification network specifically includes:

[0018] Input the support set data and query set data of a number of composite source domains into a shallow convolutional module composed of a three-dimensional convolution and a two-dimensional convolution to extract shallow features, perform convolution on the shallow features and then divide them into sub-feature one and sub-feature two along the channel dimension, input sub-feature one into the structure cascaded by a bidirectional Mamba and a Transformer in sequence to obtain non-local features, divide sub-feature two into g groups, input the sub-feature two of g groups into the bidirectional Mamba to obtain local features, splice the non-local features and local features to obtain a spliced feature, add the spliced feature and the shallow features to obtain a fused feature, and input the fused feature into the convolutional module to obtain dimensional features;

[0019] The step of constructing a network composed of a mapping layer and a Mamba Transformer, and alternately iteratively training the network composed of the mapping layer and the Mamba Transformer through the hyperspectral image data of a number of composite source domains to form a hyperspectral image classification network further includes:

[0020] Take the hyperspectral image classification network of the current iteration as a learner, and take the hyperspectral image classification network that was frozen after training on the same composite source domain last time as a prompt;

[0021] Furthermore, the step of inputting sub-feature one into the bidirectional Mamba in sequence specifically includes:

[0022] X s = S3(X in )

[0023] G = SiLu(Linear(Norm(X s )))

[0024] X l = Linear(Norm(X s ))

[0025] X1 = Flip(SSM(SiLu(Conv1d(Flip(X l )))))

[0026] X2 = SSM(SiLu(Conv1d(X l )))

[0027] X out = Reverse(Linear(X1·G + X2·G))

[0028] where X in represents the first sub - feature; S3 represents scanning the first sub - feature; X s represents the data after scanning the first sub - feature; SiLu represents the activation function; Linear represents the selective mechanism based on the linear layer; Norm represents normalization; G represents the gate in the state - space model; Flip represents the flipping operation; X1 represents the first intermediate data; SSM represents the state - space model; X2 represents the second intermediate data; Reverse represents the flipped scanning operation; X out represents the output data;

[0029] Furthermore, the cascading of the converters specifically includes:

[0030] Normalize the output data X out and input it into the multi - head self - attention module. The multi - head self - attention module linearly transforms the output data X out into three matrices: query Q, key K, and value V. Then divide the matrices Q, K, and V into n respectively, calculate the attention matrix Z, and calculate the final module output data O through the attention matrix Z. Connect the output data X out and the module output data O residually to obtain the non - local feature. The specific formula is:

[0031] Q = X out W Q , K = XW k , V = XW V ;

[0032] Q = {Q1, Q2…, Q i …, Q n};

[0033] K = {K1, K2…, K i …, K n};

[0034] V = {V1, V2…, V i …, V n};

[0035]

[0036] O = Concat(Z1, Z2,..., Z i ,..., Z n )W O ;

[0037] Where attention represents the attention mechanism; softmax represents the softmax function; T represents the transpose; Concat represents the concatenation operation; W represents the trainable projection matrix;

[0038] Furthermore, constructing the composite source domain loss function specifically includes:

[0039] Combined classification loss function L cls and the reweighted cross-entropy loss function L rce , to obtain the composite source domain loss function L overall , where the classification loss function L cls The specific formula is:

[0040]

[0041] Where y s represents the true probability that the query set data of the composite source domain belongs to each category; represents the predicted probability that the query set data of the composite source domain belongs to each category;

[0042] The specific formula of the reweighted cross-entropy loss function L rce is:

[0043]

[0044] Where P represents the original probability vector; represents the reweighted probability vector; β represents the hyperparameter used to adjust the learner's prediction weight; x represents the query set sample data of the composite source domain; W represents the network parameters of the learner; k represents the conditional selection parameter. When arg max P(y|x; W p ) = y, k = 1. When arg max P(y|x; W p ) ≠ y, k = 0;

[0045] The specific formula of the composite source domain loss function L overall is:

[0046] L overall = L cls + αL rce ;

[0047] Where α is the weight factor;

[0048] Further, training the hyperspectral image classification network with the composite source domain loss function and the hyperspectral image data of the composite source domain specifically includes:

[0049] Input the support set data and query set data of the composite source domain into the hyperspectral image classification network in a loop. Using the gradient descent method, update the parameters in the hyperspectral image classification network in a loop. Calculate the composite source domain loss function through dimensional features, update the gradient of the composite source domain loss function through backpropagation, and update the training parameters of the hyperspectral image classification network through forward propagation according to the direction of the decrease of the composite source domain loss function until the maximum number of iterations T is reached;

[0050] Further, constructing the target domain classification loss function specifically includes:

[0051]

[0052] In the formula, y t represents the true probability that the query set data of the target domain belongs to each category; represents the predicted probability that the query set data of the target domain belongs to each category;

[0053] Training the mapping layer in the trained hyperspectral image classification network with the target domain classification loss function and the hyperspectral image data of the target domain specifically includes:

[0054] Input the support set data and query set data of the target domain into the trained hyperspectral image classification network in a loop. Using the gradient descent method, update the parameters of the mapping layer in the trained hyperspectral image classification network in a loop to reduce the gradient value of the target domain classification loss function until the maximum number of iterations T is reached.

[0055] In a second aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0056] In a third aspect, the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0057] The above technical solution has the following advantages or beneficial effects:

[0058] In a first aspect, the present invention provides a few-shot domain generalization hyperspectral image classification method based on cyclic prompting. A novel MambaFormer network is designed by integrating a transformer and a bidirectional Mamba framework. This network can efficiently and effectively model local and non-local spectral-spatial features, and effectively capture basic spectral-spatial information. To achieve domain generalization during the whole-domain switching learning process, a domain-specific prompter is designed, which can mitigate overfitting to the current domain of the model and help prevent catastrophic forgetting of previously acquired knowledge. A new few-shot domain generalization framework based on cyclic prompting is introduced to enhance the generalization ability of the network. This framework utilizes multiple domains for training and performs rapid domain switching during cyclic training iterations, which helps for smoother generalization from the source domain to the target domain to alleviate the domain gap. The present invention improves the accuracy of few-shot hyperspectral image classification, enhances robustness, and can be used for mineral exploration, ecological monitoring, smart agriculture, and medical diagnosis.

[0059] In a second aspect, the present invention provides a computer device. By a processor executing a specific computer program, the steps of the method of the present invention can be efficiently implemented. When the computer device performs data processing tasks, it can accurately perform numerical calculations and logical judgments, avoiding errors caused by human factors. At the same time, due to the high stability and reliability of the computer program, the accuracy and consistency of the data processing results can be ensured.

[0060] In a third aspect, the present invention provides a computer-readable storage medium. By programming the steps of the method of the present invention into a computer program and storing it on the computer-readable storage medium, users can easily load these programs onto any compatible computer device and execute them without having to rewrite or convert the code, greatly improving the convenience and flexibility of program execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a schematic flow chart of the few-shot domain generalization hyperspectral image classification method based on cyclic prompting of the present invention;

[0062] Figure 2 is a schematic framework diagram of the network (MambaFormer) based on Mamba transformer of the present invention;

[0063] Figure 3 is a schematic diagram of the Reverse operation of the few-shot domain generalization hyperspectral image classification method based on cyclic prompting of the present invention;

[0064] FIG. 4(a) is a color composite image of the Chikusei hyperspectral image obtained from a public website in the embodiment;

[0065] FIG. 4(b) is a ground truth image of the Chikusei hyperspectral image obtained from a public website in the embodiment;

[0066] Figure 5(a) is the pseudo-color composite image of the Botswana hyperspectral image obtained from the public website in the embodiment;

[0067] Figure 5(b) is the true-color image of the Botswana hyperspectral image obtained from the public website in the embodiment;

[0068] Figure 6(a) is the pseudo-color composite image of the Salinas hyperspectral image obtained from the public website in the embodiment;

[0069] Figure 6(b) is the true-color image of the Salinas hyperspectral image obtained from the public website in the embodiment;

[0070] Figure 7(a) is the pseudo-color composite image of the Pavia Center hyperspectral image obtained from the public website in the embodiment;

[0071] Figure 7(b) is the true-color image of the Pavia Center hyperspectral image obtained from the public website in the embodiment;

[0072] Figure 8(a) is the pseudo-color composite image of the Indian Pines hyperspectral image obtained from the public website in the embodiment;

[0073] Figure 8(b) is the true-color image of the Indian Pines hyperspectral image obtained from the public website in the embodiment;

[0074] Figure 9 It is the simulation result diagram of classifying Indian Pines by the existing GCC-FSL method in the embodiment;

[0075] Figure 10 It is the simulation result diagram of classifying Indian Pines by the present invention in the embodiment;

[0076] Figure 11 It is the structural schematic diagram of the computer device of the present invention. Detailed implementation manners

[0077] The present invention will be further described in detail below in conjunction with specific embodiments, which are explanations of the present invention rather than limitations.

[0078] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0079] It should be noted that the terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0080] Embodiment:

[0081] See Figure 1 , the present invention provides a small-sample domain generalization hyperspectral image classification method based on cyclic prompting, including the following steps:

[0082] Step 1, perform data preparation for domain-switching meta-learning:

[0083] Define the Chikusei dataset as the basic source domain dataset D0, and its pseudo-color composite image is shown in Figure 4(a), and the true-color image is shown in Figure 4(b); define the Botswana, Salinas, and Pavia Center datasets as the small-sample source domain datasets D. The pseudo-color composite image of Botswana is shown in Figure 5(a), the true-color image is shown in Figure 5(b), the pseudo-color composite image of Salinas is shown in Figure 6(a), the true-color image is shown in Figure 6(b), and the pseudo-color composite image of Pavia Center is shown in Figure 7(a), the true-color image is shown in Figure 7(b); define the Indian Pines dataset as the target domain dataset, and its pseudo-color composite image is shown in Figure 8(a), the true-color image is shown in Figure 8(b). The Indian Pines dataset contains 16 types of ground objects, with a total of 42,776 labeled samples; construct a hyperspectral image source domain training set from the small-sample source domain dataset D and the basic source domain dataset D0 and perform alternating iterative training. In each iteration, randomly select 180 samples from the basic source domain dataset D0 and 5 samples of each class from the current small-sample source domain dataset D as training samples. During the training process, consider the combination of the small-sample source domain dataset D and the basic source domain dataset D0 as a single composite source domain Through several single composite source domains constitute several composite source domains; constitute the support set data and query set data of the composite source domain through the hyperspectral image data of several composite source domains, and constitute the support set data and query set data of the target domain through the hyperspectral image data of several target domains.

[0084] Step 2: Construct a hyperspectral image classification network and perform domain-switching meta-learning:

[0085] Construct a network consisting of a mapping layer and a Mamba Transformer. Iteratively train the network composed of the mapping layer and the Mamba Transformer alternately with the hyperspectral image data of several composite source domains to form a hyperspectral image classification network. Take the hyperspectral image classification network of the current iteration as the learner, and take the hyperspectral image classification network that was frozen after training on the same composite source domain last time as the prompt.

[0086] Specifically, the mapping layer is a linear layer that maps the hyperspectral image data of different composite source domains to the same dimension n, where n is taken as 100 in this example. After passing through the mapping layer, the hyperspectral image data of the composite source domain obtains a matrix with a size of (100, 9, 9); see Figure 2 , construct a network based on the Mamba Transformer, and input the matrix with a size of (100, 9, 9) into the network based on the Mamba Transformer. Specifically, the matrix with a size of (100, 9, 9) is input into a shallow convolutional module composed of a three-dimensional convolution and a two-dimensional convolution to extract shallow features, and the size of the shallow features is (64, 7, 7). After convolving the extracted shallow features, they are divided into sub-feature one and sub-feature two with a size of (32, 7, 7) along the channel dimension. Input sub-feature one into the structure cascaded by the bidirectional Mamba and the transformer to obtain non-local features with a size of (32, 7, 7). Divide sub-feature two into g groups and input them into the bidirectional Mamba in groups to obtain local features with a size of (32, 7, 7). In this example, g is taken as 4;

[0087] Specifically, input sub-feature one into the bidirectional Mamba in sequence, including:

[0088] X s = S3(X in )

[0089] G = SiLu(Linear(Norm(X s )))

[0090] X l = Linear(Norm(X s ))

[0091] X1 = Flip(SSM(SiLu(Conv1d(Flip(X l )))))

[0092] X2 = SSM(SiLu(Conv1d(X l )))

[0093] X out = Reverse(Linear(X1·G + X2·G))

[0094] In the formula, X in represents sub-feature one; S3 represents scanning sub-feature one; X s represents the data after scanning sub-feature one; SiLu represents the activation function; Linear represents the selective mechanism based on the linear layer; Norm represents normalization; G represents the gate in the state space model; Flip represents the flipping operation; X1 represents the first intermediate data; SSM represents the state space model; 2 represents the second intermediate data; Reverse represents the reverse scanning operation; X out represents the output data;

[0095] Preferably, the bidirectional Mamba network can be HyperMamba, GraphMamba;

[0096] Specifically, the transformer cascade includes: inputting the output data X out after layer normalization into the multi-head self-attention module, and the multi-head self-attention module converts the output data X out into three matrices of query Q, key K, and value V through linear transformation. The formulas are: Q = X out W Q , K = XW k , V = XW V , and then divide the matrices Q, K, and V into n respectively. The formulas are: Q = {Q1, Q2…, Q i …, Q n}, K = {K1, K2…, K i …, K n}, V = {V1, V2…, V i …, V n}, calculate the attention matrix Z, and calculate the final module output data O through the attention matrix Z. Connect the output data X out and the module output data O residually to obtain the non-local feature. The formula is:

[0097] Q = X out W Q , K = XW k , V = XW V ;

[0098] Q = {Q1, Q2…, Q i …, Q n};

[0099] K = {K1, K2…, K i …, K n};

[0100] V = {V1, V2…, V i …, Vn}

[0101]

[0102] O = Concat(Z1, Z2, ..., Z i , ..., Z n )W O ;

[0103] Wherein, attention represents the attention mechanism; softmax represents the normalized exponential function; T represents the transpose; Concat represents the concatenation operation; W represents the trainable projection matrix;

[0104] Then, the data after residual connection is sequentially input into layer normalization and a multi-layer perceptron, the input and output are subjected to residual connection, the non-local features and local features are concatenated, and the concatenated features with a size of (64, 7, 7) obtained are added to the shallow features to obtain fused features, and the fused features are input into a convolution module with a convolution kernel size of (1, 1), thereby obtaining the dimensional features with a final dimension of 128.

[0105] Step 3: Construct a composite source domain loss function and train the hyperspectral image classification network:

[0106] Construct a composite source domain loss function, and train the hyperspectral image classification network through the composite source domain loss function and the hyperspectral image data of the composite source domain to obtain the trained hyperspectral image classification network;

[0107] Specifically, constructing the composite source domain loss function includes: combining the classification loss function L cls and the reweighted cross-entropy loss function L rce to obtain the composite source domain loss function L overall , where the classification loss function L cls has the following specific formula:

[0108]

[0109] Wherein, y s represents the true probability that the query set data of the composite source domain belongs to each category; represents the predicted probability that the query set data of the composite source domain belongs to each category;

[0110] The specific formula of the reweighted cross-entropy loss function L rce is as follows:

[0111]

[0112] Wherein, P represents the original probability vector; denotes the probability vector after reweighting; β denotes the hyperparameter used to adjust the weights of the learner's predictions; x denotes the sample data of the query set in the composite source domain; W denotes the network parameters of the learner; k denotes the conditional selection parameter, when arg maxP(y|x; W p ) = y, k = 1, when arg maxP(y|x; W p ) ≠ y, k = 0;

[0113] The composite source domain loss function L overall The specific formula is:

[0114] L overall = L cls + αL rce ;

[0115] In the formula, α is the weight factor, and in this embodiment, α = 0.2;

[0116] Specifically, training the hyperspectral image classification network through the composite source domain loss function and the hyperspectral image data of the composite source domain includes: circularly inputting the support set data and query set data of the composite source domain into the hyperspectral image classification network, using the gradient descent method to circularly update the parameters in the hyperspectral image classification network to reduce the gradient value of the composite source domain loss function, setting the initial value of the learning rate, the number of meta-learning episodes, the batch size, initializing the parameters of the network model, constructing the training data for each episode according to the batch size, calculating the composite source domain loss function through dimensional features, updating the gradient of the composite source domain loss function through backpropagation, and performing forward propagation to update the training parameters of the hyperspectral image classification network according to the direction of the decrease of the composite source domain loss function, and quickly switching among 3 composite source domains in turn until the maximum number of iterations T is reached.

[0117] Step four, constructing the target domain classification loss function and training the mapping layer:

[0118] Construct the target domain classification loss function, freeze the network based on the Mamba transformer in the trained hyperspectral image classification network, and train the mapping layer in the trained hyperspectral image classification network through the target domain classification loss function and the hyperspectral image data of the target domain to obtain the hyperspectral image classification network after training the mapping layer;

[0119] Specifically, the formula of the target domain classification loss function is:

[0120]

[0121] In the formula, y t denotes the true probability that the query set data in the target domain belongs to each category; denotes the predicted probability that the query set data in the target domain belongs to each category;

[0122] Specifically, the mapping layer in the trained hyperspectral image classification network is trained using the target domain classification loss function and the hyperspectral image data of the target domain, including: cyclically inputting the support set data and query set data of the target domain into the trained hyperspectral image classification network, adopting the gradient descent method, cyclically updating the parameters of the mapping layer in the trained hyperspectral image classification network to reduce the gradient value of the target domain classification loss function, calculating the target domain classification loss function, updating the gradient of the target domain classification loss function through backpropagation, and forward propagating and updating the training parameters of the hyperspectral image classification network according to the direction of the decrease of the target domain classification loss function, and quickly switching in the target domain classification loss function in turn until the maximum number of iterations T is reached, so that the model can be quickly generalized to the target domain.

[0123] Step Five

[0124] Input the hyperspectral image data of the target domain that has not participated in the training into the hyperspectral image classification network after training the mapping layer to obtain the hyperspectral image classification result and the hyperspectral image classification map.

[0125] The effects of the present invention can be further illustrated by the following simulation results.

[0126] Simulation conditions:

[0127] Test data: Use the Chikusei dataset as the basic source domain dataset, use the Botswana, Salinas, and Pavia Center datasets as small sample source domain datasets, use the Indian Pines dataset as the target domain, and randomly select 5 labeled samples for each category in the target domain; Test environment: Use the Ubuntu 18.04LTS system, NVIDIA GeForce GTX 4090 GPU, and Pytorch deep learning framework; Evaluation metrics: The metric calculation formula is as follows:

[0128]

[0129] In the formula, TP is the number of samples correctly classified into this category; FN is the number of samples of this category that are misclassified into other categories; TN is the number of non - this - category samples classified into other categories; FP is the number of other samples misclassified into this category; x i+ represents the sum of all elements in the i - th row of the confusion matrix, that is, the sum of the actual quantities of this category; x +i represents the sum of all elements in the i - th column of the confusion matrix, that is, the sum of the quantities predicted as this category; N represents the sum of all elements; c represents the number of categories; x ii represents the elements on the diagonal of the confusion matrix, that is, the number of samples correctly predicted for each category.

[0130] Simulation content:

[0131] Simulation 1: Under the above conditions, use the existing small-sample hyperspectral image classification method GCC-FSL to classify the IndianPines data. The results are shown in Figure 9 ;

[0132] Simulation 2: Under the above conditions, use the method of the present invention to classify the Indian Pines data. The results are shown in Figure 10 ;

[0133] See Figure 9 and Figure 10 It can be found by comparison that the original method trains alternately in the source domain and the target domain, and then the domain gap between the source domain and the target domain affects the classification performance. While the method of the present invention develops a new domain-switching meta-learning strategy, which incorporates cross-domain scenarios into the training process in a "fast-switching" manner. This strategy includes a domain-specific promptor, which aims to prevent the model from overfitting to the current domain and mitigate catastrophic forgetting of previous domain knowledge, thereby enhancing the generalization ability of the model. In addition, the present invention introduces MambaFormer to enhance the discriminability of feature embeddings, combines the modeling advantages of the transformer with the computational efficiency of Mamba, and extracts representative spectral-spatial information; the classification result map obtained by the method proposed in the present invention is closer to the ground object distribution of the true value map, and the classification effect obtained in most categories is better, achieving better classification performance.

[0134] Evaluation metrics:

[0135] Repeat the above two methods 10 times respectively using the same 10 random numbers, calculate the confusion matrix, the classification accuracy PA of each category, the average classification accuracy AA, the overall classification accuracy OA, and the KAPPA coefficient. The test results are shown in Table 1.

[0136] Table 1 Simulation experiment results of the present invention and the comparative method

[0137]

[0138]

[0139] It can be seen from Table 1 that compared with the current relatively advanced GCC-FSL method, the present invention has improved the classification accuracy of the categories numbered 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16. At the same time, the average classification accuracy, the overall classification accuracy, and the KAPPA coefficient have also been significantly improved.

[0140] In the hyperspectral field, since the annotation of hyperspectral images requires a large amount of manpower and time, the labeled samples for training classification models are limited, and there is a lack of rich prior knowledge in the training process of classification networks. Therefore, it is difficult to obtain effective feature embeddings. The purpose of the present invention is to address the deficiencies of the prior art and propose a small-sample domain generalization hyperspectral image recognition method based on cyclic prompting, thereby improving the image classification effect. Different from the widely used cross-domain few-shot learning strategy that alternates training between the source domain and the target domain, in the framework of the present invention, the data of multiple source domains are quickly switched during the training process, and this method helps for smoother generalization from the source domain to the target domain; in order to alleviate overfitting to the current domain and prevent catastrophic forgetting, the present invention proposes a domain-specific promptor with reweighted cross-entropy loss, effectively enhancing the generalization ability of the model; in addition, the present invention introduces the MambaFormer network, which effectively captures local and non-local features through two branches with relatively low complexity; the present invention improves the accuracy of few-shot hyperspectral image classification, enhances the robustness, and can be used for mineral exploration, ecological monitoring, smart agriculture, and medical diagnosis.

[0141] It can be seen that, compared with the prior art, the present invention can better overcome the problem of the decline in the classification performance of hyperspectral images in the case of few samples. At the same time, compared with the efforts made by other methods in extracting domain-invariant features, the focus of our method is to capture rich spectral-spatial information while fully using the existing data to extract discriminative features, so as to provide better services for higher-level applications such as ground object monitoring and change detection.

[0142] See Figure 11, in an embodiment of the present invention, a computer device is provided. The computer device includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function. The processor described in the embodiment of the present invention can be used for the operation of the small sample domain generalization hyperspectral image classification method based on cyclic prompting.

[0143] , in an embodiment of the present invention, a computer-readable storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in the computer device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and, of course, the extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, and the operating system of the terminal is stored in this storage space. Moreover, one or more instructions suitable for being loaded and executed by the processor are stored in this storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the small sample domain generalization hyperspectral image classification method related to the embodiment.

[0144] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0145] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0146] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A small-sample domain generalization hyperspectral image classification method based on cyclic prompting, characterized in that The method includes the following steps: Obtain hyperspectral image data of a number of composite source domains and a number of target domains respectively; Construct a network composed of a mapping layer and a Mamba Transformer. Iteratively train the network composed of the mapping layer and the Mamba Transformer alternately with the hyperspectral image data of a number of composite source domains to form a hyperspectral image classification network; Construct a composite source domain loss function, and train the hyperspectral image classification network with the composite source domain loss function and the hyperspectral image data of the composite source domain to obtain a trained hyperspectral image classification network; Construct a target domain classification loss function, freeze the network based on the Mamba Transformer in the trained hyperspectral image classification network, and train the mapping layer in the trained hyperspectral image classification network with the target domain classification loss function and the hyperspectral image data of the target domain to obtain a hyperspectral image classification network after training the mapping layer; Input the hyperspectral image data of the target domain that has not participated in the training into the hyperspectral image classification network after training the mapping layer to obtain a hyperspectral image classification result and a hyperspectral image classification map.

2. The small-sample domain generalization hyperspectral image classification method based on cyclic prompting according to claim 1, wherein The obtaining of a number of composite source domains specifically includes: Using M small-sample source domain datasets D, D = {D1, D2, …, D M} and a basic source domain dataset D0 to construct a hyperspectral image source domain training set and perform training. During the training process, the combination of the small-sample source domain dataset D and the basic source domain dataset D0 is regarded as a single composite source domain Through a number of single composite source domains To form a number of composite source domains; The obtaining of hyperspectral image data of a number of composite source domains and a number of target domains respectively further includes: Respectively form a support set data and a query set data of the composite source domain through the hyperspectral image data of a number of composite source domains, and respectively form a support set data and a query set data of the target domain through the hyperspectral image data of a number of target domains.

3. A small-sample domain generalization hyperspectral image classification method based on cyclic prompting according to claim 2, characterized in that, The constructing of a network composed of a mapping layer and a Mamba Transformer, and iteratively training the network composed of the mapping layer and the Mamba Transformer alternately with the hyperspectral image data of a number of composite source domains to form a hyperspectral image classification network specifically includes: Input the support set data and the query set data of a number of composite source domains into a shallow convolutional module composed of a three-dimensional convolution and a two-dimensional convolution to extract shallow features. After convolving the shallow features, divide them into sub-feature one and sub-feature two along the channel dimension. Input sub-feature one into a structure cascaded by a two-way Mamba and a Transformer in sequence to obtain non-local features. Divide sub-feature two into g groups, and input the sub-feature two of the g groups into the two-way Mamba to obtain local features. Concatenate the non-local features and the local features to obtain a concatenated feature. Add the concatenated feature to the shallow features to obtain a fused feature. Input the fused feature into a convolutional module to obtain dimensional features; The constructing of a network composed of a mapping layer and a Mamba Transformer, and iteratively training the network composed of the mapping layer and the Mamba Transformer alternately with the hyperspectral image data of a number of composite source domains to form a hyperspectral image classification network further includes: Use the hyperspectral image classification network of the current iteration as a learner, and use the hyperspectral image classification network frozen after training on the same composite source domain last time as a promptor.

4. A small-sample domain generalization hyperspectral image classification method based on cyclic prompting according to claim 3, characterized in that The inputting of sub-feature one into the two-way Mamba in sequence specifically includes: X s = S3(X in ) G = SiLu(Linear(Norm(X s ))) X l = Linear(Norm(X s )) X1 = Flip(SSM(SiLu(Conv1d(Flip(X l ))))) X2 = SSM(SiLu(Conv1d(X l ))) X out = Reverse(Linear(X1·G + X2·G)) In the formula, X in represents sub-feature one; S3 represents scanning sub-feature one; X s represents the data after scanning sub-feature one; SiLu represents the activation function; Linear represents the selective mechanism based on the linear layer; Norm represents normalization; G represents the gate in the state space model; Flip represents the flipping operation; X1 represents the first intermediate data; SSM represents the state space model; X2 represents the second intermediate data; Reverse represents the reverse scanning operation; X out represents the output data.

5. A small-sample domain generalization hyperspectral image classification method based on cyclic prompting according to claim 4, characterized in that, The Transformer cascade specifically includes: The output data X out After layer normalization, it is input into the multi-head self-attention module. The multi-head self-attention module linearly transforms the output data X out into three matrices: query Q, key K, and value V. Then, the matrices Q, K, and V are each divided into n parts, the attention matrix Z is calculated, and the final module output data O is calculated through the attention matrix Z. The output data X out and the module output data O are subjected to residual connection to obtain non-local features. The specific formula is: Q = X out W Q , K = XW k , V = XW V ; Q = {Q1, Q2…, Q i …, Q n}; K = {K1, K2…, K i …, K n}; V = {V1, V2…, V i …, V n}; O = Concat(Z1, Z2,..., Z i ,..., Z n )W O ; In the formula, attention represents the attention mechanism; softmax represents the normalized exponential function; T represents the transpose; Concat represents the concatenation operation; W represents a trainable projection matrix.

6. A small-sample domain generalization hyperspectral image classification method based on cyclic prompting according to claim 3, characterized in that The constructing of the composite source domain loss function specifically includes: Combined classification loss function \(L\) cls and the reweighted cross-entropy loss function \(L\) rce are used to obtain the composite source domain loss function \(L\) overall where the specific formula of the classification loss function \(L\) cls is as follows: where y s represents the true probability that the query set data in the composite source domain belongs to each category; represents the predicted probability that the query set data in the composite source domain belongs to each category; The reweighted cross-entropy loss function L rce The specific formula is as follows: Wherein, P represents the original probability vector; represents the probability vector after reweighting; β represents the hyperparameter used to adjust the prediction weight of the learner; x represents the query set sample data of the composite source domain; W represents the network parameters of the learner; k represents the conditional selection parameter. When arg max P(y|x; W p ) = y, k = 1. When arg max P(y|x; W p ) ≠ y, k = 0; Composite source domain loss function L overall The specific formula is as follows: L overall = L cls + αL rce ; In the formula, α is a weight factor.

7. A small-sample domain generalization hyperspectral image classification method based on cyclic prompting according to claim 3, wherein Training the hyperspectral image classification network by using the composite source domain loss function and the hyperspectral image data of the composite source domain specifically includes: Circularly inputting the support set data and the query set data of the composite source domain into the hyperspectral image classification network, adopting the gradient descent method, circularly updating the parameters in the hyperspectral image classification network, calculating the composite source domain loss function through dimensional features, updating the gradient of the composite source domain loss function through backpropagation, and forward propagating and updating the training parameters of the hyperspectral image classification network according to the direction of the decrease of the composite source domain loss function until the maximum number of iterations T is reached.

8. A small-sample domain generalization hyperspectral image classification method based on cyclic prompting according to claim 2, wherein Constructing the target domain classification loss function specifically includes: where y t represents the true probability that the query set data in the target domain belongs to each category; represents the predicted probability that the query set data in the target domain belongs to each category; Training the mapping layer in the trained hyperspectral image classification network by using the target domain classification loss function and the hyperspectral image data of the target domain specifically includes: Circularly inputting the support set data and the query set data of the target domain into the trained hyperspectral image classification network, adopting the gradient descent method, circularly updating the parameters of the mapping layer in the trained hyperspectral image classification network to reduce the gradient value of the target domain classification loss function until the maximum number of iterations T is reached.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1-8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1-8 are implemented.

Images