A multi-modal self-supervised hybrid Mamba method for hyperspectral image classification

Abstract

The present invention relates to a hyperspectral image classification method of a multi-mode self-supervised hybrid Mamba, and the method comprises the following steps: processing a hyperspectral image classification data set of a multi-mode self-supervised hybrid Mamba, constructing a hyperspectral image classification network combining CNN and Mamba, label-free pre-training of a hyperspectral image classification network based on self-supervision, training of a multi-modal hyperspectral image classification model based on CLIP, and testing the multi-modal hyperspectral image classification model. Compared with the prior art, the method combines CNN and Mamba to extract spatial spectrum features, which is conducive to realizing accurate hyperspectral image classification, and pre-trains the model through a self-supervised learning method without labeled data, so that the model can recognize hyperspectral images in different scenes by itself, thereby improving the generalization ability of the model. At the same time, multi-modal hyperspectral image classification can be realized based on the CLIP model.

Description

Technical Field

[0001] The invention relates to the fields of optical remote sensing technology and image processing, and in particular to a hyperspectral image classification method of a multi-mode self-supervised hybrid Mamba. Background Art

[0002] With the advancement and development of remote sensing technology, hyperspectral imaging has been widely studied and applied. Because the spectral reflectance of each substance will show unique characteristics in different bands, hyperspectral imaging technology uses this feature to identify the target object, and hyperspectral usually involves dozens to hundreds of continuous spectral bands, including visible light and infrared regions. Unlike RGB images, hyperspectral images involve information in two dimensions, spatial and spectral, providing more comprehensive contextual information, and are widely used in agriculture, forestry, and environmental monitoring. Hyperspectral image classification is a basic task for these practical applications. It is used to predict the class label of each pixel in the image. This process will face the dimensionality disaster and noise interference problems brought by high-dimensional data.

[0003] In recent years, deep learning methods have made great progress in image classification, from convolutional neural networks (CNNs), to attention mechanisms (Transformers), to time series models (Mamba). Hyperspectral image data is large and difficult to label, while self-supervised learning generates labels from the data itself for training. For hyperspectral data, this method can reduce the dependence on a large amount of labeled data and improve the generalization ability of the model. CLIP is a trained self-supervised model, while Mamba solves the limitations of the local receptive field of CNN and the computational complexity of the quadratic complexity of Transformers. It has the property of long-term dependence. By combining the advantages of these two aspects, hyperspectral image classification can be better performed. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-mode self-supervised hybrid Mamba hyperspectral image classification method to solve the problems of data annotation difficulties and spectral dimensionality disaster in existing hyperspectral image classification technology.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] A multi-mode self-supervised hybrid Mamba hyperspectral image classification method includes the following steps:

[0007] 11) Processing of hyperspectral image classification dataset of multi-mode self-supervised hybrid Mamba: collect hyperspectral images, perform preprocessing of cropping, data denoising, band dimension reduction and channel normalization on all hyperspectral images, and then classify them according to whether they have labels or not. Hyperspectral images with labels are used as training sets, and hyperspectral images without labels are divided into training sets and test sets according to 8:2; collect text descriptions related to hyperspectral images, including image descriptions, category labels and geographic information, and perform preprocessing of stem extraction and word segmentation on text data. The preprocessed text is called prompt text;

[0008] 12) Construct a hyperspectral image classification network combining CNN and Mamba: construct a CNN spatial encoder to extract the spatial features of hyperspectral images, a Mamba spectral encoder to extract the spectral features of hyperspectral images, and construct a spatial-spectral information fusion module to obtain a hyperspectral image classification network combining CNN and Mamba. The output of the network is the spatial-spectral features of hyperspectral images.

[0009] 13) Pre-training of hyperspectral image classification network based on self-supervised CNN and Mamba: Using contrastive learning method, the training data is the hyperspectral images in the unlabeled training set. Each hyperspectral image in a training batch is called an original sample. The original sample is generated by data enhancement methods such as brightness adjustment, shadow generation, and Gaussian blur as a positive sample, and the hyperspectral images other than the original sample are negative samples. All samples are input into the hyperspectral image classification network for pre-training, so that the feature representation of the original sample is close to that of the positive sample in the embedding space, but far away from that of the negative sample.

[0010] 14) Training of CLIP-based multimodal hyperspectral image classification model: The constructed CLIP-based multimodal hyperspectral image classification model includes an image encoder and a text encoder. The image encoder uses a hyperspectral image classification network that combines CNN and Mamba. The text encoder uses the text encoder structure in CLIP, and the output is text features, which are initialized by the pre-trained weights of CLIP. The cosine similarity between the spatial-spectral features output by the image encoder and the text features output by the text encoder is calculated, so that the cosine similarity between the spatial-spectral features of a hyperspectral image and the text features describing the hyperspectral image is the highest.

[0011] 15) Test the CLIP-based multimodal hyperspectral image classification model: Input the hyperspectral images and prompt texts in the unlabeled test set into the trained multimodal hyperspectral image classification model for testing. The prompt text with the highest cosine similarity with the hyperspectral image is the classification result of the hyperspectral image.

[0012] The multi-modal self-supervised hybrid Mamba hyperspectral image classification dataset processing includes the following steps:

[0013] 21) Collect hyperspectral images, perform preprocessing of cropping, data denoising, band dimension reduction and channel normalization on all hyperspectral images, and the cropping size is 256×256;

[0014] 22) The preprocessed hyperspectral images are classified according to whether they have labels or not. The hyperspectral images with labels are used as training sets, and the hyperspectral images without labels are divided into training sets and test sets according to the ratio of 8:2.

[0015] 23) Collect text descriptions related to hyperspectral images, including image descriptions, category labels, and geographic information;

[0016] 24) Perform preprocessing operations of stem extraction and word segmentation on the text data. The preprocessed text is called prompt text.

[0017] The construction of a hyperspectral image classification network combining CNN and Mamba includes the following steps:

[0018] 31) Construct a CNN spatial encoder to extract the spatial features of hyperspectral images;

[0019] 311) Its spatial encoder includes a convolutional layer with a kernel size of 3 and a stride of 2, two convolutional layers with a kernel size of 3 and a stride of 1, and each convolutional layer includes a convolution, a batch normalization operation, and a ReLU activation unit;

[0020] 312) Three convolution stack blocks, each convolution stack block includes two 1×1 convolutions and one 3×3 convolution, plus a ReLU activation unit;

[0021] 313) A residual connection between two attention pooling layers and the pooling layer; the final output is the spatial features of the hyperspectral image;

[0022] 32) Construct a Mamba spectral encoder to extract spectral features of hyperspectral images. The spectral encoder includes two layer normalization operations, two Mamba modules, a SiLU activation unit, a residual connection, and an average pooling layer.

[0023] 33) Construct a spatial-spectral information fusion module. The fusion module first upsamples the spatial features twice to restore the resolution, then performs weighted average summation of the upsampled spatial and spectral features, and finally obtains the spatial-spectral features of the hyperspectral image.

[0024] The hyperspectral image classification network pre-training based on the self-supervised CNN and Mamba combination includes the following steps:

[0025] 41) The training data is the hyperspectral images in the unlabeled training set. Each hyperspectral image in a training batch is called an original sample. The original samples are generated by data enhancement methods such as brightness adjustment, shadow generation, and Gaussian blur as positive samples, and the hyperspectral images other than the original samples are negative samples.

[0026] 42) Input all samples into the hyperspectral image classification network to extract the spatial spectrum features of the hyperspectral image;

[0027] 43) Update the model parameters to minimize the loss based on the contrastive loss backpropagation;

[0028] 44) Repeat the above process until the set number of rounds is completed or the loss no longer decreases, the training is completed, and a pre-trained hyperspectral image classification network is obtained.

[0029] The training of the CLIP-based multimodal hyperspectral image classification model includes the following steps:

[0030] 51) Construct a multimodal hyperspectral image classification model based on CLIP, which includes an image encoder and a text encoder. The image encoder uses the hyperspectral image classification network that combines CNN and Mamba, and the text encoder uses the text encoder in CLIP.

[0031] 52) The input data is the hyperspectral image and prompt text in the labeled training set, which are respectively passed through the image encoder and text encoder. The initial weight of the image encoder is obtained by pre-training the hyperspectral image classification network based on the self-supervised CNN and Mamba combination. The initial weight of the text encoder is directly loaded with the pre-trained weight of the CLIP text encoder;

[0032] 53) The hyperspectral image executes an image encoder once to obtain an image feature vector, and the prompt text executes a text encoder once to obtain a text feature vector;

[0033] 54) Calculate the cosine similarity between a pair of image feature vectors and text feature vectors;

[0034] 55) InfoNCE loss between each pair of image and text feature vector is calculated using cosine similarity, and the model parameters are fine-tuned through back propagation;

[0035] 56) Repeat the above process until the set number of times or the average loss value no longer decreases, the training is completed, and a multimodal hyperspectral image classification model is obtained.

[0036] The specific steps of testing the CLIP-based multimodal hyperspectral image classification model are as follows:

[0037] 61) Read the hyperspectral image in the unlabeled test set. The hyperspectral image is cropped to 256×256 and has undergone preprocessing operations such as data denoising, band dimension reduction, and channel normalization;

[0038] 62) Read the prompt text, which has been pre-processed by stem extraction and word segmentation;

[0039] 63) Input the hyperspectral image and prompt text into the trained multimodal hyperspectral image classification model to obtain spatial spectrum features and text features respectively;

[0040] 64) Calculate the cosine similarity between the spatial spectrum feature and the text feature, using the temperature parameter It is scaled and normalized to a probability distribution through softmax. The text with the highest probability is the classification result of the hyperspectral image. Beneficial Effects

[0041] The present invention relates to a hyperspectral image classification method of a multi-mode self-supervised hybrid Mamba, which combines two deep learning algorithms, CNN and Mamba. Compared with the prior art, the present invention introduces the Mamba structure to allow the model to have long-term dependence when extracting spectral information, and also maintains linear complexity, solving the dimensional disaster problem caused by hyperspectral data; through the unlabeled pre-training of the self-supervised hyperspectral image classification network, the model is trained by generating labels from the data itself, reducing the dependence on a large amount of data annotation; through the training of the multi-modal hyperspectral image classification model, the information of the two modes of image and text is combined, which can improve the model's ability to understand hyperspectral images. The effectiveness of this method can not only improve the classification accuracy and provide a more comprehensive scene understanding, but also improve the training efficiency and enhance the robustness of the model, and is suitable for different application fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of a multi-modal self-supervised hybrid Mamba method for hyperspectral image classification;

[0043] Figure 2 This is the structural diagram of the hyperspectral image classification network combining CNN and Mamba;

[0044] Figure 3 This is a training flowchart of the multimodal hyperspectral image classification model based on CLIP; DETAILED DESCRIPTION

[0045] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used to provide a detailed description as follows:

[0046] like Figure 1As shown, the hyperspectral image classification method of a multi-mode self-supervised hybrid Mamba described in the present invention comprises the following steps:

[0047] The first step is to process the hyperspectral image classification dataset of multi-mode self-supervised hybrid Mamba:

[0048] Collect hyperspectral remote sensing images, classify them according to whether they have labels, and perform preprocessing on the images by cropping, data denoising, band dimension reduction, and channel normalization; collect hyperspectral-related text descriptions, including image descriptions, category labels, and geographic information, and perform stem extraction and word segmentation preprocessing on the text data; make the hyperspectral images and texts adapt to the input of the model and improve the generalization ability of the model. The specific steps are as follows:

[0049] (1) Collect hyperspectral images, crop all hyperspectral images to 256 × 256 size, and perform data denoising, band dimension reduction, and channel normalization preprocessing;

[0050] (2) The preprocessed hyperspectral images are classified according to whether they have labels or not. The hyperspectral images with labels are used as training sets, and the hyperspectral images without labels are divided into training sets and test sets according to the ratio of 8:2.

[0051] (3) Collect text descriptions related to hyperspectral images, including image descriptions, category labels, and geographic information, and preprocess the text data by stemming and segmenting. The preprocessed text is called prompt text.

[0052] The second step is to build a hyperspectral image classification network combining CNN and Mamba. The specific steps are as follows:

[0053] (1) Construct a CNN spatial encoder to extract the spatial features of hyperspectral images;

[0054] (1-1) Its spatial encoder consists of a convolutional layer with a kernel size of 3 and a stride of 2, and two convolutional layers with a kernel size of 3 and a stride of 1. Each convolutional layer consists of a convolution, a batch normalization operation, and a ReLU activation unit.

[0055] (1-2) Three convolutional stack blocks, each of which includes two 1×1 convolutions and one 3×3 convolution, plus a ReLU activation unit;

[0056] (1-3) A residual connection between two attention pooling layers and the pooling layer finally obtains the spatial features of the hyperspectral image;

[0057] (2) Construct a Mamba spectral encoder to extract the spectral features of hyperspectral images. The spectral encoder includes two layer normalization operations, two Mamba modules, a residual connection, and an average pooling layer to finally obtain the spectral features of the hyperspectral image.

[0058] (2-1) The Mamba module consists of a linear input layer, a convolutional layer, a SiLU activation unit, a selection state space model, a linear activation gating unit, and a linear output layer, returning an output with the same resolution as the input; the core of Mamba is the selection state space model, which can be represented by the parameter matrix ABCD;

[0059] ,

[0060] ,

[0061] in represents the input sequence, represents the hidden state, represents the update of the hidden state, Represents the predicted output sequence, the A matrix is ​​initialized with the HIPPO matrix, BC is the input data driven matrix, which is obtained by passing the input data through a linear layer, and D is a direct transfer matrix obtained by random initialization;

[0062] (3) Construct a spatial-spectral information fusion module. The fusion module first upsamples the spatial features twice to restore the resolution, then performs weighted average summation of the upsampled spatial and spectral features, and finally obtains the spatial-spectral features of the hyperspectral image.

[0063] ,

[0064] in is the weight, with a value of 0.5, is the upsampling operation, Represents spatial features, Represents the spectral characteristics, represents the fused spatial spectrum features;

[0065] First, the preprocessed hyperspectral image is passed through the spatial and spectral encoders respectively. The preprocessed hyperspectral image is the initial input. The spatial encoder first performs the first downsampling through a convolutional layer with a step size of 2, and then uses two convolutional layers with a step size of 1 and the first attention pooling for the second downsampling. After that, the output is residually connected with the feature map of the second downsampling through three convolution stacking blocks, and finally the third downsampling is performed through the second attention pooling operation to obtain the spatial features; the spectral encoder first passes through a layer normalization and Mamba module, and then passes through a layer normalization, a SiLU activation unit and Mamba module. The output is residually connected with the initial input, and finally the output spectral features are obtained through the average pooling layer; the fusion module upsamples the extracted spatial features twice to restore the resolution, and then averages the weighted sum of the upsampled spatial and spectral features to obtain the spatial-spectral features of the hyperspectral image, thereby completing the construction of the hyperspectral image classification network.

[0066] The third step is to pre-train the hyperspectral image classification network based on the self-supervised CNN and Mamba. The specific steps are as follows:

[0067] (1) The training data is the hyperspectral images in the unlabeled training set. Each hyperspectral image in a training batch is called an original sample. The original samples are generated by data enhancement methods such as brightness adjustment, shadow generation, and Gaussian blur as positive samples, and the hyperspectral images other than the original samples are negative samples.

[0068] (2) All samples are input into the hyperspectral image classification network to extract the spatial-spectral features of the hyperspectral image;

[0069] (2-1) Execute the spatial encoder once to obtain the spatial features of the sample;

[0070] (2-1-1) Perform a 3×3 convolution with a stride of 2, a batch normalization and a ReLU activation function to obtain the first down-sampled feature map;

[0071] (2-1-2) Perform two convolutional layers on the first downsampled feature map. Each convolutional layer includes a 3×3 convolution with a step size of 1, a batch normalization, and a ReLU activation function. Then, an attention pooling operation is performed to obtain the second downsampled feature map.

[0072] (2-1-3) Perform three convolution stacking blocks on the second down-sampled feature map, each convolution stacking block includes two 1×1 convolutions, one 3×3 convolution and a ReLU activation function layer;

[0073] (2-1-4) Perform a residual connection on the output of the convolution stack block and the feature map of the second downsampling, add them together and perform an attention pooling operation to obtain the spatial features of the sample;

[0074] (2-2) Execute the spectral encoder once to obtain the spectral characteristics of the sample;

[0075] (2-2-1) The initial input sample performs layer normalization once, then enters the Mamba module, and returns the first output;

[0076] (2-2-2) Perform layer normalization and SiLU activation function once, then input the Mamba module, return the second output, perform a residual connection between the second output and the initial input, and then obtain the output spectral features through average pooling;

[0077] (2-3) Execute the spatial-spectral information fusion module once, upsample the obtained spatial features twice to restore the resolution, perform weighted average summation on the upsampled spatial features and spectral features, and obtain the fused spatial-spectral features;

[0078] (3) Using contrast loss, we calculate the distance between the spatial spectrum features of the sample input into the hyperspectral image classification network and other samples in the feature space, so that the original sample is close to the positive sample and far away from the negative sample;

[0079] ,

[0080] in is the label of the sample, 1 represents a positive sample and 0 represents a negative sample. Representation sample and The distance between them, m is a hyperparameter with a value of 1.0, indicating the minimum distance between similar samples;

[0081] (4) Update the model parameters to minimize the loss based on the contrastive loss backpropagation;

[0082] (5) Determine whether the set number of rounds has been reached. If so, obtain the pre-trained hyperspectral image classification network. Otherwise, return to (1) and reload the data to continue training.

[0083] The fourth step is to train the multimodal hyperspectral image classification model based on CLIP. The specific steps are as follows:

[0084] (1) Construct a multimodal hyperspectral image classification model based on CLIP, which includes an image encoder and a text encoder. The image encoder uses the hyperspectral image classification network that combines CNN and Mamba, and the text encoder uses the text encoder in CLIP.

[0085] (2) The input data is the hyperspectral image and prompt text in the labeled training set, which are respectively passed through the image encoder and text encoder. The initial weights of the image encoder are obtained by pre-training the hyperspectral image classification network based on the self-supervised CNN and Mamba combination. The initial weights of the text encoder are directly loaded with the pre-trained weights of the CLIP text encoder.

[0086] (3) The hyperspectral image executes an image encoder once to obtain an image feature vector, and the prompt text executes a text encoder once to obtain a text feature vector;

[0087] (4) Calculate the cosine similarity between a pair of image feature vectors and text feature vectors;

[0088] ,

[0089] in and Represent hyperspectral images and prompt texts respectively, and Z represents feature embedding;

[0090] (5) Use cosine similarity to calculate the loss between each pair of image and text feature vectors. The loss function used is InfoNCE.

[0091] ,

[0092] Where N is the batch size, is the temperature parameter, with a value of 0.1, which is used to adjust the distribution of similarity. and Both represent the similarity between a pair of image and text feature vectors. and Represents the similarity between all images and text feature vectors;

[0093] (6) Use the loss function to fine-tune, back-propagate to update the model parameters, and determine whether the set number of rounds has been reached. If so, a trained multimodal hyperspectral image classification model is obtained. Otherwise, return to (2) to reload the data and continue training.

[0094] In the fifth step, the specific steps for testing the CLIP-based multimodal hyperspectral image classification model are as follows:

[0095] (1) Read the hyperspectral image in the unlabeled test set. The hyperspectral image is cropped to a size of 256×256 and preprocessed with data denoising, band dimension reduction, and channel normalization.

[0096] (2) Read the prompt text, which has been preprocessed by stem extraction and word segmentation;

[0097] (3) Input the hyperspectral image and prompt text into the trained multimodal hyperspectral image classification model to obtain spatial spectrum features and text features respectively;

[0098] (4) Calculate the cosine similarity between the spatial spectrum feature and the text feature, using the temperature parameter It is scaled and normalized to a probability distribution through softmax. The prompt text with the highest probability is the classification result of the image pixels.

[0099] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-mode self-supervised hybrid Mamba hyperspectral image classification method, characterized in that: The following steps are involved: 11) Processing of the hyperspectral image classification dataset of multi-mode self-supervised hybrid Mamba: Collect hyperspectral images, perform pre-processing of cropping, data denoising, band dimension reduction and channel normalization on all hyperspectral images, and then classify them according to whether they have labels or not, and use the labeled hyperspectral images as the training set TrainData withlabel , the unlabeled hyperspectral images are divided into the training set TrainData according to 8:2 nolabel and the test set TestData nolabel ; In addition, the text description data of the hyperspectral image is collected. The text description data includes image description, category label description and geographic information description, and the text description data is preprocessed by stem extraction and word segmentation. The preprocessed text is the prompt text; 12) Construct a hyperspectral image classification network combining CNN and Mamba. The network consists of three components: a CNN spatial encoder for extracting the spatial features of hyperspectral images, a Mamba spectral encoder for extracting the spectral features of hyperspectral images, and a spatial-spectral information fusion module. By inputting the hyperspectral image into the network, the spatial-spectral features of the hyperspectral image are obtained. 121) The CNN spatial encoder includes a convolutional layer with a convolution kernel of 3 and a stride of 2, two convolutional layers with a convolution kernel of 3 and a stride of 1, three convolutional stacking blocks, two attention pooling layers, and residual connections between the pooling layers. The final output is the spatial features of the hyperspectral image; Each convolutional layer includes a convolution, a batch normalization operation and a ReLU activation unit, and each convolution stack block includes two 1×1 convolutions, a 3×3 convolution and a ReLU activation unit; 122) The Mamba spectral encoder includes two layer normalization operations, two Mamba modules, a residual connection and an average pooling layer. The core of Mamba is the selection state space model, which can be represented by the parameter matrix ABCD. The output of the Mamba spectral encoder is the spectral feature. h′(t)=Ah(t)+Bx(t), y(t)=Ch(t)+Dx(t), Where x(t) represents the input sequence, h(t) represents the hidden state, h′(t) represents the update of the hidden state, y(t) represents the predicted output sequence, the A matrix is ​​initialized with the HIPPO matrix, BC is the input data driven matrix, which is obtained by passing the input data through a linear layer, and D is a direct transfer matrix, which is obtained by random initialization; 123) The spatial-spectral information fusion module first upsamples the spatial features twice to restore the resolution, and then performs weighted average summation of the upsampled spatial features and spectral features to finally obtain the spatial-spectral features of the hyperspectral image; F sum =w×Up(F spa )+w×F spe , Where w is the weight, the value is 0.5, Up is the upsampling operation, and F spa Represents the spatial feature, F spe represents the spectral characteristics, F sum represents the fused spatial spectrum features; 124) The hyperspectral image is passed through a CNN spatial encoder and a Mamba spectral encoder. The CNN spatial encoder first performs a first downsampling through a convolution layer with a step size of 2, and then uses two convolution layers with a step size of 1 and the first attention pooling for a second downsampling. After that, the output obtained through three convolution stacking blocks is residually connected with the feature map of the second downsampling, and finally the third downsampling is performed through the second attention pooling operation to obtain spatial features; the Mamba spectral encoder first passes through a layer normalization and a Mamba module, and then passes through a layer normalization, a SiLU activation unit and a Mamba module. The output is residually connected with the initial input, and finally the output spectral features are obtained through an average pooling layer; The fusion module upsamples the extracted spatial features twice to restore the resolution, and then averages and weights the upsampled spatial and spectral features to obtain the spatial-spectral features of the hyperspectral image, thereby completing the construction of a hyperspectral image classification network combining CNN and Mamba. 13) Pre-training of hyperspectral image classification network based on self-supervised CNN and Mamba: using contrastive learning method, the training data is TrainData nolabel , each hyperspectral image in a training batch is called an original sample. The original samples generated by brightness adjustment, shadow generation, and Gaussian blur data enhancement are positive samples, and the remaining samples in the original samples except the positive samples are negative samples; all samples are input into the hyperspectral image classification network combined with CNN and Mamba for pre-training; so that the distance between the original sample feature representation and the positive sample feature representation in the embedding space is 0, while the distance with the negative sample feature representation is greater than 1; 14) Training of multimodal hyperspectral image classification model based on CLIP: The constructed multimodal hyperspectral image classification model based on CLIP includes an image encoder and a text encoder; the image encoder uses a hyperspectral image classification network combining CNN and Mamba, and the input data of the image encoder is TrainData withlabel ; The text encoder uses the text encoder in CLIP, with prompt text as input and text features as output, and weights are initialized through the pre-trained weights of CLIP; the cosine similarity between the spatial spectral features output by the image encoder and the text features output by the text encoder is calculated, so that the cosine similarity between the spatial spectral features of a hyperspectral image and the text features describing the hyperspectral image is the highest; 15) Test the CLIP-based multimodal hyperspectral image classification model: TestData nolabel The prompt text is input into the trained CLIP-based multimodal hyperspectral image classification model for testing. The prompt text with the highest cosine similarity with the hyperspectral image is the classification result of the hyperspectral image.

2. The hyperspectral image classification method of a multi-mode self-supervised hybrid Mamba according to claim 1 is characterized in that: The hyperspectral image classification network pre-training based on the self-supervised CNN and Mamba combination includes the following steps: 21) The training data is TrainData nolabel , each hyperspectral image in a training batch is called an original sample. The original sample is generated by data enhancement methods such as brightness adjustment, shadow generation, and Gaussian blur as a positive sample, and the remaining samples except the original sample are negative samples; 22) All samples were input into the hyperspectral image classification network combining CNN and Mamba to extract spatial spectrum features; 221) Execute a CNN spatial encoder to obtain spatial features; Perform a 3×3 convolution with a stride of 2, a batch normalization and a ReLU activation function to obtain the first downsampled feature map; Perform two convolutional layers on the first downsampled feature map. Each convolutional layer includes a 3×3 convolution with a stride of 1, a batch normalization, and a ReLU activation function. Then, perform an attention pooling operation to obtain the second downsampled feature map. Perform three convolution stacking blocks on the second downsampled feature map. Each convolution stacking block includes two 1×1 convolutions, one 3×3 convolution and a ReLU activation function layer. The obtained output is residually connected with the second downsampled feature map, and then the attention pooling operation is performed after addition to obtain the spatial features of the three downsamples. 222) Execute the Mamba spectral encoder once to obtain spectral features; Perform a layer normalization, input the Mamba module, and return the first output; Perform layer normalization and SiLU activation function once, then input into the Mamba module to get the second output, perform a residual connection with the initial input, and finally get the downsampled spectral features through average pooling; 223) Execute a fusion module to fuse the two features; The obtained spatial features are upsampled twice to restore the resolution, and then the upsampled spatial features and spectral features are averaged and weighted to obtain the fused spatial-spectral features; 23) Using contrast loss, calculate the distance between the spatial spectrum features of the original sample input into the hyperspectral image classification network combined with CNN and Mamba and other samples in the feature space, so that the distance between the original sample and the positive sample in the embedding space is 0, while the distance with the negative sample is greater than 1; where y i is the label of the sample, 1 represents a positive sample and 0 represents a negative sample, D(x i , x j ) represents the sample x i With x j The distance between them, m is a hyperparameter with a value of 1.0, indicating the minimum distance between similar samples; 24) Update the model parameters to minimize the loss based on the contrastive loss backpropagation; 25) Determine whether the set number of rounds has been reached. If so, obtain the pre-trained CNN and Mamba combined hyperspectral image classification network. Otherwise, return to step 31) and reload the data to continue training.

3. The hyperspectral image classification method of a multi-mode self-supervised hybrid Mamba according to claim 1 is characterized in that: The training of the CLIP-based multimodal hyperspectral image classification model includes the following steps: 31) Construct a multimodal hyperspectral image classification model based on CLIP, which includes an image encoder and a text encoder. The image encoder uses the hyperspectral image classification network that combines CNN and Mamba, and the text encoder uses the text encoder in CLIP. 32) Input data is TrainData withlabel and prompt text, which are respectively passed through the image encoder and text encoder. The initial weights of the image encoder are obtained by pre-training the hyperspectral image classification network based on the self-supervised CNN and Mamba combination. The initial weights of the text encoder are directly loaded with the pre-trained weights of the CLIP text encoder. 33)TrainData withlabel The image encoder is executed once to obtain the image feature vector, and the text encoder is executed once to obtain the text feature vector; 34) Calculate the cosine similarity between a pair of image feature vectors and text feature vectors; Among them I i and T i Respectively represent TrainData withlabel and prompt text, Z represents feature embedding; 35) Use cosine similarity to calculate the loss between each pair of image feature vectors and text feature vectors. The loss function used is InfoNCE. Where N is the batch size, τ is the temperature parameter with a value of 0.1, which is used to adjust the distribution of similarity, S(I i , T i ) and S(T i , I i ) represents the similarity between a pair of image feature vectors and text feature vectors, S(I i , T j ) and S(T i , I j ) represents the similarity between all image feature vectors and text feature vectors; 36) Use the InfoNCE loss function to fine-tune the model, back-propagate and update the model parameters, and determine whether the set number of rounds has been reached. If so, a trained CLIP-based multimodal hyperspectral image classification model is obtained, otherwise return to step 42) and reload the data to continue training.

4. The hyperspectral image classification method of a multi-mode self-supervised hybrid Mamba according to claim 1 is characterized in that: The specific steps of testing the CLIP-based multimodal hyperspectral image classification model are as follows: 41) Read TestData nolabel , TestData nolabel Cropped to 256×256 size; read the prompt text; 42) TestData nolabel and prompt text input the trained CLIP-based multimodal hyperspectral image classification model to obtain spatial spectrum features and text features respectively; 43) Calculate the cosine similarity of the spatial spectrum feature and the text feature, scale them with the temperature parameter τ, and normalize them into a probability distribution through softmax. The prompt text with the largest probability is the classification result of the hyperspectral image.

Images