Mask Transform and contrast learning-based hyperspectral image classification method

Abstract

The invention discloses a hyperspectral image classification method based on mask Transform and contrast learning, and the method comprises the steps: obtaining and preprocessing hyperspectral image data, obtaining a sample set, and obtaining a training set based on the sample set; constructing an initial hyperspectral image classification model and a model overall loss function; dividing the training set into a labeled data set and a label-free data set, pre-training the initial hyperspectral image classification model based on the label-free data set and a model overall loss function, simplifying and recombining, and performing supervised training on the simplified and recombined model based on the labeled data set and a cross entropy loss function, and performing hyperspectral image classification through the trained model. According to the method, multi-scale mask reconstruction and local contrast learning are combined, so that the attention of the model to the ground feature in the hyperspectral image is improved, the space and channel information of the feature map is enriched, the weight of the ground feature is enhanced, and the accuracy and robustness of spectral image classification are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a hyperspectral image classification method based on mask Transformer and contrast learning. Background Art

[0002] Hyperspectral images usually contain dozens to hundreds of spectral bands, and in addition to the spatial information of the ground objects, they also have rich spectral representations. Compared with the three-channel representation of RGB images, this provides stronger support for identifying ground object information. The hyperspectral image classification task is an important task in the field of remote sensing image processing. It requires assigning a ground object true label to each pixel in the hyperspectral image, and has a wide range of applications in agriculture, environment, aerospace and other fields.

[0003] Deep learning technology has made remarkable achievements in the field of hyperspectral image classification with its excellent deep feature extraction ability, and has promoted the widespread application of this technology. However, despite the progress, current deep learning methods still face several challenges in further improving the performance of hyperspectral image classification. In particular, in certain specific application scenarios, it is difficult to collect a large amount of high-quality training data, which directly limits the training effect of deep learning models. At the same time, the high-dimensional characteristics of hyperspectral images put forward higher requirements for model training, and the model must be able to effectively process and learn these complex data. In view of this, developing an algorithm model that can efficiently process high-dimensional data and achieve excellent performance under small sample conditions is of great significance to advancing research in the field of hyperspectral image classification. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a hyperspectral image classification method based on mask Transformer and contrastive learning to solve the problems existing in the above prior art.

[0005] To achieve the above object, the present invention provides a hyperspectral image classification method based on mask Transformer and contrastive learning, comprising:

[0006] Acquire and preprocess hyperspectral image data to obtain a sample set, and obtain a training set and a validation set based on the sample set;

[0007] Construct the initial hyperspectral image classification model and the overall loss function of the model;

[0008] The training set is divided into a labeled data set and an unlabeled data set, and self-supervised iterative pre-training of the initial hyperspectral image classification model is completed based on the unlabeled data set and the overall loss function of the model;

[0009] Simplifying and reorganizing the pre-trained initial hyperspectral image classification model, and conducting supervised training on the simplified and reorganized hyperspectral image classification model based on the labeled data set and the cross entropy loss function; and using the validation set to evaluate the performance of the trained hyperspectral image classification model;

[0010] Hyperspectral image classification is performed based on the hyperspectral image classification model that has completed two training stages.

[0011] Optionally, the process of obtaining a sample set includes: removing noise and outliers in the hyperspectral image data; setting a window step size, padding pixels on the edge of the hyperspectral image with half the window step size, and then performing linear normalization; extracting a sample cube based on the window step size with the pixel as the center to obtain a sample set.

[0012] Optionally, the initial hyperspectral image classification model includes a multi-scale enhanced feature mapping module and a dual-branch multi-scale mask Transformer reconstruction; the multi-scale feature mapping module generates a low-level scale feature map and a high-level scale feature map according to the hyperspectral image data; the dual-branch multi-scale mask Transformer reconstruction obtains a corresponding first reconstructed image and a second reconstructed image based on the low-level scale feature map and the high-level scale feature map.

[0013] Optionally, the hyperspectral image data is input into the multi-scale enhanced feature mapping module, and the spatial size of the hyperspectral image data is used as a reference, a cropping interval is preset, and the hyperspectral image is cropped at multi-scale equal intervals based on the cropping interval to obtain cropped image blocks of different scales; after selecting three cropped image blocks of different scales for scaling, an image block with an intermediate scale is selected as an intermediate scale, and after convolution operations are performed on the cropped image blocks of three different scales, the cropped image block corresponding to the intermediate scale is used as a residual connection feature, and the remaining two image blocks with smaller intermediate scales are fused with the residual connection features to obtain a low-scale feature map, and the image blocks with smaller scales are fused with the residual connection features to obtain a high-scale feature map.

[0014] Optionally, the dual-branch multi-scale mask Transformer reconstruction includes a low-level scale reconstruction branch and a high-level scale reconstruction branch; the low-level scale feature map is input into the low-level scale reconstruction branch to obtain a first reconstructed image, and the high-level scale feature map is input into the high-level scale reconstruction branch to obtain a second reconstructed image; and the reconstruction loss is calculated based on the first reconstructed image, the second reconstructed image and the hyperspectral image data.

[0015] Optionally, the low-level scale reconstruction branch has the same structure as the high-level scale reconstruction branch;

[0016] The low-level scale reconstruction branch and the high-level scale reconstruction branch encode the corresponding low-level scale feature map or high-level scale feature map into several non-overlapping image blocks through image block coding embedding, and after adding the position coding, a mask strategy based on the central area is adopted to obtain a mask feature image; the mask feature image is feature-encoded by the encoder, and the learnable vector of the encoded mask feature image is filled into the mask position, and after adding the position coding for reconstruction, it is reconstructed by the decoder, and the corresponding first reconstructed image and second reconstructed image are output respectively, wherein each layer structure of the encoder and the decoder is the same.

[0017] Optionally, the central area-based mask strategy includes: using masks of different sizes to process the low-level scale feature map and the high-level scale feature map after adding position encoding, respectively, to obtain corresponding mask feature images, the mask position is the central area of ​​the low-level scale feature map and the high-level scale feature map, wherein the mask used by the low-level scale reconstruction branch is smaller than the mask used by the high-level scale reconstruction branch.

[0018] Optionally, when pre-training the initial hyperspectral image classification model, a local contrast learning module is introduced, and the local contrast learning module includes an encoding local loss unit and a decoding local loss unit with the same structure; the local contrast learning module takes the output of a dual-branch multi-scale mask Transformer reconstruction encoder and decoder as input, the output of the encoder is the input of the encoding local loss unit, and the output of the decoder is the input of the decoding local loss unit, and the NT-Xent loss is used to calculate the encoder local loss and the decoder local loss.

[0019] Optionally, the overall loss function of the model is expressed as:

[0020] L=L rec +λL con

[0021] Where, L rec represents the reconstruction loss, L con represents a local contrast loss; the local contrast loss is the sum of the encoder local loss and the decoder local loss;

[0022] The reconstruction loss calculation formula is expressed as follows:

[0023]

[0024] L rec =L ls +L hs

[0025] Where, L ls and L hsdenote the reconstruction losses of the low-scale reconstruction branch and the high-scale reconstruction branch, respectively, and m l and m h Respectively represent the number of image blocks covered by the two branches, R l and R h represent the first reconstructed image and the second reconstructed image respectively, P O Represents the original hyperspectral image data divided into image blocks.

[0026] Optionally, the process of simplifying and reorganizing the pre-trained initial hyperspectral image classification model includes:

[0027] The mask operation, decoder and multi-scale enhancement part of the multi-scale feature enhancement mapping module of the pre-trained initial hyperspectral image classification model are eliminated, a multi-scale adaptive feature fusion module and a fully connected layer are introduced, the output end of the encoder is connected to the input end of the multi-scale adaptive feature fusion module, and the output end of the multi-scale adaptive feature fusion module is connected to the fully connected layer.

[0028] Compared with the prior art, the present invention has the following advantages and technical effects:

[0029] The present invention designs a new mask strategy based on the central area of ​​the image and a local contrast learning method for reconstruction tasks, so that the model can better learn the multi-scale spatial-spectral features of the central local area in the self-supervised pre-training stage, improve the effect of the reconstruction task and the model's ability to capture key features; proposes a new multi-scale feature enhancement mapping module and a new multi-scale adaptive feature fusion module, which are respectively used to obtain feature maps of different mixed scales in the pre-training stage and to adaptively fuse feature maps of different scales in the fine-tuning stage. In this way, the model can more effectively capture and integrate multi-scale spatial-spectral information, improve the richness of feature expression and the accuracy of classification. The present invention combines multi-scale mask reconstruction with local contrast learning, which can improve the model's attention to the features of the ground objects in the hyperspectral image, enrich the spatial and channel information of the feature map, and enhance the weight of the ground object features, thereby improving the accuracy and robustness of spectral image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0031] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0032] Figure 2 This is a diagram of the overall network architecture of the pre-training phase of an embodiment of the present invention;

[0033] Figure 3 This is a structural diagram of a multi-scale feature enhancement mapping module according to an embodiment of the present invention;

[0034] Figure 4 1 is an example diagram of a central region masking strategy according to an embodiment of the present invention, wherein (a) is a spatial-spectral mask image block, and (b) is a spatial-spectral mask unit;

[0035] Figure 5 This is a diagram of the overall network architecture in the fine-tuning stage of an embodiment of the present invention;

[0036] Figure 6 It is a structural diagram of a multi-scale adaptive feature fusion module according to an embodiment of the present invention;

[0037] Figure 7 : The classification comparison result diagram of the present invention and other 8 methods on the Pavia University dataset of the embodiment of the present invention; wherein, (a) is the ground truth image, (b) is the classification result diagram of the RNN method, (c) is the classification result diagram of the 2D-CNN method, (d) is the classification result diagram of the HybridSN method, (e) is the classification result diagram of the ViT method, (f) is the classification result diagram of the morphFormer method, (g) is the classification result diagram of the C-SS-MTr method, and (h) is the classification result diagram of the CMT method; (i) is the classification result diagram of the SS-MAE method; (j) is the classification result diagram of the method proposed in the present invention;

[0038] Figure 8 : The figures are the classification comparison results of the present invention and other 8 methods on the Indian Pines dataset of an embodiment of the present invention; wherein, (a) is the ground truth image, (b) is the classification result image of the RNN method, (c) is the classification result image of the 2D-CNN method, (d) is the classification result image of the HybridSN method, (e) is the classification result image of the ViT method, (f) is the classification result image of the morphFormer method, (g) is the classification result image of the C-SS-MTr method, and (h) is the classification result image of the CMT method; (i) is the classification result image of the SS-MAE method; and (j) is the classification result image of the method proposed in the present invention.

[0039] Fig. 93. The figures are the classification comparison results of the present invention and other 8 methods on the Houston2013 data set of an embodiment of the present invention; wherein, (a) is the ground truth image, (b) is the classification result image of the RNN method, (c) is the classification result image of the 2D-CNN method, (d) is the classification result image of the HybridSN method, (e) is the classification result image of the ViT method, (f) is the classification result image of the morphFormer method, (g) is the classification result image of the C-SS-MTr method, (h) is the classification result image of the CMT method; (i) is the classification result image of the SS-MAE method; (j) is the classification result image of the method proposed in the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] Embodiment 1

[0043] like Figure 1 As shown, this embodiment provides a hyperspectral image classification method based on mask Transformer and contrast learning, including:

[0044] S1: Collect hyperspectral image data and preprocess the data. The preprocessed hyperspectral data is used as sample data and divided into training set, validation set and test set; the specific method is:

[0045] Use hyperspectral cameras or remote sensing satellites to capture and collect hyperspectral images, including the annotation of image category labels;

[0046] The data was preprocessed to remove noise and outliers from the data, and the pixels on the edges of the image were padded with 1 / 2 of the set window step. Normalization was used to map the overall value of the data set to the range of 0 to 1. The sample cube was extracted with a fixed window step with the pixel as the center, and the training set, validation set, and test set were generated by random sampling according to the set ratio. The window step was actually set to 27.

[0047] Real public hyperspectral image classification datasets, namely the Pavia University dataset, the IndianPines dataset, and the Houston2013 dataset, are selected for training and evaluation of the hyperspectral image classification model.

[0048] Pavia University Dataset: This dataset is collected by the Reflection Optical System Imaging Spectrometer (ROSIS) sensor of the University of Pavia, Italy. The spatial size of the dataset is 610×340 pixels, the spectral band range is 430~860nm, and it contains 103 spectral bands, of which 12 bands affected by water absorption have been removed. The dataset contains 9 ground object categories, and 20 samples are randomly selected from each category for training, and the rest are used for testing.

[0049] Indian Pines dataset: This dataset was collected using the AVIRIS sensor in northwest Indiana, USA. The spatial size of the image is 145×145 pixels, the spectral band ranges from 400 to 2500nm, and there are 200 spectral bands in total, of which 20 bands affected by water absorption have been removed. The dataset contains 16 ground object categories, and 20 samples of each category are randomly selected for training. If there are not enough samples, 15 samples are selected, and the rest are used for testing.

[0050] Houston2013 dataset: This dataset was collected using the CASI-1500 sensor in and around the University of Houston, USA. The spatial size is 949×1905 pixels and contains 144 spectral bands ranging from 380 to 1050nm. The cloud-free version provided by the GRSS data fusion competition is used in this paper. The dataset contains 15 ground object categories, and 20 samples are randomly selected from each category for training, and the rest are used for testing.

[0051] S2: When building a hyperspectral image classification model, the processing of the two training stages is as follows Figure 2 and Figure 5 As shown, it includes a multi-scale enhanced feature mapping module, a mask strategy based on the central area, a dual-branch multi-scale mask Transformer reconstruction, a local contrast learning mechanism and a multi-scale adaptive feature fusion module; among them, the initial hyperspectral image classification model includes a multi-scale enhanced feature mapping module, a mask strategy based on the central area, a dual-branch multi-scale mask Transformer reconstruction, and a local contrast learning mechanism.

[0052] The multi-scale enhanced feature mapping module is as follows Figure 3 As shown in , this module generates image patches of different scales through multi-scale cropping and random selection strategies. These image patches are then carefully mapped through two-dimensional convolution and residual connection, integrating spatial features and spectral information of different scales, and finally generating feature maps of mixed different scales. Specifically, this module transforms the input X∈R H×W×SRepresents an image block of the entire input hyperspectral image, where H and W represent the height and width dimensions respectively, and S represents the spectral dimension. The input image block X is cropped with different specified window interval sizes. The specific operation is to take the given spatial size H×W of the input image block as the benchmark, and crop it down five times at equal intervals with a as the interval to generate five cropped image blocks of different scales. Then, one of the two smaller image blocks is randomly selected and its spatial size is scaled to the input size H×W. Similarly, the same operation is performed on the two larger image blocks, and finally the cropped image block of the middle size is selected for scaling as the intermediate scale. In this way, P is obtained. L , P M , P H Three image blocks containing information of different scales. Subsequently, these image blocks are convolved to map their spatial-spectral features. In order to enhance feature fusion and improve the generalization ability of the model, P M As the residual connection feature, it is fused into the feature maps of different scales to obtain the output feature map F L ∈R H×W×D , F H ∈R H×W×D , where D is the spectral dimension after mapping. The MAFM module can be expressed in mathematical form as follows:

[0053] (P L ,P M ,P H )=RSR(MC(X,a))

[0054]

[0055]

[0056] Where a represents the specified interval size, MC(·) represents the multi-scale cropping operation of the input data X with a as the interval, and RSR(·) represents the random selection and resizing operation of multiple patches of different scales after cropping to obtain P L ,P M ,P H ∈R H×W×S , respectively represent the generated low-scale, medium-scale, and high-scale image blocks. represents element-wise addition for residual connection. L ,F H ∈R H×W×D , respectively represent the generated low-level and high-level feature maps.

[0057] Mask strategy based on center area:

[0058] The central pixel and its local information are crucial for HSI classification, while the information of the surrounding pixels is usually related to the central pixel and is important information to assist the model in feature modeling. However, some existing random masking strategies will lead to a certain degree of redundant information modeling in the model. In order to enable the model to better learn the association between the central local information and the surrounding feature information, this embodiment provides a spatial-spectral masking strategy based on the central area, such as Figure 4 As shown. Figure 4 In (a), the dark area in the center represents the masked area, and the specific representation of the mask unit is shown in Figure 4 (b). By adjusting the hyperparameters, the size of the mask window can be flexibly adjusted. This masking strategy masks the central pixel and its surrounding area, forcing the model to learn the potential correlation between the central and surrounding feature information during the reconstruction process. It is worth noting that different mask sizes are used in the two branches, which enables the model to learn at different scales and improve its understanding and modeling capabilities of multi-scale features.

[0059] The dual-branch multi-scale mask Transformer reconstruction part shown includes:

[0060] The multi-scale spatial-spectral Transformer reconstruction includes two reconstruction branches, namely the low-level scale reconstruction branch and the high-level scale reconstruction branch. The two branches use a central region-based mask strategy with different mask ratios to reconstruct the input hyperspectral image block X as the reconstruction target, and obtain the first reconstructed image corresponding to the low-level scale feature map and the second reconstructed image corresponding to the high-level scale feature map. In this way, richer multi-scale spatial-spectral information can be learned during the reconstruction process, and the association between the central local information and the surrounding information can be learned to enhance the model's understanding of the image. Taking the low-level scale reconstruction branch as an example, the input feature map F L ∈R H ×W×D First, the input feature map is encoded into a series of non-overlapping image blocks through image block encoding embedding, and position encoding is added to obtain Where P is the image block size, N = (H × W) / P 2 is the number of coded image blocks. Then it is subjected to a central area mask operation according to a given mask window to obtain Where T represents the dimension after masking. The above process can be expressed in mathematical form as:

[0061]

[0062] Where PE(·) represents the image block encoding embedding operation, E posrepresents the position encoding used, CPM(·) represents the mask operation based on the center area, and m represents the specified mask window size.

[0063] Then, the masked data It is sent to the low-level scale spatial-spectral encoder based on Transformer for feature encoding to obtain features containing long-range dependencies. Then will The feature mapping is performed through the linear mapping layer, and the learnable vector is filled into the masked position, the position encoding for reconstruction is added, and then it is put into the low-level scale space-spectral decoder for reconstruction, and finally the reconstruction loss is calculated by mapping and the original input data. It is worth noting that each layer of the encoder and decoder consists of the same structure. The i-th layer formula of the encoder and decoder is as follows:

[0064] F′ i =MSA(LN(F i-1 ))+F i-1

[0065] F i =MLP(LN(F′ i ))+F i

[0066] where F i-1 represents the output of the i-1th layer of the encoder and decoder, LN(·) represents layer normalization, MSA(·) represents multi-head self-attention, MLP(·) represents multi-layer perceptron, and F i represents the output of the i-th layer.

[0067] Furthermore, the local contrastive learning mechanism focuses on calculating features near the central area. This approach allows the network to concentrate on capturing the most representative regional information, thereby reducing the impact of redundant features. This approach significantly enhances the network's ability to model local central information during training and promotes mutual learning between the two branches in feature modeling. The model uses the encoder outputs of the two branches as contrast pairs, enabling the network to learn multi-scale representations of central pixels during feature modeling. In order to ensure that the details of multi-scale information are captured during feature extraction and reconstruction, local contrastive learning is also applied to the decoder outputs of the two branches. This further enhances the network's ability to capture multi-scale information and understand central local features during reconstruction. The local contrastive learning loss is calculated using the NT-Xent loss.

[0068] Taking the output features of the decoder as an example, the decoder outputs of the low-level scale reconstruction branch and the high-level scale reconstruction branch are and This process can be expressed mathematically as:

[0069]

[0070] Among them, PH(·) represents the mapping head, which consists of a linear layer and an average pooling layer, c represents the position parameter, and the average pooling layer only calculates the features of the given position. and They represent the local features obtained after the mapping head processing, L CDe represents the local contrastive loss calculated for the decoder output and τ represents the temperature parameter.

[0071] The multi-scale adaptive feature fusion module is as follows: Figure 6 As shown in Figure 2, this module aims to efficiently extract and integrate multi-scale spatial-spectral features through convolutional mapping and adaptive attention mechanism to enhance the interaction of features between different scales. Specifically, it is assumed that the low-scale spatial-spectral encoder and the high-scale spatial-spectral encoder after the unsupervised pre-training stage output the features F LS and F HS The process first independently LS and F HS Apply the convolution operation and get F' LS and F' HS . Then, adaptive attention calculations are performed on these two sets of features through the adaptive average pooling layer and the convolution layer, and F” is obtained respectively. LS and F” HS This step not only emphasizes the most relevant patterns at each scale, but also further enhances the feature recognition ability by highlighting the complementary information between features at different scales. Finally, these two features are fused to form a unified output map. The mathematical expression of this module can be expressed as:

[0072] F′ LS =Conv1D(F LS ),F′ HS =Conv1D(F HS )

[0073]

[0074]

[0075] F = Conv1D(Concat(F″ LS ,F″ HS ))

[0076] Among them, AvgPool represents the adaptive average pooling layer, represents element-wise multiplication, and Concat represents the concatenation of features in a specified dimension. F represents the final output of the module. After F is mapped by the fully connected layer, the classification graph output of the model can be obtained.

[0077] S3: Use the unlabeled training set to perform self-supervised pre-training on the hyperspectral image classification model, such as Figure 2 As shown in the figure, the input data is masked and then sent to the Encoder for feature extraction, which is reconstructed by the Decoder. Contrastive learning is performed simultaneously, multiple iterations of training are performed, the loss function is calculated, and finally the training parameters of the model are saved. The loss calculation of the model is as follows:

[0078] The mean square loss is used to calculate the reconstruction loss of the two branches respectively. Its mathematical form can be expressed as:

[0079]

[0080] L rec =L ls +L hs

[0081] Where L ls and L hs denote the reconstruction losses of the low-scale reconstruction branch and the high-scale reconstruction branch respectively. We only calculate the loss of the mask part. m l and m h Respectively represent the number of image blocks covered by the two branches, R l and R h Represent the reconstruction data obtained by the two branches, P O represents the original input hyperspectral image data after image block division, L rec represents the total reconstruction loss.

[0082] The NT-Xent loss is used to calculate the local contrast loss L of the two branch encoder outputs. CEn NT-Xent loss can be expressed mathematically as:

[0083]

[0084] Where N represents the number of samples in a mini-batch, (x i ,x j ) means that for the sample pair of the kth sample, s(p,q)=p T q / ||p||||q|| represents the l2-normalized inner product between p and q. is an indicator function that returns 1 if and only if k≠i.

[0085] The overall loss of the model can be expressed as:

[0086] L con =L CEn +L CDe

[0087] L=L rec +λL con

[0088] Where L con represents the overall contrast loss, λ represents the parameter that controls the proportion of local contrast learning loss, and L represents the overall loss of the model.

[0089] S4: Figure 5 As shown, a small amount of labeled data is used to perform supervised fine-tuning on the hyperspectral image classification model, update parameters and optimize the model; and the performance of the hyperspectral image classification model is verified using the validation set;

[0090] After the unsupervised pre-training stage, the model has excellent multi-scale spatial-spectral feature modeling capabilities. On this basis, the model is further supervised fine-tuned to enable the model to accurately classify hyperspectral images. In the fine-tuning stage, the mask operation is cancelled and the decoder part is removed. In addition, the multi-scale enhancement part of the multi-scale feature enhancement mapping module is cancelled, and only the feature mapping part is retained.

[0091] The model uses the cross entropy loss to calculate the classification loss after the output F of the adaptive multi-scale feature fusion module is mapped through a linear layer. F is input into the classification head for softmax classification probability prediction, the cross entropy loss is calculated, and all weight parameters of the refinement model are adjusted.

[0092] The present invention evaluates the results produced by the model in the experiment through overall accuracy (OA) and Kappa coefficient (K). Overall accuracy is the average accuracy of each category. Kappa coefficient measures whether the classification result is consistent with the actual underlying category. The specific calculation formula for calculating the above evaluation criteria is as follows:

[0093]

[0094] Where TP stands for true positives, TN stands for true negatives, FP stands for false positives, and FN stands for false negatives. n is the number of categories, and N is the total number of data samples. ii Represents the values ​​on the diagonal of the confusion matrix, x i+ and x +i Represent the total values ​​of the i-th row and i-th column of the confusion matrix respectively.

[0095] S5: Use the trained hyperspectral image classification model to classify the test set data;

[0096] The effectiveness of the embodiments of the present invention is verified by experiments as follows:

[0097] The embodiment of the present invention is compared with the existing hyperspectral image classification method on the Pavia University dataset, the Indian Pines dataset and the Houston 2013 dataset, and the experimental results shown in Table 1 can be obtained.

[0098] Table 1

[0099]

[0100] As can be seen from Table 1, the method proposed in the present invention outperforms all the comparison methods in terms of OA and Kappa evaluation indicators. The OA results on the Pavia University, Indian Pines and Houston 2013 datasets reached 94.49%, 88.78% and 93.73% respectively. For example, in the experimental results of the Indian Pines dataset with imbalanced categories, the method proposed in the present invention still showed the best results, and OA was 1.07% and 12.39% higher than the best performing CMT and morphFormer, a method based on the combination of CNN and Transformer, among other pre-training methods. The network model of the present invention shows excellent robustness and generalization ability under the condition of limited training samples, and the classification results prove that it can maintain high classification accuracy in various environments and conditions.

[0101] like Figure 7 As shown in the visualization results on the Pavia University dataset, the classification map produced by the method proposed in the present invention has the least noise and is closest to the true label. In contrast, other methods, such as morphFormer, have more misclassifications. This may be because it is difficult to train a high-performance classifier relying solely on supervised learning when there are fewer training samples. In addition, although the C-SS-MTr method uses a masked Transformer based on contrastive learning for pre-training, there is still a lot of noise in the classification map, which may be because the model introduces a large amount of redundant information during the modeling process, interfering with the classification results. The method proposed in the present invention reduces the modeling of redundant information by designing a central region-based mask strategy and local contrastive learning, making it easier for the model to capture important features.

[0102] like Figure 8 and Fig. 9 As shown in Figure 2, on the Indian Pines and Houston 2013 datasets, the method proposed in this invention also achieves the best visualization results, generates the least noise, and is closest to the true label. Figure 8As shown in the figure, the SS-MAE method produces a large number of misclassified pixels on the Indian Pines dataset with imbalanced categories. This may be because although SS-MAE uses a pre-training strategy that combines spatial masking and spectral masking strategies, it does not make full use of multi-scale information in the modeling process, resulting in the inability to further improve the model classification performance.

[0103] from Figure 8 It can be seen that the RNN and ViT methods generate a lot of noise, which may be because these two methods are not sufficient in modeling spatial information and fail to fully extract the complex structural information between image pixels, resulting in unsatisfactory classification results. In contrast, methods using pre-training-fine-tuning strategies, such as C-SS-MTr, CMT and SS-MAE, show relatively excellent performance, which may be because they have learned rich feature representations through the pre-training stage, so that they can better adapt to specific tasks in the fine-tuning stage. However, even so, these methods still have a certain amount of noise compared with the method proposed in the present invention. The reason why the method proposed in the present invention performs excellently is due to its innovative use of a multi-scale feature enhancement mapping module to capture multi-scale input information. In addition, in the fine-tuning stage, the feature maps of different scales are effectively integrated through the adaptive multi-scale feature fusion module, which further improves the accuracy of classification, thereby ensuring excellent classification results.

[0104] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A hyperspectral image classification method based on masked Transformer and contrastive learning, characterized in that: The following steps are involved: Acquire and preprocess hyperspectral image data to obtain a sample set, and obtain a training set and a validation set based on the sample set; Construct the initial hyperspectral image classification model and the overall loss function of the model; The training set is divided into a labeled data set and an unlabeled data set, and self-supervised iterative pre-training of the initial hyperspectral image classification model is completed based on the unlabeled data set and the overall loss function of the model; Simplifying and reorganizing the pre-trained initial hyperspectral image classification model, and conducting supervised training on the simplified and reorganized hyperspectral image classification model based on the labeled data set and the cross entropy loss function; and using the validation set to evaluate the performance of the trained hyperspectral image classification model; Hyperspectral image classification is performed based on the hyperspectral image classification model that has completed two training stages.

2. The hyperspectral image classification method based on masked Transformer and contrastive learning according to claim 1, characterized in that: The process of obtaining a sample set includes: removing noise and outliers in the hyperspectral image data; setting a window step size, filling pixels on the edge of the hyperspectral image with half the window step size, and then performing linear normalization; taking pixels as the center, extracting a sample cube based on the window step size to obtain a sample set.

3. The hyperspectral image classification method based on masked Transformer and contrastive learning according to claim 1, characterized in that: The initial hyperspectral image classification model includes a multi-scale enhanced feature mapping module and a dual-branch multi-scale mask Transformer reconstruction; the multi-scale feature mapping module generates a low-level scale feature map and a high-level scale feature map according to the hyperspectral image data; the dual-branch multi-scale mask Transformer reconstruction obtains the corresponding first reconstructed image and second reconstructed image based on the low-level scale feature map and the high-level scale feature map.

4. The hyperspectral image classification method based on masked Transformer and contrastive learning according to claim 3 is characterized in that: The hyperspectral image data is input into the multi-scale enhanced feature mapping module, based on the spatial size of the hyperspectral image data, a cropping interval is preset and the hyperspectral image is cropped at multi-scale equal intervals based on the cropping interval to obtain cropped image blocks of different scales; After selecting three cropped image blocks of different scales for scaling, an image block with an intermediate scale size is selected as the intermediate scale. After performing convolution operations on the three cropped image blocks of different scales, the cropped image block corresponding to the intermediate scale is used as the residual connection feature, and the remaining two image blocks with smaller intermediate scales are fused with the residual connection features to obtain a low-level scale feature map, and the image blocks with smaller scales are fused with the residual connection features to obtain a high-level scale feature map.

5. The hyperspectral image classification method based on masked Transformer and contrastive learning according to claim 4 is characterized in that: The dual-branch multi-scale mask Transformer reconstruction includes a low-level scale reconstruction branch and a high-level scale reconstruction branch; the low-level scale feature map is input into the low-level scale reconstruction branch to obtain a first reconstructed image, and the high-level scale feature map is input into the high-level scale reconstruction branch to obtain a second reconstructed image; A reconstruction loss is calculated based on the first reconstructed image, the second reconstructed image and the hyperspectral image data.

6. The hyperspectral image classification method based on masked Transformer and contrastive learning according to claim 5, characterized in that: The low-level scale reconstruction branch has the same structure as the high-level scale reconstruction branch; The low-level scale reconstruction branch and the high-level scale reconstruction branch encode the corresponding low-level scale feature map or high-level scale feature map into several non-overlapping image blocks through image block coding embedding, and after adding the position coding, a mask strategy based on the central area is adopted to obtain a mask feature image; the mask feature image is feature-encoded by the encoder, and the learnable vector of the encoded mask feature image is filled into the mask position, and after adding the position coding for reconstruction, it is reconstructed by the decoder, and the corresponding first reconstructed image and second reconstructed image are output respectively, wherein each layer structure of the encoder and the decoder is the same.

7. The hyperspectral image classification method based on masked Transformer and contrastive learning according to claim 6, characterized in that: The central region-based mask strategy includes: using masks of different sizes to process the low-level scale feature map and the high-level scale feature map after adding position encoding, respectively, to obtain corresponding mask feature images, the mask position is the central region of the low-level scale feature map and the high-level scale feature map, wherein the mask used by the low-level scale reconstruction branch is smaller than the mask used by the high-level scale reconstruction branch.

8. The hyperspectral image classification method based on masked Transformer and contrastive learning according to claim 6, characterized in that: When pre-training the initial hyperspectral image classification model, a local contrast learning module is introduced, and the local contrast learning module includes an encoding local loss unit and a decoding local loss unit with the same structure; the local contrast learning module uses the output of a dual-branch multi-scale mask Transformer reconstruction encoder and a decoder as input, the output of the encoder is the input of the encoding local loss unit, and the output of the decoder is the input of the decoding local loss unit, and the NT-Xent loss is used to calculate the encoder local loss and the decoder local loss.

9. The hyperspectral image classification method based on masked Transformer and contrastive learning according to claim 8, characterized in that: The overall loss function of the model is expressed as: L=L rec +λL con Where, L rec represents the reconstruction loss, L con represents a local contrast loss; the local contrast loss is the sum of the encoder local loss and the decoder local loss; The reconstruction loss calculation formula is expressed as follows: L rec =L ls +L hs Where, L ls and L hs denote the reconstruction losses of the low-scale reconstruction branch and the high-scale reconstruction branch, respectively, and m l and m h Respectively represent the number of image blocks covered by the two branches, R l and R h represent the first reconstructed image and the second reconstructed image respectively, P O Represents the original hyperspectral image data divided into image blocks.

10. The hyperspectral image classification method based on masked Transformer and contrastive learning according to claim 7, characterized in that: The process of simplifying and reorganizing the pre-trained hyperspectral image classification model includes: The mask operation, decoder and multi-scale enhancement part of the multi-scale feature enhancement mapping module of the pre-trained initial hyperspectral image classification model are eliminated, a multi-scale adaptive feature fusion module and a fully connected layer are introduced, the output end of the encoder is connected to the input end of the multi-scale adaptive feature fusion module, and the output end of the multi-scale adaptive feature fusion module is connected to the fully connected layer.

Images