Hyperspectral image classification method based on multi-scale feature pyramid

Abstract

The application discloses a hyperspectral image classification method based on a multi-scale feature pyramid, and specifically comprises the following steps: step 1, performing dimension reduction processing on a hyperspectral image to obtain a dimension-reduced hyperspectral image, sampling the dimension-reduced hyperspectral image, and dividing the dimension-reduced hyperspectral image into a training set and a test set; step 2, constructing an improved ViT model; step 3, training the improved ViT model constructed in step 2 by using the training set divided in step 1 to obtain a trained improved ViT model; and step 4, inputting the test set divided in step 1 into the trained improved ViT model to obtain a classification result. The method solves the problems of low classification efficiency and low classification accuracy of the prior art.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a hyperspectral image classification method based on a multi-scale feature pyramid. Background Technology

[0002] Hyperspectral images typically possess hundreds of spectral channels and unique properties of one spectrum per object, leading to their widespread application in agriculture, military surveillance, object recognition, and environmental monitoring. In recent years, the continuous development and widespread use of hyperspectral image acquisition equipment have resulted in a continuously increasing demand for hyperspectral image processing.

[0003] Traditional hyperspectral classification algorithms are limited by the specialized domain knowledge required for manual feature design, resulting in significant research limitations. Deep learning algorithms have addressed the challenges of manual feature design to some extent, but their application in hyperspectral image classification tasks still suffers from high computational complexity and low efficiency. In recent years, ViT has demonstrated the powerful capabilities of Transformer models in image classification tasks, achieving promising results, but it also suffers from limitations such as acquiring information at a single scale and computational complexity. Furthermore, the diverse acquisition devices and scenes used for hyperspectral images lead to varying data distributions and significant spatial variability. Consequently, most existing deep learning or Transformer-based hyperspectral image classification algorithms cannot acquire features at multiple scales. However, hyperspectral images possess rich spectral and spatial information, both of which greatly contribute to the final classification results. Therefore, how to increase the number of feature representations at more scales in the model to improve its performance is a crucial issue for the practical application of hyperspectral image classification algorithms.

[0004] Recent research has focused on multi-scale information fusion in networks to improve model performance. Multi-scale information fusion refers to using certain operations or mechanisms to enable a model to acquire feature representations from different scales and aspects of an image, increasing the model's robustness and generalization ability, and achieving better network performance than traditional networks. By fusing multi-scale information into a network, classification performance and network efficiency can be improved.

[0005] In the hyperspectral image HSI land cover classification task, it has also been confirmed that multi-scale learning can improve the performance of the network.

[0006] The paper "Fan H, Xiong B, Mangalam K, et al. Multiscale vision transformers[C]. Proceedings of the IEEE / CVF international conference on computer vision. 2021:6824-6835." proposes a multi-scale VisionTransformer (MViT) for video and image recognition, which has multiple channel resolution scale stages. Early layers operate with high spatial resolution, simulating simple, low-level visual information; deeper layers operate with spatially coarse but complex high-dimensional features. By combining multi-scale features, the model's performance and operational efficiency are improved.

[0007] The paper "Chen CFR, Fan Q, Panda R. Crossvit: Cross-attention multi-scale vision transformer for image classification [C]. Proceedings of the IEEE / CVF international conference on computer vision. 2021:357-366." proposes a multi-scale cross-attention model, ViT, which implements a two-branch architecture based on standard ViT, with branches for processing images of different sizes. Small images allow for the learning of fine-grained features, while large images enable the learning of global representations. The cross-learning of information from the two branches adds more image information across multiple scales. Cross-attention achieves efficient information exchange in linear time, significantly saving computational resources and yielding better classification results than ViT.

[0008] However, the aforementioned MViT and cross-attention ViT networks still have limitations such as computational complexity and incomplete information capture, which lead to low classification efficiency and low classification accuracy. Summary of the Invention

[0009] The purpose of this invention is to provide a hyperspectral image classification method based on multi-scale feature pyramids, which solves the problems of low classification efficiency and low classification accuracy of existing methods.

[0010] The technical solution adopted in this invention is a hyperspectral image classification method based on a multi-scale feature pyramid, which is implemented according to the following steps:

[0011] Step 1: Perform dimensionality reduction on the hyperspectral image to obtain the dimensionality-reduced hyperspectral image. Then, sample the dimensionality-reduced hyperspectral image and divide it into a training set and a test set.

[0012] Step 2, construct the improved ViT model;

[0013] Step 3: Use the training set divided in Step 1 to train the improved ViT model constructed in Step 2, and obtain the trained improved ViT model.

[0014] Step 4: Input the test set divided in Step 1 into the trained improved ViT model to obtain the classification results.

[0015] The invention is further characterized in that,

[0016] Furthermore, in step 1, principal component analysis is used for dimensionality reduction.

[0017] Furthermore, in step 1, the specific process of sampling and dividing the training and test sets is as follows:

[0018] Step 1.1: Set the sampling window of the dimensionality-reduced hyperspectral image to the spatial size of the patch block, and sample samples containing spectral and spatial dimensionality information;

[0019] Step 1.2: Using the label of the center pixel of a patch block as the label of that patch block, randomly divide all labeled patch blocks in the dimensionality-reduced hyperspectral image into training sets. and test set in, and Let represent the training set and test set of the i-th dimension-reduced hyperspectral image, respectively.

[0020] Furthermore, in step 2, the improved ViT model introduces a feature pyramid structure from the convolutional network into the ViT model, which has four stages.

[0021] Each stage includes a Patch Embdding module, a Position Embdding module, and an Encoder module;

[0022] The Patch Embdding module contains a Linear layer and a Norm layer. The Linear layer involves the input image size, patch size, input dimension, and output dimension.

[0023] The Position Embdding module generates a row vector whose length is equal to the number of patches in the Patch Embdding module;

[0024] The Encoder module consists of a Norm layer, an SRA layer, and a Feed Forward layer. The SRA layer includes Spatial Reduction and multi-head attention operations, which involve spatial reduction ratio and the number of attention heads.

[0025] Furthermore, the specific processes of the four stages set in the improved ViT model are as follows:

[0026] Step 2.1, in the first stage, the input dimensionality-reduced hyperspectral image of size H×W×C is divided into... Each patch block is 4×4×C in size. The resulting patch blocks are linearly projected into the Patch Embdding module onto a shape of size 4×4×C. The embedded patch, along with the position vector generated in the Position Embedding module, is fed into the Encoder module, outputting a first-stage size of [size missing]. Feature map;

[0027] Step 2.2, in the i-th stage, assuming the patch block size in the i-th stage is P. i In this stage, the size generated in the previous stage is H. i-1 ×W i-1 ×C i-1 Feature maps divided into Each patch block is fed into the Patch Embedding module of this stage, and linear projection is used to obtain the embedding patch for this stage. The embedding patch for this stage and the position vector generated in the Position Embedding module are then fed into the Encoder module, and the output has... Feature diagram of the shape.

[0028] Furthermore, the Spatial Reduction and multi-head attention operations at each stage in the improved ViT model are as follows:

[0029]

[0030] head j =Attention(QW j Q ,SR(K)W j K ,SR(V)W j V (2)

[0031] In equations (1) and (2), Q, K, and V represent the input vectors for the attention operation, respectively; Concat(·) represents the concatenation operation; head i This represents the attention result produced by the i-th attention head; and Represents the projection parameters, n i d represents the number of attention heads in stage i. head The size is

[0032] Where SR(·) represents the operation of reducing the spatial dimension of the input sequence, then:

[0033] SR(x)=Norm(Reshape(x,r i W S (3)

[0034] In equation (3), Let r represent the input sequence. i Reshape(x,r) represents the space reduction ratio in the i-th stage. i The input sequence x will first be reduced to a shape of size . Then through the projection matrix Reduce the dimension of the input sequence back to C. i Norm(·) represents the layer normalization operation;

[0035] Attention(·) represents the attention operation in the improved ViT model:

[0036]

[0037] In equation (4), Softmax(·) represents the normalization function, which is used to map the output vector of the improved ViT model to a probability distribution in the range [0,1] with a sum of 1.

[0038] Furthermore, the specific process of step 3 is as follows:

[0039] Step 3.1: Before training, perform random center pruning and random horizontal flipping operations on the training set to enhance the diversity of data in the training set;

[0040] Step 3.2: During training, data augmentation methods are used to further expand the number of training set samples;

[0041] Step 3.3: Use the "SoftTargetCrossEntropy"() function to calculate the cross-entropy loss during the training process of the improved ViT model, i.e.:

[0042]

[0043] In equation (5), y i This represents the true label of the i-th sample. Let c represent the probability of class i predicted by the improved ViT model, where c is the total number of classes.

[0044] Step 3.4: Update the network parameters using the AdamW algorithm. For each parameter θ, first calculate the gradient of the loss function with respect to that parameter. Then, the parameter θ is updated using weight decay, as expressed by the formula:

[0045]

[0046] In equation (6), α is the learning rate, and η is the learning rate. t This represents a custom scaling factor, ∈ is a constant to prevent division by zero, and λ is the weight decay coefficient. and Let represent the first-order momentum and second-order momentum after the deviation correction in stage t, respectively;

[0047] Step 3.5: As the iterations continue, the total loss will gradually decrease and then stabilize. Save the network weights of the iteration with the smallest total loss as the final weights of the improved ViT model, and obtain the trained improved ViT model.

[0048] Furthermore, in step 3.2, the data augmentation method adopts the Cutmix or Mixup method.

[0049] The beneficial effects of this invention are:

[0050] (1) The method of the present invention is based on the unique properties of hyperspectral images, which have multiple spectral dimensions and one object, one spectrum. It uses a multi-scale learning method to extract rich spectral spatial information in hyperspectral images, which can better fuse information of different dimensions and increase the understanding of the input by the improved ViT model.

[0051] (2) The method of this invention draws on the advantages of convolutional models and introduces the feature pyramid structure of convolutional networks into the ViT model, eliminating the limitation of high feature map resolution at different stages of the ViT model. It can obtain feature maps of different resolutions at different stages and reduce the number of model parameters.

[0052] (3) The method of the present invention uses a space reduction attention mechanism to replace the self-attention mechanism in the original model, which reduces the spatial dimension of the attention module input, further reduces the training time and number of parameters of the model, and can reduce the complexity of the model inference process and improve the running efficiency without affecting the classification accuracy.

[0053] (4) The improved ViT model involved in the method of the present invention is an improvement on the ViT model. It does not contain any convolution operation and is a general framework. It not only performs well in hyperspectral image classification tasks with high classification accuracy, but can also serve as a backbone network for application in other fields, and has strong versatility. Attached Figure Description

[0054] Figure 1 This is a flowchart of the method of the present invention;

[0055] Figure 2 This is a schematic diagram of the improved ViT model in the method of the present invention;

[0056] Figure 3 This is a comparison diagram of the spatially reduced attention mechanism involved in the improved ViT model in the method of this invention and the self-attention mechanism in ViT.

[0057] Figure 4 This is the result of land cover classification obtained from hyperspectral image data of Pavia University in Embodiment 1 of the present invention;

[0058] Figure 5 This is the result image obtained by classifying land cover from Salinas hyperspectral image data in Embodiment 2 of the present invention;

[0059] Figure 6 This is the result of land cover classification obtained from WHU-Hi-LongKou hyperspectral image data in Embodiment 3 of the present invention. Detailed Implementation

[0060] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0061] This invention relates to a hyperspectral image classification method based on a multi-scale feature pyramid, such as... Figure 1 As shown, please follow these steps:

[0062] Step 1: Perform dimensionality reduction on the hyperspectral image to obtain the dimensionality-reduced hyperspectral image. Then, sample the dimensionality-reduced hyperspectral image and divide it into a training set and a test set.

[0063] Among them, principal component analysis was used to reduce the dimensionality of the hyperspectral images, retaining the main spectral channels and reducing the redundancy of the hyperspectral images.

[0064] The dimensionality-reduced hyperspectral image is a dataset containing all ground features. It needs to be sampled first to obtain data and labels related to each pixel before it can be input into the model for inference operations. The specific process of sampling and dividing the training and test sets is as follows:

[0065] Step 1.1: Set the sampling window of the dimensionality-reduced hyperspectral image to the spatial size of the patch block, and sample samples containing spectral and spatial dimensionality information;

[0066] Step 1.2: Using the label of the center pixel of a patch block as the label of that patch block, randomly divide all labeled patch blocks in the dimensionality-reduced hyperspectral image into training sets. and test set in, and Let represent the training set and test set of the i-th dimension-reduced hyperspectral image, respectively;

[0067] Step 2, construct the improved ViT model;

[0068] like Figure 2 As shown, the improved ViT model introduces the feature pyramid structure from the convolutional network into the ViT model. That is, the improved ViT model has four stages. The spatial dimension of the feature map generated by each stage gradually decreases while the spectral dimension gradually increases, thereby generating feature maps of different sizes at different stages.

[0069] Each stage in the improved ViT model includes a Patch Embedding module, a Position Embedding module, and an Encoder module. The Patch Embedding module contains Linear and Norm layers. The Linear layer addresses the input image size, patch size, input dimension, and output dimension. The Position Embedding module generates a row vector with a length equal to the number of patches in the Patch Embedding module. The Encoder module contains Norm, SRA, and Feed Forward layers. The SRA layer includes Spatial Reduction and multi-head attention operations, primarily involving the spatial reduction ratio and the number of attention heads.

[0070] The four stages set in the improved ViT model are as follows:

[0071] Step 2.1, in the first stage, the input dimensionality-reduced hyperspectral image of size H×W×C is divided into... Each patch block is 4×4×C in size. The resulting patch blocks are linearly projected into the Patch Embdding module onto a shape of size 4×4×C. The embedded patch, along with the position vector generated in the Position Embedding module, is fed into the Encoder module, outputting a first-stage size of [size missing]. Feature map;

[0072] Step 2.2, in the i-th stage, assuming the patch block size in the i-th stage is P. i In this stage, the size generated in the previous stage is H. i-1 ×W i-1 ×C i-1 Feature maps divided into Each patch block is fed into the Patch Embedding module of this stage, and linear projection is used to obtain the embedding patch for this stage. The embedding patch for this stage and the position vector generated in the Position Embedding module are then fed into the Encoder module, and the output has... Feature diagram of the shape;

[0073] The specific processes for stages two through four are as follows:

[0074] Step 2.2.1, in the second stage, the size obtained in step 2.1 is... Feature maps divided into Each patch block is fed into the Patch Embedding module and linearly projected into an embedding patch of dimension C2. The embedding patch and the position vector generated in the Position Embedding module are then fed into the Encoder module, outputting a second-stage output of size C2. Feature map;

[0075] Step 2.2.2, in the third stage, the size obtained in step 2.2.1 is... Feature maps divided into Each patch block is fed into the Patch Embedding module and linearly projected into an embedding patch of dimension C3. The embedding patch and the position vector generated in the Position Embedding module are then fed into the Encoder module, outputting the second-stage output of size C3. Feature map;

[0076] Step 2.2.3, in the fourth stage, the size obtained in step 2.2.2 is... Feature maps divided into Each patch block is fed into the Patch Embedding module and linearly projected into an embedding patch of dimension C4. The embedding patch and the position vector generated in the PositionEmbdding module are then fed into the Encoder module, outputting a second-stage size of [value missing]. Feature map;

[0077] In the improved ViT model, the Spatial Reduction and multi-head attention operations at each stage are as follows:

[0078]

[0079] head j =Attention(QW j Q ,SR(K)W j K ,SR(V)W j V (2)

[0080] In equations (1) and (2), Q, K, and V represent the input vectors for the attention operation, respectively; Concat(·) represents the concatenation operation; head i This represents the attention result produced by the i-th attention head; and Represents the projection parameters, n i Let d represent the number of attention heads in the i-th stage, and therefore the dimension of each head is d. head The size is

[0081] Where SR(·) represents the operation of reducing the spatial dimension of the input sequence, then:

[0082] SR(x)=Norm(Reshape(x,r i W S (3)

[0083] In equation (3), Let r represent the input sequence. i Reshape(x,r) represents the space reduction ratio in the i-th stage. i The input sequence x will first be reduced to a shape of size . Then through the projection matrix Reduce the dimension of the input sequence back to C. i Norm(·) represents the layer normalization operation;

[0084] Attention(·) represents the attention operation in the improved ViT model:

[0085]

[0086] In equation (4), Softmax(·) represents the normalization function, which is used to map the output vector of the improved ViT model to a probability distribution in the range [0,1] with a sum of 1;

[0087] like Figure 3 As shown, the improved ViT model, by employing a space-reduced attention operation, can save space compared to the multi-head self-attention operation in the original ViT model. This reduces computational consumption by a factor of two, saves computing resources, and improves the model's running efficiency.

[0088] Step 3: Use the training set divided in Step 1 to train the improved ViT model constructed in Step 2, and obtain the trained improved ViT model.

[0089] The specific process is as follows:

[0090] Step 3.1: Before training, perform random center pruning and random horizontal flipping operations on the training set to enhance the diversity of data in the training set;

[0091] Step 3.2: During training, data augmentation methods are used to further expand the number of training set samples. The data augmentation methods used are Cutmix or Mixup.

[0092] Step 3.3: Use the "SoftTargetCrossEntropy"() function to calculate the cross-entropy loss during the training process of the improved ViT model, i.e.:

[0093]

[0094] In equation (5), y i This represents the true label of the i-th sample. Let c represent the probability of class i predicted by the improved ViT model, where c is the total number of classes.

[0095] Step 3.4: Update the network parameters using the AdamW algorithm. For each parameter θ, first calculate the gradient of the loss function with respect to that parameter. Then, the parameter θ is updated using weight decay, as expressed by the formula:

[0096]

[0097] In equation (6), α is the learning rate, and η is the learning rate. t This represents a custom scaling factor, ∈ is a small constant to prevent division by zero, and λ is the weight decay coefficient. and Let represent the first-order momentum and second-order momentum after the deviation correction in stage t, respectively;

[0098] Step 3.5: As the iterations continue, the total loss will gradually decrease and then tend to stabilize. Save the network weights of the iteration with the smallest total loss as the final weights of the improved ViT model, and obtain the trained improved ViT model.

[0099] Step 4: Input the test set into the trained improved ViT model to obtain the classification results, and output the classification map of the image based on the classification results.

[0100] The effectiveness of the method of the present invention can be further illustrated by the following experiments:

[0101] I. Experimental conditions:

[0102] Hardware environment: Intel i7-9700F processor, 3.00GHz, 16GB RAM, NVIDIA GeForce GTX2070SUPER GPU, 15.9GB RAM.

[0103] Software environment: 64-bit Windows 10 system and PyTorch 1.12.1 deep learning framework.

[0104] The evaluation metrics include overall accuracy (OA), average accuracy (AA), and Kappa coefficient (KAPPA), where:

[0105] Overall accuracy (OA) represents the proportion of correctly classified samples out of all samples; the higher the value, the better the classification effect.

[0106] Average precision (AA) represents the average classification precision for each class; the higher the value, the better the classification performance.

[0107] The Kappa coefficient represents the different weights in the confusion matrix; the closer the value is to 1, the better the classification effect.

[0108] Sampling was performed using 13×13 patch blocks. 10% of the labeled samples from each class were used as the training set on the PaviaU and Salinas datasets, while 50 labeled samples from each class were used as the training set on the WHU-Hi-LongKou dataset. The remaining labeled samples were used as the test set for hyperspectral image classification.

[0109] During model training, the teacher training algebra was set to 300, the single sample input was 64, the loss function was the soft-label cross-entropy loss function, and the initial learning step size was 0.0005.

[0110] Perform three independent inferences on the test set, and take the average of the classification accuracy of the three inferences as the final classification accuracy.

[0111] II. Experiment Content

[0112] Example 1: On the hyperspectral image data from Pavia University, the method of this invention and other existing efficient multi-scale methods (DeiT, SwinT, and TNT) were used to classify the hyperspectral images, respectively. Classification result images were obtained, and evaluation metrics for each classification were calculated. The results are shown in Table 1 and... Figure 4 As shown.

[0113] Table 1 Comparison of results from Pavia UniversityDeiT, SwinT, TNT, and the method of this invention.

[0114]

[0115] As shown in Table 1, the accuracy of the method of the present invention is higher than that of other existing multi-scale learning methods on all categories of the Pavia University data. Compared with the three models DeiT, SwinT and TNT, the OA of the method of the present invention is 0.83%, 0.42% and 0.86% higher, respectively; the AA is 2.27%, 1.29% and 1.82% higher, respectively; and the KAPPA is 1.0%, 0.56% and 1.15% higher, respectively.

[0116] Depend on Figure 4 As can be seen, the classification result image of the present invention has a smoother classification result and higher edge recognition accuracy.

[0117] Example 2: On Salinas hyperspectral image data, the method of this invention and other existing efficient multi-scale methods (DeiT, SwinT, and TNT) were used to classify the hyperspectral images, respectively. Classification result images were obtained, and evaluation metrics for each classification were calculated. The results are shown in Table 2 and... Figure 5 .

[0118] Table 2 Comparison of results from SalinasDeiT, SwinT, TNT, and the method of this invention.

[0119]

[0120] As shown in Table 2, the method of the present invention performs better on the Salinas dataset, with an accuracy of 100% across all categories, which is higher than or equivalent to the accuracy of other existing multi-scale methods. Compared with DeiT, SwinT, and TNT methods, the OA of the method of the present invention is 0.28%, 0.06%, and 0.01% higher, respectively; the AA is 0.40%, 0.12%, and 0.02% higher, respectively; and the KAPPA is 0.28%, 0.06%, and 0.02% higher, respectively.

[0121] Depend on Figure 5As can be seen, the classification result image of the method of the present invention has a smoother classification result and higher edge recognition accuracy.

[0122] Example 3: On the WHU-Hi-LongKou hyperspectral image data, the hyperspectral images were classified using the method of this invention and other existing efficient multi-scale methods (DeiT, SwinT, and TNT). Classification result images were obtained, and evaluation metrics for each classification were calculated. The results are shown in Table 3. Figure 6 .

[0123] Table 3 Comparison of results from WHU-Hi-LongKouDeiT, SwinT, TNT, and the method of this invention.

[0124]

[0125] As shown in Table 3, the accuracy of the method of the present invention is higher than that of other existing multi-scale methods in various categories of WHU-Hi-LongKou data. Compared with the three comparison methods DeiT, SwinT and TNT, the OA of the method of the present invention is 2.63%, 2.9% and 0.35% higher, respectively; the AA is 5.03%, 4.24% and 1.89% higher, respectively; and the KAPPA is 3.42%, 3.76% and 0.45% higher, respectively.

[0126] Depend on Figure 6 As can be seen, the classification result image of the present invention has a smoother classification result and higher edge recognition accuracy.

[0127] Example 4: On the hyperspectral image data of Pavia University, the method of the present invention was compared with the existing multi-scale models SwinT and TNT in terms of parameter number and training time to verify the effect of the method of the present invention on reducing the number of model parameters and improving the running efficiency. The results are shown in Table 4.

[0128] Table 4 Comparison of results from SwinT, TNT, and the method of this invention.

[0129] Parameters 48844179 23973801 22988240 Training time (h) 38.5 16.5 12.25

[0130] As shown in Table 4, compared with the existing SwinT and TNT, the method of the present invention reduces the number of parameters by 112.47% and 4.29% respectively, and the training time by 214.29% and 32% respectively. While reducing the model complexity, it improves the classification accuracy and running efficiency of the model.

[0131] Experimental results show that the method of this invention introduces the feature pyramid structure from the convolutional model into the ViT model, which solves the problem of the single scale of the feature map in the ViT model, enabling the model to obtain feature maps of different resolutions at different stages and improving the classification performance of the model. In order to further reduce the amount of computation, a spatial reduction attention mechanism is used to reduce the spatial dimension of the input data of the attention module, saving computational resources while achieving better results than the existing technology.

Claims

1. A hyperspectral image classification method based on multi-scale feature pyramids, characterized in that, The specific steps are as follows: Step 1: Perform dimensionality reduction on the hyperspectral image to obtain the dimensionality-reduced hyperspectral image. Then, sample the dimensionality-reduced hyperspectral image and divide it into a training set and a test set. Step 2, construct the improved ViT model; In step 2, the improved ViT model introduces the feature pyramid structure from the convolutional network into the ViT model, which has four stages. Each stage includes a Patch Embdding module, a Position Embdding module, and an Encoder module; The Patch Embdding module contains a Linear layer and a Norm layer. The Linear layer involves the input image size, patch size, input dimension, and output dimension. The Position Embdding module generates a row vector whose length is equal to the number of patches in the Patch Embdding module; The Encoder module consists of a Norm layer, an SRA layer, and a Feed Forward layer. The SRA layer includes Spatial Reduction and multi-head attention operations, which involve spatial reduction ratio and the number of attention heads. The four stages set in the improved ViT model are as follows: Step 2.1, in the first stage, the input size is... The dimensionality-reduced hyperspectral image is divided into There are 1, each with a size of 1 The C patch block is linearly projected into the Patch Embdding module onto a shape of size [size missing]. The embedded patch, along with the position vector generated in the Position Embedding module, is fed into the Encoder module, outputting a first-stage size of [size missing]. Feature map; Step 2.2, in the i Phase, assuming the first The patch block size for a stage is In this stage, the size generated in the previous stage is... Feature maps divided into Each patch block is fed into the Patch Embedding module of this stage, and linear projection is used to obtain the embedding patch for this stage. The embedding patch for this stage and the position vector generated in the Position Embedding module are then fed into the Encoder module, and the output has... Feature diagram of the shape; The Spatial Reduction and multi-head attention operations at each stage in the improved ViT model are as follows: （1） （2） In equations (1) and (2), , and These represent the input vectors for the attention operation; Indicates a splicing operation; Indicates the first The attention results generated by each attention head; Indicates projection parameters, Indicates the first Number of attention heads in a phase The size is ; in, If the operation represents reducing the spatial dimension of the input sequence, then: （3） In equation (3), Represents the input sequence. Indicates the first The space reduction ratio of the stage The input sequence will be processed first. Shrink to shape size Then through the projection matrix Reduce the dimension of the input sequence back ; Presentation layer normalization operation; This represents the attention operation in the improved ViT model: （4） In equation (4), represents the normalization function used to map the output vector of the improved ViT model to a probability distribution in the range [0,1] with a sum of 1; Step 3: Use the training set divided in Step 1 to train the improved ViT model constructed in Step 2, and obtain the trained improved ViT model. Step 4: Input the test set divided in Step 1 into the trained improved ViT model to obtain the classification results.

2. The hyperspectral image classification method based on multi-scale feature pyramids according to claim 1, characterized in that, In step 1, principal component analysis is used for dimensionality reduction.

3. The hyperspectral image classification method based on multi-scale feature pyramids according to claim 1, characterized in that, In step 1, the specific process of sampling and dividing the training and test sets is as follows: Step 1.1: Set the sampling window of the dimensionality-reduced hyperspectral image to the spatial size of the patch block, and sample samples containing spectral and spatial dimensionality information; Step 1.2: Using the label of the center pixel of a patch block as the label of that patch block, randomly divide all labeled patch blocks in the dimensionality-reduced hyperspectral image into training sets. and test set ,in, and They represent the first The training and test sets of the dimensionality-reduced hyperspectral images.

4. The hyperspectral image classification method based on multi-scale feature pyramids according to claim 1, characterized in that, The specific process of step 3 is as follows: Step 3.1: Before training, perform random center pruning and random horizontal flipping operations on the training set to enhance the diversity of data in the training set; Step 3.2: During training, data augmentation methods are used to further expand the number of training set samples; Step 3.3: Use the "SoftTargetCrossEntropy()" function to calculate the cross-entropy loss during the training process of the improved ViT model, i.e.: （5） In equation (5), Indicates the first The true label of each sample This indicates the category predicted by the improved ViT model. The probability, c It is the total number of categories; Step 3.4: Update the network parameters using the AdamW algorithm. For each parameter... First, calculate the gradient of the loss function with respect to this parameter. Then update the parameters using weight decay. The formula is expressed as: （6） In equation (6), It is the learning rate. This represents a custom scaling factor. It is a constant that prevents division by zero. It is the weight decay coefficient. and Let represent the first-order momentum and second-order momentum after the deviation correction in stage t, respectively; Step 3.5: As the iterations continue, the total loss will gradually decrease and then stabilize. Save the network weights of the iteration with the smallest total loss as the final weights of the improved ViT model, and obtain the trained improved ViT model.

5. The hyperspectral image classification method based on multi-scale feature pyramids according to claim 4, characterized in that, In step 3.2, the data augmentation method uses Cutmix or Mixup.

Images