A hyperspectral image classification method and device of a lightweight transformer

Abstract

The application provides a hyperspectral image classification method and device of a lightweight Transformer, and the method comprises the following steps: obtaining a public hyperspectral image dataset; performing image preprocessing on the hyperspectral image dataset to obtain a hyperspectral image data block that can be input into a classification model; constructing an initial network model; inputting the hyperspectral image data block into the initial network model to obtain a classification result, determining a loss value of a loss function of the classification result and an image classification label, obtaining a model parameter training completion condition that is determined when a training frequency reaches a set threshold, obtaining a classification accuracy of the model in a model iteration process, and measuring the lightweight of the model according to a network parameter quantity and a floating point operation number in the model iteration process; and taking an iteration model with the highest classification accuracy and the lightest model as the final classification model that is trained completely. The application can improve the classification accuracy of the hyperspectral image and reduce the calculation amount of the model.

Description

Technical Field

[0001] This invention relates to the field of image classification technology, and in particular to a lightweight Transformer-based hyperspectral image classification method and apparatus. Background Technology

[0002] In the field of hyperspectral remote sensing, hyperspectral image data can more accurately reflect ground cover information through spectral information. Because the spectral information of different ground covers has subtle differences, it is easier to classify ground cover information, thus making the application of hyperspectral images in the field of remote sensing more extensive.

[0003] Hyperspectral images, due to their rich spectral information, possess stronger feature recognition capabilities and more accurate segmentation results compared to ordinary 3D images. However, it is precisely this abundant spectral and spatial information that increases the computational load and complexity of the network.

[0004] Convolutional neural networks (CNNs) are widely used in the field of hyperspectral imaging. However, CNNs can only capture local information, while attention mechanisms can capture global information. Therefore, the entire model only considers the attention mechanism to improve image classification accuracy. However, the attention mechanism has the problem of high computational cost. Therefore, we adopt a model reduction approach and dynamic token sparsification to reduce the computational complexity of the model and achieve a lightweight model.

[0005] "CoAtNet: Marrying Convolution and Attention for All Data Sizes" explores how convolutional neural networks and self-attention mechanisms can be effectively combined, proposing the CCTT model structure, where C represents Convolution and T represents Transformer. However, this model performs poorly in classification on hyperspectral image data.

[0006] Therefore, a lightweight Transformer-based hyperspectral remote sensing image classification method is needed to solve the above problems. Summary of the Invention

[0007] This invention provides a lightweight Transformer-based hyperspectral image classification method to address the shortcomings of existing models in classifying hyperspectral image data, thereby improving the accuracy of hyperspectral image classification and reducing the computational load of the model.

[0008] This invention provides a lightweight Transformer-based hyperspectral image classification method, comprising:

[0009] Acquire the hyperspectral image to be classified;

[0010] The hyperspectral image to be classified is input into the trained classification model, and the classification result of the hyperspectral image to be classified is output.

[0011] The classification model is a lightweight model built on the self-attention mechanism, and it is trained based on the hyperspectral image dataset and the image classification labels corresponding to the hyperspectral image dataset.

[0012] According to the present invention, a lightweight Transformer-based hyperspectral image classification method is provided, wherein the training method of the classification model includes:

[0013] Obtain publicly available hyperspectral image datasets;

[0014] The hyperspectral image dataset is preprocessed to obtain hyperspectral image data blocks that can be input into the classification model;

[0015] Construct the initial network model;

[0016] The hyperspectral image data block is input into the initial network model to obtain the classification result. The loss value is determined by the loss function of the classification result and the image classification label. The training of the model parameters is considered complete when the number of training times reaches a set threshold. The classification accuracy of the model is obtained during the model iteration process, and the lightweight nature of the model is measured based on the number of network parameters and floating-point operations during the model iteration process.

[0017] The iterative model with the highest classification accuracy and the most lightweight model is used as the final trained classification model.

[0018] According to a lightweight Transformer-based hyperspectral image classification method provided by the present invention, the step of preprocessing the hyperspectral image dataset to obtain hyperspectral image data blocks that can be input into the classification model includes:

[0019] The hyperspectral image dataset is subjected to PCA data dimensionality reduction processing to obtain dimensionality-reduced images;

[0020] The reduced-dimensional image is used to construct cube samples to obtain the hyperspectral image data block.

[0021] According to a lightweight Transformer-based hyperspectral image classification method provided by the present invention, the step of performing PCA data dimensionality reduction processing on the hyperspectral image dataset to obtain a dimensionality-reduced image includes:

[0022] The three dimensions of the original hyperspectral image of the hyperspectral image dataset are determined, including the first image width, the first image height, and the number of original image bands.

[0023] Based on principal component analysis, the first image width, first image height, and first original image band number of the original hyperspectral image are reduced in dimensionality to obtain the dimensionality-reduced image, and the second image width, second image height, and number of channels of the dimensionality-reduced image are determined.

[0024] According to a lightweight Transformer-based hyperspectral image classification method provided by the present invention, the step of constructing cube samples from the dimensionality-reduced image to obtain the hyperspectral image data block includes:

[0025] The dimensionality-reduced image is cropped based on the width of the second image, the height of the second image, and the number of channels after dimensionality reduction to determine the window size of the cube sample;

[0026] The three-dimensional image block of the cube sample is determined based on the window size and the number of channels after dimensionality reduction.

[0027] According to a lightweight Transformer hyperspectral image classification method provided by the present invention, the initial network model includes a convolutional module, a first self-attention mechanism module, a sparse token module, and a second self-attention mechanism module; both the first and second self-attention mechanism modules include a feedforward neural network submodule and a self-attention submodule.

[0028] The convolution module includes two convolutional layers. The convolution module is used to divide the three-dimensional image block into blocks to obtain image data that can be input to the first self-attention mechanism module.

[0029] The first self-attention mechanism module is used to downsample the image data to obtain a three-dimensional image block with the first downsampled value;

[0030] The sparse token module is used to perform sparsification processing on the data;

[0031] The second self-attention mechanism module is used to downsample the first downsampled 3D image patch after sparsification to obtain a second downsampled 3D image patch, and obtain the classification result based on the second downsampled 3D image patch.

[0032] According to the present invention, a lightweight Transformer-based hyperspectral image classification method is provided, wherein the loss function is the cross-entropy loss function, and the evaluation index of classification accuracy is determined based on overall accuracy, evaluation accuracy, and consistency.

[0033] The present invention also provides a lightweight Transformer hyperspectral image classification device, comprising:

[0034] The acquisition unit is used to acquire the hyperspectral image to be classified.

[0035] The classification unit is used to input the hyperspectral image to be classified into the trained classification model and output the classification result of the hyperspectral image to be classified.

[0036] The classification model is a lightweight model built on the self-attention mechanism, and it is trained based on the hyperspectral image dataset and the image classification labels corresponding to the hyperspectral image dataset.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lightweight Transformer hyperspectral image classification method as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the lightweight Transformer hyperspectral image classification method as described above.

[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a hyperspectral image classification method of the lightweight Transformer as described above.

[0040] The present invention provides a lightweight Transformer-based hyperspectral image classification method and apparatus. By inputting the hyperspectral image to be classified into a trained classification model, the classification result of the hyperspectral image to be classified is output. The classification model of the present invention uses the transformer module as the principal component of the model. It is a lightweight model built based on the self-attention mechanism, which solves the problem of high computational cost when using only the transformer model. At the same time, it improves the accuracy of hyperspectral image classification and reduces the computational cost of the model. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the lightweight Transformer hyperspectral image classification method provided by the present invention.

[0043] Figure 2 This is a flowchart illustrating the training process of the classification model of the present invention;

[0044] Figure 3 This is a schematic diagram of the stem stage architecture of the present invention;

[0045] Figure 4 This is a schematic diagram of the architecture of the self-attention mechanism of the present invention;

[0046] Figure 5 This is a schematic diagram of the feedforward neural network architecture of the present invention;

[0047] Figure 6 This is a schematic diagram illustrating the specific components of token sparsity in this invention;

[0048] Figure 7 This is a flowchart illustrating the specific process of model training in this invention;

[0049] Figure 8 A schematic diagram of the architecture of the lightweight Transformer hyperspectral image classification device provided by the present invention;

[0050] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] Reference Figure 1 The present invention provides a lightweight Transformer-based hyperspectral image classification method, comprising the following steps:

[0053] Step 110: Obtain the hyperspectral image to be classified;

[0054] Step 120: Input the hyperspectral image to be classified into the trained classification model and output the classification result of the hyperspectral image to be classified.

[0055] The classification model is a lightweight model built on the self-attention mechanism, and it is trained based on the hyperspectral image dataset and the image classification labels corresponding to the hyperspectral image dataset.

[0056] This embodiment describes the specific classification process of hyperspectral images, i.e., the application process of the model.

[0057] The obtained hyperspectral image to be classified is input into the trained classification model to output the classification result of the image.

[0058] It's important to note that a hyperspectral image is a type of image data acquired across multiple narrow bands within a spectral range. Compared to ordinary color images, hyperspectral images provide richer spectral information, containing more subtle spectral features. Traditional color images consist of three channels: red, green, and blue, while hyperspectral images typically contain tens to hundreds of consecutive spectral bands. Each band corresponds to a different wavelength or frequency band, allowing the acquisition of the reflection, radiation, or emission characteristics of ground objects at different frequency levels.

[0059] It should be further noted that the classification model in this embodiment is a lightweight transformer model based on the attention mechanism. This classification model uses transformer modules as principal components, which solves the problem of high computational cost when using only transformer models, while improving the accuracy of hyperspectral image classification and reducing the computational cost of the model.

[0060] Reference Figure 2 , Figure 2 The training process of the classification model of this invention includes the following steps:

[0061] Step 210: Obtain a publicly available hyperspectral image dataset;

[0062] Step 220: Perform image preprocessing on the hyperspectral image dataset to obtain hyperspectral image data blocks that can be input into the classification model;

[0063] Step 230: Construct the initial network model;

[0064] Step 240: Input the hyperspectral image data block into the initial network model to obtain the classification result. Determine the loss value by combining the classification result and the loss function of the image classification label. When the number of training times reaches the set threshold, the training of the model parameters is considered complete. During the model iteration process, obtain the classification accuracy of the model and measure the lightweight nature of the model based on the number of network parameters and floating-point operations during the model iteration process.

[0065] Step 250: The iterative model with the highest classification accuracy and the model being lightweight is taken as the final training classification model.

[0066] Specifically, the dataset is prepared in step 210 by using a publicly available hyperspectral remote sensing image dataset.

[0067] Optionally, the publicly available datasets used are: Indian Pine Dataset (IN), University of Pavia Dataset (UP), and Salinas Dataset (SV).

[0068] The Indian Pine Dataset (IN), acquired by the AVIRIS sensor, has a size of 145×145 pixels and contains 224 bands, of which 200 are valid bands; it includes 16 crop categories. The University of Pavia Dataset (UP), acquired by the ROSIS sensor, has a size of 610×340 pixels and contains 115 bands, which were processed to yield 103 bands. The Pavia Center data contains 102 bands and includes 9 land cover categories. The Salinas Dataset (SV), also acquired by the AVIRIS sensor, has a spatial resolution of 3.7 meters and a size of 512×217 pixels. The original data contained 224 bands; after removing bands with severe moisture absorption, 204 bands remained, representing 16 crop categories.

[0069] Image preprocessing is performed in step 220: PCA data dimensionality reduction is performed on the dataset prepared in step 210, and cube samples are constructed from the dimensionality-reduced images to obtain a hyperspectral image data block that can be input into the model.

[0070] It's important to note that PCA (Principal Component Analysis) is a commonly used data dimensionality reduction technique. It uses linear transformations to map the original data to a new feature space, maximizing the variance among the new features. Simply put, PCA aims to find a new set of mutually orthogonal eigenvectors, called principal components, which retain the most information from the original data.

[0071] The initial network model is constructed through step 230.

[0072] Furthermore, the initial network model includes a convolutional module, a first self-attention mechanism module, a sparse token module, and a second self-attention mechanism module; both the first and second self-attention mechanism modules include a feedforward neural network submodule and a self-attention submodule.

[0073] The convolution module includes two convolutional layers. The convolution module is used to divide the three-dimensional image block into blocks to obtain image data that can be input to the first self-attention mechanism module.

[0074] The first self-attention mechanism module is used to downsample the image data to obtain a three-dimensional image block with the first downsampled value;

[0075] The sparse token module is used to perform sparsification processing on the data;

[0076] The second self-attention mechanism module is used to downsample the first downsampled 3D image patch after sparsification to obtain a second downsampled 3D image patch, and obtain the classification result based on the second downsampled 3D image patch.

[0077] Specifically, the initial network model framework in this embodiment consists of four parts: stem stage, relative self-attention mechanism module, feedforward neural network and sparse token module stacked together. The stacked relative self-attention module and feedforward neural network respectively constitute the first self-attention mechanism module group and the second self-attention mechanism module group.

[0078] In other words, the entire model can be divided into four stages:

[0079] Reference Figure 3 , Figure 3 This is a schematic diagram of the stem stage architecture of the present invention.

[0080] Stage 1 convolutional module consists of two convolutional layers. It should be noted that the convolutional layer is conv3×3.

[0081] Phase 2, the first self-attention mechanism module, consists of a relative self-attention module and a feedforward neural network. In this phase, the image needs to be downsampled and then input into the module 5 times in a loop. In addition, Dropkey is used in the feedforward neural network.

[0082] Reference Figure 4 , Figure 4 This is a schematic diagram of the architecture of the self-attention mechanism of the present invention.

[0083] First, it should be noted that Q (Query), K (Key), and V (Value) in the diagram are three key input variables, while matmul (matrix multiplication), scale (scaling factor), mask (optional), and relative bias (relative bias term) are a series of operations performed to calculate attention weights.

[0084] Q (Query): Represents the query vector, used to calculate the similarity with other locations and determine which locations need to be noticed.

[0085] K(Key): Represents the key vector, used to calculate the similarity with other locations and participates in the calculation of attention weights.

[0086] V(Value): Represents a value vector containing the feature representation of each position, which is used to sum the values ​​according to the attention weights to obtain the final self-attention representation.

[0087] Through the self-attention mechanism in this embodiment, the similarity between each input variable and other variables is calculated, and the final attention weight is calculated based on the similarity, so as to achieve better feature aggregation and semantic understanding.

[0088] Reference Figure 5 , Figure 5 This is a schematic diagram of the feedforward neural network architecture of the present invention.

[0089] The feedforward neural network consists of the following layers from top to bottom: linear layer, GELU activation function, Dropout, linear layer, and Dropkey.

[0090] Phase 3 sparse token module uses a dynamically sparse token module.

[0091] Reference Figure 6 , Figure 6 This is a schematic diagram illustrating the specific components of token sparsity;

[0092] Its specific components include: features, policy, decisions, and sparse tokens.

[0093] 1. Features: In machine learning, features refer to attributes or variables used to represent data. In PredictorLG, features refer to various attributes or characteristics of the input data, such as words, sentence length, and part-of-speech tags in text. These features are used to build predictive models.

[0094] 2. Policy: In machine learning, a policy refers to the rules or methods for making decisions based on input data. In PredictorLG, the policy refers to how the model predicts the output given input features. The model learns how to choose the best decision-making strategy based on the input features.

[0095] 3. Decisions: In machine learning, decision-making refers to choosing the best action or outcome from multiple options based on input data and the model's predictions. In PredictorLG, decision-making is the best action or output predicted by the model based on given features and policies.

[0096] 4. Sparse tokens: In natural language processing, sparse tokens refer to tokens or words that occur infrequently. In PredictorLG, sparse tokens are those that appear less frequently in the training data. They may contain specific information or patterns, but because they occur infrequently, the model may have weaker predictive power for them.

[0097] Phase 4, the second self-attention mechanism module, consists of a relative self-attention module and a feedforward neural network. In this phase, the image needs to be downsampled again and then input into the module twice in a loop. Dropkey is also used in the feedforward neural network.

[0098] It's important to note that in self-attention modules, Dropkey refers to a technique that randomly discards key representations during attention score calculation. This is done to increase model robustness and reduce the risk of overfitting. Specifically, Dropkey randomly sets some key representations to zero, thus reducing their contribution to the final attention score. This allows the model to focus more on other keys, thereby improving its generalization ability when processing input sequences.

[0099] Reference Figure 7 , Figure 7 This describes the specific process of model training in this invention.

[0100] After image processing, the original image goes through a stem stage layer, then enters a five-layer repetitive self-attention submodule (Rel-Attention) and feedforward neural network submodule (FNN), then homomorphic token sparsification is performed, and finally it goes through two layers of repetitive self-attention submodule (Rel-Attention) and feedforward neural network submodule (FNN).

[0101] Step 240 involves selecting a loss function and evaluation metrics: the loss function between the classification result image and the label is calculated until the number of training iterations reaches a set threshold, at which point the model parameters are considered to have been trained successfully; at the same time, evaluation metrics are selected to measure the accuracy of the algorithm and evaluate the performance of the system, and the number of network parameters and floating-point operations are selected to measure the lightweight nature of the network model;

[0102] Finally, in step 250, the model is saved: the model is saved when the classification accuracy is highest and the computational cost and parameter count are lightweight. When performing hyperspectral image classification, the hyperspectral data can be directly input into the model for classification.

[0103] The lightweight Transformer hyperspectral remote sensing image classification method provided by this invention uses only the self-attention mechanism to extract the spatial spectral features of hyperspectral images. By dynamically sparsifying tokens and gradually and dynamically deleting redundant tokens, the computational load during inference is reduced. Dropkey is used to prevent overfitting of the Transformer, thereby reducing the model size, reducing computation time, and improving the classification accuracy of hyperspectral images, resulting in good classification performance.

[0104] Further, the step of preprocessing the hyperspectral image dataset to obtain hyperspectral image data blocks that can be input into the classification model includes:

[0105] The hyperspectral image dataset is subjected to PCA data dimensionality reduction processing to obtain dimensionality-reduced images;

[0106] The reduced-dimensional image is used to construct cube samples to obtain the hyperspectral image data block.

[0107] Furthermore, the step of performing PCA data dimensionality reduction processing on the hyperspectral image dataset to obtain a dimensionality-reduced image includes:

[0108] The three dimensions of the original hyperspectral image of the hyperspectral image dataset are determined, including the first image width, the first image height, and the number of original image bands.

[0109] Based on principal component analysis, the first image width, first image height, and first original image band number of the original hyperspectral image are reduced in dimensionality to obtain the dimensionality-reduced image, and the second image width, second image height, and number of channels of the dimensionality-reduced image are determined.

[0110] The step of constructing the hyperspectral image data block by constructing cube samples from the dimensionality-reduced image includes:

[0111] The dimensionality-reduced image is cropped based on the width of the second image, the height of the second image, and the number of channels after dimensionality reduction to determine the window size of the cube sample;

[0112] The three-dimensional image block of the cube sample is determined based on the window size and the number of channels after dimensionality reduction.

[0113] Specifically, the dimensionality reduction process and the cubic sample construction process are as follows:

[0114] The three dimensions of the original hyperspectral image I1 are determined as W, H and C1. The original hyperspectral image I1 with dimensions W×H×C1 is reduced to a new hyperspectral image I2 with dimensions W×H×C2 by PCA, where W is the image width, H is the image height, C1 is the number of bands in the original image and C2 is the number of channels after dimensionality reduction.

[0115] Then, based on principal component analysis, the new hyperspectral image I2 is cropped into a three-dimensional image patch of size w×w×C2 and input into the network model, where w is the window size.

[0116] Referring to the Indian pine dataset described in the above embodiment, the original hyperspectral image I1 is first reduced to a 145×145×30 hyperspectral image I2; then, cube samples are constructed from the reduced image I2 to obtain a 28×28×30 three-dimensional image patch.

[0117] This embodiment uses PCA to reduce the dimensionality of hyperspectral image data, which helps reduce the dimensionality of the dataset, remove redundant information and noise, and preserve as many of the main features of the original data as possible. The dimensionality-reduced data is easier to visualize, analyze, and understand, and can improve the efficiency and performance of machine learning algorithms.

[0118] Optionally, the loss function is the cross-entropy loss function, and the evaluation index of classification accuracy is determined based on overall accuracy, evaluation accuracy, and consistency.

[0119] During the model iteration process, the cross-entropy loss function is selected as the loss function, and the overall accuracy, average accuracy, and consistency evaluation are chosen for classification accuracy. The number of network parameters and floating-point operations are chosen to measure the lightweight nature of the network model. The calculation formulas for overall accuracy, average accuracy, consistency evaluation, number of network parameters, and floating-point operations are shown below:

[0120]

[0121]

[0122]

[0123]

[0124]

[0125] Param=(k×k×C in +1)·C out

[0126] FLOPs = W × H × (k × k × C) in +1)·C out

[0127] Where TP is the positive sample correctly classified by the model, FN is the positive sample misclassified by the model, FP is the negative sample misclassified by the model, and TN is the negative sample correctly classified by the model; C is the total number of classes, Ti is the number of samples correctly classified in each class, ai is the number of true samples in each class, bi is the number of predicted samples in each class, and n is the total number of samples.

[0128] The training iterations were set to 300, with each input image size to the network being 64. The upper limit for the number of input images was determined based on the accuracy of the final classification result. The learning rate during training was set to 0.0005 to maintain fast network fitting without causing overfitting. The Adamw optimizer was chosen as the network optimizer. Its advantages are simple implementation, high computational efficiency, and the ability to adaptively adjust the learning rate of each parameter, while also exhibiting good convergence performance and robustness.

[0129] This invention constructs a lightweight Transformer-based hyperspectral remote sensing image classification method, which can classify hyperspectral images, reducing model complexity and computational load, and shortening testing time while improving classification accuracy. By calculating the correlation indicators of the images with existing methods, the feasibility and superiority of this method are further verified.

[0130] The lightweight Transformer hyperspectral image classification device provided by the present invention is described below. The lightweight Transformer hyperspectral image classification device described below and the lightweight Transformer hyperspectral image classification method described above can be referred to in correspondence.

[0131] Reference Figure 8 The present invention provides a lightweight Transformer hyperspectral image classification device, comprising:

[0132] Acquisition unit 810 is used to acquire hyperspectral images to be classified;

[0133] The classification unit 820 is used to input the hyperspectral image to be classified into the trained classification model and output the classification result of the hyperspectral image to be classified.

[0134] The classification model is a lightweight model built on the self-attention mechanism, and it is trained based on the hyperspectral image dataset and the image classification labels corresponding to the hyperspectral image data.

[0135] The lightweight Transformer hyperspectral image classification device provided by this invention inputs the hyperspectral image to be classified into a trained classification model and outputs the classification result of the hyperspectral image to be classified. The classification model of this invention uses the transformer module as the principal component of the model and is a lightweight model built based on the self-attention mechanism. It solves the problem of high computational cost when using only the transformer model, while improving the accuracy of hyperspectral image classification and reducing the computational cost of the model.

[0136] In some embodiments, the training method of the classification model includes:

[0137] Obtain publicly available hyperspectral image datasets;

[0138] The hyperspectral image dataset is preprocessed to obtain hyperspectral image data blocks that can be input into the classification model;

[0139] Construct an initial network model, the classification model including a convolutional module, a relative self-attention module, a feedforward neural network, and a sparse token module;

[0140] The hyperspectral image data block is input into the initial network model to obtain the classification result. The loss value is determined by the loss function of the classification result and the image classification label. The training of the model parameters is considered complete when the number of training times reaches a set threshold. The classification accuracy of the model is obtained during the model iteration process, and the lightweight nature of the model is measured based on the number of network parameters and floating-point operations during the model iteration process.

[0141] The iterative model with the highest classification accuracy and the most lightweight model is used as the final trained classification model.

[0142] In some embodiments, the step of preprocessing the hyperspectral image dataset to obtain hyperspectral image data blocks that can be input into the classification model includes:

[0143] The hyperspectral image dataset is subjected to PCA data dimensionality reduction processing to obtain dimensionality-reduced images;

[0144] The reduced-dimensional image is used to construct cube samples to obtain the hyperspectral image data block.

[0145] In some embodiments, performing PCA data dimensionality reduction on the hyperspectral image dataset to obtain a dimensionality-reduced image includes:

[0146] The three dimensions of the original hyperspectral image of the hyperspectral image dataset are determined, including the first image width, the first image height, and the number of original image bands.

[0147] Based on principal component analysis, the first image width, first image height, and first original image band number of the original hyperspectral image are reduced in dimensionality to obtain the dimensionality-reduced image, and the second image width, second image height, and number of channels of the dimensionality-reduced image are determined.

[0148] In some embodiments, constructing the hyperspectral image data block by constructing cube samples from the dimensionality-reduced image includes:

[0149] The dimensionality-reduced image is cropped based on the width of the second image, the height of the second image, and the number of channels after dimensionality reduction to determine the window size of the cube sample;

[0150] The three-dimensional image block of the cube sample is determined based on the window size and the number of channels after dimensionality reduction.

[0151] In some embodiments, the classification model includes a convolutional module, a first self-attention mechanism module, a sparse token module, and a second self-attention mechanism module; both the first and second self-attention mechanism modules include a feedforward neural network submodule and a family attention submodule.

[0152] The convolution module includes two convolutional layers. The convolution module is used to divide the three-dimensional image block into blocks to obtain image data that can be input to the first self-attention mechanism module.

[0153] The first self-attention mechanism module is used to downsample the image data to obtain a three-dimensional image block with the first downsampled value;

[0154] The sparse token module is used to perform sparsification processing on the data;

[0155] The second self-attention mechanism module is used to downsample the first downsampled 3D image patch after sparsification to obtain a second downsampled 3D image patch, and obtain the classification result based on the second downsampled 3D image patch.

[0156] In some embodiments, the loss function is the cross-entropy loss function, and the evaluation metric for classification accuracy is determined based on overall accuracy, evaluation accuracy, and consistency.

[0157] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a lightweight Transformer hyperspectral image classification method, which includes:

[0158] Acquire the hyperspectral image to be classified;

[0159] The hyperspectral image to be classified is input into the trained classification model, and the classification result of the hyperspectral image to be classified is output.

[0160] The classification model is a lightweight model built on the self-attention mechanism, and it is trained based on the hyperspectral image dataset and the image classification labels corresponding to the hyperspectral image data.

[0161] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the lightweight Transformer hyperspectral image classification method provided by the above methods, the method comprising:

[0163] Acquire the hyperspectral image to be classified;

[0164] The hyperspectral image to be classified is input into the trained classification model, and the classification result of the hyperspectral image to be classified is output.

[0165] The classification model is a lightweight model built on the self-attention mechanism, and it is trained based on the hyperspectral image dataset and the image classification labels corresponding to the hyperspectral image data.

[0166] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a lightweight Transformer hyperspectral image classification method provided by the methods described above, the method comprising:

[0167] Acquire the hyperspectral image to be classified;

[0168] The hyperspectral image to be classified is input into the trained classification model, and the classification result of the hyperspectral image to be classified is output.

[0169] The classification model is a lightweight model built on the self-attention mechanism, and it is trained based on the hyperspectral image dataset and the image classification labels corresponding to the hyperspectral image data.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight Transformer-based hyperspectral image classification method, characterized in that, include: Acquire the hyperspectral image to be classified; The hyperspectral image to be classified is input into a classification model trained by an initial network model, and the classification result of the hyperspectral image to be classified is output. The initial network model includes a convolutional module, a first self-attention mechanism module, a sparse token module, and a second self-attention mechanism module. Both the first and second self-attention mechanism modules include a feedforward neural network submodule and a self-attention submodule. The feedforward neural network has a specific hierarchical structure, including linear layers, activation functions, and random deactivation regularization components. The sparse token module is used to perform sparsification processing on the data. The classification model is a lightweight Transformer model built on the self-attention mechanism, and it is trained based on the hyperspectral image dataset and the image classification labels corresponding to the hyperspectral image dataset.

2. The lightweight Transformer hyperspectral image classification method according to claim 1, characterized in that, The training methods for the classification model include: Obtain publicly available hyperspectral image datasets; The hyperspectral image dataset is preprocessed to obtain hyperspectral image data blocks that can be input into the classification model; Construct the initial network model; The hyperspectral image data block is input into the initial network model to obtain the classification result. The loss value is determined by the loss function of the classification result and the image classification label. The training of the model parameters is determined to be completed when the number of training times reaches a set threshold. The classification accuracy of the model is obtained during the model iteration process, and the lightweight nature of the model is measured based on the number of network parameters and floating-point operations during the model iteration process. The iterative model with the highest classification accuracy and the most lightweight model is used as the final trained classification model.

3. The lightweight Transformer hyperspectral image classification method according to claim 2, characterized in that, The step of preprocessing the hyperspectral image dataset to obtain hyperspectral image data blocks that can be input into the classification model includes: The hyperspectral image dataset is subjected to PCA data dimensionality reduction processing to obtain dimensionality-reduced images; The reduced-dimensional image is used to construct cube samples to obtain the hyperspectral image data block.

4. The lightweight Transformer hyperspectral image classification method according to claim 3, characterized in that, The step of performing PCA data dimensionality reduction processing on the hyperspectral image dataset to obtain a dimensionality-reduced image includes: The three dimensions of the original hyperspectral image of the hyperspectral image dataset are determined, including the first image width, the first image height, and the number of original image bands. Based on principal component analysis, the first image width, first image height, and first original image band number of the original hyperspectral image are reduced in dimensionality to obtain the dimensionality-reduced image, and the second image width, second image height, and number of channels of the dimensionality-reduced image are determined.

5. The lightweight Transformer hyperspectral image classification method according to claim 4, characterized in that, The step of constructing the hyperspectral image data block by constructing cube samples from the dimensionality-reduced image includes: The dimensionality-reduced image is cropped based on the width of the second image, the height of the second image, and the number of channels after dimensionality reduction to determine the window size of the cube sample; The three-dimensional image block of the cube sample is determined based on the window size and the number of channels after dimensionality reduction.

6. The lightweight Transformer hyperspectral image classification method according to claim 5, characterized in that, The convolution module includes two convolutional layers. The convolution module is used to divide the three-dimensional image block into blocks to obtain image data that can be input to the first self-attention mechanism module. The first self-attention mechanism module is used to downsample the image data to obtain a three-dimensional image block of the first downsampled version. The second self-attention mechanism module is used to downsample the first downsampled 3D image patch after sparsification to obtain a second downsampled 3D image patch, and obtain the classification result based on the second downsampled 3D image patch.

7. The lightweight Transformer hyperspectral image classification method according to claim 2, characterized in that, The loss function is the cross-entropy loss function, and the evaluation index of classification accuracy is determined based on overall accuracy, evaluation accuracy, and consistency.

8. A lightweight Transformer-based hyperspectral image classification device, characterized in that, include: The acquisition unit is used to acquire the hyperspectral image to be classified. A classification unit is used to input the hyperspectral image to be classified into a classification model trained by an initial network model, and output the classification result of the hyperspectral image to be classified. The initial network model includes a convolutional module, a first self-attention mechanism module, a sparse token module, and a second self-attention mechanism module. Both the first and second self-attention mechanism modules include a feedforward neural network submodule and a self-attention submodule. The feedforward neural network has a specific hierarchical structure, including linear layers, activation functions, and random deactivation regularization components. The sparse token module is used to perform sparsification processing on the data. The classification model is a lightweight Transformer model built on the self-attention mechanism, and it is trained based on the hyperspectral image dataset and the image classification labels corresponding to the hyperspectral image dataset.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the lightweight Transformer hyperspectral image classification method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the lightweight Transformer hyperspectral image classification method as described in any one of claims 1 to 7.

Images