Hyperspectral image transformer network training and classification method

Abstract

The application discloses a hyperspectral image Transformer network training and classification method, which first divides a hyperspectral image sample based on a spectral dimension to obtain a plurality of spectral subbands, wherein the hyperspectral image sample is an unlabeled training sample; each spectral subband is input into an embedding module to obtain local space-spectrum embedding features output by each embedding module, wherein the embedding module extracts space-spectrum features of the spectral subband at multiple scales; all local space-spectrum embedding features are fused according to the positions of the spectral subbands to obtain global space-spectrum embedding features; the global space-spectrum embedding features are input into a Transformer encoder, and a center region token is masked and reconstructed in a Transformer decoder to train the Transformer encoder in a self-supervised manner. Compared with the prior art, the unlabeled sample can be effectively utilized to improve the training effect of the Transformer network, and the hyperspectral image can be classified with high precision.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral image classification technology, and in particular to a method for training and classifying hyperspectral image Transformer networks. Background Technology

[0002] Hyperspectral remote sensing images can extract ground feature information from hundreds of consecutive spectral bands, giving them a powerful ability to distinguish ground targets. Over the past few decades, hyperspectral images have played a crucial role in military target detection, marine monitoring, and disaster prevention. The identification and classification of hyperspectral images is a key issue in hyperspectral image analysis and plays a vital role in the advancement and development of hyperspectral remote sensing technology.

[0003] With the widespread application of deep learning in various fields, various deep learning classification methods have emerged for hyperspectral image classification, such as autoencoders (AE) and convolutional neural networks (CNN). However, due to the small sample size of hyperspectral images, it is even more difficult to achieve ideal training results with only labeled samples on small datasets. This results in poor training performance of existing hyperspectral image classification models, making it difficult to fully extract spatial spectral feature information and leading to low classification accuracy in practical applications of hyperspectral image classification.

[0004] Therefore, existing technologies need to be improved and enhanced. Summary of the Invention

[0005] The main objective of this invention is to provide a method for training and classifying hyperspectral image Transformer networks, aiming to solve the problem of poor training performance of existing hyperspectral image classification models.

[0006] To achieve the above objectives, the present invention provides a training method for a Transformer network for hyperspectral images, wherein an adaptive multi-scale embedding layer is set in the Transformer network, the adaptive multi-scale embedding layer comprising multiple embedding modules, and the training method includes:

[0007] Hyperspectral image samples are segmented based on the spectral dimension to obtain multiple spectral sub-bands. The hyperspectral image samples are unlabeled training samples.

[0008] Each spectral sub-band is input into an embedding module to obtain the local spatial spectral embedding features output by each embedding module. The embedding module extracts the spatial spectral features of the spectral sub-band at multiple scales.

[0009] By fusing all the local spatial spectrum embedding features according to the position of the spectral sub-bands, a global spatial spectrum embedding feature is obtained.

[0010] The global spatial spectrum embedding features are input into the Transformer encoder, and the central region token is masked and reconstructed in the Transformer decoder to train the Transformer encoder in a self-supervised manner.

[0011] Optionally, the embedding module includes a small-scale branch for embedding small-scale features, a medium-scale branch for embedding medium-scale features, and a large-scale branch for embedding large-scale features. The step of inputting each spectral sub-band into an embedding module to obtain the local spatial-spectral embedding features output by each embedding module includes:

[0012] The spectral sub-bands are input into the embedding module to obtain the small-scale embedding features output by the small-scale branch, the medium-scale embedding features output by the medium-scale branch, and the large-scale embedding features output by the large-scale branch, respectively.

[0013] The local spatial spectrum embedding feature is obtained by fusing the large-scale embedding feature, the medium-scale embedding feature, and the small-scale embedding feature.

[0014] Optionally, obtaining the large-scale embedding features of the large-scale branch output includes: performing a three-dimensional convolution operation on the spectral sub-band to extract a first spatial spectral feature, performing a three-dimensional convolution operation on the first spatial spectral feature to extract a second spatial spectral feature, and stitching and flattening the second spatial spectral feature in the spectral dimension to obtain the large-scale embedding features.

[0015] The process of obtaining the mesoscale embedding features of the mesoscale branch output includes: performing a three-dimensional convolution operation on the spectral sub-bands, extracting the third spatial spectral features, fusing the third spatial spectral features and flattening them to obtain the mesoscale embedding features.

[0016] Optionally, fusing the large-scale embedding features, the mesoscale embedding features, and the small-scale embedding features to obtain the local spatial spectrum embedding features includes:

[0017] The local spatial spectrum embedding feature is obtained by fusing the large-scale embedding feature, the medium-scale embedding feature, and the small-scale embedding feature using learnable weight parameters.

[0018] Optionally, after training the Transformer encoder, the following may also be included:

[0019] The weight parameters are reset, the output of the trained Transformer encoder is input into the classifier, and the trained Transformer encoder is retrained using labeled hyperspectral image samples.

[0020] Optionally, the step of masking and reconstructing the central region token in the Transformer decoder to train the Transformer encoder in a self-supervised manner includes:

[0021] Replace the center pixel of the training sample with a learnable vector;

[0022] Reconstruct the center pixel and training samples in the Transformer decoding layer;

[0023] The Transformer encoder is trained based on the loss value reconstructed from the center pixel and the loss value reconstructed from the training samples.

[0024] To achieve the above objectives, the present invention also provides a hyperspectral image classification method based on a multi-scale convolutional Transformer network. The multi-scale convolutional Transformer network includes an adaptive multi-scale embedding layer and a Transformer encoder. The adaptive multi-scale embedding layer includes multiple embedding modules. The classification method includes:

[0025] The multi-scale convolutional Transformer network is trained in advance using any of the above-mentioned training methods for Transformer networks used for hyperspectral images;

[0026] The hyperspectral image is input into the multi-scale convolutional Transformer network to obtain classification features;

[0027] The classification features are input into the classifier to obtain the hyperspectral image classification results.

[0028] To achieve the above objectives, the present invention also provides a training apparatus for a Transformer network for hyperspectral images, wherein the Transformer network includes an adaptive multi-scale embedding layer, the adaptive multi-scale embedding layer including multiple embedding modules, and the apparatus includes:

[0029] The spectral sub-band module is used to segment a hyperspectral image based on the spectral dimension to obtain multiple spectral sub-bands, wherein the hyperspectral image is an unlabeled training sample of the Transformer network;

[0030] The local spatial spectrum embedding feature module is used to input each spectral sub-band into an embedding module to obtain the local spatial spectrum embedding feature output by each embedding module. The embedding module extracts the spatial spectrum features of the spectral sub-band at multiple scales.

[0031] The global spatial spectrum embedding feature module is used to fuse all the local spatial spectrum embedding features according to the position of the spectral sub-bands to obtain the global spatial spectrum embedding feature;

[0032] The self-supervised training module is used to input the global spatial spectrum embedding features into the Transformer encoder, and to mask and reconstruct the central region token in the Transformer decoder to train the Transformer encoder in a self-supervised manner.

[0033] To achieve the above objectives, the present invention also provides a smart terminal, which includes a memory, a processor, and a training program for a Transformer network for hyperspectral images stored in the memory and executable on the processor. When the training program for the Transformer network for hyperspectral images is executed by the processor, it implements any of the steps of the training method for the Transformer network for hyperspectral images described above.

[0034] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a training program for a Transformer network for hyperspectral images, wherein the training program for a Transformer network for hyperspectral images, when executed by a processor, implements any of the steps of the above-described training method for a Transformer network for hyperspectral images.

[0035] As can be seen from the above, this invention effectively reduces the difficulty of feature extraction and fully extracts sample information by grouping hyperspectral image bands and then extracting the spatial spectral features of the spectral sub-bands at multiple scales. Based on the masking and reconstruction of the central region token, it effectively utilizes unlabeled samples and improves the Transformer network's ability to model relationships within its neighborhood through self-supervised learning, further enhancing the Transformer network's performance in small sample scenarios. A Transformer network with good training results can classify hyperspectral images with high accuracy. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic flowchart of an embodiment of the training method for a Transformer network for hyperspectral images provided by the present invention;

[0038] Figure 2 yes Figure 1Schematic diagram of the network model architecture of the multi-scale embedding layer in the embodiment;

[0039] Figure 3 yes Figure 2 A schematic diagram of the embedded module;

[0040] Figure 4 yes Figure 1 A schematic diagram illustrating the process of obtaining large-scale embedded features in the embodiment;

[0041] Figure 5 yes Figure 1 A schematic diagram of the process for obtaining mesoscale embedding features in the embodiment;

[0042] Figure 6 This is a schematic diagram of the model architecture during Transformer network training;

[0043] Figure 7 This is a schematic diagram of the process of self-supervised training of a Transformer network;

[0044] Figure 8 This is a schematic diagram of the structure of a training device for a Transformer network for hyperspectral images provided in an embodiment of the present invention;

[0045] Figure 9 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0046] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0047] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0048] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0049] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0050] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0053] Hyperspectral images are formed over long distances, and numerous interference factors exist during the imaging process, leading to widespread phenomena such as different spectra for the same object and different objects with the same spectrum. Traditional classification models, during training, cannot achieve long-range modeling of input samples, resulting in insufficient extraction of sample information. In scenarios with small sample sizes, relying solely on labeled samples makes it even more difficult to achieve ideal training results. Therefore, traditional classification models perform poorly and struggle to accurately classify hyperspectral images.

[0054] To address the issues of poor training performance and low classification accuracy in existing hyperspectral image classification models, particularly their poor performance with limited sample sizes, this invention provides a training method for Transformer networks used in hyperspectral images. Based on a band-group-based spatial-spectral feature embedding technique, local spatial-spectral embeddings for each spectral band are first constructed, and then combined to form a global spatial-spectral embedding. Since the divide-and-conquer band-group-based feature extraction effectively reduces the difficulty of feature embedding and fully extracts spatial-spectral features, and through masking and reconstruction based on central region tokens, unlabeled samples are effectively utilized, and self-supervised learning enhances the Transformer network's ability to model relationships within its neighborhood, further improving its performance with small sample sizes. Therefore, the spatial-spectral feature information of hyperspectral images can be fully extracted, improving the training effect of the Transformer network and achieving accurate classification of hyperspectral images.

[0055] Exemplary methods

[0056] like Figure 1 As shown, this embodiment of the invention provides a training method for a Transformer network for hyperspectral images, deployed on a smart terminal. In terms of network architecture, an adaptive multi-scale embedding layer comprising multiple embedding modules is first set before the Transformer encoder in the Transformer network. The spatial-spectral features extracted by the adaptive multi-scale embedding layer are input into the Transformer encoder. Each embedding module is used to extract spatial-spectral features of spectral sub-bands from multiple scales.

[0057] Specifically, the above training method includes the following steps:

[0058] Step S100: Segment the hyperspectral image samples based on the spectral dimension to obtain multiple spectral sub-bands, wherein the hyperspectral image samples are unlabeled training samples;

[0059] Specifically, in this embodiment, the training samples are unlabeled hyperspectral images. The Transformer network is trained using unlabeled hyperspectral images to fully learn the spatial spectral features of the hyperspectral images and improve the classification performance of the Transformer network.

[0060] Assuming the hyperspectral image is Where X and Y represent the length and width of the spatial dimension of the hyperspectral image, and B represents the spectral dimension (i.e., spectral band) of the hyperspectral image. By dividing the hyperspectral image into equal-length segments along the spectral dimension, we can obtain images such as... Figure 2 The multiple spectral sub-bands shown.

[0061] Since hyperspectral images are high-dimensional data, to avoid the curse of dimensionality, this embodiment first uses Principal Component Analysis (PCA) to reduce the dimensionality of the original hyperspectral image before segmentation. Assume the dimensionality-reduced hyperspectral image is H∈R. X×Y×K Where X and Y are the length and width of the hyperspectral image's spatial dimension, respectively, generally X = Y, and K is the spectral dimension retained after PCA dimensionality reduction. Specifically, in this embodiment, X = Y = 13, K = 80, meaning 80 spectral bands are retained after dimensionality reduction. Then, the dimensionality-reduced hyperspectral image H∈R is divided into equal-length segments along the spectral dimension. X×Y×K After equal-length segmentation, the image is divided into k spectral sub-bands. In this embodiment, the value of k is 8, resulting in a set of spectral sub-bands (H′1, H′2, H′3, ..., H′8). The specific value of k is not limited and can be adjusted according to the actual situation.

[0062] Step S200: Input each spectral sub-band into an embedding module to obtain the local spatial spectrum embedding features output by each embedding module. The embedding module extracts the spatial spectrum features of the spectral sub-band at multiple scales.

[0063] Specifically, the number of embedding modules in the adaptive multi-scale embedding layer is set according to the number of spectral sub-bands, so that each embedding module extracts the spatial-spectral features of one spectral sub-band. While the Transformer network excels at extracting global information and achieving long-range modeling, its extraction of local information is insufficient. To more fully extract the spatial-spectral features of spectral sub-bands, the embedding module of this invention extracts these features at multiple scales and fuses all the features to obtain local spatial-spectral embedding features, which are then output. This approach is more suitable for feature extraction from hyperspectral images and achieves excellent model performance under both small-sample and arbitrary-sample training conditions. The method for extracting spatial-spectral features of spectral sub-bands at multiple scales is not limited; for example, convolution operations at different scales can be used to extract spatial-spectral features of spectral sub-bands at different scales.

[0064] In this embodiment, each embedding module extracts spatial-spectral features from three scales: a small-scale branch for embedding small-scale features, a medium-scale branch for embedding medium-scale features, and a large-scale branch for embedding large-scale features. After inputting the spectral sub-bands into the embedding module, the small-scale embedding features output by the small-scale branch, the medium-scale embedding features output by the medium-scale branch, and the large-scale embedding features output by the large-scale branch are obtained respectively. Then, the large-scale embedding features, medium-scale embedding features, and small-scale embedding features are fused to obtain local spatial-spectral embedding features.

[0065] In the small-scale branch, for the input spectral sub-band First, linear embedding is used to embed features into each spectrum in the spectral sub-bands. Then, the embedding results are flattened in the spatial dimension to obtain small-scale embedding features. The specific expression is:

[0066]

[0067]

[0068] Among them, the linear embedding results Result after flattening Emb small ∈R EmbNum×EmbDim EmbNum is X×Y, and EmbDim is the model embedding dimension.

[0069] Considering the partial feature dependencies between the mesoscale and large-scale branches, their feature embeddings differ from those of the small-scale branch, such as... Figure 3 As shown, it also includes 3D convolution operations.

[0070] like Figure 4 As shown, the specific steps to obtain the large-scale embedding features of the large-scale branch output include:

[0071] Step A210: Perform a three-dimensional convolution operation on the spectral sub-bands to extract the first spatial spectral features;

[0072] Step A220: Perform a three-dimensional convolution operation on the first spatial spectral feature to extract the second spatial spectral feature;

[0073] Step A230: Concatenate and flatten the second spatial spectral features along the spectral dimension to obtain large-scale embedded features.

[0074] Specifically, the first spatial spectral feature is obtained by first performing a 3D convolution operation with a kernel size of (3,3,3) to extract local spatial spectral features. Performing another 3D convolution operation with a kernel size of (3,3,3) yields the second spatial spectral feature. After stitching along the spectral dimension, the data is flattened along the spatial dimension to obtain large-scale embedded features. The specific expression is:

[0075]

[0076]

[0077]

[0078] like Figure 5 As shown, the specific steps to obtain the mesoscale embedding features of the mesoscale branch output include:

[0079] Step B210: Perform a three-dimensional convolution operation on the spectral sub-bands to extract the third spatial spectral features;

[0080] Step B220: Fuse the third spatial spectral features and flatten them to obtain mesoscale embedding features.

[0081] Specifically, a third spatial-spectral feature is first obtained by performing a 3D convolution operation with a kernel size of (3,3,3) to extract local spatial-spectral features. Then, one-dimensional convolution is used to fuse and flatten the four third spatial spectral features to obtain the mesoscale embedding features. The specific expression is:

[0082]

[0083]

[0084] Among them, spatial spectral features Mesoscale embedding features EmbNum is X×Y, and EmbDim is the model embedding dimension.

[0085] Furthermore, considering the differences in spatial resolution of the datasets, learnable weight parameters were used in the multi-scale embedding feature fusion. small ,w mid ,w large To fuse large-scale embedded features Mesoscale embedding features and small-scale embedding features This allows for the adjustment of embedding weights for each branch feature, thereby obtaining local spatial spectral embedding features. The specific expression is:

[0086]

[0087] k spectral sub-bands After inputting into the parallel embedding module, the local spatial-spectral embedding features corresponding to each spectral sub-band are obtained. Specifically, this is represented as follows: Among them, Emb local ∈R k ×X′×Y′×z′ / k X′ and Y′ are the length and width of the spatial dimension after local spatial spectral feature extraction, and Z′ is the spectral dimension after local spatial spectral feature extraction.

[0088] It should be noted that the multi-scale branches of the embedding module can be freely selected based on the data features of the hyperspectral image, giving the embedding module better adaptability. The embedding module can be specifically represented as:

[0089]

[0090] in, SSEM represents the local spatial spectral embedding feature corresponding to the i-th spectral sub-band. i The function represents the local spatial spectral embedding function corresponding to the i-th spectral sub-band. Local Spatial Spectral Embedding Function (SSEM) i (i = 1, 2, ..., k) can be freely selected based on the features of the sample data, which brings great freedom to feature embedding.

[0091] Step S300: Fuse all local spatial spectrum embedding features according to the position of the spectral sub-bands to obtain global spatial spectrum embedding features;

[0092] Specifically, such as Figure 2 As shown, the above-mentioned local spatial spectrum embedding features are used. The fusion is performed according to the positional order of the spectral sub-bands. For example, all local spatial spectral embedding features are simply concatenated according to the positional order of the spectral sub-bands, and then flattened in the spatial dimension through two flattening operations to obtain the global spatial spectral embedding feature Emb. global .

[0093] Emb′ local =Reshape_1(Emb local Emb global =Reshape_2(Emb′ local ),

[0094] Where Reshape_1 and Reshape_2 represent flattening operations, and the flattened local spatial spectral embedding feature Emb′ is... kocal ∈R X′×Y′×Z′ Flattened global spatial spectrum embedding feature Emb global ∈R EmbSize×Z′ The number of feature embeddings, EmbSize = X′ × Y′. Flattening is a common technique in neural network models and will not be elaborated upon here.

[0095] Step S400: Input the global spatial spectrum embedding features into the Transformer encoder, and mask and reconstruct the central region token in the Transformer decoder to train the Transformer encoder in a self-supervised manner.

[0096] Specifically, to address the problem that existing classification models do not effectively utilize unlabeled samples, this invention employs masking and reconstruction based on central region tokens, which can effectively utilize unlabeled samples. Through self-supervised learning, it enhances the model's ability to model relationships in its neighborhood, further improving the performance of the classification model in small sample situations. By effectively utilizing unlabeled samples, it can also enhance the effect of labeled sample classification training.

[0097] refer to Figure 6 As shown, in this embodiment, the decoder consists of two layers of standard Transformer encoders. The global spatial spectrum is embedded into the feature input decoder, and the central region token is masked and reconstructed in the decoder to train the Transformer encoder in a self-supervised manner.

[0098] like Figure 7 As shown, the central region token is masked and reconstructed in the decoder to train the Transformer encoder in a self-supervised manner. The specific steps include:

[0099] Step S410: Replace the center pixel of the training sample with a learnable vector;

[0100] Step S420: Reconstruct the center pixel and training samples in the decoder;

[0101] Step S430: Train the Transformer encoder based on the loss value reconstructed from the center pixel and the loss value reconstructed from the training samples.

[0102] Specifically, unlike self-supervised training in computer vision, RGB images cannot directly identify the regions of interest, while the neighborhood regions of hyperspectral (HSI) image samples exist to enrich the feature representation of the central pixel. Therefore, the masking target can select an important part of the training samples, namely the central pixel, and embed the global spatial spectrum into the sequence marker L in the middle of the feature. center Replace with a learnable vector V learn This yields a new feature representation sequence, which is then used to reconstruct the training samples. Specifically, it is labeled as follows:

[0103] L = (L1, L2, ..., L center ,…L k → L′=(L1,L2,…,V learn ,…L k ).

[0104] The masked sequence L′ is input into the Transformer decoder, where the MLP head (MLP: a fully connected network with one hidden layer) performs pixel-level sequence reconstruction to obtain the overall reconstruction result R(R1,R2,…,R) of the center pixel and the training samples. center ,…,R k ).

[0105] R = Decoder(L′).

[0106] Among them, R i Corresponding sequence L i The pixel sequence reconstruction result.

[0107] In other words, the training process in this embodiment is divided into two tasks: Task 1 is the main task—sequence reconstruction of the center pixel—and Task 2 is the auxiliary task—reconstruction of neighboring samples. This multi-task training strategy allows the training model to better learn the relationship between the center pixel and its neighboring pixels even without labels. Simultaneously, the auxiliary task also serves as a regularization mechanism, preventing the main task from falling into pattern collapse. The loss function of the training model can be expressed as:

[0108] Loss center =MSE(R) center ,Spectral center ),

[0109] Loss sample =MSE(R,H),

[0110] Loss total =w center *Loss center +w sample *Loss sample ,

[0111] Among them, Spectral center w represents the center pixel sequence of the input sample H. center and w sample The Loss value represents the weight of the center pixel reconstruction task and the sample reconstruction task in the total loss function. center Loss is the loss value for the center pixel reconstruction task. sample The loss value for the sample reconstruction task.

[0112] Most hyperspectral image classification methods are patch-based. However, the input to a classification model includes not only the spectral curve of the center pixel but also its neighboring regions (typically square regions), making the input more distinctive. Therefore, the training method in this embodiment is easier to implement and more suitable for hyperspectral data.

[0113] After the above training is completed, in practical applications, the Transformer decoder is removed and used only during the training phase. The Transformer encoder containing the adaptive multi-scale embedding layer becomes the classification model, which can be used to classify and recognize hyperspectral images. The above training of the Transformer network using unlabeled samples can also be called pre-training. Based on pre-training, the classification model (i.e., the Transformer network without the Transformer decoder) can be further trained using labeled samples to improve the training effect and classification accuracy.

[0114] In one embodiment, the specific steps of the above classification training are as follows: resetting the weight parameters used when fusing large-scale, medium-scale, and small-scale embedding features; inputting labeled hyperspectral image samples into the Transformer network; and inputting the output of the Transformer encoder into the classifier to retrain the Transformer encoder. By resetting the multi-branch weight parameters and using a small number of labeled samples to fine-tune the classification model, the training effect is further enhanced, resulting in higher classification accuracy of the trained model. Using labeled samples to train neural network models is a conventional technique in this field and will not be elaborated further here.

[0115] As described above, this embodiment first divides unlabeled hyperspectral image samples into multiple spectral sub-bands based on the spectral dimension. Each spectral sub-band is then input into an embedding module to extract spatial-spectral features from multiple scales to obtain local spatial-spectral embedding features. All local spatial-spectral embedding features are then fused to obtain global spatial-spectral embedding features, which are then input into a Transformer encoder. The Transformer's long-range modeling capability is utilized to reconstruct the global spatial-spectral embedding features. Furthermore, the token in the central region is masked and reconstructed to self-supervisedly train the Transformer encoder, resulting in a trained Transformer network. This approach effectively extracts feature information from unlabeled samples, enhances the training effect of the Transformer network, and achieves higher classification accuracy.

[0116] It should be noted that the specific type of Transformer encoder described above is not limited and can be selected according to the actual application. In this embodiment, only the position encoding part of the Transformer encoder has been removed.

[0117] Based on the Transformer network model constructed using the above training method, this embodiment of the invention also provides a hyperspectral image classification method based on a multi-scale convolutional Transformer network. The multi-scale convolutional Transformer network is the classification model for hyperspectral images, including an adaptive multi-scale embedding layer and a Transformer encoder. The adaptive multi-scale embedding layer is the same as the adaptive multi-scale embedding layer in the training method, including multiple embedding modules. The specific steps of the classification method include: pre-training the multi-scale convolutional Transformer network using the above training method; then inputting the hyperspectral image into the multi-scale convolutional Transformer network; extracting spatial spectral features at multiple scales to obtain the classification features output by the multi-scale convolutional Transformer network; and inputting the classification features into a classifier to obtain the hyperspectral image classification result. The classifier is not limited and can be various classifiers such as logistic regression decision trees, random forests, and support vector machines (SVM).

[0118] As described above, this embodiment segments the hyperspectral image into equal-length segments along the spectral dimension, inputs each segment into a parallel embedding module to obtain local spatial-spectral embedding features, then concatenates these features according to their original positions before inputting them into a Transformer encoding layer to extract features and further classify them, thus obtaining the hyperspectral image classification result. By employing a divide-and-conquer approach to extract global spatial-spectral embedding features from the hyperspectral image, achieving both divide-and-conquer feature embedding and global spatial-spectral embedding feature reconstruction, the difficulty of spatial-spectral feature extraction can be effectively reduced, feature information can be fully extracted, and the classification accuracy of the hyperspectral image can be improved.

[0119] The complete implementation process is as follows: First, pre-training is performed. Principal component analysis is used to reduce the dimensionality of the original hyperspectral images, and the hyperspectral image dataset is divided into training and testing sets. After the hyperspectral image samples are divided into equal-length segments, they are input into parallel embedding modules to obtain the local spatial spectral embedding features corresponding to the hyperspectral image samples. These features are then concatenated. Tokens generated from the center region are removed and replaced with learnable vectors. All sequences are input into a decoding module consisting of two Transformer decoders. By capturing the relationships between tokens and the information interaction of the self-attention module, the learnable vectors can restore the spectral features of the center region as much as possible. The spectral curve of the center pixel is used as the supervision target, and the relationship between the neighborhood and the center pixel is learned in this process. Then, the Transformer encoder is used as a classification model. The multi-branch weight parameters are reset, and the classification model is fine-tuned using a small number of samples to obtain the trained classification model. This model can then be used to classify ground features.

[0120] Tables 1 and 2 below show the experimental results of various hyperspectral image classification methods on the Pavia University dataset and the Gaofen-5 dataset, respectively, with five training samples for each category. It can be seen that, compared to existing hyperspectral image classification methods, the classification method of this invention achieves better classification accuracy.

[0121] Table 1. Pavia University Dataset

[0122]

[0123]

[0124] Table 2. Gaofen-5 dataset

[0125] category CNNHSI HybridSN VIT SpectralFormer SSFTT SPRLT This method 1 73.13 73.10 49.52 66.96 80.76 90.61 88.83 2 98.91 99.39 88.06 99.65 99.46 98.60 99.09 3 60.89 63.83 51.45 82.79 89.74 80.13 85.27 4 64.17 65.84 79.18 90.43 91.84 93.49 95.79 5 70.94 76.11 81.96 98.94 94.60 97.23 98.40 6 82.00 81.30 74.08 84.59 87.06 83.34 86.38 7 49.24 53.39 56.59 63.93 57.77 64.97 59.19 8 74.03 71.46 71.86 82.80 81.37 85.58 84.67 9 87.24 86.38 71.48 69.56 85.94 83.88 88.63 10 93.38 78.79 67.99 76.94 82.89 81.67 91.21 11 79.57 83.79 47.55 83.07 92.92 81.42 92.47 12 79.10 79.43 63.60 73.97 80.98 77.28 85.04 13 75.31 83.64 70.84 84.24 91.49 87.26 92.38 14 77.43 72.18 62.29 88.33 79.60 86.76 83.12 15 80.42 62.96 59.97 92.86 70.18 88.63 63.28 16 89.64 88.24 74.55 77.82 84.09 87.70 90.66 17 93.76 92.49 78.40 86.28 94.44 84.65 97.98 18 56.53 55.43 50.83 66.32 69.16 57.62 69.09 19 74.82 81.22 59.47 61.99 82.97 75.79 84.21 20 95.14 81.23 75.08 77.69 90.46 72.31 85.85 OA 72.96 74.86 72.60 87.78 89.07 88.57 91.38 AA 77.78 76.51 66.74 80.46 84.39 82.95 86.08 Kappa 69.30 71.54 68.46 85.86 87.37 86.78 90.02

[0126] Exemplary device

[0127] like Figure 8 As shown, corresponding to the training method of the Transformer network for hyperspectral images, this embodiment of the invention also provides a training device for the Transformer network for hyperspectral images. The Transformer network includes an adaptive multi-scale embedding layer, which includes multiple embedding modules. Specifically, the device includes:

[0128] The spectral sub-band module 600 is used to segment a hyperspectral image based on the spectral dimension to obtain multiple spectral sub-bands, wherein the hyperspectral image is an unlabeled training sample of the Transformer network.

[0129] The local spatial spectrum embedding feature module 610 is used to input each spectral sub-band into an embedding module to obtain the local spatial spectrum embedding feature output by each embedding module. The embedding module extracts the spatial spectrum features of the spectral sub-band at multiple scales.

[0130] The global spatial spectrum embedding feature module 620 is used to fuse all the local spatial spectrum embedding features according to the position of the spectral sub-bands to obtain the global spatial spectrum embedding feature.

[0131] The self-supervised training module 630 is used to input the global spatial spectrum embedding features into the Transformer encoder and to mask and reconstruct the central region token in the Transformer decoder to train the Transformer encoder in a self-supervised manner.

[0132] In this embodiment, the training apparatus for the Transformer network for hyperspectral images described above can be referred to the corresponding description in the training method for the Transformer network for hyperspectral images described above, and will not be repeated here.

[0133] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 9 As shown. The aforementioned smart terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a training program for a Transformer network for hyperspectral images. The internal memory provides an environment for the operation of the operating system and the training program for the Transformer network for hyperspectral images stored in the non-volatile storage medium. The network interface of the smart terminal is used for communication with external terminals via a network connection. When the training program for the Transformer network for hyperspectral images is executed by the processor, it implements the steps of any of the aforementioned training methods for the Transformer network for hyperspectral images. The display screen of the smart terminal can be a liquid crystal display (LCD) or an e-ink display.

[0134] Those skilled in the art will understand that Figure 9 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0135] In one embodiment, a smart terminal is provided, the smart terminal including a memory, a processor, and a training program for a Transformer network for hyperspectral images stored in the memory and executable on the processor, wherein the training program for the Transformer network for hyperspectral images, when executed by the processor, performs the following operation instructions:

[0136] Hyperspectral image samples are segmented based on the spectral dimension to obtain multiple spectral sub-bands. The hyperspectral image samples are unlabeled training samples.

[0137] Each spectral sub-band is input into an embedding module to obtain the local spatial spectral embedding features output by each embedding module. The embedding module extracts the spatial spectral features of the spectral sub-band at multiple scales.

[0138] By fusing all the local spatial spectrum embedding features according to the position of the spectral sub-bands, a global spatial spectrum embedding feature is obtained.

[0139] The global spatial spectrum embedding features are input into the Transformer encoder, and the central region token is masked and reconstructed in the Transformer decoder to train the Transformer encoder in a self-supervised manner.

[0140] Optionally, the embedding module includes a small-scale branch for embedding small-scale features, a medium-scale branch for embedding medium-scale features, and a large-scale branch for embedding large-scale features. The step of inputting each spectral sub-band into an embedding module to obtain the local spatial-spectral embedding features output by each embedding module includes:

[0141] The spectral sub-bands are input into the embedding module to obtain the small-scale embedding features output by the small-scale branch, the medium-scale embedding features output by the medium-scale branch, and the large-scale embedding features output by the large-scale branch, respectively.

[0142] The local spatial spectrum embedding feature is obtained by fusing the large-scale embedding feature, the medium-scale embedding feature, and the small-scale embedding feature.

[0143] Optionally, obtaining the large-scale embedding features of the large-scale branch output includes: performing a three-dimensional convolution operation on the spectral sub-band to extract a first spatial spectral feature, performing a three-dimensional convolution operation on the first spatial spectral feature to extract a second spatial spectral feature, and stitching and flattening the second spatial spectral feature in the spectral dimension to obtain the large-scale embedding features.

[0144] The process of obtaining the mesoscale embedding features of the mesoscale branch output includes: performing a three-dimensional convolution operation on the spectral sub-bands, extracting the third spatial spectral features, fusing the third spatial spectral features and flattening them to obtain the mesoscale embedding features.

[0145] Optionally, fusing the large-scale embedding features, the mesoscale embedding features, and the small-scale embedding features to obtain the local spatial spectrum embedding features includes:

[0146] The local spatial spectrum embedding feature is obtained by fusing the large-scale embedding feature, the medium-scale embedding feature, and the small-scale embedding feature using learnable weight parameters.

[0147] Optionally, after training the Transformer encoder, the following may also be included:

[0148] The weight parameters are reset, the output of the trained Transformer encoder is input into the classifier, and the trained Transformer encoder is retrained using labeled hyperspectral image samples.

[0149] Optionally, the step of masking and reconstructing the central region token in the Transformer decoder to train the Transformer encoder in a self-supervised manner includes:

[0150] Replace the center pixel of the training sample with a learnable vector;

[0151] Reconstruct the center pixel and training samples in the Transformer decoding layer;

[0152] The Transformer encoder is trained based on the loss value reconstructed from the center pixel and the loss value reconstructed from the training samples.

[0153] This invention also provides a computer-readable storage medium storing a training program for a Transformer network for hyperspectral images. When executed by a processor, the training program implements the steps of any of the training methods for a Transformer network for hyperspectral images provided in this invention.

[0154] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0156] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0157] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0158] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of the above modules or units is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0159] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0160] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not mean that the essence of the corresponding technical solutions deviates from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for training Transformer networks for hyperspectral images, characterized in that, An adaptive multi-scale embedding layer is set in the Transformer network, the adaptive multi-scale embedding layer comprising multiple embedding modules, and the training method includes: Hyperspectral image samples are segmented based on the spectral dimension to obtain multiple spectral sub-bands. The hyperspectral image samples are unlabeled training samples. Each spectral sub-band is input into an embedding module to obtain the local spatial spectral embedding features output by each embedding module. The embedding module extracts the spatial spectral features of the spectral sub-band at multiple scales. By fusing all the local spatial spectrum embedding features according to the position of the spectral sub-bands, a global spatial spectrum embedding feature is obtained. The global spatial spectrum embedding features are input into the Transformer encoder, and the central region token is masked and reconstructed in the Transformer decoder to train the Transformer encoder in a self-supervised manner. The embedding module includes a small-scale branch for embedding small-scale features, a medium-scale branch for embedding medium-scale features, and a large-scale branch for embedding large-scale features. Each spectral sub-band is input into one embedding module to obtain the local spatial-spectral embedding features output by each embedding module, including: The spectral sub-bands are input into the embedding module to obtain the small-scale embedding features output by the small-scale branch, the medium-scale embedding features output by the medium-scale branch, and the large-scale embedding features output by the large-scale branch, respectively. The local spatial spectrum embedding feature is obtained by fusing the large-scale embedding feature, the medium-scale embedding feature, and the small-scale embedding feature; The method for obtaining the large-scale embedded features of the large-scale branch output includes: performing a three-dimensional convolution operation on the spectral sub-band to extract the first spatial spectral feature, performing a three-dimensional convolution operation on the first spatial spectral feature to extract the second spatial spectral feature, and splicing and flattening the second spatial spectral feature in the spectral dimension to obtain the large-scale embedded features. The method for obtaining the mesoscale embedding feature of the mesoscale branch output includes: performing a three-dimensional convolution operation on the spectral sub-band, extracting the third spatial spectral feature, fusing the third spatial spectral feature and flattening it to obtain the mesoscale embedding feature. The process of obtaining small-scale embedding features of small-scale branch inputs includes: using linear embedding to embed features into each spectrum in the spectral sub-band, flattening the embedding results in the spatial dimension, and obtaining small-scale embedding features.

2. The training method for Transformer networks for hyperspectral images as described in claim 1, characterized in that, The process of fusing the large-scale embedding features, the medium-scale embedding features, and the small-scale embedding features to obtain the local spatial spectrum embedding features includes: The local spatial spectrum embedding feature is obtained by fusing the large-scale embedding feature, the medium-scale embedding feature, and the small-scale embedding feature using learnable weight parameters.

3. The training method for Transformer networks used in hyperspectral images as described in claim 2, characterized in that, After training the Transformer encoder, the following steps are also included: The weight parameters are reset, the output of the trained Transformer encoder is input into the classifier, and the trained Transformer encoder is retrained using labeled hyperspectral image samples.

4. The training method for a Transformer network for hyperspectral images as described in claim 1, characterized in that, The process of masking and reconstructing the central region token in the Transformer decoder to train the Transformer encoder in a self-supervised manner includes: Replace the center pixel of the training sample with a learnable vector; Reconstruct the center pixel and training samples in the Transformer decoding layer; The Transformer encoder is trained based on the loss value reconstructed from the center pixel and the loss value reconstructed from the training samples.

5. A hyperspectral image classification method based on multi-scale convolutional Transformer networks, characterized in that, The multi-scale convolutional Transformer network includes an adaptive multi-scale embedding layer and a Transformer encoder. The adaptive multi-scale embedding layer includes multiple embedding modules. The classification method includes: The multi-scale convolutional Transformer network is trained in advance using the training method for Transformer networks for hyperspectral images as described in any one of claims 1 to 4; The hyperspectral image is input into the multi-scale convolutional Transformer network to obtain classification features; The classification features are input into the classifier to obtain the hyperspectral image classification results.

6. A training apparatus for a Transformer network for hyperspectral images, characterized in that, The Transformer network includes an adaptive multi-scale embedding layer, which includes multiple embedding modules. The device includes: The spectral sub-band module is used to segment a hyperspectral image based on the spectral dimension to obtain multiple spectral sub-bands, wherein the hyperspectral image is an unlabeled training sample of the Transformer network; The local spatial spectrum embedding feature module is used to input each spectral sub-band into an embedding module to obtain the local spatial spectrum embedding feature output by each embedding module. The embedding module extracts the spatial spectrum features of the spectral sub-band at multiple scales. The embedding module includes a small-scale branch for embedding small-scale features, a medium-scale branch for embedding medium-scale features, and a large-scale branch for embedding large-scale features. Each spectral sub-band is input into one embedding module to obtain the local spatial-spectral embedding features output by each embedding module, including: The spectral sub-bands are input into the embedding module to obtain the small-scale embedding features output by the small-scale branch, the medium-scale embedding features output by the medium-scale branch, and the large-scale embedding features output by the large-scale branch, respectively. The local spatial spectrum embedding feature is obtained by fusing the large-scale embedding feature, the medium-scale embedding feature, and the small-scale embedding feature; The method for obtaining the large-scale embedded features of the large-scale branch output includes: performing a three-dimensional convolution operation on the spectral sub-band to extract the first spatial spectral feature, performing a three-dimensional convolution operation on the first spatial spectral feature to extract the second spatial spectral feature, and splicing and flattening the second spatial spectral feature in the spectral dimension to obtain the large-scale embedded features. The method for obtaining the mesoscale embedding feature of the mesoscale branch output includes: performing a three-dimensional convolution operation on the spectral sub-band, extracting the third spatial spectral feature, fusing the third spatial spectral feature and flattening it to obtain the mesoscale embedding feature. The method for obtaining the small-scale embedding features of the small-scale branch input includes: using linear embedding to embed features into each spectrum in the spectral sub-band, flattening the embedding results in the spatial dimension, and obtaining small-scale embedding features. The global spatial spectrum embedding feature module is used to fuse all the local spatial spectrum embedding features according to the position of the spectral sub-bands to obtain the global spatial spectrum embedding feature; The self-supervised training module is used to input the global spatial spectrum embedding features into the Transformer encoder, and to mask and reconstruct the central region token in the Transformer decoder to train the Transformer encoder in a self-supervised manner.

7. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a training program for a Transformer network for hyperspectral images stored in the memory and executable on the processor. When the training program for the Transformer network for hyperspectral images is executed by the processor, it implements the steps of the training method for a Transformer network for hyperspectral images as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a training program for a Transformer network for hyperspectral images, which, when executed by a processor, implements the steps of the training method for a Transformer network for hyperspectral images as described in any one of claims 1-4.

Images