End-to-end transformer-based hyperspectral image classification method and device and medium
By using a method of grouping words and self-attention mechanism to filter features, combined with a cascaded encoder and classifier, the problem of high computational cost and long inference time in hyperspectral image classification of the Transformer architecture is solved, achieving efficient end-to-end dimensionality reduction and refined classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-08
- Publication Date
- 2026-03-03
AI Technical Summary
Existing hyperspectral image classification methods based on the Transformer architecture suffer from high computational and inference time burdens and struggle to effectively learn the spatial-spectral joint features of hyperspectral data, resulting in poor classification performance.
Effective features are selected by grouping and tokenizing words and using self-attention mechanisms. By combining cascaded encoders and classifiers and setting a retention rate, end-to-end dimensionality reduction and classification are achieved. Spatial-spectral joint features are extracted using multi-head attention computing layers, word selection and fusion layers and multilayer perceptrons.
It significantly reduces computational burden and inference time, improves classification robustness, enables refined classification of ground features, and lowers the computational cost of the Transformer architecture.
Smart Images
Figure CN115731410B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of remote sensing image processing technology, and in particular to an end-to-end hyperspectral image classification method, apparatus and medium based on a deep self-attention transform network (Transformer). Background Technology
[0002] Hyperspectral remote sensing images, containing continuous and detailed spectral information of ground features, are widely used in numerous Earth observation missions. Different spectral characteristics help people accurately distinguish different types of ground features based on images. With the development of hyperspectral imaging technology, an increasing number of hyperspectral resolution images based on spaceborne or airborne platforms provide stable and abundant data sources for ground missions. However, the obtained hyperspectral data exhibits high-dimensional characteristics, especially with enhanced correlation between bands, non-linear spectra of individual pixels, and spectral variability caused by ground features under different environments. Therefore, deep learning-based hyperspectral classification methods are increasingly dominating classification tasks. Among them, convolutional neural networks have become reliable feature extraction networks for deep learning algorithms, and their specially designed network structures can simultaneously fit the spatial and spectral features of hyperspectral data.
[0003] However, convolutional neural networks focus excessively on the spatial information of images, causing distortion of the spectral characteristics of the original pixels during convolution operations, which is detrimental to refined classification tasks. To better fit the spectral features of hyperspectral data, current conventional solutions introduce the Transformer model from natural language processing into hyperspectral classification tasks. This is because the Transformer model has a global receptive field even in shallow networks. However, directly using a Transformer-based model in conventional solutions faces several drawbacks: First, due to the special operation methods of self-attention and multilayer perceptrons in the Transformer, the computational cost and the number of learnable parameters are enormous when fitting high-dimensional data, potentially far exceeding that of a convolutional neural network of the same scale. Second, the Transformer requires tokenizing the original image data into sequence information, which fragments the original spatial information, hindering the effective learning of joint spatial-spectral features. Third, due to the high-dimensionality of hyperspectral data, the Transformer architecture is more prone to the Hughes phenomenon when learning data features, while traditional data dimensionality reduction methods compromise the end-to-end characteristics of deep learning algorithms, making them unsuitable for future implementation in embedded or mobile devices. Summary of the Invention
[0004] In view of this, embodiments of the present invention aim to provide an end-to-end Transformer-based hyperspectral image classification method, apparatus, and medium; capable of achieving dimensionality reduction and classification of hyperspectral images in an end-to-end process; furthermore, significantly reducing the computational burden and inference time of deep learning methods based on the Transformer architecture; and compared with conventional convolutional architectures, exhibiting better robustness in the face of spectral variability problems, thus achieving refined classification of ground features.
[0005] The technical solution of this invention is implemented as follows:
[0006] In a first aspect, embodiments of the present invention provide an end-to-end Transformer-based hyperspectral image classification method, the method comprising:
[0007] For each of the multiple patch data obtained from the segmentation of the original hyperspectral image, generate the corresponding word group;
[0008] A hyperspectral image classification model is constructed based on Transformer; wherein the model includes at least multiple cascaded encoders and classifiers, and each encoder corresponds to a preservation law;
[0009] For all patch data obtained from the segmentation of the original hyperspectral image, the corresponding word groups are input into the hyperspectral image classification model in a preset batch. Backpropagation is performed based on the loss value output by the classifier to update the parameters in the hyperspectral image classification model until all patch data obtained from the segmentation of the original hyperspectral image have been input into the hyperspectral image classification model and the parameter learning has been completed.
[0010] The hyperspectral image to be classified is input into the trained hyperspectral image classification model in the form of patch data to obtain the classification result of the hyperspectral image to be classified.
[0011] Secondly, embodiments of the present invention provide an end-to-end Transformer-based hyperspectral image classification device, the device comprising: an image segmentation part, a construction part, a hyperspectral image classification model, a learning part, and an input part; wherein,
[0012] The image segmentation part is configured to generate a corresponding word group for each of the multiple patch data obtained by segmenting the original hyperspectral image.
[0013] The construction part is configured to build the hyperspectral image classification model based on Transformer; wherein the model includes at least multiple cascaded encoders and classifiers, and each encoder corresponds to a preservation law;
[0014] The learning part is configured to input the corresponding word groups into the hyperspectral image classification model in a preset batch for all patch data obtained by segmenting the original hyperspectral image, perform backpropagation calculation based on the loss value output by the classifier, and update the parameters in the hyperspectral image classification model until all patch data obtained by segmenting the original hyperspectral image have been input into the hyperspectral image classification model and the parameter learning has been completed.
[0015] The input section is configured to input the hyperspectral image to be classified as patch data into the trained hyperspectral image classification model to obtain the classification result of the hyperspectral image to be classified.
[0016] Thirdly, embodiments of the present invention provide a computer storage medium, characterized in that the computer storage medium stores an end-to-end Transformer-based hyperspectral image classification program, wherein the end-to-end Transformer-based hyperspectral image classification program, when executed by at least one processor, implements the steps of the end-to-end Transformer-based hyperspectral image classification method described in the first aspect.
[0017] This invention provides an end-to-end Transformer-based hyperspectral image classification method, apparatus, and medium. It achieves an end-to-end process by setting a hold-out rate for the encoder, and performs dimensionality reduction and classification on hyperspectral images. Furthermore, it significantly reduces the computational burden and inference time of deep learning methods based on the Transformer architecture. Moreover, compared to convolutional architectures, this method exhibits better robustness in the face of spectral variability, achieving refined classification of ground features. Attached Figure Description
[0018] Figure 1 A schematic diagram of an end-to-end Transformer-based hyperspectral image classification method provided in an embodiment of the present invention;
[0019] Figure 2 This is a schematic diagram of the process for obtaining spectral terms from patch data according to an embodiment of the present invention;
[0020] Figure 3 This is a schematic diagram of the components of a hyperspectral image classification model based on Transformer provided in an embodiment of the present invention;
[0021] Figure 4 This is a schematic diagram of the encoder composition provided in an embodiment of the present invention;
[0022] Figure 5 A schematic diagram of an end-to-end Transformer-based hyperspectral image classification device provided in an embodiment of the present invention;
[0023] Figure 6 This is a schematic diagram of the specific hardware structure of a computing device provided in an embodiment of the present invention. Detailed Implementation
[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0025] To address the shortcomings of current hyperspectral image classification schemes based on the Transformer architecture, this invention aims to solve these problems through the following means:
[0026] First, by tokenizing grouped tokens, each token contains information from adjacent bands, enabling the learning of richer spatial-spectral joint features;
[0027] Secondly, by utilizing the self-attention mechanism, effective and redundant features are selected based on attention scores, and redundant words are continuously eliminated proportionally, thus achieving end-to-end dimensionality reduction.
[0028] Next, word filtering is performed for each encoder, thereby gradually reducing the computational load of the encoder as the network depth increases;
[0029] Finally, an adaptive fusion mechanism is used to extract useful information from redundant symbols.
[0030] Based on this, see Figure 1 This illustrates an end-to-end Transformer-based hyperspectral image classification method provided by an embodiment of the present invention, the method comprising:
[0031] S101: For each of the multiple patch data obtained from the segmentation of the original hyperspectral image, generate the corresponding word group;
[0032] S102: Construct a hyperspectral image classification model based on Transformer; wherein the model includes at least multiple cascaded encoders and classifiers, and each encoder corresponds to a preservation law;
[0033] S103: For all patch data obtained by segmenting the original hyperspectral image, the corresponding word groups are input into the hyperspectral image classification model in a preset batch. Backpropagation calculation is performed based on the loss value output by the classifier to update the parameters in the hyperspectral image classification model until all patch data obtained by segmenting the original hyperspectral image are input into the hyperspectral image classification model and the parameter learning is completed.
[0034] S104: Input the hyperspectral image to be classified into patch data into the trained hyperspectral image classification model to obtain the classification result of the hyperspectral image to be classified.
[0035] pass Figure 1 The technical solution shown achieves an end-to-end process by setting a retention rate for the encoder, and performs dimensionality reduction and classification on hyperspectral images. Secondly, it significantly reduces the computational burden and inference time of deep learning methods based on the Transformer architecture. Moreover, compared with the convolutional architecture, this method has better robustness in the face of spectral variability problems, and achieves refined classification of ground features.
[0036] for Figure 1 In some possible implementations of the technical solution shown, generating a corresponding token group for each of the multiple patch data obtained from the segmentation of the original hyperspectral image includes:
[0037] The original hyperspectral image is segmented according to the set center pixel and side length to obtain multiple patch data;
[0038] For each patch of data, a feature sequence is obtained by splitting it based on the channel dimension;
[0039] Multiple groups of data are obtained by dividing the feature sequence into groups according to a set grouping size; wherein, there is at least one common feature between adjacent groups of data;
[0040] For each group of data, project it onto a preset dimension to obtain the spectral symbols corresponding to each group of data;
[0041] A classification term is added to each patch data, and together with the spectral terms corresponding to all groups of data, they form a term group corresponding to each patch data.
[0042] Specifically, the above implementation method can divide the original hyperspectral data into a series of patch data based on pixels carrying label information. For example, each patch data is a square patch of odd length centered on a pixel carrying label information. The size of the patch can be determined according to different data, and the label of its central pixel determines the label of the entire patch data. In some examples, such as... Figure 2 As shown, a patch of data can be represented as {X∈R}. w·h·nLet Y be the spatial dimension of the patch data, w·h, n be the number of spectral channels, and Y be the label value corresponding to its center pixel. In other words, the dimension of this patch data is w·h·n. Next, the patch data is split along the spectral channel dimension and flattened into a feature sequence. The dimension of this sequence then becomes wh·n. Furthermore, the feature sequence obtained from the splitting is padded according to the set grouping size g, for example... Figure 2 The gray-filled blocks in the data are divided into multiple groups, such as... Figure 2 As shown, g=3, and there is only one difference between adjacent groups of data. This ensures that each token includes the channel information of adjacent tokens, preventing the fragmentation of original information during serialization. This is beneficial for learning spatial-spectral joint features and also improves robustness. Next, a fully connected layer is used to project each group of data onto a set dimension d to obtain the spectral tokens corresponding to each group of data. In some examples, each spectral token corresponds to a positional encoding information to indicate its position in the patch data. Each patch data also has a corresponding classification token to record classification-related information during the calculation process, thus forming a token group corresponding to each patch data. In some examples, the classification token can also have encoding information set, and set to an encoding start value, such as 0.
[0043] against Figure 1 In some possible implementations of the technical solution shown, the construction of the hyperspectral image classification model based on Transformer includes:
[0044] Construct one or more cascaded encoders; wherein, for each encoder, the product of the number of input words and the retention rate is the number of output words for each encoder; each encoder includes a multi-head attention computation layer, a word filtering and fusion layer, and a multilayer perceptron;
[0045] During the construction of each encoder:
[0046] The multi-head attention layer is constructed to calculate and normalize the corresponding attention score for each input word.
[0047] The word filtering and fusion layer is constructed to extract the K words with the highest attention scores according to the number of output words corresponding to the encoder using the Top-K algorithm, and to generate supplementary words based on the remaining words with attention scores lower than the Kth word.
[0048] The multilayer perception layer is constructed to extract spatial-spectral joint features from the tokens output by the fusion layer based on the token selection, and the normalized features are used as the output tokens of the encoder.
[0049] A classifier is built to classify words based on the output of the last encoder, the classification result is output, and the loss value is calculated based on the classification result.
[0050] In some examples, the generation of supplementary tokens based on the remaining tokens with attention scores lower than the Kth token includes:
[0051] For the i-th encoder, based on the set hyperparameter b i Obtain the quantile q corresponding to the i-th encoder. i =sin(b) i ); where the hyperparameter b i The parameter is selected between 0 and 1;
[0052] From the remaining words whose attention scores are lower than the Kth word, select words whose attention scores are greater than the quantile corresponding to the i-th encoder;
[0053] The attention scores of the selected lexical characters are weighted and summed to obtain the supplementary lexical character x. fuse .
[0054] In some examples, the classifier includes two fully connected layers and a softmax layer;
[0055] The calculation of the loss value based on the classification result includes:
[0056] The loss value is calculated according to the following formula:
[0057]
[0058] Where M represents the number of categories, y ic p is a symbolic function ic Let N represent the probability that observed sample i belongs to category c, and let N represent the total number of observed samples.
[0059] Specifically, regarding the above implementation methods and examples, let's take... Figure 3 Taking the hyperspectral image classification model 300 based on Transformer as an example, the number of cascaded encoders is N, as shown below. Figure 3 As shown, encoders are labeled Encoder-1, Encoder-2, ..., Encoder-N. Each encoder corresponds to a retention rate, and each token is represented by a square block. That is, for each encoder, the product of its input token count and the retention rate is the output token count for that encoder. This gradually reduces the encoding process, achieving end-to-end dimensionality reduction. As the network depth increases, the computational load of the encoder is gradually reduced. Continuing... Figure 3 As shown, the output of the last encoder-N is connected to the input of the classifier.
[0060] against Figure 3 For any encoder in the model 300 shown, such as Figure 4 As shown, it includes: a multi-head attention computation layer, a word filtering and fusion layer, and a multilayer perceptron. In some examples, for the multi-head attention computation layer, ... Figure 4 The encoder shown is an example of a first-level encoder, whose input word groups are obtained through... Figure 2 The example shown obtains a token group generated from the original hyperspectral image, which includes multiple spectral tokens and one classification token. The encoding information of the classification token is set to the encoding start value, such as 0, so the encoding of the spectral tokens starts from 1. Taking the number of spectral tokens as n as an example, in the process of calculating the attention score for each token in the token group, the final calculation result will form a (n+1) order symmetric matrix. The first column of this matrix represents the correlation value obtained by calculating the attention score between the classification token and itself and other spectral tokens. In this embodiment of the invention, the elements of the first column of this symmetric matrix are used as the attention score corresponding to each token in the token group, and normalization processing is performed. In some examples, for the word selection and fusion layer, the top K words with the highest attention scores are first selected using the Top-K algorithm based on the attention scores of each word. Among the remaining words, there are also words that provide information helpful for classification. Therefore, words are selected according to the quantiles described in the previous examples and weighted summation is performed to generate new supplementary words. These supplementary words, along with the aforementioned top K words, serve as the words output by the encoder. This supplements the information needed for classification from redundant words, improving the classification progress. Since the attention score is the first column element in the aforementioned symmetric matrix, the attention score of the classification word must be the largest. Therefore, the classification word will definitely be one of the top K words with the highest attention scores. However, the classification word does not contain any image information and cannot provide sharing in the multilayer perception process. Therefore, the supplementary words obtained by weighted summation need to replace the classification word in subsequent image information processing calculations. Understandably, the word selection and fusion layer achieves the preservation rate for each encoder and reduces the data dimensionality. As the network depth increases, the computational load of the encoder is gradually reduced. In some examples, the tokens output from the token filtering fusion layer are passed through a multilayer perceptron to extract richer spatial-spectral joint features, and then normalized before entering the next level encoder.
[0061] against Figure 3 In the example of the classifier in model 300, backpropagation is performed based on the calculated loss value to update the learnable parameters in model 300, thereby completing a learning cycle.
[0062] Through the above implementation method and examples, not only is the composition structure of the hyperspectral image classification model built based on Transformer illustrated, but also the learning process of one round of the model is explained in detail. It can be understood that if one patch of data is input into the model 300 in batches, the learning and training process of the model 300 will be completed when all patch data is input.
[0063] In practical applications, the hyperspectral image to be classified can be input into the trained hyperspectral image classification model as patches, and the prediction results can be output, thereby realizing the end-to-end rapid and refined classification of land objects.
[0064] Based on the same inventive concept as the aforementioned technical solution, see [link to inventive concept]. Figure 5 This illustration shows an end-to-end Transformer-based hyperspectral image classification device 50 provided by an embodiment of the present invention. The device 50 includes: an image segmentation part 501, a construction part 502, a hyperspectral image classification model 503, a learning part 504, and an input part 505; wherein,
[0065] The image segmentation part 501 is configured to generate a corresponding word group for each of the multiple patch data obtained by segmenting the original hyperspectral image.
[0066] The construction part 502 is configured to construct the hyperspectral image classification model 503 based on Transformer; wherein the model includes at least a cascaded plurality of encoders and a classifier, and each encoder corresponds to a retention law;
[0067] The learning part 504 is configured to input the corresponding word groups into the hyperspectral image classification model 503 in a preset batch for all patch data obtained by segmenting the original hyperspectral image, perform backpropagation calculation based on the loss value output by the classifier, and update the parameters in the hyperspectral image classification model 503 until all patch data obtained by segmenting the original hyperspectral image have been input into the hyperspectral image classification model 503 and the parameter learning has been completed.
[0068] The input section 505 is configured to input the hyperspectral image to be classified as patch data into the trained hyperspectral image classification model 503 to obtain the classification result of the hyperspectral image to be classified.
[0069] In some examples, the image segmentation portion 501 is configured as follows:
[0070] The original hyperspectral image is segmented according to the set center pixel and side length to obtain multiple patch data;
[0071] For each patch of data, a feature sequence is obtained by splitting it based on the channel dimension;
[0072] Multiple groups of data are obtained by dividing the feature sequence into groups according to a set grouping size; wherein, there is at least one common feature between adjacent groups of data;
[0073] For each group of data, project it onto a preset dimension to obtain the spectral symbols corresponding to each group of data;
[0074] A classification term is added to each patch data, and together with the spectral terms corresponding to all groups of data, they form a term group corresponding to each patch data.
[0075] In some examples, the build section 502 is configured as follows:
[0076] Construct one or more cascaded encoders; wherein, for each encoder, the product of the number of input words and the retention rate is the number of output words for each encoder; each encoder includes a multi-head attention computation layer, a word filtering and fusion layer, and a multilayer perceptron;
[0077] During the construction of each encoder:
[0078] The multi-head attention layer is constructed to calculate and normalize the corresponding attention score for each input word.
[0079] The word filtering and fusion layer is constructed to extract the K words with the highest attention scores according to the number of output words corresponding to the encoder using the Top-K algorithm, and to generate supplementary words based on the remaining words with attention scores lower than the Kth word.
[0080] The multilayer perception layer is constructed to extract spatial-spectral joint features from the tokens output by the fusion layer based on the token selection, and the normalized features are used as the output tokens of the encoder.
[0081] A classifier is built to classify words based on the output of the last encoder, the classification result is output, and the loss value is calculated based on the classification result.
[0082] In some examples, the build section 502 is configured as follows:
[0083] For the i-th encoder, based on the set hyperparameter b i Obtain the quantile q corresponding to the i-th encoder. i =sin(b) i ); where the hyperparameter bi The parameter is selected between 0 and 1;
[0084] From the remaining words whose attention scores are lower than the Kth word, select words whose attention scores are greater than the quantile corresponding to the i-th encoder;
[0085] The attention scores of the selected lexical characters are weighted and summed to obtain the supplementary lexical character x. fuse .
[0086] In some examples, the classifier includes two fully connected layers and a softmax layer;
[0087] The construction part 502 is configured as follows:
[0088] The loss value is calculated according to the following formula:
[0089]
[0090] Where M represents the number of categories, y ic p is a symbolic function ic Let N represent the probability that observed sample i belongs to category c, and let N represent the total number of observed samples.
[0091] Understandably, in this embodiment, "part" can be a part of a circuit, a part of a processor, a part of a program or software, etc., or it can be a unit, a module, or a non-modular one.
[0092] Furthermore, in this embodiment, the components can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.
[0093] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0094] Therefore, this embodiment provides a computer storage medium storing an end-to-end Transformer-based hyperspectral image classification device program. When the end-to-end Transformer-based hyperspectral image classification device program is executed by at least one processor, it implements the steps of the end-to-end Transformer-based hyperspectral image classification device method described in the above technical solution.
[0095] Based on the aforementioned end-to-end Transformer-based hyperspectral image classification device 50 and computer storage medium, see [link to relevant documentation]. Figure 6 This illustration shows the specific hardware structure of a computing device 60 capable of implementing the aforementioned end-to-end Transformer-based hyperspectral image classification device 50, provided by an embodiment of the present invention. The computing device 60 can be a wireless device, mobile or cellular phone (including so-called smartphones), personal digital assistant (PDA), video game console (including video display, mobile video game device, mobile video conferencing unit), laptop computer, desktop computer, set-top box, tablet computing device, e-book reader, fixed or mobile media player, etc. The computing device 60 includes: a communication interface 601, a memory 602, and a processor 603; the various components are coupled together through a bus system 604. It is understood that the bus system 604 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general designates all buses as Bus System 604. Among them,
[0096] The communication interface 601 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;
[0097] The memory 602 is used to store computer programs that can run on the processor 603;
[0098] The processor 603 is configured to execute, when running the computer program,
[0099] The steps of the end-to-end Transformer-based hyperspectral image classification method described in the aforementioned technical solution will not be repeated here.
[0100] It is understood that the memory 602 in this embodiment of the invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 602 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0101] The processor 603 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 603 or by instructions in software form. The processor 603 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 602, and the processor 603 reads the information in memory 602 and, in conjunction with its hardware, completes the steps of the above method.
[0102] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.
[0103] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.
[0104] It is understood that the exemplary technical solutions of the end-to-end Transformer-based hyperspectral image classification device 50 and computing device 60 described above belong to the same concept as the technical solutions of the aforementioned end-to-end Transformer-based hyperspectral image classification method. Therefore, all details not described in detail above regarding the technical solutions of the end-to-end Transformer-based hyperspectral image classification device 50 and computing device 60 can be found in the description of the technical solutions of the aforementioned end-to-end Transformer-based hyperspectral image classification method. This embodiment of the invention will not elaborate further on these details.
[0105] It should be noted that the technical solutions described in the embodiments of the present invention can be combined arbitrarily without conflict.
[0106] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. An end-to-end Transformer-based hyperspectral image classification method, characterized in that, The method comprises: For each of the multiple patch data obtained by segmenting the original hyperspectral image, a corresponding word symbol group is generated; A hyperspectral image classification model is constructed based on Transformer. The model includes at least multiple cascaded encoders and a classifier, with each encoder corresponding to a retention rate. Each encoder includes a multi-head attention calculation layer, a word selection and fusion layer, and a multilayer perceptron. During the construction of each encoder: the multi-head attention calculation layer is constructed to calculate and normalize the corresponding attention score for each input word; the word selection and fusion layer is constructed to extract the K words with the highest attention scores according to the number of output words corresponding to the encoder, using the Top-K algorithm, and generate supplementary words based on the remaining words with attention scores lower than the Kth word; wherein, generating supplementary words based on the remaining words with attention scores lower than the Kth word includes: for the i-th encoder, according to the set hyperparameters... b i Obtain the first i The quantile corresponding to each encoder q i =sin( b i ); whereby the hyperparameters b i The parameters are selected between 0 and 1; from the remaining words whose attention scores are lower than the Kth word, words with attention scores greater than the quantile corresponding to the i-th encoder are selected; the attention scores of the selected words are weighted and summed to obtain the supplementary words. x fuse The multilayer perceptron is constructed to extract spatial-spectral joint features from the tokens output by the fusion layer based on the token selection, and the normalized features are used as the output tokens of the encoder; a classifier is constructed to classify the tokens based on the output tokens of the last encoder, output the classification results, and calculate the loss value based on the classification results. For all patch data obtained by segmenting the original hyperspectral image, the corresponding word symbol groups are input into the hyperspectral image classification model in a preset batch, back propagation calculation is performed according to the loss value output by the classifier, the parameters in the hyperspectral image classification model are updated, and the process is repeated until all patch data obtained by segmenting the original hyperspectral image are input into the hyperspectral image classification model and the parameter learning is completed; The hyperspectral image to be classified is input into the trained hyperspectral image classification model in the form of patch data, and the classification result of the hyperspectral image to be classified is obtained.
2. The method of claim 1, wherein, For each of the multiple patch data obtained by segmenting the original hyperspectral image, a corresponding word symbol group is generated, comprising: The original hyperspectral image is segmented according to a set center pixel point and a side length to obtain multiple patch data; For each patch data, a feature sequence is obtained based on channel dimension splitting; According to the feature sequence, multiple group data are obtained by dividing according to a set grouping size; wherein there is at least one same feature between adjacent group data; For each group data, a spectral word symbol corresponding to each group data is obtained by projecting to a preset dimension; For each patch data, a classification word symbol is added, and the spectral word symbol corresponding to all group data constitutes a word symbol group corresponding to each patch data.
3. The method of claim 1, wherein, The classifier comprises two fully connected layers and a softmax layer; The loss value is calculated according to the classification result, comprising: The loss value is calculated according to the following formula: wherein, M denotes the number of classification classes, y ic is the indicator function, p ic denotes the observation sample i belongs to the classification class c with probability, N denotes the total number of observation samples.
4. An end-to-end Transformer-based hyperspectral image classification apparatus, characterized in that, The device comprises an image segmentation part, a construction part, a hyperspectral image classification model, a learning part, and an input part; wherein, The image segmentation part is configured to generate a corresponding word symbol group for each of the multiple patch data obtained by segmenting the original hyperspectral image; The construction part is configured to construct the hyperspectral image classification model based on a Transformer; wherein the model at least includes a plurality of cascaded encoders and a classifier, and each encoder corresponds to a retention rate; wherein each encoder includes a multi-head attention calculation layer, a word symbol screening and fusion layer, and a multi-layer perceptron; in the process of constructing each encoder: the multi-head attention calculation layer is constructed to calculate a corresponding attention score for each input word symbol and normalize it; the word symbol screening and fusion layer is constructed to extract the K word symbols with the highest attention scores according to the number of output word symbols corresponding to the encoder by using a Top-K algorithm, and generate supplementary word symbols according to the remaining word symbols with attention scores lower than the Kth word symbol; wherein the generation of the supplementary word symbols according to the remaining word symbols with attention scores lower than the Kth word symbol includes: for the ith encoder, generating the supplementary word symbols according to a set hyperparameter b i , wherein the hyperparameter i q i = sin( b i ); wherein the hyperparameter b i is a parameter selected between 0 and 1; screening the word symbols with attention scores greater than the quantile number corresponding to the ith encoder from the remaining word symbols with attention scores lower than the Kth word symbol; performing weighted summation on the attention scores of the screened word symbols to obtain the supplementary word symbols x fuse ; the multi-layer perceptron is constructed to extract spatial-spectral joint features according to the word symbols output by the word symbol screening and fusion layer, and normalize them as the output word symbols of the encoder; the classifier is constructed to classify according to the output word symbols of the last encoder, output a classification result, and calculate a loss value according to the classification result; The learning part is configured to input the corresponding word symbol groups into the hyperspectral image classification model in a preset batch for all patch data obtained by segmenting the original hyperspectral image, perform back propagation calculation according to the loss value output by the classifier, update the parameters in the hyperspectral image classification model, and repeat the process until all patch data obtained by segmenting the original hyperspectral image are input into the hyperspectral image classification model and the parameter learning is completed; The input part is configured to input the hyperspectral image to be classified into the trained hyperspectral image classification model in the form of patch data, and obtain the classification result of the hyperspectral image to be classified.
5. The apparatus of claim 4, wherein, The image segmentation part is configured to: The original hyperspectral image is segmented according to a set center pixel point and a side length to obtain multiple patch data; For each patch data, a feature sequence is obtained based on channel dimension splitting; According to the feature sequence, multiple group data are obtained by dividing according to a set grouping size; wherein there is at least one same feature between adjacent group data; For each group data, a spectral word symbol corresponding to each group data is obtained by projecting to a preset dimension; A classification word symbol is added for each of the patch data, and a corresponding word symbol group of each patch data is formed with all the spectral word symbols corresponding to the group data.
6. A computer storage medium, characterized in that The computer storage medium stores an end-to-end Transformer-based hyperspectral image classification program, and the end-to-end Transformer-based hyperspectral image classification program, when executed by at least one processor, implements the steps of the end-to-end Transformer-based hyperspectral image classification method in any one of claims 1 to 3.