A hyperspectral image classification method based on hybrid Fourier operator Transformer network

Abstract

A hyperspectral image classification method based on a hybrid Fourier operator Transformer network, which divides a hyperspectral image data set into a training set and a test set; establishes a hybrid Fourier operator Transformer network; uses the training set to train to obtain a trained classification model, and uses the test set to test its classification performance; and uses the classification model that meets the classification performance to classify the hyperspectral image. The Fourier operator Transformer network of the present invention includes a grouped spectral embedding module, a hybrid Fourier operator Transformer encoder network, and a cross-layer feature fusion module; it can learn spatial spectral sequence information from adjacent bands of hyperspectral images, reduce the computational complexity of the network, and reduce the possibility of losing valuable information in the layer-by-layer propagation process. Experimental results show that the present invention is superior to the detection performance and computational efficiency of convolutional neural networks, and meets the needs of fine classification of hyperspectral images.

Description

Technical Field

[0001] The invention belongs to the technical field of hyperspectral remote sensing images, and in particular relates to a hyperspectral image classification method based on a hybrid Fourier operator Transformer network. Background Art

[0002] Hyperspectral images are characterized by approximately continuous spectral information, which can achieve fine identification of materials by capturing subtle spectral differences. Due to its excellent local context modeling ability, convolutional neural networks (CNNs) have been proven to be powerful feature extractors in image classification. However, CNNs fail to mine and represent the sequence properties of spectral features well due to the inherent limitations of their network backbone. In addition, dimensionality reduction methods need to be added to the commonly used CNN preprocessing, while linear dimensionality reduction methods will lose some spectral information. To solve these problems, existing researchers have reconsidered hyperspectral image classification from a sequence perspective and introduced a cutting-edge backbone network, Transformer, into hyperspectral classification. However, the self-attention calculation in Transformer has a quadratic complexity problem, and the computational efficiency is low when processing hyperspectral images with high resolution or more spectral channels. The performance of Transformer in hyperspectral image classification needs to be improved. Summary of the invention

[0003] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a hyperspectral image classification method based on a hybrid Fourier operator Transformer network, so as to solve the problems of low computational efficiency and low classification accuracy of deep learning methods in processing high-resolution hyperspectral images in the prior art.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A hyperspectral image classification method based on a hybrid Fourier operator Transformer network comprises the following steps:

[0006] Step 1), obtain a hyperspectral image dataset and divide it into a training set and a test set;

[0007] Step 2), establish a hybrid Fourier operator Transformer network;

[0008] The hybrid Fourier operator Transformer network includes a grouped spectral embedding module, a hybrid Fourier operator Transformer encoder network and a cross-layer feature fusion module; the grouped spectral embedding module overlaps and groups the hyperspectral image to be classified according to a set number of adjacent spectral bands, and performs feature embedding to generate a spectral feature sequence; the hybrid Fourier operator Transformer encoder network takes the spectral feature sequence as input, and includes a total of four layers, namely encoder one, encoder two, encoder three and encoder four, wherein encoder one and encoder two are standard Transformer encoders, and encoder three and encoder four are Fourier operator Transformer encoders; the cross-layer feature fusion module fuses the output of encoder one with the output of encoder three as the input of encoder four, fuses the output of encoder two with the output of encoder four, and passes through a fully connected layer as the output of the hybrid Fourier operator Transformer network;

[0009] Step 3), using the training set to train the hybrid Fourier operator Transformer network to obtain a trained weight model, and using the test set to test its classification performance;

[0010] Step 4), classify the hyperspectral image using a weight model that meets the classification performance.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: when the present invention classifies hyperspectral images, the hybrid Fourier operator Transformer network used includes three parts: a grouped spectral embedding module, a hybrid Fourier operator Transformer encoder network, and a cross-layer feature fusion module; the grouped spectral embedding module and the cross-layer feature fusion module respectively enhance the model's ability to mine spectral adjacent information and more effectively convey memory information across layers; the Fourier operator Transformer encoder with linear computational complexity improves the computational efficiency of high-resolution hyperspectral images; the classification model proposed in the present invention does not require dimensionality reduction of the spectrum, is more flexible and efficient, and the classification results are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is the overall framework of the hyperspectral classification method based on the hybrid Fourier Transformer network.

[0013] Figure 2 The standard Transformer encoder structure used in this invention

[0014] Figure 3 This is the Fourier operator Transformer encoder structure of the present invention.

[0015] Figure 4 It is a schematic diagram of the cross-layer feature fusion module of the present invention fusing the output of encoder one with the output of encoder three. DETAILED DESCRIPTION

[0016] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0017] Based on the self-attention mechanism and Fourier operator, this paper designs a hyperspectral image classification method that can more effectively mine fine spectral information. The overall idea is: the hyperspectral image containing spatial spectral information is serialized through the grouped spectral embedding module, and then input into the hybrid Fourier operator Transformer encoder network. The network has four layers in total, the first two layers are standard Transformer encoders, and the last two layers are Fourier operator Transformer encoders. The cross-layer fusion module is used between the network layers to improve the information fusion and utilization capabilities.

[0018] like Figure 1 As shown, the present invention comprises the following steps:

[0019] Step 1), obtain the hyperspectral image dataset and divide it into training set and test set, and the dataset has annotation results.

[0020] Step 2), establish a hybrid Fourier operator Transformer network.

[0021] Step 3), use the training set to train the hybrid Fourier operator Transformer network to obtain the trained weight model, and use the test set to test its classification performance.

[0022] Step 4), classify the hyperspectral image using a weight model that meets the classification performance.

[0023] The hybrid Fourier operator Transformer network of the present invention comprises a grouped spectrum embedding module, a hybrid Fourier operator Transformer encoder network and a cross-layer feature fusion module.

[0024] The grouped spectral embedding module groups the hyperspectral images to be classified into overlapping groups according to a set number of adjacent spectral bands, and performs feature embedding to generate a spectral feature sequence. Based on this module, spatial spectral sequence information can be learned from adjacent bands of hyperspectral images.

[0025] The hybrid Fourier operator Transformer encoder network takes the spectral feature sequence as input, and includes four layers, namely encoder 1, encoder 2, encoder 3 and encoder 4, wherein encoder 1 and encoder 2 are standard Transformer encoders, and encoder 3 and encoder 4 are Fourier operator Transformer encoders. Based on this network, the computational complexity of the network can be reduced.

[0026] The cross-layer feature fusion module fuses the output of encoder 1 with the output of encoder 3 as the input of encoder 4, and fuses the output of encoder 2 with the output of encoder 4, and passes through a fully connected layer as the output of the hybrid Fourier operator Transformer network. Based on this module, the possibility of losing valuable information in the layer-by-layer propagation process can be reduced.

[0027] Specifically, through the grouped spectrum embedding module, the hyperspectral image can be overlapped and grouped according to adjacent bands, and then converted into feature vectors to finally obtain a spectral feature sequence. The serialization of ordinary hyperspectral image transformers is to serialize a single band as a token input, while the overlapping grouping operation of the present invention integrates several bands into a token input.

[0028] In the present invention, the grouped spectrum embedding module learns feature embedding from adjacent bands, and its feature embedding matrix A is expressed as follows:

[0029] A=WX=Wg(x)

[0030] x is the pixel of the hyperspectral image to be classified, x=[x1,x2,...,x m ]∈R 1×m , x m represents the pixel of the mth band, m is the number of bands, the pixel is the intensity value of all bands at the (x, y) spatial point, the dimension is 1*m, and the dimension of A is d*m; the function g(·) represents the overlapping grouping operation; X∈R n×m Corresponding to the grouped representation of x, n represents the number of nearest neighbor bands, W is a variable, which is a matrix, which can be simply regarded as a layer of the input layer of the hybrid Fourier operator Transformer network, and can be optimized by updating the hybrid Fourier operator Transformer network. W updates x to a feature embedding matrix A through overlapping grouping operations;

[0031] In the present invention, the number of adjacent spectral bands is set to an odd number greater than 1, for example, 3. The feature vector obtained after overlapping grouping is added to the position code, and the code is combined with the feature vector to form a token, that is, a spectral feature sequence y is generated, y = [y1, y2, ..., y m], the length of y is m, y1,y2,...,y m A serialized feature vector representing each band of a hyperspectral image.

[0032] Through the grouped spectral embedding module, the hyperspectral images of a set number of adjacent spectral bands are flattened into spatial spectral feature vectors, which largely retains the spectral sequence information in network learning while considering the spatial context information to input the hybrid Fourier operator Transformer encoder network.

[0033] The following focuses on introducing the standard Transformer encoder and the Fourier operator Transformer encoder in the method of the present invention.

[0034] like Figure 2 As shown in the figure, the standard Transformer encoder structure consists of two main modules, a multi-head attention module and a fully connected layer (MLP). The core of the standard Transformer encoder is the multi-head attention mechanism, in which LayerNorm and Dropout layers can be set to help the network converge better and prevent network overfitting. Among them, the input of the fully connected layer is the output of the multi-head attention module.

[0035] The specific calculation steps of the multi-head attention module are as follows:

[0036] (1) Input spectral feature sequence y.

[0037] (2) For each element y in y i , feature embedding is obtained through variable W, expressed as a i , all a i The combination is the feature embedding matrix A.

[0038] (3) Each feature is embedded in a i Multiply by three different transformation matrices W respectively q , W k and W v , we get Q vector, K vector and K vector, that is, Q=[q1,...,q m ], K=[k1,...,k m ] and V=[v1,...,v m ].

[0039] (4) Calculate the attention score s between each Q vector and each K vector in the form of inner product, for example, q i ·k j , in order to stabilize the gradient, the attention score needs to be normalized and scaled, that is where d and q i or k j is related to the dimension.

[0040] (5) Calculate the softmax activation function on the attention score s, for example,

[0041] (6) Generate attention representation z = [z1, ..., z m ], for example, z = ∑ i s i v i .

[0042] The self-attention calculation method can be generally expressed as:

[0043]

[0044] like Figure 3 As shown in the figure, the Fourier operator Transformer encoder structure contains two main modules, the Fourier operator and a fully connected layer (MLP). The core of the Fourier operator Transformer encoder is the discrete Fourier operator. The LayerNorm and Dropout layers consistent with the standard Transformer encoder can be set, and the input of the fully connected layer is the output of the Fourier operator.

[0045] The representation of the Fourier operator is as follows:

[0046]

[0047] In the formula, r n represents the input of the Fourier operator Transformer encoder, {r n} represents the spectral feature sequence, n∈[0,N-1], n represents the number of spectral bands, and k represents the frequency domain sampling interval after Fourier transform. For each k, the discrete Fourier transform generates a new representation R k , R k is all input r n The sum of; Since the multiplication in the frequency domain is equal to the convolution in the time domain, the Fourier operator can be regarded as a large kernel convolution with linear computational complexity. It is a single-layer network without bias and activation function, and has a specific form of weights.

[0048] The cross-layer feature fusion module of the present invention realizes fusion through residual connection. Specifically, the outputs from the standard Transformer encoder and the Fourier operator Transformer encoder are fused in the channel dimension, and a 1×2 convolution kernel is used for feature extraction. Figure 4 , taking the first part of the fusion, that is, the fusion of the output of encoder one and the output of encoder three as an example, the fusion formula is as follows:

[0049]

[0050] In the formula, represents the fused output, i.e., the input of encoder 4 of the present invention, z (l) represents the output of encoder three, z (l-2) represents the output of encoder 1, Represents a 1*2 convolution kernel.

[0051] Accordingly, the second part of the fusion, that is, the fusion of the output of encoder 2 and the output of encoder 4, is taken as an example. The fusion formula is as follows:

[0052]

[0053] In the formula, represents the fused output, i.e., the output of encoder 4 of the present invention, z (l-1) Represents the output of encoder two.

[0054] In the present invention, steps 1), 2), and 3) are for training the network model. It is not necessary to repeat steps 1), 2), and 3) every time a hyperspectral image is classified. After obtaining the trained model, only step 4) needs to be executed.

[0055] In the present invention, there is no need to perform dimensionality reduction processing on the hyperspectral image, and all bands of the original image are used as network input.

[0056] In a specific embodiment of the present invention, it is assumed that the target hyperspectral image data is the commonly used Houston 2013, which has a spatial dimension of 349×1905, 144 bands, and a spectral range of 364-1046 nm, including 15 types of classified objects and 15029 labeled pixels to be classified.

[0057] The classification method randomly selects about 190 samples from each category of the data set as training samples, and the rest as test samples to train the hybrid Fourier operator Transformer network. The training process is as follows: the training samples are data cubes describing the characteristics of the hyperspectral image. They are input into the classification network in batches. The network will extract the data features and output them. The loss function is calculated by the difference between the network output and the true value label, and the network parameters are updated through the back propagation algorithm to minimize the loss function, so that the network output is close to the true value label. Specifically, it includes the attention matrix parameters and the fully connected layer parameters in the Transformer encoder. The entire network is iterated 600 times, and one iteration traverses the entire data set once to obtain a trained classification model. The test process is as follows: after the training is completed, the parameters are fixed, the test sample is input into the network to obtain the predicted value, the predicted value is compared with the known test sample category label, the classification accuracy is calculated, and the classification model is evaluated according to the classification accuracy.

[0058] When in use, an unlabeled hyperspectral image to be classified is input into the classification model, and the classification result is output.

[0059] The effectiveness of the present invention is verified by simulation experiments. The hardware equipment of the simulation experiment is as follows: processor Intel Xeon Gold 6226R CPU, memory 64-GB RAM, NVIDIA RTX 2080Ti 11-GB GPU.

[0060] About 190 samples were randomly selected from each category of the data set as training samples, and the rest were used as test samples to train the hybrid Fourier operator Transformer hyperspectral image classification network. Table 1 shows the classification accuracy of different classification methods for the Houston 2013 hyperspectral data set. Among them, 1D-CNN represents a one-dimensional convolutional neural network, RNN represents a recurrent neural network, MSPN represents a multi-scale three-dimensional and two-dimensional hybrid convolutional neural network, and SpectralFormer represents an ordinary Transformer network. OA, Kappa and AA represent the overall accuracy, consistency coefficient and average accuracy of the classification method, respectively. It can be seen from Tables 1 and 2 that compared with the existing hyperspectral image classification methods, the method proposed in the present invention has higher accuracy and faster calculation speed.

[0061] Table 1

[0062] Classification method 1D-CNN RNN MSPN SpectralFormer The present invention OA / % 78.99 82.26 84.43 86.03 87.17 Kappa / % 77.17 80.74 83.11 84.84 86.08 AA / % 81.50 83.81 86.41 87.51 88.53

[0063] Table 2

[0064] Training time / s Test time / s MSPN 4443.4 1667.3 SpectralFormer 971.3 55.2 The present invention 753.1 48.6

[0065] In summary, from the perspective of the spectrum, the present invention designs a hybrid Fourier coding network based on Transformer in view of the inherent limitation that CNN is only good at extracting local information and is not good at mining the sequence information of the spectrum. The network backbone uses a mixture of standard Transformer encoders and Fourier operator Transformer encoders that reduce computational complexity. In addition, it also includes a grouped spectrum embedding module that can integrate the information of spectral neighboring bands, and a cross-layer feature fusion module that transmits and fuses information. The proposed classification method breaks through the limitations of CNN and demonstrates superior performance and computing speed that surpasses models such as CNN and RNN without using dimensionality reduction methods.

Claims

1. A hyperspectral image classification method based on a hybrid Fourier operator Transformer network, characterized in that: The following steps are involved: Step 1), obtain a hyperspectral image dataset and divide it into a training set and a test set; Step 2), establish a hybrid Fourier operator Transformer network; The hybrid Fourier operator Transformer network includes a grouped spectral embedding module, a hybrid Fourier operator Transformer encoder network and a cross-layer feature fusion module; The grouped spectral embedding module overlaps and groups the hyperspectral image to be classified according to a set number of adjacent spectral bands, and performs feature embedding to generate a spectral feature sequence; the hybrid Fourier operator Transformer encoder network takes the spectral feature sequence as input, and includes a total of four layers, namely encoder one, encoder two, encoder three and encoder four, wherein encoder one and encoder two are standard Transformer encoders, and encoder three and encoder four are Fourier operator Transformer encoders; the cross-layer feature fusion module fuses the output of encoder one with the output of encoder three as the input of encoder four, fuses the output of encoder two with the output of encoder four, and passes through a fully connected layer as the output of the hybrid Fourier operator Transformer network; The feature embedding matrix A of the grouped spectrum embedding module is expressed as follows: A=WX=Wg(x) x is the pixel of the hyperspectral image to be classified, x=[x1,x2,...,x m ]∈R 1×m , x m represents the pixel of the mth band, where m is the number of bands; the function g(·) represents the overlapping grouping operation; X∈R n×m Corresponding to the grouping representation of x, n represents the number of bands of the nearest neighbors, and W is a variable; The set number is an odd number greater than 1. The feature vector obtained after overlapping grouping is added to the position code, and the code and the feature vector are combined into a token to generate a spectral feature sequence y, y = [y1, y2, ..., y m ], the length of y is m, y1,y2,...,y m Represents the serialized feature vector of each band of the hyperspectral image; The core of the standard Transformer encoder is the multi-head attention mechanism, and the core of the Fourier operator Transformer encoder is the discrete Fourier operator; The Fourier operator Transformer encoder includes a Fourier operator and a fully connected layer, and the input of the fully connected layer is the output of the Fourier operator; The Fourier operator is expressed as follows: In the formula, r n represents the input of the Fourier operator Transformer encoder, {rn} represents the spectral feature sequence, n∈[0,N-1], n represents the number of spectral bands, k represents the frequency domain sampling interval after Fourier transform, N is the period of the periodic sequence in Fourier transform, for each k, the discrete Fourier transform generates a new representation R k , R k is all input r n The sum of In the cross-layer feature fusion module, fusion is achieved through residual connection; Step 3), using the training set to train the hybrid Fourier operator Transformer network to obtain a trained classification model, and using the test set to test its classification performance; Step 4), classify the hyperspectral image using a classification model that meets the classification performance.

2. The hyperspectral image classification method based on hybrid Fourier operator Transformer network according to claim 1, characterized in that: The standard Transformer encoder includes a multi-head attention module and a fully connected layer, and the input of the fully connected layer is the output of the multi-head attention module.

3. The hyperspectral image classification method based on hybrid Fourier operator Transformer network according to claim 1, characterized in that: In the cross-layer feature fusion module, the outputs are fused in the channel dimension, and a 1×2 convolution kernel is used for feature extraction.

4. The hyperspectral image classification method based on hybrid Fourier operator Transformer network according to claim 1, characterized in that: There is no need to perform dimensionality reduction on the hyperspectral image, and all bands of the original image are used as network input.

Images