Hyperspectral image classification method based on Transform parallel GRU algorithm

Abstract

The invention aims to provide a hyperspectral image classification method based on a Transform parallel GRU algorithm, and the method comprises the following steps: A, constructing a deep neural network model which comprises an improved Spectral Former framework and a GRU extraction time sequence feature network; b, training the improved deep neural network model to obtain a trained deep neural network model; c, combining the basic wave band data in the hyperspectral remote sensing image data to be measured with the SWIR wave band data, and then sequentially nesting the basic wave band data and the SWIR wave band data in pairs to obtain a two-dimensional data set; d, the two-dimensional data set is processed through an improved SpectraFormer frame, a time sequence feature network is extracted through GRU, and two branch processing results are processed through tensor flattening; and E, adding and fusing the processing results of the two branches, and performing 1 * 1 convolution to obtain a final output result. According to the method, the problem of sample imbalance can be better solved, and high-performance pixel-level hyperspectral image classification can be better realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a hyperspectral image classification method based on a Transformer parallel GRU algorithm. Background Art

[0002] Spectral data information can be used for object recognition and classification. Object recognition and classification based on remote sensing hyperspectral images rely on image classification. With the rapid advancement of aerospace technology and sensor technology, researchers can easily obtain a large number of high-quality hyperspectral remote sensing images, which reflect the status of the ecological environment and signs of human activities. Effectively learning and extracting the information contained in these images has become a key task for intelligent interpretation of remote sensing images. Semantic segmentation has attracted much attention as a feasible solution, and its core goal is to identify the semantic category of each pixel in the image. Remote sensing image semantic segmentation has been widely used in multiple practical scenarios, such as urban planning, disaster assessment, and agricultural production. In hyperspectral image classification, although traditional convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have made some progress, they still have some limitations. CNN performs well in spatial feature extraction, but when processing hyperspectral images, it is difficult to effectively capture the temporal features in the spectral dimension. RNN has advantages in processing temporal data, but its computational complexity is high and it is difficult to parallelize, which limits its application in hyperspectral image classification. In order to solve these problems, the present invention proposes a hyperspectral image classification method based on Transformer parallel GRU algorithm. This method achieves efficient classification of hyperspectral images by combining an improved Transformer framework with a GRU network for extracting temporal features. Specifically, the improved Transformer framework can process hyperspectral data in parallel and reduce computational complexity, while the GRU network can effectively extract temporal features in the spectral dimension and enhance the performance of the classification model. By fusing the processing results of these two models, the present invention can better achieve high-performance pixel-level hyperspectral image classification.

[0003] In summary, land object identification and classification technology based on hyperspectral data has higher accuracy and reliability, and provides more useful surface information, which makes it an important tool for modern land management and environmental monitoring, and further research is necessary. Summary of the invention

[0004] The present invention aims to provide a hyperspectral image classification method based on the Transformer parallel GRU algorithm, which can better handle the sample imbalance problem and improve the recognition ability of rare categories. The method has the advantages of efficient parallel processing, multi-level feature fusion, adaptability to high-dimensional data, noise robustness, sample imbalance processing and high-performance classification, and can better achieve high-performance pixel-level hyperspectral image classification.

[0005] The technical solution of the present invention is as follows:

[0006] The hyperspectral image classification method based on the Transformer parallel GRU algorithm comprises the following steps:

[0007] Among them, Wz and Uz are the weight matrices from input and hidden state to update gate, respectively, and b Z is the bias term.

[0008] The hidden state The calculation formula is:

[0009]

[0010] Among them, tan h is the hyperbolic tangent activation function, w h and u h are the weight matrices from the hidden state adjusted by the input and reset gates to the candidate hidden state, and b h is the bias term, and ⊙ represents element-wise multiplication.

[0011] The beneficial effects of the present invention are as follows:

[0012] The present invention is good at capturing local temporal features through the GRU network, and mainly focuses on global dependencies by using the improved Transformer framework, combining them to give full play to the strengths of both, further improving the accuracy of image processing, and obtaining better feature extraction capabilities.

[0013] The Transformer framework of the present invention can process sequence data in parallel through the self-attention mechanism, significantly reducing the computational complexity. The GRU (Gated Recurrent Unit) network has advantages in processing time series data and can effectively capture local time series features in the spectral dimension. Through parallel processing, the GRU network can accelerate the feature extraction process and improve classification efficiency.

[0014] The present invention combines the advantages of both the Transformer network and the GRU framework, so the model can achieve high-performance hyperspectral image classification tasks under the same accuracy compared to using the GRU network or the Transformer framework alone.

[0015] Based on the problem that deep learning methods are difficult to handle non-two-dimensional structured data, this paper proposes a new hyperspectral image pixel-level classification network framework, which has the advantages of efficient parallel processing, multi-level feature fusion, adaptability to high-dimensional data, noise robustness, sample imbalance processing and high-performance classification, and can better realize high-performance pixel-level hyperspectral image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the structure of the deep neural network model of Example 1;

[0017] Figure 2 This is a test comparison diagram obtained in Example 2 on the WHU-Hi-HanChuan dataset (the comparison algorithm experiments are as follows: a is the original image, b is the gt label map, c is AB-LSTM, d is 3D-CNN, e is SpectralFormer, f is SSFTT, g is M3D-DCNN, h is DFFN, i is RSSAN, and j is the method of Example 1);

[0018] Figure 3 This is a test comparison chart obtained in Example 2 on the WHU-Hi-HongHu dataset (the comparison algorithm experiments are as follows: a is the original image, b is the gt label map, c is AB-LSTM, d is 3D-CNN, e is SpectralFormer, f is SSFTT, g is M3D-DCNN, h is DFFN, i is RSSAN, and j is the method of Example 1). DETAILED DESCRIPTION

[0019] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0020] Example 1

[0021] The hyperspectral image classification method based on the Transformer parallel GRU algorithm includes the following steps:

[0022] A. Build a deep neural network model, which includes: an improved SpectralFormer framework and a GRU network for extracting temporal features;

[0023] B. Train the improved deep neural network model to obtain a trained deep neural network model;

[0024] C. Combine the R, G, B, and NiR band data of Sentinel 2 in the hyperspectral remote sensing image data to be tested with the SWIR band data, and then nest them in pairs to obtain a two-dimensional data set;

[0025] D. Divide the two-dimensional data group into two branches. The first branch is input into the improved SpectralFormer framework to obtain the processing result of the first branch. The second branch is processed by GRU to extract the temporal feature network and tensor flattening to obtain the processing result of the second branch.

[0026] The processing process in the Transformer framework is as follows:

[0027] Each two-dimensional data in the input two-dimensional data group is flattened into a row vector, and then processed to the set dimension through a fully connected layer to obtain a mapped two-dimensional data group; a vector of the set dimension is generated based on the Transformer framework, and it is concatenated with each two-dimensional data in the mapped two-dimensional data group to obtain a concatenated two-dimensional data group, which is input into the Transformer Encoder module for processing to obtain the first branch processing result.

[0028] The processing process in the GRU temporal feature extraction network is as follows:

[0029] Initialize the hidden state, calculate the reset gate and update gate for each time step, adjust the hidden state of the previous moment based on the reset gate and calculate the candidate hidden state, perform a weighted combination of the hidden state of the previous moment and the candidate hidden state through the update gate, obtain the final hidden state at the current moment and output it. The output result is the temporal characteristics and spatial dependencies in the hyperspectral remote sensing image.

[0030] The reset gate r t Determine which information in the previous hidden state should be ignored. The calculation formula is as follows:

[0031] r t =σ(w r x t +u r h t -1+b r ) (1)

[0032] Where σ is the sigmoid activation function, Wr and Ur are the weight matrices from the input and hidden state to the reset gate, br is the bias term, and x t is the input at the current moment, h t-1 is the hidden state at the previous moment.

[0033] The update gate z t Determine how much information in the hidden state of the previous moment should be retained and passed to the current moment. The calculation formula is as follows:

[0034] z t =σ(w z x t +uz h t-1 +b z ) (2)

[0035] Among them, Wz and Uz are the weight matrices from input and hidden state to update gate, respectively, and b Z is the bias term.

[0036] The hidden state The calculation formula is:

[0037]

[0038] Among them, tan h is the hyperbolic tangent activation function, w h and u h are the weight matrices from the hidden state adjusted by the input and reset gates to the candidate hidden state, and b h is the bias term, and ⊙ represents element-wise multiplication.

[0039] E. After adding and fusing the results of the first two branches and the processing results of the second branch, a 1*1 convolution is performed to obtain the final output result.

[0040] Example 2

[0041] Selection of satellite remote sensing training samples in remote sensing hyperspectral image database WHU-Hi-HanChuan: The WHU-Hi-HanChuan dataset was collected in Hanchuan City, Hubei Province, China from 17:57 to 18:46 on June 17, 2016. A 17mm focal length headwall Nano-Hyperspec imaging sensor was mounted on the Leica AibotX6 drone V1 platform. When the data was collected, the weather was clear, the temperature was about 30℃, and the relative humidity was about 70%. The study area is a rural-urban fringe area with buildings, water, and cultivated land, and there are seven crops: strawberry, cowpea, soybean, sorghum, water spinach, watermelon, and green vegetables. The drone was flown at an altitude of 250m, the image size was 1217×303 pixels, there were 274 bands from 400 to 1000nm, the spatial resolution of the drone-carried hyperspectral image was about 0.109m, and WHU-Hi-HongHu: The WHU-Hi-HongHu dataset was collected on November 20, 2017 at 16:23-17:37 in Honghu City, Hubei Province, China, with a headwall Nano-Hyperspec imaging sensor with a focal length of 17mm on a DJI Matrice 600 Pro drone platform. During the data collection process, the weather was cloudy, the temperature was about 8℃, and the relative humidity of the air was about 55%. The experimental area is a complex agricultural scene with a wide variety of crops. Different varieties of the same crop are also planted in the area, including Chinese cabbage and cabbage, as well as Brassica and Brassica minor. It is worth noting that different varieties of the same crop type are planted in the area; for example, Chinese cabbage / cabbage and Brassica / Brassica minor. The flying altitude of the drone is 100m, the image size is 940×475 pixels, there are 270 bands from 400 to 1000nm, and the spatial resolution of the hyperspectral image carried by the drone is about 0.043m. The training set is used to train the network model for the task of object recognition and classification, the validation set is used to adjust the parameters of the network model, and the test set is used to test whether the network model is accurate.

[0042] The test results are shown in Table 1, Table 2, Figure 2 and Figure 3 As shown:

[0043] Table 1 Comparison of classification results of different classification methods on the WHU-Hi-HanChuan dataset

[0044]

[0045] Table 2 Comparison of classification results of different classification methods on the WHU-Hi-HongHu dataset

[0046]

[0047] Figure 2 and Figure 3 Classification results of different algorithms for the WHU-Hi-HanChuan dataset and the WHU-Hi-HongHu dataset.

[0048] From Table 1, Table 2 and Figure 2 , Figure 3 It can be seen that the results of the method of Example 1 of the present invention are better both globally and locally. The method of Example 1 of the present invention can better distinguish the classes with smaller differences within the group, and it can be clearly seen that the method of Example 1 is clearer than other algorithms.

Claims

1. A hyperspectral image classification method based on Transformer parallel GRU algorithm, characterized in that: The following steps are involved: A. Build a deep neural network model, which includes: an improved SpectralFormer framework and a GRU network for extracting temporal features; B. Train the improved deep neural network model to obtain a trained deep neural network model; C. Combine the basic band data in the hyperspectral remote sensing image data to be tested with the SWIR band data, and then nest them in pairs to obtain a two-dimensional data set; D. Divide the two-dimensional data group into two branches. The first branch is input into the improved SpectralFormer framework to obtain the processing result of the first branch. The second branch is processed by GRU to extract the temporal feature network and tensor flattening to obtain the processing result of the second branch. E. After adding and fusing the results of the first two branches and the processing results of the second branch, a 1*1 convolution is performed to obtain the final output result.

2. The hyperspectral image classification method based on the Transformer parallel GRU algorithm as claimed in claim 1, characterized in that: In the step C, the basic band data in the hyperspectral remote sensing image data to be measured is the R, G, B, and NiR band data of Sentinel 2.

3. The hyperspectral image classification method based on the Transformer parallel GRU algorithm as claimed in claim 1, characterized in that: In step D, the processing process in the Transformer framework is as follows: Each two-dimensional data in the input two-dimensional data group is flattened into a row vector, and then processed to the set dimension through a fully connected layer to obtain a mapped two-dimensional data group; a vector of the set dimension is generated based on the Transformer framework, and it is concatenated with each two-dimensional data in the mapped two-dimensional data group to obtain a concatenated two-dimensional data group, which is input into the Transformer Encoder module for processing to obtain the first branch processing result.

4. The hyperspectral image classification method based on Transformer parallel GRU algorithm as claimed in claim 1, characterized in that: In the step D, the processing process in the GRU temporal feature extraction network is as follows: Initialize the hidden state, calculate the reset gate and update gate for each time step, adjust the hidden state of the previous moment based on the reset gate and calculate the candidate hidden state, perform a weighted combination of the hidden state of the previous moment and the candidate hidden state through the update gate, obtain the final hidden state at the current moment and output it. The output result is the temporal characteristics and spatial dependencies in the hyperspectral remote sensing image.

5. The hyperspectral image classification method based on Transformer parallel GRU algorithm as claimed in claim 4, characterized in that: The reset gate r t Determine which information in the previous hidden state should be ignored. The calculation formula is as follows: r t =σ(w r x t +u r h t-1 +b r ) (1) Where σ is the sigmoid activation function, Wr and Ur are the weight matrices from the input and hidden state to the reset gate, br is the bias term, and x t is the input at the current moment, h t-1 is the hidden state at the previous moment.

6. The hyperspectral image classification method based on Transformer parallel GRU algorithm as claimed in claim 5, characterized in that: The update gate z t Determine how much information in the hidden state of the previous moment should be retained and passed to the current moment. The calculation formula is as follows: z t =σ(W z x t +u z h t-1 +b z ) (2) Among them, Wz and Uz are the weight matrices from input and hidden state to update gate, respectively, and b Z is the bias term.

7. The hyperspectral image classification method based on Transformer parallel GRU algorithm as claimed in claim 5, characterized in that: The hidden state The calculation formula is: Among them, tan h is the hyperbolic tangent activation function, w h and u h are the weight matrices from the hidden state adjusted by the input and reset gates to the candidate hidden state, and b h is the bias term, and ⊙ represents element-wise multiplication.

Images