Hyperspectral classification method based on reconstruction convolution Transform

Abstract

The invention discloses a hyperspectral classification method based on reconstruction convolution Transform, and belongs to the field of hyperspectral remote sensing detection, and the method comprises the steps: obtaining original image data, introducing a spectral space reconstruction network, and carrying out the separation and reconstruction operation; evaluating information contents of different feature maps by combining a scaling factor in the group normalization layer so as to obtain a calculation feature map; by means of redundancy of the features, a spectral feature reconstruction network is introduced, and a segmentation transformation and fusion strategy is adopted; a space attention module is used to overcome limitation brought by a convolution kernel grid structure, and meanwhile, a new activation scheme is introduced; establishing a relation of spectrums of different pixels in the hyperspectral cube by using a spectrum correlation module, and reconstructing an input feature map; and carrying out linear output by using Pooling to obtain a classification result. According to the method, high-precision and high-speed hyperspectral image classification is realized, and stable classification performance can be kept in various complex scenes.

Description

Technical Field

[0001] The present invention belongs to the field of hyperspectral remote sensing detection, and specifically relates to a hyperspectral classification method based on reconstructed convolutional Transformer. Background Art

[0002] Hyperspectral imaging technology can acquire spectral information from hundreds of continuous bands. Combining spatial and spectral information provides powerful object resolution capabilities, playing an irreplaceable role in numerous fields, including precision agriculture, environmental monitoring, pollution identification, geological exploration, and urban planning. However, hyperspectral image data is characterized by high dimensionality and small sample sizes, leading to the "curse of dimensionality" problem. As the number of spectral bands increases, the accuracy of object classification decreases sharply after reaching a certain number of spectra. The fundamental reason is that the algorithm's feature space grows exponentially with the spectral dimension, requiring extensive computational resources to process and effectively analyze hyperspectral data. Designing efficient hyperspectral classification algorithms has become a highly challenging task in this field.

[0003] Traditional hyperspectral image classification algorithms typically utilize the image's spectral data, employing manual dimensionality reduction methods such as principal component analysis, independent component analysis, and linear discriminant analysis. These algorithms then extract features through manual design and use various classifiers for classification. However, these algorithms have numerous limitations. For example, they assume large variance between variables, making it difficult to effectively describe subtle features or nonlinear relationships in complex data. They also exhibit high algorithmic complexity, require parameters to be specified or manually set, and are significantly affected by noise. Furthermore, they often require solving the inverse of high-order matrices, making them difficult to deploy on edge devices and limiting their general applicability.

[0004] In recent years, the application of deep learning in computer vision has achieved remarkable results, and many studies have attempted to transfer the feature learning capabilities of neural networks to hyperspectral classification tasks. Autoencoders and convolutional neural networks were among the earliest neural networks used for hyperspectral image classification, offering certain advantages over traditional classifiers. Subsequently, numerous neural network-based hyperspectral image classification methods have been proposed, such as using recurrent neural networks to learn spectral feature sequences and using convolutional neural networks to extract joint spatial and spectral features. However, the development of neural networks in hyperspectral classification still faces two major obstacles. First, widely used feature extraction schemes such as PCA and convolution extract a large number of redundant features during the dimensionality reduction of hyperspectral images, which is detrimental to subsequent classification operations and consumes a lot of computational resources. Second, during the classification stage, traditional convolutional neural networks have a limited receptive field. While the Transformer overcomes the geometric limitations of convolution kernels in classification, these attention-based networks, while achieving good classification results, compute spectral attention as correlation between feature channels, lacking correlation between spatial locations. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a hyperspectral classification method based on reconstructed convolutional Transformer, which greatly reduces the complexity of the classification algorithm, effectively reduces information redundancy in the feature extraction stage, and makes full use of the joint spatial-spectral information in the classification processing stage to effectively extract the features of the ground objects, achieving higher accuracy and faster hyperspectral image classification, and can maintain robust classification performance in various complex scenarios.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A hyperspectral classification method based on reconstructed convolutional Transformer includes the following steps:

[0008] Step 1: Input a scene of hyperspectral image data As the original training image data, data from the hyperspectral image data are randomly selected as the training set, and the remaining data are used as the test set;

[0009] Step 2: Establish a hyperspectral data compression network, introduce a spectral space reconstruction network and a spectral feature reconstruction network; use separation and reconstruction operations to fully utilize spatial redundancy; combine the scaling factor in the group normalization layer to evaluate the information content of different feature maps, add features with rich information to features with less information to generate features with richer information and save storage space; the spectral feature reconstruction network uses segmentation transformation and fusion strategies to compress the original hyperspectral image data to generate hyperspectral compressed data;

[0010] Step 3: Based on the generated hyperspectral compressed data, a Transformer network combined with a fast spatial attention mechanism is constructed to perform image classification operations. The spatial attention mechanism overcomes the geometric limitations of the convolution kernel in human-computer interaction classification.

[0011] Step 4: The mask generated by the three-dimensional convolution operation is integrated with spatial information through the spectral correlation module and reconstructed, and finally the classification result is output through the pooling operation.

[0012] Beneficial effects:

[0013] The present invention introduces a spectral space reconstruction network and a spectral feature reconstruction network, fully utilizes the redundancy of spatial features, combines the scaling factor in the normalization layer to evaluate the information content of different feature maps, adds information-rich features to information-less features to generate more information-rich features and saves storage space, greatly reduces the complexity of the classification algorithm, and can greatly reduce the amount of calculation; by constructing a Transformer network combined with a fast spatial attention mechanism for image classification operations, it overcomes the geometric limitations of convolution kernels in human-computer interaction classification, fully utilizes the joint spatial-spectral information to effectively extract ground feature features, and effectively improves the accuracy of hyperspectral image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of the hyperspectral classification method based on reconstructed convolutional Transformer of the present invention; Figure 2 This is a schematic diagram of the spectral space reconstruction network structure of the present invention; Figure 3 This is a schematic diagram of the spectral feature reconstruction network of the present invention; Figure 4 Schematic diagram of the attention mechanism of the present invention; Figure 5 This is a schematic diagram of the spectrum correlation module of the present invention; Figure 6 Schematic diagram of the classification results of the present invention on the PaviaU dataset, where (a) is the PaviaU pseudo-color image, (b) is the PaviaU true value image, (c) is the SSRN classification result, (d) is the SSTN classification result, and (e) is the classification result diagram of the present invention. DETAILED DESCRIPTION

[0015] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0016] like Figure 1 As shown in the figure, the present invention proposes a hyperspectral image classification method based on reconstructed convolution and Transformer, which achieves higher accuracy and faster hyperspectral image classification and can maintain robust classification performance in various complex scenes. The method includes the following steps:

[0017] Step 1: Input data and segment the dataset: Input a scene of hyperspectral image data As the original training image data, randomly select about The data is used as the training set, and the rest of the data is used as the test set;

[0018] Step 2. Establish a hyperspectral data compression network to generate hyperspectral compressed data: Establish a hyperspectral data compression network. On the basis of conventional convolution, introduce a spectral space reconstruction network and a spectral feature reconstruction network: First, use separation and reconstruction operations to make full use of spatial redundancy. At the same time, combine the scaling factor in the group normalization layer to evaluate the information content of different feature maps, add information-rich features to less information-rich features to generate more information-rich features and save storage space. In order to utilize the redundancy of features, introduce a spectral feature reconstruction network, adopt segmentation transformation and fusion strategies to further compress the original hyperspectral data and generate hyperspectral compressed data;

[0019] Step 3: Image classification: Based on the hyperspectral compressed data generated above, a Transformer network combined with a fast spatial attention mechanism is constructed to perform image classification operations. This mainly overcomes the geometric limitations of convolution kernels in human-computer interaction classification through the spatial attention mechanism.

[0020] Step 4: Output classification results: Through the spectral correlation module, the mask generated by the three-dimensional convolution operation is integrated with spatial information and the data obtained in step 3 is reconstructed. Finally, the classification results are output through the pooling layer.

[0021] Specifically, if Figure 2 As shown, the step 2 includes:

[0022] The information content of different feature maps is evaluated by combining the scaling factors in the group normalization layer to obtain a calculated feature map. The original image is ,in is the original hyperspectral image data, is the batch size, is the number of channels of the hyperspectral image, and is the height and width of the hyperspectral image. First, by subtracting the image mean and divided by the image standard deviation To standardize the input features , as shown below:

[0023] (1)

[0024] in, is batch normalization, It is an additional parameter added to improve the stability of the algorithm. and are trainable neural network parameters.

[0025] Afterwards, the trainable neural network parameters in the GN (batch normalization) layer are used Measures the spatial pixel variance of each batch and channel. A larger value of the trainable neural network parameter γ comes from richer spatial information, which is specifically manifested as the change in spatial pixels. The weight after batch normalization is shown in Equation (2):

[0026] (2)

[0027] in, is the weight after batch normalization, i, j represents the i-th and j-th channels. Then it is mapped to The range gets the weight value and is controlled by the established threshold.

[0028] Figure 2 middle, Represents the calculated weight, Represents a channel data in the original data.

[0029] Finally, get the weight The whole process can be expressed as formula (3):

[0030] (3)

[0031] in, is the threshold function, is the input data. After that, it is the threshold function Set a certain threshold (usually 0.5) and set the threshold function The weight of the parameter that is equal to or greater than the threshold is set to 1, and the information weight is obtained , the weight of the parameter less than the threshold is set to 0, and the information weight is obtained Finally, the input features are multiplied by the information weights and information weight , we get two weighted features: features with large information content and features with small information content. We successfully divide the input features into two parts: spatial content with large information content and strong expressiveness. and spatial content with little or no information, which is considered redundant .

[0032] (4)

[0033] (5)

[0034] Afterwards, the features with rich information are added to the features with less information to generate more informative features and save storage space, and the feature map output of this module is obtained. (Right now Figure 2 Output feature map in ( ):

[0035] (6)

[0036] in, Indicates a series operation. Preferably, Figure 2 In the output feature map C=48.

[0037] Specifically, if Figure 3 As shown, in step 2, the segmentation transformation and fusion strategies adopted by the spectral feature reconstruction network include:

[0038] For a given module feature map output (Right now Figure 3 ), , , are the number of channels, height, and width of the features (preferably, C=48). First, the module feature map is output The channel is divided into two parts, namely Channel and aisle, is the threshold set. Subsequently, 1×1 convolution (Conv) is used to transform the channel of the compressed feature map to improve computational efficiency. After this operation, the spatial refinement features can be divided into the upper layer and lower .

[0039] (7)

[0040] (8)

[0041] in, It is a convolution operation with a convolution kernel size of 1. Since it contains higher information, the upper layer In the upper layer conversion stage, efficient convolution operations, namely GWC (group convolution) and PWC (point convolution), are used to replace expensive standard convolution to extract high-level representative information and reduce computational costs. Due to the sparse convolution connection, GWC reduces the number of parameters and computation, but cuts off the information flow between channel groups. PWC compensates for the information loss and helps information flow across feature channels. Therefore, in the same upper layer Execute and operate, is the convolution kernel size.

[0042] Sum the output to get the feature map, which is the output of the up-conversion stage , which can be expressed as:

[0043] (9)

[0044] in, and are the learnable weight matrices of GWC and PWC respectively.

[0045] is input to the lower conversion stage, with lower application complexity Operations are used to generate feature maps with shallow hidden details as a complement to the rich feature extractor, while connecting the generated and reused features to form the output of the down-conversion stage. , as shown below:

[0046] (10)

[0047] After performing the transform operation, the output from the up-conversion stage is adaptively merged using a simplified SKNet method. and the output of the down-conversion stage , and then use global average pooling (i.e. Figure 3 Pooling) to obtain global spatial information with channel statistics (including the global channel descriptor , global channel descriptor ), the calculation formula is:

[0048] (11)

[0049] Among them, Pooling represents the global average pooling operation, Represents the output, that is, the output of the up-conversion stage and the output of the down-conversion stage ; Represents the output of the image pixel position (p, q); m represents the number of the up-conversion stage and the down-conversion stage. , This is the height and width of the feature map at this time.

[0050] Afterwards, the global channel descriptor , global channel descriptor Stacked together and using channel soft attention operation to generate feature importance vector , as shown below:

[0051] (12)

[0052] (13)

[0053] The compressed hyperspectral image output is (Right now Figure 3 The output feature map of is:

[0054] (14)

[0055] Preferably, Figure 3 In , the output feature map has C=36.

[0056] Specifically, if Figure 4 As shown, the spatial attention mechanism in step 3 is:

[0057] (15)

[0058] (16)

[0059] (17)

[0060] in, , , are the three vectors in the attention mechanism, 、 and Represents the trainable parameters in the tensor convolution operation. In addition, 、 、 and Represent the compressed hyperspectral image output Channel size, width, height and and The channel size. The convolution kernel is Two-dimensional convolution operation. and Reshape into The size of the attention map is generated as follows:

[0061] (18)

[0062] in, is the intermediate parameter, represent Activation operation, the superscript T indicates the transpose of the matrix.

[0063] Output of the spatial attention module The calculation is as follows:

[0064] (19)

[0065] Among them, the output of the spatial attention module Each position is obtained by summing up the weighted value tensor right The reconstruction of , while skip connections are used to regularize the spatial attention module to achieve smooth training.

[0066] Specifically, if Figure 5 As shown, the spectral correlation module in step 4 integrates the spatial information using the mask generated by the three-dimensional convolution operation and constructs it to ensure that the information is fully utilized:

[0067] (20)

[0068] (twenty one)

[0069] in, is the spectral correlation kernel, is a nonlinear activation function, Indicates that a three-dimensional convolution is performed using a kernel size of k×1×C×1×1 (i.e. Figure 5 ), generating a 3D convolution of size Tensor of , is the image mask, is the associated output. Then the tensor is applied Function. Then, take the generated mask To integrate input features The spatial information of spectral correlation kernel.

[0070] Finally, the output of the spectral correlation module It can be calculated as follows:

[0071] (twenty two)

[0072] (twenty three)

[0073] Among them, the output of the spectral correlation module is a 3D convolution The regularized output of is the image mask. and Shared trainable parameters. Finally, the output After processing by the Linear layer, the classification results are obtained.

[0074] Example:

[0075] The Pavia University dataset selected for testing consists of hyperspectral images of the University of Pavia, Italy, acquired in 2003 by the German ROSIS-03 airborne reflective optical spectrographic imaging system. The image size is 610 × 340. After removing noise, 103 valid bands remain.

[0076] A hyperspectral image classification method based on reconstructed convolution and Transformer in this embodiment includes the following steps:

[0077] Step 1. Input a scene of hyperspectral image data As the original training image data, approximately The data is used as the training set, and the rest of the data is used as the test set.

[0078] Step 2. Establishing a hyperspectral data compression network: In this embodiment, , , .

[0079] Step 3. Based on the hyperspectral compressed data generated above, a Transformer network combined with a fast spatial attention mechanism is constructed to perform image classification operations. The spatial attention mechanism is mainly used to overcome the geometric limitations of the convolution kernel in classification. , .

[0080] Step 4. Through the spectral correlation module, the mask generated by the three-dimensional convolution operation is integrated with the spatial information and reconstructed, and finally the classification result is output through Pooling. .set up and Shared trainable parameters, where the convolution kernel size is 18×1×1. Finally, the output After processing by the Linear layer, a vector of length 10 is obtained, and the classification result is obtained.

[0081] The classification results were evaluated by OA (overall accuracy), AA (average accuracy) and Kappa coefficient, and compared with the mainstream SSRN (spectral-spatial residual network for HSI classification), SSTN (spectral-spatial convolutional network), and SPA (successive projection algorithm). The results are as follows: Figure 6As shown in Table 1, the present invention significantly improves categories such as trees, gravel, and masonry in the Pavia University dataset, demonstrating the advantages of spatial-spectral joint features. Categories such as Bitumen (concrete) perform at the same or slightly lower levels, likely due to small sample sizes or spectral variation. OA: The present invention achieved 97.97%, an improvement of 1.30% over the suboptimal model SSTN (96.67%), and significantly outperformed traditional methods such as SPA (86.33%). The Kappa coefficient of the present invention was 97.31, far exceeding SSRN (95.01) and SPA (77.98), indicating higher classification consistency and better results.

[0082] Table 1 Classification results of University of Pavia

[0083]

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hyperspectral classification method based on reconstructed convolutional Transformer, characterized in that: The steps include: Step 1: Input a scene of hyperspectral image data As the original training image data, data from the hyperspectral image data are randomly selected as the training set, and the remaining data are used as the test set; Step 2: Establish a hyperspectral data compression network, introduce a spectral space reconstruction network and a spectral feature reconstruction network; Separation and reconstruction operations are used to fully utilize spatial redundancy. The scaling factor in the group normalization layer is combined to evaluate the information content of different feature maps, and features with rich information are added to features with less information to generate more informative features and save storage space. The spectral feature reconstruction network uses segmentation transformation and fusion strategies to compress the original hyperspectral image data to generate hyperspectral compressed data. The segmentation transformation and fusion strategies adopted by the spectral feature reconstruction network include: For a given module feature map output , first output the module feature map The channel is divided into two parts, namely Channel and aisle, is the threshold set; then, the 1×1 convolution is used to transform the channel of the compressed feature map, and the spatial refinement features are divided into the upper layer and lower : ； ； in, It is a 1×1 convolution operation; , , are the number of channels, height, and width of the feature respectively; upper layer Send it to the upper conversion stage, perform convolution operation, sum the outputs, and get the output of the up-conversion stage ; Step 3: Based on the generated hyperspectral compressed data, a Transformer network combined with a fast spatial attention mechanism is constructed to perform image classification operations. The spatial attention mechanism overcomes the geometric limitations of the convolution kernel in human-computer interaction classification. Step 4: The mask generated by the three-dimensional convolution operation is integrated with spatial information through the spectral correlation module and reconstructed, and finally the classification result is output through the pooling operation.

2. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 1 is characterized in that: In step 2, the original hyperspectral image data is compressed to generate hyperspectral compressed data, which includes the following steps: The information content of different feature maps is evaluated by combining the scaling factors in the group normalization layer to obtain a computational feature map; the original hyperspectral image data is ,in is the original hyperspectral data, is the batch size, is the number of channels of the hyperspectral image, and are the width and height of the image respectively; first, by subtracting the image mean and divided by the image standard deviation To standardize the input features ; Leveraging trainable parameters in batch normalization layers Measure the spatial pixel variance for each batch and channel; By mapping the Sigmoid function to Get the weight value in the range and control it through the established threshold; get the weight The process is: ； in, is the threshold function, For input data, is the batch normalization operation, is the weight after batch normalization; The Sigmoid function is the activation function.

3. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 2 is characterized in that: is the threshold function The parameter setting threshold is set, and the weight of the parameter equal to or greater than the threshold is set to 1 to obtain the information weight , the weight of the parameter less than the threshold is set to 0, and the information weight is obtained ; Multiply the input features by and , we get two weighted features: features with large information content and features with small information content, which are respectively the spatial content with large information content and the spatial content with small information content. , spatial content with little or no information ; The spatial content with large amount of information and spatial content with little or no information Add together to get the module feature map output .

4. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 3 is characterized in that: Lower level is input to the lower conversion stage to form the output of the down conversion stage .

5. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 4 is characterized in that: Adaptively merge the outputs from the up-conversion stage using a simplified SKNet approach and the output of the down-conversion stage , and then global average pooling is used to obtain global spatial information with channel statistics , the calculation formula is: ； in, and are the width and height of the image respectively; Pooling represents the global average pooling operation, Represents the output, that is, the output of the up-conversion stage and the output of the down-conversion stage ; Represents the output of the image pixel position (p, q); global spatial information with channel statistics Includes the global channel descriptor , global channel descriptor , m represents the number of the up-conversion stage and the down-conversion stage.

6. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 5 is characterized in that: Global Channel Descriptor , global channel descriptor Stacked together and using channel soft attention operation to generate feature importance vector , as shown below: ； 。 7. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 6 is characterized in that: Compressed hyperspectral image output for: 。 8. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 7 is characterized in that: The step 4 comprises: ； ； in, is the spectral correlation kernel, is a nonlinear activation function, Indicates that a three-dimensional convolution is performed using a kernel size of k×1×C×1×1, generating a size of Tensor of , is the image mask, is the associated output; is the output of the spatial attention module; the superscript T represents the transpose of the matrix; is the convolution kernel size; Take the generated mask To integrate input features The spatial information of spectral correlation kernel.

9. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 8, characterized in that: Output of the spectral correlation module It is calculated as follows: ； ； Among them, the output of the spectral correlation module is a three-dimensional convolution The regularized output of is the image mask; and Shared trainable parameters, is the output of the spectral correlation module.

10. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 9 is characterized in that: Finally, the output of the spectral correlation module After processing by the linear layer, the classification results are obtained.

Images