Hyperspectral classification method based on reconstructed convolutional transformer

Abstract

The application discloses a hyperspectral classification method based on a reconstructed convolutional Transformer, and belongs to the field of hyperspectral remote sensing detection, and comprises the following steps: after original image data is acquired, a spectral space reconstruction network is introduced, and separation and reconstruction operations are adopted; a scaling factor in a group normalization layer is combined to evaluate the information content of different feature maps, so that a calculation feature map is acquired; a spectral feature reconstruction network is introduced by using the redundancy of features, and segmentation transformation and fusion strategies are adopted; a spatial attention module is used to overcome the limitation caused by the grid structure of a convolution kernel, and a new activation scheme is introduced; a spectral correlation module is used to establish the connection of different pixel spectra in a hyperspectral cube, and an input feature map is reconstructed; and a linearized output is obtained by using Pooling, so that a classification result is obtained. The application realizes hyperspectral image classification with higher precision and faster speed, and can maintain stable classification performance in various complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of hyperspectral remote sensing, specifically relating to a hyperspectral classification method based on reconstructed convolutional Transformer. Background Technology

[0002] Hyperspectral imaging technology can acquire spectral information from hundreds of consecutive bands. Combining spatial and spectral information, it provides powerful ground feature resolution capabilities, playing an irreplaceable role in numerous fields such as precision agriculture, environmental monitoring, pollution identification, geological exploration, and urban planning. However, hyperspectral image data is characterized by high dimensionality and small sample size, leading to the "curse of dimensionality" problem. As the number of spectral bands increases, the accuracy of ground feature classification drops sharply after reaching a certain number of spectra. The fundamental reason is that the feature space of the algorithm grows exponentially with the spectral dimension, making the computational processing and effective analysis of hyperspectral data require a large amount of computational resources. Designing efficient hyperspectral classification algorithms has become a highly challenging issue in this field.

[0003] Traditional hyperspectral image classification algorithms typically utilize the spectral data of images, employing manual dimensionality reduction methods such as principal component analysis, independent component analysis, and linear discriminant analysis to extract features through manual design, and then using various classifiers for classification. However, these algorithms have many limitations. For example, they require the assumption of large variance between variables, making it difficult to effectively describe subtle ground features or nonlinear relationships in complex data; the algorithms are highly complex, requiring parameters to be specified in advance or manually set, and are significantly affected by noise; they usually require solving the inverse of high-order matrices, making them difficult to deploy on edge devices and lacking versatility.

[0004] In recent years, deep learning has achieved remarkable results in the field of computer vision, and many studies have attempted to transfer the feature learning capabilities of neural networks to hyperspectral classification tasks. Autoencoders and convolutional neural networks were among the earliest neural networks used for hyperspectral image classification, offering certain advantages over traditional classifiers. Subsequently, numerous neural network-based hyperspectral image classification methods have been proposed, such as using recurrent neural networks to learn spectral feature sequences and employing convolutional neural networks to extract joint spatial spectral features. However, the development of neural networks in hyperspectral classification still faces two major obstacles: First, widely used feature extraction schemes such as PCA and convolutional networks extract a large number of redundant features during hyperspectral image dimensionality reduction, which is detrimental to subsequent classification operations and consumes significant computational resources. Second, in the classification stage, traditional convolutional neural networks have limited receptive fields, while Transformers overcome the geometric limitations of convolutional kernels in classification. However, although these attention-based networks have achieved good classification results, they calculate spectral attention as the correlation between feature channels, lacking the correlation between spatial locations. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a hyperspectral classification method based on reconstructed convolutional Transformer, which significantly reduces the complexity of the classification algorithm, effectively reduces information redundancy in the feature extraction stage, and fully utilizes spatial-spectral joint information to effectively extract ground feature characteristics in the classification processing stage. This achieves higher accuracy and faster speed hyperspectral image classification, and maintains robust classification performance in various complex scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A hyperspectral classification method based on reconstructed convolutional Transformer includes the following steps:

[0008] Step 1: Input hyperspectral image data. As the original training image data, data from the hyperspectral image data were randomly selected as the training set, and the remaining data were used as the test set.

[0009] Step 2: Establish a hyperspectral data compression network, introducing a spectral spatial reconstruction network and a spectral feature reconstruction network; employ separation and reconstruction operations to fully utilize spatial redundancy; combine the scaling factor in the group normalization layer to evaluate the information content of different feature maps, adding information-rich features with less information to generate more information-rich features and save storage space; the spectral feature reconstruction network uses segmentation transformation and fusion strategies to compress the original hyperspectral image data, generating hyperspectral compressed data;

[0010] Step 3: Based on the generated hyperspectral compressed data, construct a Transformer network that combines a fast spatial attention mechanism to perform image classification operations. The spatial attention mechanism overcomes the geometric limitations of convolutional kernels in human-computer interaction classification.

[0011] Step 4: Integrate the spatial information from the mask generated by the 3D convolution operation through the spectral correlation module and reconstruct it. Finally, output the classification result through pooling operation.

[0012] Beneficial effects:

[0013] This invention introduces a spectral spatial reconstruction network and a spectral feature reconstruction network, making full use of spatial feature redundancy. It combines a scaling factor in the normalization layer to evaluate the information content of different feature maps, adding information-rich features to those with less information to generate more information-rich features while saving storage space. This significantly reduces the complexity of the classification algorithm and greatly alleviates computational burden. Furthermore, by constructing a Transformer network incorporating a fast spatial attention mechanism for image classification, it overcomes the geometric limitations of convolutional kernels in human-computer interactive classification, fully utilizing joint spatial-spectral information to effectively extract ground feature characteristics and significantly improve the accuracy of hyperspectral image classification. Attached Figure Description

[0014] Figure 1 A flowchart of the hyperspectral classification method based on reconstructed convolutional Transformer of the present invention;

[0015] Figure 2 This is a schematic diagram of the spectral spatial reconstruction network structure of the present invention;

[0016] Figure 3 This is a schematic diagram of the spectral feature reconstruction network of the present invention;

[0017] Figure 4 This is a schematic diagram of the attention mechanism of the present invention;

[0018] Figure 5 This is a schematic diagram of the spectral correlation module of the present invention;

[0019] Figure 6 The diagram shows the classification results of the present invention on the PaviaU dataset, where (a) is the PaviaU pseudo-color image, (b) is the PaviaU ground truth image, (c) is the SSRN classification result, (d) is the SSTN classification result, and (e) is the classification result diagram of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0021] like Figure 1 As shown, this invention proposes a hyperspectral image classification method based on reconstructed convolution and Transformer, achieving higher accuracy and faster speed hyperspectral image classification, and maintaining robust classification performance in various complex scenes. The method includes the following steps:

[0022] Step 1: Input data and segment the dataset: Input a hyperspectral image data. As the original training image data, approximately [number] images were randomly selected from the hyperspectral image data. The data used is the training set, and the remaining data is the test set;

[0023] Step 2: Establish a hyperspectral data compression network to generate compressed hyperspectral data. A hyperspectral data compression network is established by introducing a spectral spatial reconstruction network and a spectral feature reconstruction network on top of conventional convolution. First, separation and reconstruction operations are used to fully utilize spatial redundancy. Simultaneously, the scaling factor in the group normalization layer is used to evaluate the information content of different feature maps, adding information-rich features with less information to generate more information-rich features and save storage space. To utilize feature redundancy, a spectral feature reconstruction network is introduced, employing segmentation transformation and fusion strategies to further compress the original hyperspectral data, generating compressed hyperspectral data.

[0024] Step 3, Image Classification: Based on the hyperspectral compressed data generated above, a Transformer network combining a fast spatial attention mechanism is constructed to perform image classification. It mainly overcomes the geometric limitations of convolutional kernels in human-computer interaction classification through the spatial attention mechanism.

[0025] Step 4: Classification result output: Through the spectral association module, the spatial information is integrated from the mask generated by the 3D convolution operation and the data obtained in Step 3 is reconstructed. Finally, the classification result is output through the pooling layer.

[0026] Specifically, such as Figure 2 As shown, step 2 includes:

[0027] The scaling factor in the group normalization layer is used to evaluate the information content of different feature maps, thereby obtaining a computed feature map. The original image is... ,in For raw hyperspectral image data, For batch size, The number of channels in the hyperspectral image. and This represents the height and width of the hyperspectral image. First, by subtracting the image average... Divide by the image standard deviation To standardize input features As shown below:

[0028] (1)

[0029] in, For batch normalization, It is an additional parameter added to improve the stability of the algorithm. and These are the parameters of a trainable neural network.

[0030] Then, the trainable neural network parameters in the GN (batch normalization) layer are utilized. The spatial pixel variance of each batch and channel is measured. Larger values ​​of trainable neural network parameters γ stem from richer spatial information, specifically the variation in spatial pixels. The batch-normalized weights are shown in equation (2):

[0031] (2)

[0032] in, Here, the weights are the batch-normalized weights, where i and j represent the i-th and j-th channels, respectively. They are then mapped to... using the Sigmoid function (activation function). The weight values ​​are obtained within a range and controlled by a set threshold.

[0033] Figure 2 middle, This represents the calculated weight. This represents data from a specific channel in the original data.

[0034] Finally, obtain the weight. The entire process can be expressed as equation (3):

[0035] (3)

[0036] in, For threshold function, The input data is used. Next, the threshold function is used. Set a certain threshold (usually 0.5), and then use the threshold function. The weights of parameters that are equal to or greater than the threshold are set to 1, thus obtaining the information weights. The weights of parameters less than the threshold are set to 0, thus obtaining the information weights. Finally, the input features are multiplied by the information weights. and information weight This yields two weighted features: features with high information content and features with low information content. The input features are successfully divided into two parts: spatial content with high information content and strong expressive power. Spatial content that is considered redundant, has little or no information .

[0037] (4)

[0038] (5)

[0039] Next, the information-rich features are added to the less information-rich features to generate more information-rich features and save storage space, resulting in the feature map output of this module. (Right now Figure 2 Output feature map in (the output feature map)

[0040] (6)

[0041] in, This indicates a cascade operation. Preferably, Figure 2 In the middle, output feature map C=48.

[0042] Specifically, such as Figure 3 As shown, in step 2, the segmentation transformation and fusion strategies employed by the spectral feature reconstruction network include:

[0043] For a given module feature map output (Right now Figure 3 (input feature map) , , These represent the number of channels, height, and width of the feature (preferably C=48). First, output the module feature map. The channel is divided into two parts, namely Channels and aisle, The threshold is set. Then, a 1×1 convolution (Conv) is used to transform the feature map into compressed channels to improve computational efficiency. After this operation, the spatially refined features can be divided into upper layers. and lower level .

[0044] (7)

[0045] (8)

[0046] in, This is a convolution operation with a kernel size of 1. Because it contains a high level of information, the upper layers... The data is then fed into the upper-level transformation stage, where efficient convolutional operations, namely GWC (Grouped Convolution) and PWC (Pointed Convolution), replace expensive standard convolutions to extract high-level representative information and reduce computational costs. Due to sparse convolutional connections, GWC reduces the number of parameters and computational cost, but it disrupts the information flow between channel groups. PWC, on the other hand, compensates for information loss, helping information flow across feature channels. Therefore, in the same upper layer... Execution and operate, is the kernel size.

[0047] Summing the outputs yields the feature map, which is the output of the up-conversion stage. , can be represented as:

[0048] (9)

[0049] in, and These are the learnable weight matrices for GWC and PWC, respectively.

[0050] The input is passed to the next layer of transformation, and the application complexity is relatively low. The operation generates feature maps with shallow hidden details, complementing the rich feature extractor, while concatenating the generated and reused features to form the output of the down-transformation stage. As shown below:

[0051] (10)

[0052] After performing the transformation operation, the outputs from the up-transformation stage are adaptively merged using a simplified SKNet method. and the output of the downconversion stage Then use global average pooling (i.e. Figure 3 Pooling) is used to obtain global spatial information with channel statistics. (including the upper global channel descriptor) Global Channel Descriptor The calculation formula is:

[0053] (11)

[0054] Where Pooling represents the global average pooling operation. This indicates the output, specifically the output of the upconversion stage. and the output of the downconversion stage ; This represents the output at pixel position (p, q) in the image; m represents the number of the up-conversion stage and the down-conversion stage. , This refers to the height and width of the feature map at this point.

[0055] Next, the global channel descriptor will be used. Global Channel Descriptor Stack them together and use channel soft attention operations to generate feature importance vectors. As shown below:

[0056] (12)

[0057] (13)

[0058] The compressed hyperspectral image output (Right now Figure 3 The output feature map is:

[0059] (14)

[0060] Preferred, Figure 3 In the output feature map, C=36.

[0061] Specifically, such as Figure 4 As shown, the spatial attention mechanism in step 3 is as follows:

[0062] (15)

[0063] (16)

[0064] (17)

[0065] in, , , These are the three vectors in the attention mechanism. , and This represents the trainable parameters in tensor convolution operations. Furthermore, , , and These represent the compressed hyperspectral image outputs. The channel size, width, height and and The channel size. Indicates that the convolution kernel is Two-dimensional convolution operation. and Remodeling The size of the attention map and the generation process are as follows:

[0066] (18)

[0067] in, For intermediate parameters, represent The activation operation, with the superscript T indicating the transpose of the matrix.

[0068] Output of the spatial attention module The calculation is as follows:

[0069] (19)

[0070] Among them, the output of the spatial attention module Each position is represented by a weighted tensor. right The reconstruction, in which Meanwhile, skip connections are used to regulate the spatial attention module, thereby achieving smooth training.

[0071] Specifically, such as Figure 5 As shown, the spectral correlation module in step 4 integrates spatial information using a mask generated by three-dimensional convolution operations and constructs it to ensure full utilization of the information:

[0072] (20)

[0073] (twenty one)

[0074] in, For spectral correlation kernel, It is a non-linear activation function. This indicates that a 3D convolution is performed using a kernel size of k×1×C×1×1 (i.e., ... Figure 5 3D convolution), generating a size of tensor, For image mask, This is the associated output. Then, an application was applied to this tensor. The function. Then, the generated mask is used. To integrate input features Spatial information, resulting in a size of Spectral correlation kernel.

[0075] Finally, the output of the spectral correlation module It can be calculated as follows:

[0076] (twenty two)

[0077] (twenty three)

[0078] Among them, the output of the spectral correlation module For 3D convolution The regularized output, For image mask. and Shared trainable parameters. Finally, the output... The classification results are obtained after processing through a Linear layer.

[0079] Example:

[0080] The Pavia University dataset used for testing is a hyperspectral image of the Pavia University region in Italy, acquired in 2003 by the German ROSIS-03 airborne Reflective Optics Spectrographic Imaging System. The image size is 610×340. After removing noise, 103 valid bands remain.

[0081] This embodiment of a hyperspectral image classification method based on reconstructed convolution and Transformer includes the following steps:

[0082] Step 1. Input hyperspectral image data. As the original training image data, approximately [number] images were randomly selected from the Pavia U hyperspectral data. The data in question is used as the training set, and the remaining data is used as the test set.

[0083] Step 2. Establish a hyperspectral data compression network: In this embodiment, , , .

[0084] Step 3. Based on the generated hyperspectral compressed data, a Transformer network combining a fast spatial attention mechanism is constructed for image classification. This network primarily overcomes the geometric limitations of convolutional kernels in classification through spatial attention. In this embodiment, [the following is taken]... , .

[0085] Step 4. Using the spectral association module, the spatial information generated by the 3D convolution operation is integrated and reconstructed. Finally, the classification result is output through Pooling. In this embodiment, the following steps are taken: .set up and Shared trainable parameters, with a kernel size of 18×1×1. Finally, the output... After processing by the Linear layer, a vector of length 10 is obtained, and the classification result is obtained.

[0086] The classification results were evaluated using OA (overall accuracy), AA (average accuracy), and Kappa coefficient. The results were compared with mainstream algorithms such as SSRN (Spectral-Spatial Residual Network for HSI classification), SSTN (Spectral-Spatial Convolutional Network), and SPA (Continuous Projection Algorithm). Figure 6As shown in Table 1, this invention significantly improves the performance of categories such as trees, gravel, and bricks in the Pavia University dataset, demonstrating the advantages of the spatial-spectral joint features. Categories such as Bitumen (concrete) show similar or slightly lower performance, possibly due to small sample sizes or spectral variations. OA: This invention achieves 97.97%, a 1.30% improvement over the suboptimal SSTN model (96.67%), and significantly better than traditional methods such as SPA (86.33%). The Kappa coefficient of this invention is 97.31, far exceeding SSRN (95.01) and SPA (77.98), indicating higher classification consistency and superior performance.

[0087] Table 1. Classification results of the University of Pavia

[0088]

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hyperspectral classification method based on reconstructed convolutional Transformer, characterized in that, Includes the following steps: Step 1: Input hyperspectral image data. As the original training image data, data from the hyperspectral image data were randomly selected as the training set, and the remaining data were used as the test set. Step 2: Establish a hyperspectral data compression network and introduce a spectral spatial reconstruction network and a spectral feature reconstruction network; The system employs separation and reconstruction operations to fully utilize spatial redundancy; it combines scaling factors in the group normalization layer to evaluate the information content of different feature maps, adding information-rich features with less information to generate more information-rich features and save storage space; the spectral feature reconstruction network uses segmentation transformation and fusion strategies to compress the original hyperspectral image data and generate hyperspectral compressed data. The segmentation, transformation, and fusion strategies employed by the spectral feature reconstruction network include: For a given module feature map output First, output the module feature map. The channel is divided into two parts, namely Channels and aisle, A threshold was set; subsequently, 1×1 convolutions were used to transform the channels into compressed feature maps, and the spatially refined features were divided into upper layers. and lower level : ； ； in, This is a 1×1 convolution operation; , , These represent the number of channels, height, and width of the feature, respectively. upper layer The output is fed into the upper-level transformation stage, where a convolution operation is performed, and the outputs are summed to obtain the output of the up-transformation stage. ; lower level The input is fed into the next lower conversion stage to form the output of the next conversion stage. ; Adaptively merge the outputs from the up-transformation stage using a simplified SKNet method. and the output of the downconversion stage Then, global average pooling is used to obtain global spatial information with channel statistics. The calculation formula is: ； in, and These represent the width and height of the image, respectively; Pooling indicates global average pooling operation. This indicates the output, specifically the output of the upconversion stage. and the output of the downconversion stage ; The output represents the pixel position (p, q) of the image; it contains global spatial information with channel statistics. Including the upper global channel descriptor Global Channel Descriptor , m represents the number of the up-conversion stage and the down-conversion stage; Upload global channel descriptor Global Channel Descriptor Stack them together and use channel soft attention operations to generate feature importance vectors. As shown below: ； Compressed hyperspectral image output for: ； Step 3: Based on the generated hyperspectral compressed data, construct a Transformer network that combines a fast spatial attention mechanism to perform image classification operations. The spatial attention mechanism overcomes the geometric limitations of convolutional kernels in human-computer interaction classification. Step 4: Integrate the spatial information from the mask generated by the 3D convolution operation through the spectral correlation module and reconstruct it. Finally, output the classification result through pooling operation.

2. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 1, characterized in that, Step 2 involves compressing the original hyperspectral image data to generate compressed hyperspectral data, including the following steps: The scaling factor in the group normalization layer is used to evaluate the information content of different feature maps, thereby obtaining a computed feature map; the original hyperspectral image data is... ,in For raw hyperspectral data, For batch size, The number of channels in the hyperspectral image. and These are the width and height of the image, respectively; first, by subtracting the average value of the image... Divide by the image standard deviation To standardize input features ; Utilizing trainable parameters in the batch normalization layer Measure the spatial pixel variance for each batch and channel; By mapping the Sigmoid function to The weight value is obtained within a range and controlled by a set threshold; weight is obtained. The process is as follows: ； in, For threshold function, For input data, For batch normalization operations, These are the weights after batch normalization. The Sigmoid function is an activation function.

3. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 2, characterized in that, Threshold function The parameter setting threshold is used to assign a weight of 1 to parameters that are equal to or greater than the threshold, thus obtaining the information weight. The weights of parameters less than the threshold are set to 0, thus obtaining the information weights. Multiply the input features by ; and This yields two weighted features: features with high information content and features with low information content, which are spatial contents with high information content, respectively. Space content with little or no information ; to store large amounts of information in a space Space content with little or no information Add them together to get the module feature map output. .

4. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 1, characterized in that, Step 4 includes: ； ； in, For spectral correlation kernel, It is a non-linear activation function. This indicates that a 3D convolution is performed using a kernel size of k×1×C×1×1, generating a result of size k×1×C×1×1. tensor, For image mask, For associated output quantities; This is the output of the spatial attention module; the superscript T indicates the transpose of the matrix. The kernel size; Use the generated mask To integrate input features Spatial information, resulting in a size of Spectral correlation kernel.

5. A hyperspectral classification method based on reconstructed convolutional Transformer according to claim 4, characterized in that, Output of the spectral correlation module Calculate as follows: ； ； Among them, the output of the spectral correlation module For 3D convolution The regularized output, For image mask; and Shared trainable parameters This is the output of the spectral correlation module.

6. The hyperspectral classification method based on reconstructed convolutional Transformer according to claim 5, characterized in that, Finally, the output of the spectral correlation module is... The classification results are obtained after processing with a linear layer.

Images