Adaptive kernel selection fusion Transform network and method for hyperspectral image classification

Abstract

The invention discloses a self-adaptive kernel selection fusion Transform network and method for hyperspectral image classification, and belongs to the technical field of hyperspectral image classification, and the network comprises three innovative modules: (1) a self-adaptive convolution module (ACBlock) improves the calculation efficiency while guaranteeing the feature extraction effect through a cascade structure of convolution and separable convolution; (2) a kernel selection fusion attention module (KSFA) adopts multi-scale convolution and dynamic attention fusion design, and feature enhancement and dynamic weighted integration are realized through a kernel selection strategy; and (3) the dynamic Tanh (DyT) replaces the traditional normalization layer, and two-dimensional convolution is introduced to compensate the information loss of the Transform layer. According to the innovative design, the AKSFormer keeps the advantages of Transform global modeling, and meanwhile, the problems that local feature extraction and small sample generalization are insufficient in ability and the like are effectively solved. Experiments on three common data sets and one homemade data set show that compared with the existing advanced method, the AKSFormer classification precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hyperspectral image classification, and particularly relates to an adaptive kernel selection fusion Transformer network and method for hyperspectral image classification. BACKGROUND

[0002] Hyperspectral remote sensing has become a research hotspot in the field of remote sensing due to its advantages of multiple bands, high resolution and continuous spectrum, and plays an important role in the fields of agriculture, ocean observation, environmental monitoring, address exploration, etc. With the launch of GF-5 in May 2018, China's spaceborne hyperspectral capabilities have made another leap. As a key component of China's high-resolution earth observation system, the satellite carries a 330-band visible and short-wave infrared hyperspectral imager, and the high-quality hyperspectral data acquired by it provides support for remote sensing scientific research.

[0003] Hyperspectral image classification, as a core link of remote sensing information extraction, its essence is to realize the accurate division of ground object classes at the pixel level through spectral feature analysis. Traditional hyperspectral image classification methods rely on hand-crafted features extracted from spectral and spatial information, such as principal component analysis (PCA), local binary pattern (LBP) and support vector machine (SVM) methods. Traditional methods are simple to train and require relatively less data, but they have limitations when facing the complex and high-dimensional characteristics of hyperspectral data, ignoring the spectral spatial dependence between adjacent pixels.

[0004] Compared with traditional methods, deep learning methods, especially convolutional neural networks (CNN), can learn hierarchical feature representations extracted from raw data, and have become one of the mainstream methods for hyperspectral classification. CNN improves classification performance by extracting spectral and spatial features between adjacent pixels, but CNN also has some limitations, such as the repeated use of convolution kernels in different spatial positions, which makes CNN have difficulty in capturing spatial diversity, especially in understanding long-distance associations between pixels, CNN does not have such good classification effect.

[0005] Unlike CNN, Transformer was mainly applied in the field of natural language processing before. The core of Transformer is self-attention mechanism, which can capture long-distance dependencies by processing input as a series of tokens. Based on this, some people designed Vision Transformer to process images. Although Transformer has shown strong ability in processing long-distance dependencies and complex feature extraction, it is weak in local detail extraction and cannot efficiently capture fine-grained local features like CNN. SUMMARY

[0006] The embodiment of the application provides a self-adaptive kernel selection fusion Transformer network and method for hyperspectral image classification, to solve the technical problems that local detail information is easy to be ignored in long sequence modeling of the Transformer itself, and the model generalization ability is insufficient in a small sample scene, and the content of the application is as follows:

[0007] The first object of the application is to provide a self-adaptive kernel selection fusion Transformer network for hyperspectral image classification, and the technical point is to include an adaptive convolution module, a kernel selection fusion attention module and a FlowFusion Transformer module.

[0008] The adaptive convolution module adopts a convolution main branch combined with 3DCNN and 2DCNN and a residual structure capable of autonomously judging channels, and combines residual connection design, to preliminarily extract spectral and spatial features of the hyperspectral image.

[0009] The kernel selection fusion attention module realizes enhancement and dynamic weighted integration of spectral and spatial features through a parallel multi-scale convolution branch and a dynamic attention fusion design.

[0010] The FlowFusion Transformer module adopts a dynamic Tanh (DyT) module to replace a traditional normalization layer, and introduces two-dimensional convolution compensation information loss, to extract global context information.

[0011] In order to better realize the above technical scheme, the residual connection of the adaptive convolution module in the self-adaptive kernel selection fusion Transformer network for hyperspectral image classification includes: when the number of input and output channels is consistent, an identity mapping is adopted as a fast connection; when the number of channels is inconsistent, a projection path containing a 1x1 convolution and batch normalization is used to adjust the dimension.

[0012] In order to better realize the above technical scheme, the mathematical expression of the adaptive convolution module in the self-adaptive kernel selection fusion Transformer network for hyperspectral image classification is as follows:

[0013] Y = ReLU [BN (2) (Conv 2d (ReLU(BN (1) (Conv 3d (X)))))+Γ(X)];

[0014] wherein Conv 3d represents a standard convolution operation of 3DCNN, BN (i)Batch Normalization, ReLU is a rectified linear unit activation function, and Gamma (X) is a dimension adaptation function, and the formula is:

[0015]

[0016] where C in and C out represent the number of channels of the input and output feature maps, respectively.

[0017] In order to better realize the above technical scheme, the kernel selection fusion attention module in the adaptive kernel selection fusion Transformer network for hyperspectral image classification comprises:

[0018] Parallel local receptive field and dilated convolution branch are used to capture spatial context information at different scales.

[0019] The spatial attention branch generates a position weight map through average pooling, maximum pooling and Sigmoid activation.

[0020] The channel attention branch generates a spectral selection mask through global average pooling, a fully connected layer and Softmax activation.

[0021] The kernel selection fusion strategy dynamically weights and fuses the features of the multi-scale convolution branch through the spectral mask.

[0022] In order to better realize the above technical scheme, the FlowFusion Transformer module in the adaptive kernel selection fusion Transformer network for hyperspectral image classification divides the input features into fixed size patches, and after linear mapping and position encoding, it enters the Transformer encoder, and combines the multi-head self-attention mechanism and the feedforward neural network to extract global features.

[0023] 1. The adaptive kernel selection fusion Transformer network for hyperspectral image classification according to claim 1, wherein the mathematical expression of the dynamic Tanh (DyT) module in the FlowFusion Transformer module is:

[0024] DyT (x) = gamma * tanh (alpha * x) + beta

[0025] Where alpha can dynamically adjust its parameters according to the size of the input x, and gamma and beta are consistent with the scaling and bias parameters in the normalization layer LayerNorm, which can be flexibly scaled to any value range.

[0026] The second objective of this invention is to provide a hyperspectral image classification method using an adaptive kernel selection fusion Transformer network, the technical point of which is that it includes the following steps:

[0027] The hyperspectral image is input into the adaptive convolution module to initially extract spectral and spatial features;

[0028] The initially extracted features are input into the kernel selection and fusion attention module to perform multi-scale convolution and dynamic attention fusion to enhance spectral spatial features;

[0029] The enhanced features are input into the FlowFusion Transformer module, and the DyT module is used to replace the normalization layer to extract global context information.

[0030] The output of the FlowFusion Transformer module is classified to obtain the predicted land cover categories of the hyperspectral image.

[0031] Compared with the prior art, the technical solution of the present invention can achieve the following beneficial effects:

[0032] 1. This invention proposes a novel adaptive and selective fusion Transformer network (AKSFormer), which can fully fuse the contextual information of spectral spatial features while reducing unnecessary information interference, and achieve accurate classification.

[0033] 2. To initially extract hyperspectral features, this application designs an ACBlock module. This module employs a residual design with main path and shortcut connections, perfectly solving the gradient vanishing problem and automatically handling changes in the number of channels to accelerate processing. This application also introduces a KSFA module, which achieves feature enhancement through a fusion design of multi-scale convolution and dynamic attention, and integrates them using kernel fusion.

[0034] 3. Replace the normalization layer with the DyT module and add two-dimensional convolutions to avoid information loss caused by too many Transformer layers.

[0035] 4. A new dataset was created, and the classification performance of the model was validated on both public and private datasets with good results. Attached Figure Description

[0036] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0037] Figure 1 This is a structural diagram of the AKSFormer model of the present invention;

[0038] Figure 2 This is the complete structure of the core selection fusion attention module of the present invention;

[0039] Figure 3 Transformer layer structure;

[0040] Figure 4 Schematic diagram of multi-head attention mechanism;

[0041] Figure 5 Pseudo-color images of the University of Pavia dataset;

[0042] Figure 6 Pseudo-color images from the IndianPine dataset;

[0043] Figure 7 Pseudo-color images of the Xuzhou dataset;

[0044] Figure 8 Pseudo-color images of the Wangshan Park dataset;

[0045] Figure 9 Classification results of the Pavia University dataset;

[0046] Figure 10 Classification results of the IndianPine dataset;

[0047] Figure 11 The classification results of the Xuzhou dataset;

[0048] Figure 12 Classification results of the Wangshan Park dataset;

[0049] Figure 13 The impact of training sample size on accuracy;

[0050] Figure 14 A graph showing the impact of the number of attention heads on overall accuracy.

[0051] Figure 15 Visualization of the impact of input dimensions on accuracy and time;

[0052] Figure 16 Accuracy curves of the proposed model on various datasets;

[0053] Figure 17 Loss plots for each dataset of the proposed model. Detailed Implementation

[0054] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in connection with the embodiments thereof and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0055] 1. Residual network

[0056] Generally, the increase of network model depth can extract deeper features to improve network performance, but this operation may also cause the possibility of model gradient vanishing or explosion. In order to solve this problem, He et al. introduced the idea of residual learning by adding "skip connection" in the network, which allows the network to directly pass the input signal to the deeper layer without going through the nonlinear transformation of each layer, thereby helping the network to learn the residual mapping, i.e. the difference between the input and the output.

[0057] The basic structure of the residual network is composed of input X, convolution operation F and final output Y, and the formula is as follows:

[0058] Y = F(X) + X

[0059] This way can make the network easier to optimize, even if the transformation of a certain layer is not ideal, the network can still pass the signal to the next layer through the skip connection to avoid the problem of gradient vanishing.

[0060] 2. Normalization layer

[0061] The internal covariate shift in the training of neural network will cause the problem of training instability, and the normalization layer is proposed by Jimmy Ba et al. in 2016 to solve this problem, which normalizes the feature dimension of each sample to make the Transformer process variable length sequences and accelerate the training process.

[0062] The formula of layer normalization is as follows:

[0063]

[0064] Where x is the feature vector of the input sample, μ is the mean of the input feature vector, σ 2 is the variance of the input feature vector, is a constant to prevent division by zero error, and γ and β are learnable parameters for scaling and shifting, respectively.

[0065] Figure 1The hyperspectral classification model is proposed in the application, and the proposed hyperspectral classification model is described in detail in this section. Three modules are introduced, namely an adaptive convolution module, a kernel selection and fusion attention module, and a FlowFusionTransformer.

[0066] 3. Adaptive convolution module

[0067] The adaptive convolution module designed by the application is a lightweight convolution module specially designed for hyperspectral image processing. Its main function is to balance the computational efficiency and feature extraction capability, preliminarily extract spectral and spatial features, and simplify the data to facilitate further extraction by the next module. This module innovatively uses a 3x3 convolution and a depth separable cascade structure. The former is responsible for capturing local spatial features, and the latter innovatively judges the input and output channels. When the channels are consistent, the residual network design idea is used for identity mapping as a fast connection. When the channels are inconsistent, the conditional branch automatically constructs a projection path containing a 1x1 convolution and a batch normalization to adaptively adjust the dimension of the shortcut connection, ensuring consistency between the main branch and the residual branch when the features are fused. Compared with the fixed projection connection scheme, this design can save computational efficiency under the premise of preliminary feature extraction.

[0068] The mathematical expression of the adaptive convolution module is as follows:

[0069] Y = ReLU [BN (2) (Conv 2d (ReLU(BN (1) (Conv 3d (X)))))+Γ(X)]

[0070] where Conv 3d is a standard convolution operation of 3DCNN, BN (i) represents a batch normalization processing layer, ReLU is a rectified linear unit activation function, and Γ(X) is a dimension adaptation function, and the formula is:

[0071]

[0072] where C in and C out represent the number of channels of the input and output feature maps, respectively.

[0073] The design of this module can reduce the computational complexity from the traditional O(C in C out HWK 2 ) to O(C in C out HW+C out HWK 2), which is particularly suitable for the preliminary extraction of hyperspectral feature data, saving the efficiency of calculation.

[0074] 4. Kernel selection fusion attention module

[0075] The kernel selection fusion attention module cited in the present application is designed through the cooperation of multi-scale convolution path and dynamic attention fusion to realize the enhancement of spectral spatial features. The module uses parallel local receptive field and dilated convolution branch to capture spatial context information of different scales and adaptively calibrates the importance distribution of spectral dimension through channel attention mechanism. Finally, the kernel selection fusion strategy is used to dynamically weight and integrate the features.

[0076] The complete structure of the kernel selection fusion attention module is shown in Figure 2 .

[0077] Considering that different receptive fields have different abilities to extract features, the module introduces two decoupled deep convolution F 1 ,F 2 to model local details and global context in parallel. The given feature input is The formulas of the two outputs are as follows:

[0078]

[0079] Where d represents the down-sampling rate.

[0080] In order to highlight the key areas in space, the module introduces a spatial attention branch to generate a position weight map. V1 and V2 obtained previously are mapped to a unified dimension through linear mapping, and the formula is as follows:

[0081] U1 = W1 * V1, U2 = W2 * V2

[0082] The fusion of the two can be represented as:

[0083]

[0084] Then average pooling and max pooling are used to compress U, and the formulas of the two are as follows:

[0085]

[0086] After splicing, the two are mapped to O spatial attention maps, and the formula is as follows:

[0087] Z = Conv 1×1 (Concat(S avg ,S max ))

[0088] B i = σ(Z :,:,i ), i ∈ {1, 2}

[0089] where σ(·) is the Sigmoid activation, and the two convolution branches correspond to two sets of spatial masks B1 and B2.

[0090] Since different land cover types have different dependencies on spectral dimensions in hyperspectral images, the fused feature maps are compressed by channel-wise global average pooling, and two fully connected layers are used to model spectral attention, denoted as:

[0091]

[0092] where δ(·) represents the ReLU activation, and O is the number of spectral selection channels. Then the softmax function is used to normalize the attention vector to obtain the spectral selection mask:

[0093]

[0094] After assigning the selection weights to the spectral mask, the two branch features B1 and B2 are multiplied and fused, as follows:

[0095]

[0096] where ⊙ represents element-wise multiplication, i and w is applied to all spatial positions and channels. Then the two enhanced features are added to obtain the fused feature:

[0097] S = F1 + F2

[0098] To ensure smooth information flow, residual connections are introduced to add the fused feature S to the input feature Y, and the final output is:

[0099] Y ′ = Y + S

[0100] 5. FlowFusion Transformer

[0101] The Transformer builds a bottom-up hierarchical structure, with each layer processing features of different scales, as shown in Figure 3 .

[0102] This application uses dynamic Tanh (DyT) instead of the normalization layer, in order to achieve and exceed the effect of normalization without using the normalization layer. The formula is as follows:

[0103] DyT(x) = γ * tanh(αx) + β

[0104] Where a can dynamically adjust its parameters according to the input x size, and g and b can be flexibly scaled to any value range as the scaling and bias parameters in the normalization layer LayerNorm.

[0105] The pseudo-code implementation of DyT is shown as follows.

[0106]

[0107]

[0108] After that, the input features are divided into windows of different sizes, and each window is calculated separately for self-attention. Given the input feature The Transformer calculates the attention within each window by dividing the input into local regions of window size:

[0109]

[0110] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively, d k is the dimension of the key. The division of the local window makes each attention calculation only within the window, reducing the complexity. The core of the Transformer module is the multi-head attention mechanism, as shown in Figure 4 .

[0111] The computational complexity of multi-head attention is:

[0112] Ω(MSA) = 4HWC 2 + 2(HW) 2 C

[0113] The FlowFusion Transformer designed in this application is composed of multiple Transformer layers, each of which contains a multi-head attention mechanism and a feedforward neural network, effectively extracting the context information of the feature map and enhancing the multi-level semantic fusion expression.

[0114] To give the Transformer the ability to perceive the positions in the image sequence, the input features need to be processed by a specific embedding fusion before entering the Transformer layer. The feature map is divided into fixed-size patches, and each patch is flattened and mapped to a high-dimensional representation space through a linear transformation to obtain the initial input feature T in , which is in sequence form. Then a learnable classification label vector is introduced and added element-wise to the learnable position encoding PE pos to form the final input of the Transformer encoder, as shown below:

[0115]

[0116] For input features X, the processing procedure of the l-th layer can be represented as:

[0117]

[0118] where MSA (l) denotes the multi-head self-attention mechanism, is the operation of the DyT layer. To further enhance the feature representation, the outputs of the first three Transformer layers are concatenated with the original input to form a multi-scale representation. The fused features are then rearranged and input into a 1x5 convolutional kernel to learn the dependencies between layers and reduce the information loss of the previous Transformer layers. The formula is as follows:

[0119] X fused =Conv 1×5 (Concat(X,X (1) ,X (2) ,X (3) ));

[0120] After fusion, a Transformer layer is used again, and the output is as follows:

[0121]

[0122] Experimental Example

[0123] This experimental example mainly introduces three public data sets and one self-made data set, namely the University of Pavia data set, the Indian Pine data set, the Xuzhou data set, and the Wangshan Park data set, and describes the experimental setup and evaluation indicators. Seven deep learning models are compared with the proposed model to evaluate the classification results. In addition, the influence of different parameters on the experimental results is analyzed. Finally, the influence of the added module on the experimental accuracy is analyzed through the ablation experiment.

[0124] 1. Data set description

[0125] (1) The University of Pavia data was obtained by the German airborne reflective optical spectral imaging system ROSIS-03 in 2003, which is part of the hyperspectral image of Pavia City, Italy. The spectral imager continuously images 115 bands in the wavelength range of 0.43-0.86 μm, with a spatial resolution of 1.3 m. Of these, 12 bands are eliminated due to noise, so the images of the remaining 103 spectral bands are generally used. The size of the data is 610x340, so a total of 2207400 pixels are included, but it contains a large number of background pixels, while the total number of pixels containing surface objects is only 42776, which includes 9 types of surface objects, including trees, asphalt, bricks, grass, etc.Figure 5 The training and testing samples are shown in Table 1.

[0126] Table 1. Number of labeled training and testing samples in Pavia University dataset

[0127]

[0128] (2) The IndianPine dataset was collected in 1992 at the Indian Pine Test Site in Northwest Indiana, using the AVIRIS device, with a size of 145x145, 224 bands, containing 16 categories such as forest, vegetation, housing, etc., and the pseudo-color map is as shown in Figure 6 The training and testing samples are shown in Table 2.

[0129] Table 2. Number of labeled training and testing samples in IndianPine dataset

[0130]

[0131] (3) The Xuzhou dataset is located at a suburban site in Xuzhou City, collected by an airborne HYSPEX hyperspectral camera in November 2014. The size of the dataset is 260x500, with 436 bands. The spatial resolution is 0.73m. The dataset has 9 categories, including crops, vegetation, man-made structures, and coalfields, as shown in Figure 7 The training and testing samples are shown in Table 3.

[0132] Table 3. Number of labeled training and testing samples in Xuzhou dataset

[0133]

[0134] (4) The Wangshan Park dataset is a self-made dataset located in Wangshan Park near Zilang Lake in Nantong City, Jiangsu Province, China. The data was collected by a DJI hyperspectral unmanned aerial vehicle, with a shooting time of December 2024. The self-made dataset has a size of 963x788, with 123 bands, and a spatial resolution of 5cm. According to field investigation, the dataset includes 14 categories, marked by flower leaf greenwood, southern bamboo, magnolia, deciduous shrubs, etc. The pseudo-color map is as shown in Figure 8 The training and testing samples are shown in Table 4.

[0135] Table 4. Number of labeled training and testing samples in Wangshan Park dataset

[0136]

[0137] 2. Parameter settings

[0138] (1) Parameter details

[0139] The proposed model is implemented on an Ubuntu server with Intel Gold 6240x 2-GPU768GB memory, two NVIDIA RTX6000 GPUs. In this experiment, the learning rate is set to 1e^-4, the ratio of training set to test set is 1:9, and the Adam optimizer is used for training. Except for the IndianPine dataset which is trained for 100 epochs, the proposed network is trained for 50 epochs for other datasets.

[0140] (2) Evaluation index

[0141] The overall accuracy (OA), average accuracy (AA), Kappa coefficient (K) and confusion matrix are used to evaluate the classification performance of all methods.

[0142] The overall accuracy is the proportion of correctly classified samples to the total samples. The formula is:

[0143]

[0144] Where TP, TN, FP and FP represent true positive samples, true negative samples, false positive samples and false negative samples, respectively.

[0145] The average accuracy is the average accuracy of each class, which reflects the classification effect of each class. The formula is:

[0146]

[0147] Where N is the number of classes, TP i is the sum of true and false positive cases of the ith class.

[0148] The Kappa coefficient is used to measure the consistency of the classification results and the random classification results. The formula is:

[0149]

[0150] Where P o represents the observed accuracy, and P e represents the expected random accuracy.

[0151] 3、Results and discussion

[0152] In order to test the effectiveness of the proposed model, several advanced deep learning models are used as comparison, including: FullycontNet, ChostNet, HybridSN, VIT, SSRNet, UNet and SynFormer. In order to ensure the fairness of the experiment, the training sample ratio and input block size of other deep learning models are set to be the same as this model.

[0153] (1) Quantitative analysis

[0154] The records of the Pavia University dataset, the Indian Pine dataset, the Xuzhou dataset, and the Wangshan Park dataset are shown in Tables 5 to 8, and the results of the experiment have the f1-score of each category, the overall accuracy, the average accuracy, and the Kappa coefficient (K), wherein the bold part represents the method with the best accuracy of the same category. As can be seen from the table, the model proposed in the present application has more bold results in each dataset, because FullyContNet and GhostNet are models that rely purely on convolutional structures, and their limited feature receptive field and weak feature fusion capability lead to the inability to establish long spectral spatial dependency, and the performance of the heterogeneous features such as the Red-title classification in the table is insufficient, wherein FullyContNet is 78.21%, and GhostNet is 94.60%; and for the HybridSN and SSRN models, although 3D convolution is introduced to capture spectral dimension features, due to the lack of a dynamic feature selection mechanism, the performance fluctuates greatly on small sample categories, as shown in Table 5 for Trees, HybridSN is 86.43%, and SSRN is 83.83%; VIT is a pure Transformer architecture and is not sensitive enough to local spatial features, and in the texture complex category such as Bare Soil in Table 5, it is only 56.17%, and UNet has information loss in deep feature transmission due to the encoding and decoding structure, resulting in poor category consistency, such as Gravel in Table 5, which is only 7.41%. SynFormer has better performance, and in some categories, the classification effect is even better than the model proposed in the present application, but it does not use adaptive multi-scale feature interaction, and there is still room for improvement in the overall classification.

[0155] The model proposed in the present application uses an adaptive convolution module and a kernel selective fusion attention module to extract the features of hyperspectral, the former adaptively adjusts the feature dimension of the spectrum and space, simplifies the data and facilitates further extraction by the next module, and the latter realizes the feature enhancement of the spectrum and space through the collaborative design of the multi-scale convolution path and the dynamic attention fusion, and then the designed FlowFusion Transformer effectively extracts the context information of the feature map and enhances the multi-level semantic fusion expression, realizing the effective combination of CNN and Transformer. Through comparison with each experiment, except that the average accuracy on the Indian Pine dataset does not achieve the best result, the present application confirms that the model proposed in the present application has better classification effect on each dataset.

[0156] Table 5. Results of different deep learning methods on the Pavia University dataset, including OA, AA, and KAPPA.

[0157]

[0158]

[0159] Table 6. Results of different deep learning methods on the IndianPine dataset, including OA, AA, and KAPPA

[0160]

[0161] Table 7. Results of different deep learning methods on the Xuzhou dataset, including OA, AA, and KAPPA

[0162]

[0163] Table 8. Results of different deep learning methods on the Wangshan Park dataset, including OA, AA, and KAPPA

[0164]

[0165]

[0166] (2) Visual Analysis

[0167] Figures 9-12 This paper presents a comparison of the classification performance of various models on different datasets. Sub-image (a) is a pseudo-color image, (b) is the ground truth image, and the remaining images (c)-(i) correspond to the classification results of the FullycontNet, HostNet, HybridSN, VIT, SSRNet, UNet, and SynFormer deep learning models, respectively. Image (j) is the model proposed in this application. A specific area is highlighted in the lower right corner for comparison.

[0168] For the Pavia University dataset, while FullycontNet can roughly distinguish the regions of each major category, its performance in handling boundary details is mediocre. Some of the dark green Self-Blocking Bricks are incorrectly predicted as dark blue Gravel. The HostNet, HybridSN, VIT, and UNet models all exhibit significant class confusion in the magnified regions, indicating that these models have certain issues with handling small structures and edge categories. Although SSRNet is consistent with the ground truth in the magnified regions and has the best prediction, its weak ability to distinguish some categories leads to the incorrect prediction of yellow Trees in a large area of ​​light green Meadows, resulting in an overall unsatisfactory classification performance. SynFormer further integrates the advantages of convolution and Transformer, achieving good classification accuracy and performance, but it still suffers from detail loss in local regions and incomplete recognition of small sample categories, making it inferior to the model proposed in this application in this respect. On the IndianPine dataset, apart from the SynFormer model, other models exhibited excessive noise in their classification within the magnified bounding box. Some models even showed more noticeable class overlap in other categories. SynFormer and the model proposed in this application performed well on the IndianPine dataset, making it difficult to intuitively determine which model yielded the best result. This classification effect was even more pronounced on the Xuzhou dataset. The magnified region consisted of Bareland1 and Crops-1, and other models predicted the dark red Bareland1 poorly. Only the model proposed in this application remained consistent with the ground truth map. For the dataset created in this application, the magnified image identified two categories: Lotus magnolia at the top and magnolia at the bottom. Only SynFormer and the model proposed in this application were completely consistent with the ground truth map. Other models provided inconsistent predictions for Lotus magnolia at the top and incorrectly predicted Cinnamomumcamphor for the magnolia portion at the bottom. These models failed to fully learn the spectral and spatial features of the two tree species, leading to classification errors. SynFormer's error stemmed from incomplete feature extraction of small sample classes, causing it to predict "way" instead of "branch."

[0169] In these datasets, the small land features in this application are clearly segmented with clear boundary contours and no obvious breaks or confusion, demonstrating excellent small sample recognition ability and fine-grained feature modeling ability.

[0170] (3) The impact of training sample size on accuracy

[0171] To verify the impact of the ratio of training samples on classification accuracy, this application uses the Pavia University dataset as the training samples. Figure 13 As shown in Figures (a), (b), and (c), they represent the OA, AA, and Kappa results for different training models, respectively. It is evident from the figures that when the training sample ratio is only 2%, most deep learning models exhibit low classification accuracy. Only SynFormer and the model in this application achieve approximately 90% classification accuracy, demonstrating their high feature extraction and learning capabilities with a small number of samples. The other models fall short. As the ratio increases, the model's classification accuracy improves with a larger training sample ratio. For most models, a 10% training sample ratio yields the best accuracy. However, some models, like FullyContNet, achieve better classification accuracy with 8% than with 10%. To ensure consistency across most learning models, this application selects 10% as the training sample ratio.

[0172] (4) The impact of the number of attention points on overall accuracy

[0173] Table 9 below and Figure 14 This table shows the model's performance on different datasets using different numbers of attention heads. The data presented is the overall accuracy, with the bolded portion representing the best results for that dataset. As the table shows, the model's accuracy generally improves with increasing the number of attention heads, until it reaches its optimal performance. On most datasets, 8 attention heads are the best performing option; further increases in attention heads have less noticeable effects and may even slightly decrease the accuracy. Therefore, the number of attention heads is set to 8.

[0174] Table 9. Effect of different numbers of attention heads on classification accuracy

[0175]

[0176] (5) The impact of input size on accuracy and time

[0177] This section primarily explores the impact of input size on model classification accuracy. Using the Pavia University dataset again, with other parameters remaining constant, this application evaluates the model's performance on the Pavia University dataset using different image patch sizes (3×3, 7×7, 11×11, 15×15, and 19×19) and records the training and testing times. The results are shown in Table 10 below. Figure 15 As shown.

[0178] As shown in the table, the spectral and spatial information provided by the 3×3 image patch is relatively limited, resulting in low accuracy because the model cannot make accurate judgments from insufficient data. However, for the model proposed in this application, increasing the size means extracting more features and utilizing more information. Multi-scale feature fusion cannot always achieve better results by simply increasing the size. The figure intuitively reflects the performance results of the model, where (a) is the training time and (b) is the testing time. The different training colors and the size of the points also represent the length of time. The closer to the upper right corner, the better the classification performance of the model. The best classification performance is achieved with an 11×11 size. Therefore, this application uniformly sets the size to 11×11 in the experiments to obtain better classification results.

[0179] Table 10. Results of the impact of input size on classification performance

[0180]

[0181] (6) Accuracy curve and loss graph

[0182] Since this application uses DyT to replace the LayerNorm layer, and LayerNorm accelerates model convergence, improves stability, and enhances generalization ability, the effectiveness of DyT as a replacement for the normalization layer is highly tested. This application conducted four sets of experiments, with subscripts a, b, c, and d representing the four datasets: Pavia University, IndianPine, Xuzhou, and Wangshan Park. The first set shows the accuracy curves for the four datasets, and the second set shows the visualization of the loss values, as shown below. Figure 16 , 17 As shown, with increasing training epochs, both the accuracy curve and the loss graph eventually stabilize, except for slight fluctuations on the IndianPine dataset. This is because DyT can automatically adjust the input activation values ​​using a scaling factor α and compress them using an S-shaped Tanh function. This simple function effectively aligns with the effect of LayerNorm, making it a viable alternative.

[0183] (7) Ablation test

[0184] To verify the effectiveness of each module and DyT, this application designed five sets of comparative experiments using the IndianPine dataset, with FlowFusion Transformer as the baseline model. In the first set of experiments, this application only arranged the baseline model. The second to fourth sets of experiments investigated the impact of each module on classification accuracy. This application also separately set up a fifth set of experiments to confirm the experimental results of DyT and the normalization layer, as shown in Table 11.

[0185] When using only the baseline model, with only four Transformer layers to extract features, the Transformer cannot extract rich hyperspectral features, especially local details of hyperspectral images, resulting in less than ideal results. Using the ACBlock and KSFA modules can significantly improve the model's classification accuracy. However, the improvement effect of the ACBlock module is not as significant as that of the KSFA module. This is because the ACBlock module only performs preliminary extraction of hyperspectral data, unlike the KSFA module, which enhances the features of spectral spatial data through multi-scale kernel selection.

[0186] The experiments in groups 4 and 5 show that DyT can be used as a module to replace LayerNorm. Although OA, AA, and KAPPA only improved performance by 0.33%, 0.65%, and 0.43% respectively, such a module can refresh people's understanding of the indispensability of normalization layers in modern neural networks.

[0187] Table 11. Ablation Test

[0188]

[0189] 4. Conclusion

[0190] This invention proposes the AKSFormer model for hyperspectral classification, which fully extracts spectral spatial information. Specifically, it consists of three modules: the ACBlock module primarily extracts initial spectral spatial features, simplifying the computation process; the KSFA module uses parallel-designed local receptive fields and dilated convolutional paths to extract spatial context features and adaptively optimizes spectral feature weights using a channel attention mechanism; finally, a dynamic kernel selection fusion method is used to achieve adaptive weighted fusion of features. These fused features are then fed into the FlowFusion Transformer for classification. The FlowFusion Transformer uses DyT to replace the normalization layer for better results and introduces two-dimensional convolutions to reduce information loss caused by excessive Transformer layers. Experiments on three public datasets and one private dataset show good classification results, validating the effectiveness of the AKSFormer model.

[0191] The AKSFormer model demonstrates powerful feature extraction and classification capabilities in hyperspectral classification. However, it also has some limitations. For example, its training and testing times are still significantly longer compared to simpler deep learning models. This is because the model employs a complex multi-branch parallel structure, which increases the overhead of parameter computation. Furthermore, the dynamic feature fusion and DyT modules require real-time weight calculation and adjustment during runtime. These design features, while improving model performance, also lead to higher computation time. Future work in this application will further explore how to reduce time complexity while maintaining hyperspectral classification performance using the Transformer as a baseline model, designing a simple and efficient model.

[0192] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. An adaptive kernel selection fusion Transformer network for hyperspectral image classification, characterized in that, It includes an adaptive convolution module, a kernel selection fusion attention module, and a FlowFusionTransformer module; The convolutional main branch combining 3DCNN and 2DCNN, along with the residual structure that can autonomously determine channels and the residual connection design, is used to initially extract the spectral and spatial features of hyperspectral images. The kernel selection fusion attention module achieves enhancement and dynamic weighted integration of spectral spatial features through parallel multi-scale convolutional branches and dynamic attention fusion design. The FlowFusionTransformer module replaces the traditional normalization layer with a dynamic Tanh (DyT) module and introduces two-dimensional convolution to compensate for information loss, in order to extract global context information.

2. The adaptive kernel selection fusion Transformer network for hyperspectral image classification according to claim 1, characterized in that, The residual connections of the adaptive convolution module include: when the number of input and output channels is the same, identity mapping is used as a fast connection; when the number of channels is different, the dimension is adjusted by a projection path containing 1×1 convolution and batch normalization.

3. The adaptive kernel selection fusion Transformer network for hyperspectral image classification according to claim 1, characterized in that, The mathematical expression for the adaptive convolution module is as follows: Y=ReLU[BN (2) (Conv. 2d (ReLU(BN (1) (Conv. 3d (X)))))+Γ(X)]; Where Conv 3d This is represented as the standard convolution operation in 3DCNN, BN. (i) Represents the batch normalization layer, ReLU is the modified linear unit activation function, and Γ(X) is the dimension adaptation function, as shown in the formula: Where C in and C out These represent the number of channels in the input and output feature maps, respectively.

4. The adaptive kernel selection fusion Transformer network for hyperspectral image classification according to claim 1, characterized in that, The kernel selection fusion attention module includes: Parallel local receptive fields and dilated convolution branches are used to capture spatial context information at different scales; The spatial attention branch generates a location weight map through average pooling, max pooling, and sigmoid activation; The channel attention branch generates a spectral selection mask through global average pooling, a fully connected layer, and Softmax activation. The kernel-selective fusion strategy dynamically weights and fuses features from multi-scale convolutional branches using spectral masks.

5. The adaptive kernel selection fusion Transformer network for hyperspectral image classification according to claim 1, characterized in that, The FlowFusion Transformer module divides the input features into fixed-size patches, which are then linearly mapped and positionally encoded before entering the Transformer encoder. It then combines a multi-head self-attention mechanism and a feedforward neural network to extract global features.

6. The adaptive kernel selection fusion Transformer network for hyperspectral image classification according to claim 5, characterized in that, The mathematical expression for the dynamic Tanh(DyT) module in the FlowFusion Transformer module is: DyT(x)=γ*tanh(αx)+β α can dynamically adjust its own parameters according to the size of the input x, while γ and β are consistent with the scaling and bias parameters in the normalization layer LayerNorm and can be flexibly scaled to any numerical range.

7. A hyperspectral image classification method based on the adaptive kernel selection fusion Transformer network for hyperspectral image classification as described in claims 1-6, characterized in that, Includes the following steps: The hyperspectral image is input into the adaptive convolution module to initially extract spectral and spatial features; The initially extracted features are input into the kernel selection and fusion attention module to perform multi-scale convolution and dynamic attention fusion to enhance spectral spatial features; The enhanced features are input into the FlowFusion Transformer module, and the DyT module is used to replace the normalization layer to extract global context information. The output of the FlowFusion Transformer module is classified to obtain the predicted land cover categories of the hyperspectral image.

Images