A lightweight hyperspectral image classification method based on multi-scale attention

Abstract

The present invention proposes a lightweight hyperspectral image classification method based on multi-scale attention, comprising: step 1: first inputting HSI data into a convolutional layer for preprocessing, and performing preliminary feature extraction and dimensionality reduction on the input HSI data through preprocessing; step 2: inputting the preprocessed data into a first branch; step 3: inputting the preprocessed data into a second branch; step 4: performing feature fusion on the outputs of steps 1, 2 and 3 and inputting the fusion result into a dynamic grouping convolutional layer; the present invention effectively considers the changes in the spatial scale of the objects, makes full use of the spectral spatial characteristics, reduces the spectral differences and the lack of spatial context information, and improves the classification accuracy.

Description

Technical Field

[0001] The present invention relates to the field of hyperspectral image classification, and in particular to a lightweight hyperspectral image classification method based on multi-scale attention. Background Art

[0002] Deep learning has been successfully applied to hyperspectral image classification and has achieved significant performance improvements. In particular, methods based on convolutional neural networks (CNNs) have made significant progress in this field. However, existing CNN-based hyperspectral image (HSI) classification methods have some shortcomings: (1) There is a lack of efficient and simple feature representation methods in CNNs, and the influence of spectral differences and spatial context information is not taken into account; (2) The model has a large number of training parameters and high computational complexity, resulting in high network complexity. (3) The category samples in the HSI dataset have a serious long-tail distribution phenomenon, which affects the HSI classification performance. Summary of the Invention

[0003] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes a lightweight hyperspectral image classification method based on multi-scale attention.

[0004] In order to achieve the above-mentioned object of the present invention, the present invention provides a lightweight hyperspectral image classification method based on multi-scale attention, comprising:

[0005] Step 1: First, the HSI data is input into the convolutional layer for preprocessing. Preprocessing can perform preliminary feature extraction and dimensionality reduction on the input HSI data.

[0006] Step 2: The preprocessed data is input into the first branch, which includes the MsCA module and the grouped convolutional layer. The preprocessed data passes through the MsCA module to selectively enhance or suppress spectral information through channel weights; then, the output features of the MsCA module are input into the grouped convolutional layer for feature extraction;

[0007] Step 3: The preprocessed data is input into the second branch, which includes a PAM module, an SFEF module, and a grouped convolutional layer. The preprocessed data passes through the PAM module, which highlights important spectral features through position features and reduces unnecessary redundant information. The output of the PAM module is then input into the SFEF module to further refine the spectral and spatial features of the data and enhance the feature extraction capability. Finally, the output feature map of the SFEF module is input into the grouped convolutional layer, which enables the model to better extract spectral information while reducing the parameters of the convolutional neural network.

[0008] Step 4: The outputs of Step 1, Step 2, and Step 3 are subjected to feature fusion and the fusion results are input into the dynamic grouping convolution layer. The convolution kernel used in this layer will change with the number of hyperspectral data bands to adapt to different HSI data.

[0009] Step 5: The feature map obtained in step 4 is reduced in dimension through adaptive average pooling, and finally the classification result is output through the fully connected layer.

[0010] Furthermore, the pre-processed data is subjected to the MsCA module to selectively enhance or suppress spectral information through channel weights, comprising the following steps:

[0011] S1-1, the pre-processed HSI data is processed by adaptive average pooling to obtain a one-dimensional spectral channel vector

[0012]

[0013] in: Indicates the current channel number, C indicates the total number of channels;

[0014] is a one-dimensional spectral channel vector;

[0015] Represents the one-dimensional spectral channel vector under the current channel;

[0016] H o Represents the length of the output tensor after the adaptive average pooling operation;

[0017] W o Indicates the width of the output tensor after the adaptive average pooling operation;

[0018] The convolution kernel and step size of the adaptive average pooling layer are obtained by the following formula:

[0019]

[0020]

[0021] stride indicates the step length;

[0022] floor(.) means round down;

[0023] Indicates the size of the output data space H o ×W o ;

[0024] U s Indicates the spatial size of the input data H×W;

[0025] kernel s represents the convolution kernel;

[0026] S1-2, set a multi-scale kernel to capture the relevant features of different distances, and define a set of multi-scale convolution kernels D = {D1, D2, D3, ..., D K′}, the data obtained by the adaptive average pooling layer After convolution with multi-scale convolution kernels respectively, the extracted multi-scale features are fused to obtain the given C k ×C k ×C k The kernel size is used to extract the adjacent k Spectral correlation characteristics of T k , the formula is as follows:

[0027]

[0028] Where: ⊙ represents the convolution operation;

[0029] represents a one-dimensional spectral channel vector;

[0030] D k represents the convolution kernel;

[0031] Represents the output data calculated by the adaptive average pooling layer;

[0032] Denotes the convolution kernel Dk subconvolution;

[0033] C represents the total number of channels;

[0034] S1-3, will have adjacent distance C k Spectral correlation characteristics of T k Fusion, fusion feature T s It is expressed as follows:

[0035]

[0036] Indicates the Spectral correlation features under convolution kernels T k ;

[0037] K′ represents the total amount of convolution kernel;

[0038] S1-4, then the fused features T s Input into the fully connected layer with nonlinear mapping to obtain the channel mask τ, the formula is as follows:

[0039] τ=σ(W2×δ(W1×T s )) (6)

[0040] Where: σ represents the ReLu function;

[0041] δ represents the Sigmoid function;

[0042] W1 and W2 are weights;

[0043] S1-5, readjust the input data U of the MsCA module, and the readjusted data is U new , the adjustment strategy is as follows:

[0044]

[0045] Indicates the channel masks on the channels,

[0046] Indicates the input data channels.

[0047] Through the MsCA module, the network can calculate a channel weight τ based on the input data U and assign the channel weight to the data U. Different input data have different weights, and the channel weights of multiple input data can be adaptively adjusted.

[0048] Furthermore, the pre-processed data is processed by a PAM module to highlight important spectral features through position features and reduce unnecessary redundant information, including the following steps:

[0049] First, the input feature map U∈R is transformed through a 1x1 convolution {C×H×W} Mapped to three feature maps Q, K, V, respectively, ({Q, K, V}∈R C×H×W ), the shapes of Q, K, and V are C×H×W, and then reshaped to C×g, where g=H×W, g is the number of pixels, and the specific formula is as follows:

[0050]

[0051] Where: ⊙ represents the convolution operation;

[0052] Δ q , Δ k , Δ v They are 1×1 convolution kernels respectively;

[0053] Next, multiply the transpose of Q by K and perform softmax normalization to obtain the attention weight A, A∈R {g ×g} The specific formula is as follows:

[0054]

[0055] Among them: A j，i is each element of the attention weight A, which represents the influence of position i on position j;

[0056] K j represents the j-th element of K;

[0057] Q T i represents the i-th element after transposing Q;

[0058] Then, multiply the feature V by the attention weight A and then multiply it by the scale coefficient γ to obtain the weighted eigenvalue matrix Z. The specific formula is as follows:

[0059]

[0060] V i is the i-th element of V;

[0061] Z j is the j-th element of the eigenvalue matrix Z;

[0062] γ is the scale factor, initialized to 0;

[0063] Finally, the weighted eigenvalue matrix Z is reshaped and concatenated with the original eigenvalue matrix U to obtain the final output feature map

[0064]

[0065] Among them: [;] represents splicing;

[0066] U j Represents the j-th element of the input data U.

[0067] PAM calculates the attention weight for each position of the input feature map, thereby performing weighted fusion of features at different positions to obtain a more accurate feature representation, learn features at different positions, better understand image content, and improve classification accuracy and stability.

[0068] Furthermore, the step of inputting the output of the PAM module into the SFEF module to further refine the spectral and spatial characteristics of the data includes:

[0069] The features of all layers of the SFEF module are circulated with the help of feedback connections and residual connections. Therefore, each layer now obtains the output features from other layers as its input in turn; formulas (12) and (13) represent the input and output formulas of the SFEF module respectively:

[0070]

[0071]

[0072] Where: C m Represents the output obtained after the mth convolution training;

[0073] Indicates the output obtained after passing through the PAM module;

[0074] Represents the ReLU activation function;

[0075] α represents the batch normalization layer;

[0076] ⊙ represents convolution operation;

[0077] X m+1 Represents the input of the m+1th convolution;

[0078] W and B represent the corresponding weights and biases;

[0079] {} i Indicates the index of the sample.

[0080] The SFEF module combines feedback connection, residual connection and feature fusion method. The module can extract multi-scale spectral features from hyperspectral data and improve the ability to understand the data through feedback mechanism and residual connection.

[0081] Furthermore, the calculation formula of the group convolution is as follows:

[0082]

[0083] Where: w, h and c are the height, width and number of channels of the convolution kernel respectively;

[0084] C i Indicates the number of channels of the input packet convolution layer data;

[0085] C o Indicates the number of channels of output data from the grouped convolutional layer;

[0086] β is the number of groups to be grouped;

[0087] Moreover, after the group convolution, the data is subjected to batch normalization and ReLU activation function in sequence.

[0088] By using grouped convolution instead of traditional convolution, the number of parameters in the convolutional neural network is reduced, speeding up model training. Specifically, grouped convolution divides the input feature map into groups, performs an independent convolution operation on each group of feature maps, and finally concatenates the results of all groups. This can reduce the number of model parameters to a certain extent while better capturing the characteristic information of the spectrum. In addition, the combination of grouped convolution + BN + ReLU for feature extraction can better extract spectral information.

[0089] Furthermore, it also includes the loss function:

[0090] FL(P, Y) = -π(1-P) λ ×Ylog(P)-(1-π)P λ ×(1-Y)log(1-P) (19)

[0091] Where: π is the weighting parameter for adjusting the number of samples;

[0092] P represents the predicted probability of the model;

[0093] Y is the probability of being true;

[0094] λ is the adjustment factor for adjusting the weight of difficult and easy samples.

[0095] In summary, by adopting the above technical solutions, the present invention effectively considers the changes in the spatial scale of the ground objects, fully utilizes the spectral spatial characteristics, alleviates the lack of spectral differences and spatial context information, and improves classification accuracy. In addition, while reducing network complexity, it does not reduce classification performance and can handle the sample imbalance caused by long-tail distribution, improving the efficiency of training and convergence, as well as the classification performance and generalization ability of the model. The advantages are specifically as follows:

[0096] (1) In order to reduce the number of model parameters and computational complexity and promote spectral-spatial feature fusion, a spectral feature extraction and fusion module (SFEF) is designed, which uses grouped convolutions with different kernel sizes for residual connection and feature fusion to achieve efficient feature representation.

[0097] (2) In order to fully utilize the spectral-spatial correlation of HSI data, a multi-scale convolutional activation-guided attention mechanism (MsCA) is designed and a position attention module (PAM) is introduced. These attention mechanisms can capture the spectral differences and spatial contextual relationships between objects.

[0098] (3) The focal loss (FL) in computer vision tasks is applied to the LSSMA model to deal with the sample imbalance caused by the long-tail distribution, improve the efficiency of training and convergence, as well as the classification performance and generalization ability of the model.

[0099] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0101] Figure 1 Schematic diagram of the structure of the LSSMA model of the present invention.

[0102] Figure 2 This is a schematic diagram of the multi-scale convolution activation-guided attention mechanism model of the present invention.

[0103] Figure 3 It is a schematic diagram of the PAM structure of the present invention.

[0104] Figure 4 Schematic diagram of group convolution of the present invention.

[0105] Figure 5 It is a schematic diagram of the classification results obtained by different methods of the IP dataset and the pseudo-color map and the real labels of the ground objects.

[0106] Figure 6 This is a schematic diagram of the classification results obtained by different methods for the H2013 dataset, as well as pseudo-color images and true labels of objects.

[0107] Figure 7 This is a schematic diagram of the ablation experiment under the IP dataset. Figure 7 (a) is a false color image; Figure 7 (b) SFEF+PAM; Figure 7 (c) MsCA+SFEF; Figure 7 (d) is the chemical combination of MsCA+PAM; Figure 7 (e) is the LSSMA method.

[0108] Figure 8 This is a schematic diagram of the ablation experiment under the H2013 dataset. Figure 8 (a) is a false color image; Figure 8 (b) SFEF+PAM; Figure 8 (c) MsCA+SFEF; Figure 8 (d) MsCA+PAM; Figure 8 (e) is the LSSMA method. DETAILED DESCRIPTION

[0109] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0110] 1. Structure of LSSMA

[0111] The overall framework of LSSMA proposed in this paper is as follows Figure 1 In the figure, (a), (b), (c) and (d) are convolution layers. (e) and (g) are group convolution layers. (f) is a dynamic group convolution layer. First, the HSI data is preprocessed, and then the feature maps generated after preprocessing are branched 1( Figure 1 The black dotted box with Branch 1 in the Figure 1 For Branch 1, it first passes through the MsCA module to selectively enhance or suppress spectral information, and then the output features are input into the grouped convolution layer ( Figure 1 e in Figure 1) performs group convolution to fuse spectral and spatial information. For Branch 2, the pixel features at different positions are first weighted by the PAM module, and the important spectral features are highlighted to reduce unnecessary redundant information. The output is then input into the SFEF module to further refine the spectral and spatial features of the data and enhance the feature extraction capability. For the SFEF module, the output of the PAM module is sequentially input into the convolutional layers (a) and (b), and then residually connected with the output of the PAM. The results are then sequentially input into the convolutional layers (c) and (d). Figure 1 The residual connection is performed in the direction of the arrow in the figure, and finally the feature map is input into the group convolution layer ( Figure 1 The g figure in Figure 1 performs group convolution to enable the model to better extract spectral information while reducing the parameters of the convolutional neural network. In order to reuse the previous features, the features of Branch1, Branch2 and the pre-processed data are fused and the fusion results are input into the dynamic group convolution layer ( Figure 1 As shown in Figure f, the convolution kernel used in this layer will change with the number of hyperspectral data bands to adapt to different HIS data.

[0112] The LSSMA method effectively accounts for variations in the spatial scale of objects and fully utilizes spectral spatial features, alleviating spectral differences and insufficient spatial context information, while improving classification accuracy. Furthermore, it reduces network complexity without compromising classification performance and can handle sample imbalance caused by long-tail distributions, improving training and convergence efficiency, as well as the model's classification performance and generalization capabilities.

[0113] 1.1 Multi-scale Convolutional Activation-guided Attention Mechanism (MsCA)

[0114] We propose a multi-scale convolutional activation guided attention mechanism model (MsCA), such as Figure 2 As shown in Figure 2, MsCA introduces multi-scale convolution to capture the correlation characteristics between different spectral bands, thereby more effectively extracting features from different spectral bands. Although HSI contains a large amount of information, its high dimensionality easily leads to data redundancy. Through convolution operations and activation functions, the attention weights of different features are guided and adjusted to capture feature information at multiple scales and enhance the model's perception of objects at different scales. This attention mechanism can improve the performance and accuracy of classification tasks, allowing the model to focus more on features that are more meaningful to the task.

[0115] Assume that the input of the model is HSI data U∈R {C×H×W} , C is the number of channels, H and W are the length and width of the image respectively. First, we specify the size of the output tensor after the adaptive average pooling operation as (H o ×W o ), the image (pre-processed HSI data U) is processed by adaptive average pooling to obtain a one-dimensional spectral channel vector The operation of the adaptive average pooling layer is expressed as:

[0116]

[0117] in: Indicates the current channel number, C indicates the total number of channels;

[0118] is a one-dimensional spectral channel vector, All sizes are C×H o ×W o ;

[0119] Represents the one-dimensional spectral channel vector under the current channel.

[0120] pass and U computing kernel s The default value of padding is 0.

[0121]

[0122]

[0123] Among them: kernel s Indicates the convolution kernel size;

[0124] stride indicates the step length;

[0125] padding means filling;

[0126] Indicates the size of the output data space H o ×W o ,Right now

[0127] U s Indicates the spatial size of the input data H×W, that is, U s =H×W;

[0128] floor(.) means round down.

[0129] Since the information of HSI in the spectral dimension is continuous, the size of the single-size convolution kernel is fixed, and it can only obtain relevant features within a fixed range, and cannot effectively capture feature information of different scales. In order to better capture features of different scales, we set a multi-scale kernel to capture relevant features at different distances. A set of multi-scale convolution kernels D = {D1, D2, D3, ..., D K′}, K′ represents the total number of defined convolution kernels, and the size of the kth convolution kernel is C k ×C k ×C k , C k =(1, 3, 5) (this paper sets three convolution kernels, i.e. k=3, and the kernel sizes are 1×1×1, 3×3×3, and 5×5×5 respectively). The data obtained by the adaptive average pooling layer is After convolution with multi-scale convolution kernels respectively, the extracted multi-scale features are fused to obtain the given C k ×C k ×C k The kernel size is used to extract the adjacent k Spectral correlation characteristics of T k , the formula is as follows:

[0130]

[0131] Where: ⊙ represents the convolution operation;

[0132] represents a one-dimensional spectral channel vector;

[0133] D k represents the convolution kernel;

[0134] Represents the output data calculated by the adaptive average pooling layer;

[0135] Denotes the convolution kernel D k No. Second convolution.

[0136] will have adjacent distances C k Spectral correlation characteristics of T k Fusion, fusion feature T s It is expressed as follows:

[0137]

[0138] Indicates the Spectral correlation features under convolution kernels T k ;

[0139] Then the fused feature T s Input into the fully connected layer with nonlinear mapping to obtain the channel mask τ, the formula is as follows:

[0140] τ=σ(W2×δ(W1×T s )) (6)

[0141] Where: σ represents the ReLu function;

[0142] δ represents the Sigmoid function;

[0143] W1 and W2 are weights.

[0144] Finally, the input data U of the MsCA module is re-adjusted, and the re-adjusted data is U new , the adjustment strategy is as follows:

[0145]

[0146] Indicates the channel masks on the channels,

[0147] Indicates the input data channels;

[0148] The above strategy is the formation process of the MsCA module. Through the MsCA module, the network can calculate a channel weight τ based on the input data U and assign the channel weight to the data U. Different input data have different weights, and the channel weights of multiple input data are adaptively adjusted. This strategy is more flexible and efficient in data processing and feature learning, improving model performance and generalization ability while reducing computational complexity and redundant information.

[0149] 1.2 Position Attention Module (PAM)

[0150] In order to further explore the spectral and spatial position information, the Position Attention Module (PAM) is introduced to build an attention module to enrich the position information of the spectral space. Figure 3 This is the structure diagram of PAM. PAM calculates attention weights for each position in the input feature map, thereby weightedly fusing features at different positions to obtain more accurate feature representations. By learning features at different positions, PAM can better understand image content and improve classification accuracy and stability.

[0151] First, the input feature map U∈R is transformed through a 1x1 convolution {C×H×W} Mapped to three feature maps Q, K, V, respectively, ({Q, K, V}∈R C×H×W ), the shapes of Q, K, and V are C×H×W, and then reshaped to C×g, where g=H×W, g is the number of pixels, and the specific formula is as follows:

[0152]

[0153] Where: ⊙ represents the convolution operation;

[0154] Δ q , Δ k , Δ v They are 1×1 convolution kernels respectively.

[0155] Next, multiply the transpose of Q by K and perform softmax normalization to obtain the attention weight A, A∈R {g ×g} The specific formula is as follows:

[0156]

[0157] Among them: A j，i is each element of the attention weight A, which represents the influence of position i on position j;

[0158] K j represents the j-th element of K;

[0159] Q T i Represents the i-th element after transposing Q.

[0160] Then, multiply the feature V by the attention weight A and then multiply it by the scale coefficient γ to obtain the weighted eigenvalue matrix Z. The specific formula is as follows:

[0161]

[0162] V i is the i-th element of V;

[0163] Z j is the j-th element of the eigenvalue matrix Z;

[0164] γ is the scaling factor, which is initialized to 0.

[0165] Finally, the weighted eigenvalue matrix Z is reshaped and concatenated with the original eigenvalue matrix U to obtain the final output feature map

[0166]

[0167] Among them: [;] represents splicing;

[0168] U j Represents the j-th element of the input data U.

[0169] 1.3. Spectral Feature Extraction and Fusion Module (SFEF)

[0170] In order to fully utilize the information of remote sensing hyperspectral images and extract accurate and reliable feature representations, we propose a spectral feature extraction and fusion SFEF module. SFEF is a module applied to remote sensing image hyperspectral classification tasks. By extracting and fusing effective features, the model can reduce complexity while maintaining high-precision classification results. By combining feedback connections, residual connections and feature fusion methods, the module can extract multi-scale spectral features from hyperspectral data and improve the ability to understand the data through feedback mechanisms and residual connections. The features of all layers of the SFEF module are circulated with the help of feedback / residual connections. Therefore, each layer now obtains the output features from other layers as its input in turn. Formulas (12) and (13) represent the input and output formulas of the SFEF module respectively.

[0171]

[0172]

[0173] Where: C m Represents the output obtained after the mth convolution training;

[0174] Indicates the output obtained after passing through the PAM module;

[0175] α represents the batch normalization layer;

[0176] Represents the ReLU activation function;

[0177] X m+1 Represents the input of the m+1th convolution;

[0178] W and B represent the corresponding weights and biases;

[0179] {} i Indicates the index of the sample.

[0180] 1.4 Grouped Convolution

[0181] In order to better extract spectral information, we use a combination of grouped convolution + BN + ReLU for feature extraction. By using grouped convolution instead of traditional convolution, the number of parameters of the convolutional neural network is reduced and the training speed of the model is accelerated. Specifically, grouped convolution divides the input feature map into channels according to the number of groups, performs an independent convolution operation on each group of feature maps, and finally splices the results of all groups together. This can reduce the number of model parameters to a certain extent, while better capturing the characteristic information of the spectrum. The parameter calculation formula of the standard convolution layer is as follows:

[0182] w×h×c×C i ×C o (14)

[0183] Where: w, h, and c are the height, width, and number of channels of the convolution kernel, respectively.

[0184] C i Indicates the number of channels of the input packet convolution layer data;

[0185] C o Indicates the number of channels of the output data from the grouped convolutional layer.

[0186] The parameters of the grouped convolutional layer in this paper are calculated as follows Figure 4 As shown, the calculation formula is as follows:

[0187]

[0188] Where: β is the number of groups to be grouped.

[0189] 1.5 Loss Function

[0190] When training a hyperspectral image classification model, choosing an appropriate loss function is crucial to the performance of the model. Currently, commonly used loss functions include Focal Loss and cross entropy loss functions. This article uses the Focal Loss loss function. Compared with the cross entropy loss function, Focal Loss can solve the problem of data imbalance, because in hyperspectral image classification, the number of different types of land objects varies greatly, and the problem of data imbalance is more prominent. In addition, Focal Loss can also balance the data by reducing the weight of easy-to-classify samples and increasing the weight of difficult-to-classify samples, thereby improving the model's classification accuracy for difficult samples. Since the sample imbalance of the hyperspectral image dataset is more obvious, we used Focal Loss instead of the cross entropy loss function. In order to solve the class imbalance, two parameters π and 1-π are set to balance the weights of positive and negative samples. If π is defined s In the following form:

[0191]

[0192] π s The general term for parameters π and 1-π, {π, 1-π}∈π s ;

[0193] Y∈{-1, 1}, where Y is the true probability (sample label value, 1 is a positive sample). Then the Focal Loss can be defined as:

[0194] CE(P, Y) = -π s (-Ylog(P)-(1-Y)log(1-P)) (17)

[0195] P represents the predicted probability of the model.

[0196] Although formula (17) solves the problem of sample imbalance, it does not distinguish between easy and difficult samples. In order to solve this problem, a coefficient (1-P s ) λ ,in

[0197] At this point, the Focal Loss loss can be defined as:

[0198] CE(P, Y) = -(1-P s ) λ (-Ylog(P)-(1-Y)log(1-P)) (18)

[0199] We combine formula (17) and formula (18). Since Y is unknown, the formula of the Focal Loss loss function is as follows:

[0200] FL(P, Y) = -π(1-P) λ ×Ylog(P)-(1-π)P λ ×(1-Y)log(1-P) (19)

[0201] Where: π is the weighting parameter for adjusting the number of samples;

[0202] λ is the adjustment factor for adjusting the weight of difficult and easy samples.

[0203] 2. Effects of the invention:

[0204] 2.1 Model comparison experiment

[0205] In order to evaluate the classification performance of the proposed LSSMA, seven deep learning-based HSI classification methods were compared with LSSMA on the agricultural remote sensing image dataset IP and the urban area remote sensing dataset H2013, including FDSSC, HybridSN, SSRN, DBDA, DBMA, LMFN, and DmuCA. Table 1-2 summarizes the detailed results of different methods under different datasets, and the corresponding classification diagrams are shown in Figure 1-2. Figure 5-6 As shown in Figure 2, we randomly select a small number of labeled samples from each class to form the training dataset, and the remaining samples are used for testing. For both the IP and H2013 datasets, we use 10% of the data for training. We use several broad metrics for evaluation, including overall accuracy (OA), average accuracy (AA), kappa coefficient (kappa), floating point operations (Flops), training time, and test time.

[0206] Table 1 Classification results of IP dataset

[0207] Class No. FDSSC HybridSN SSRN DBDA DBMA LMFN DMuCA KRD1 1 68.22±28.98 46.82±23.83 80.62±16.89 84.40±12.05 91.24±4.51 83.38±7.55 56.66±25.10 94.86±6.94 2 96.90±3.08 85.40±5.96 97.58±1.11 92.80±2.84 91.06±3.47 90.68±3.67 92.88±4.89 97.82±0.74 3 91.84±2.34 70.38±3.45 91.40±2.70 88.54±1.67 82.10±18.61 82.64±5.16 86.84±5.89 91.98±0.38 4 95.36±1.06 68.54±10.66 95.72±2.16 91.86±2.89 90.74±13.86 86.54±7.20 86.62±7.41 95.72±2.46 5 93.74±1.42 85.52±8.85 94.06±1.34 90.16±2.87 90.30±1.92 91.34±2.17 92.66±1.01 95.20±0.81 6 98.52±0.87 93.52±3.16 98.60±1.69 97.78±0.46 97.84±1.74 96.62±1.39 97.22±1.53 99.20±0.48 7 95.52±4.68 17.78±20.50 94.12±6.78 72.30±18.75 94.16±4.55 79.52±19.07 80.98±11.95 99.60±0.89 8 98.44±1.67 96.46±1.22 98.74±1.77 99.12±0.70 99.54±0.43 98.24±1.45 97.16±1.65 99.94±0.13 9 55.64±5.33 9.42±17.37 80.92±18.13 73.18±15.75 84.02±14.94 78.84±13.64 70.54±39.86 91.30±6.65 10 91.44±5.48 83.10±5.78 95.30±0.29 88.62±1.18 87.68±4.39 87.68±3.24 91.00±3.30 95.06±1.03 11 94.32±4.61 88.04±1.62 96.78±0.44 91.88±2.86 91.02±5.53 91.50±1.74 94.28±2.03 96.96±0.23 12 96.60±0.58 76.96±4.15 95.18±2.17 93.16±2.04 88.68±9.45 83.06±7.23 87.36±7.61 96.74±1.33 13 99.70±0.00 97.90±1.17 99.28±0.94 99.60±0.31 98.72±1.23 97.60±1.68 98.74±0.78 99.40±0.47 14 98.28±1.23 95.58±1.72 98.12±0.57 97.50±1.27 97.86±1.51 97.14±0.73 97.94±0.83 98.82±0.72 15 83.62±4.41 65.26±5.58 81.14±3.94 80.94±4.87 83.68±7.61 78.42±3.18 76.90±5.18 84.44±2.97 16 95.62±2.96 96.48±1.70 99.14±0.72 96.16±2.14 94.94±4.64 96.36±1.48 93.64±6.10 98.20±1.66 OA (%) 93.67±2.43 84.33±2.29 94.63±0.53 91.39±0.91 90.58±4.27 89.51±2.24 91.36±2.47 95.28±0.21 AA (%) 90.86±2.87 73.57±3.10 93.54±1.07 89.88±2.40 91.47±5.48 88.72±3.40 87.59±1.94 95.95±0.75 Kappa (%) 92.80±2.75 82.16±2.62 93.90±0.58 90.22±1.01 89.08±4.98 88.08±2.54 90.20±2.77 94.66±0.23 FLOPs 13.737G 2.067G 12.235G 19.046G 19.017G 134.484M 1.330G 1.560G Parameters(k) 1227.6 337.9 364.2 607.0 609.9 36.9 210.5 8.7 Train time(s) 60.27 16.82 31.46 98.45 100.52 21.53 22.97 61.72 Test time(s) 2.21 0.32 0.70 1.17 1.16 0.34 0.85 1.07

[0208] Table 2 Classification results of the H2013 dataset

[0209]

[0210]

[0211] 2.2 Ablation Experiment

[0212] The LSSMA method proposed in this paper consists of three very important modules, namely MsCA, PAM and SFEF. In order to study the effectiveness of these three modules and their contribution to the model, we conducted ablation experiments on two datasets (IP and H2013). Table 3 shows the detailed results of the experiment.

[0213] (1) Detailed data table obtained from the ablation experiments on the IP and H2013 datasets. As shown in Table 3, the network combining the SFEF module and the PAM module can basically achieve reasonable classification performance, but the accuracy is low. After replacing the PAM module with the MsCA module (MsCA+SFEF), the classification accuracy is further improved for the H2013 dataset. This shows that the strategy of integrating multi-scale related features into the spectral attention mechanism is effective. After replacing the SFEF module with the MsCA (MsCA+PAM), the classification accuracy is greatly improved. This shows from another perspective that the PAM module is effective in extracting spatial features. At the same time, the combination of the MsCA and PAM modules also makes the spatial features more closely linked to the spectral features. From the perspective of improving classification accuracy, the MsCA and PAM modules contribute more to the model. This is because the MsCA module combines the ideas of multi-scale convolution and attention mechanism. When processing spectral features, MsCA can perform convolution operations on features in different bands to capture the spectral information of different bands, thereby improving the expression ability of spectral features. When processing spatial features, MsCA can capture feature information at different spatial scales by performing convolution operations on features within different receptive ranges, thereby improving perception of different regions within the image. PAM, on the other hand, strengthens focus on specific regions, fully utilizing the spatial structure and spectral information in the image, improving classification accuracy. Therefore, judging by the data results, the MsCA and PAM modules contribute more significantly, but in reality, the three modules, MsCA, PAM, and SFEF, are inseparable.

[0214] Table 3 Ablation experiment results on IP and H2013 datasets

[0215]

[0216]

[0217] (2) Figure 7 The classification results of some areas of the IP dataset are shown. We select some areas of the IP dataset (white solid line frame) for comparative analysis. Figure 7 The classification boundary of (b) is relatively clear, but after adding the MsCA module, Figure 7 As shown in (e), the classification boundaries are clearer. This shows that the MsCA module helps extract edge information in the IP dataset. Although the combination of SFEF and PAM can basically distinguish the classification boundaries of samples, adding MsCA achieves better classification results and clearer classification boundaries.

[0218] (3) Figure 8The classification results of some areas of the H2013 dataset are shown. We select some areas of the H2013 dataset (white solid line frame) for comparative analysis. Figure 8 (b) Figure 8 (c) and Figure 8 The classification boundary of (d) is relatively vague, especially Figure 8 (d) The combination lacks the SFEF module, and its classification boundary is less clear, while the combination with the SFEF module Figure 8 (b) and Figure 8 (c) The classification boundary is relatively clear, which shows that the SFEF module is helpful for extracting edge information in the H2013 dataset and can better capture the feature information of different levels in the H2013 dataset. At the same time, after combining the three modules of MsCA, PAM and SFEF, as shown in the figure, Figure 8 As shown in (e), the classification boundaries are clearer.

Claims

1. A lightweight hyperspectral image classification method based on multi-scale attention, characterized by: include: Step 1: First, the HSI data is input into the convolutional layer for preprocessing. Preprocessing can perform preliminary feature extraction and dimensionality reduction on the input HSI data. Step 2: Input the preprocessed data into the first branch, which includes an MsCA module and a grouped convolutional layer, and the preprocessed data passes through the MsCA module; Then, the output features of the MsCA module are input into the grouped convolutional layer for feature extraction; The MsCA module includes the following steps: S1-1, the pre-processed HSI data is processed by adaptive average pooling to obtain a one-dimensional spectral channel vector in: Indicates the current channel number, C indicates the total number of channels; is a one-dimensional spectral channel vector; Represents the one-dimensional spectral channel vector under the current channel; H o Represents the length of the output tensor after the adaptive average pooling operation; W o Indicates the width of the output tensor after the adaptive average pooling operation; The convolution kernel and step size of the adaptive average pooling layer are obtained by the following formula: stride indicates the step length; floor(.) means round down; Indicates the size of the output data space H o ×W o ; U s Indicates the spatial size of the input data H×W; kernel s represents the convolution kernel; S1-2, set up multi-scale kernels to capture relevant features at different distances, and define a set of multi-scale convolution kernels D = {D1, D2, D3, ..., D K′ }, the data obtained by the adaptive average pooling layer After convolution with multi-scale convolution kernels respectively, the extracted multi-scale features are fused to extract features with adjacent distance C k Spectral correlation characteristics of T k , the formula is as follows: Where: ⊙ represents the convolution operation; represents a one-dimensional spectral channel vector; D k represents the convolution kernel; Represents the output data calculated by the adaptive average pooling layer; Denotes the convolution kernel D k c-th convolution; C represents the total number of channels; S1-3, will have adjacent distance C k Spectral correlation characteristics of T k Fusion, fusion feature T s It is expressed as follows: Indicates the Spectral correlation features under convolution kernels T k ; K ′ Represents the total amount of convolution kernel; S1-4, then the fused features T s Input into the fully connected layer with nonlinear mapping to obtain the channel mask τ, the formula is as follows: τ=σ(W2×δ(W1×T s )) (6) Where: σ represents the ReLu function; δ represents the Sigmoid function; W1 and W2 are weights; S1-5, readjust the input data U of the MsCA module, and the readjusted data is U new , the adjustment strategy is as follows: Indicates the channel masks on the channels, Indicates the input data channels; Step 3: The preprocessed data is input into the second branch, which includes a PAM module, an SFEF module and a grouped convolutional layer. The preprocessed data passes through the PAM module to highlight important spectral features through position features, and then the output of the PAM module is input into the SFEF module. Finally, the output feature map of the SFEF module is input into the grouped convolutional layer; the SFEF module includes: The features of all layers of the SFEF module are circulated with the help of feedback connections and residual connections. Therefore, each layer now obtains the output features from other layers as its input in turn; formulas (12) and (13) represent the input and output formulas of the SFEF module respectively: Where: C m Represents the output obtained after the mth convolution training; Indicates the output obtained after passing through the PAM module; Represents the ReLU activation function; α represents the batch normalization layer; ⊙ represents convolution operation; X m+1 Represents the input of the m+1th convolution; W and B represent the corresponding weights and biases; {} i Indicates the index of the sample; Step 4: The outputs of Step 1, Step 2, and Step 3 are subjected to feature fusion and the fusion results are input into the dynamic grouping convolution layer. The convolution kernel used in this layer will change with the number of hyperspectral data bands to adapt to different HSI data. Step 5: The feature map obtained in step 4 is reduced in dimension through adaptive average pooling, and finally the classification result is output through the fully connected layer.

2. A lightweight hyperspectral image classification method based on multi-scale attention according to claim 1, characterized in that: The pre-processed data is passed through the PAM module to highlight important spectral features through position features, including the following steps: First, the input feature map U∈R is transformed through a 1x1 convolution {C×H×W} Mapped to three feature maps Q, K, V respectively, {Q, K, V}∈R C×H×W In [15], the shapes of Q, K, and V are C×H×W, and then reshaped to C×g, where g=H×W, g is the number of pixels, as shown in the following formula: Q=reshape(U⊙Δ q ) K=reshape(U⊙Δ k ) (8) V=reshape(U⊙Δ v ) Where: ⊙ represents the convolution operation; Δ q , Δ k , Δ v They are 1×1 convolution kernels respectively; Next, multiply the transpose of Q by K and perform softmax normalization to obtain the attention weight A,A∈R {g×g} The specific formula is as follows: Among them: A j,i is each element of the attention weight A, which represents the influence of position i on position j; K j represents the j-th element of K; Q T i represents the i-th element after transposing Q; Then, multiply the feature V by the attention weight A and then multiply it by the scale coefficient γ to obtain the weighted eigenvalue matrix Z. The specific formula is as follows: V i is the i-th element of V; Z j is the j-th element of the eigenvalue matrix Z; γ is the scale factor, initialized to 0; Finally, the weighted eigenvalue matrix Z is reshaped and concatenated with the original eigenvalue matrix U to obtain the final output feature map Among them: [;] represents splicing; U j Represents the j-th element of the input data U.

3. The lightweight hyperspectral image classification method based on multi-scale attention according to claim 1 is characterized in that The calculation formula of the group convolution is as follows: Where: w, h and c are the height, width and number of channels of the convolution kernel respectively; C i Indicates the number of channels of the input packet convolution layer data; C o Indicates the number of channels of output data from the grouped convolutional layer; β is the number of groups to be grouped; Moreover, after the group convolution, the data is subjected to batch normalization and ReLU activation function in sequence.

4. The lightweight hyperspectral image classification method based on multi-scale attention according to claim 1 is characterized in that Also includes the loss function: FL(P,Y)=-π(1-P) λ ×Ylog(P)-(1-π)P λ ×(1-Y)log(1-P) (19) Where: π is the weighting parameter for adjusting the number of samples; P represents the predicted probability of the model; Y is the probability of being true; λ is the adjustment factor for adjusting the weight of difficult and easy samples.

Images