Hyperspectral image classification method based on bidirectional interactive fusion space-spectrum multi-order gating aggregation network

Abstract

The invention relates to a hyperspectral image classification method based on a bidirectional interactive fusion space-spectrum multi-order gating aggregation network, and belongs to the field of remote sensing image processing. The method comprises the steps of determining a data set; data preprocessing: performing sample block extraction on the hyperspectral data set, and dividing the hyperspectral data set into a training set, a verification set and a test set; network construction: constructing a space-spectrum multi-order gating aggregation network based on bidirectional interactive fusion, wherein the space-spectrum multi-order gating aggregation network is used for hyperspectral image classification; the hyperspectral samples in the training set are input into the constructed network in batches for training, and after each training batch is completed, the classification performance is evaluated by using the verification set samples; and sample classification: inputting the hyperspectral samples in the test set into the trained classification network to obtain a final classification result. According to the method, efficient feature extraction can be realized, high-accuracy classification is performed on the hyperspectral image, and the method can be widely applied to remote sensing application fields such as hyperspectral image surface feature category detection and recognition.

Description

Technical Field

[0001] The present invention relates to a hyperspectral image classification method based on a spatial-spectral multi-order gated aggregation network based on bidirectional interactive fusion, and in particular to a hyperspectral image classification method combining deep learning and remote sensing technology, belonging to the technical field of remote sensing image processing. Background Art

[0002] Hyperspectral images provide rich spectral information across the visible spectrum and a wider range of electromagnetic bands. Unlike natural images, hyperspectral images can capture spatial information for each pixel while preserving the spectral characteristics of objects across multiple bands. These unique properties have led to their widespread application in numerous practical scenarios, such as geological exploration, environmental monitoring, urban planning, and precision agriculture. As a key step in hyperspectral image processing and analysis, hyperspectral image classification aims to accurately identify and assign each pixel to a predefined category or set of categories, and has therefore garnered significant attention.

[0003] In early research, traditional machine learning methods such as support vector machines, principal component analysis, and random forests were widely used for hyperspectral image classification tasks. Although these methods can extract basic image features and have achieved certain results in classification performance, they still have many shortcomings. First, manual feature extraction relies on professional knowledge and experience, and the extracted features often fail to fully represent the essential attributes of the image. Second, hyperspectral images are generally subject to a large amount of noise and redundant information, which can easily weaken the classification effect. Finally, due to the limitations of the artificially designed features and the incompleteness of the extraction methods, the accuracy and robustness of the classifier are significantly restricted.

[0004] In recent years, deep learning-based methods have made significant progress in hyperspectral image classification. Deep learning models can automatically learn image feature representations, avoiding the problems of manual feature extraction algorithms. Among deep learning models, convolutional neural networks (CNNs) are a widely used model. CNNs can automatically extract features from images and have good performance and computational efficiency. However, due to the local nature of convolution operations, they may not fully capture contextual features in hyperspectral images. Even by increasing the convolution kernel size, effective encoding of multi-level expressive features remains insufficient. Furthermore, CNNs typically apply convolution kernels across spatial dimensions and treat all spectral features equally, which may weaken their ability to extract diagnostic information from the spectral domain. The self-attention mechanism, originally designed for natural language processing tasks, treats two-dimensional images as one-dimensional sequences. This approach breaks down the two-dimensional structure of images and can lead to a loss of neighborhood associations. Furthermore, the self-attention mechanism excels at modeling long-range dependencies and capturing global contextual information, while the convolution operation excels at extracting local features. However, naively combining these two operations can lead to an over-reliance on either global or local information, lacking explicit mechanisms to model interactions at intermediate levels. If these key interactions cannot be effectively captured, the obtained feature representation will be limited, thereby weakening the classification performance.

[0005] Based on this background, by utilizing deep learning networks to achieve interactive feature fusion, the spatial-spectral characteristics of hyperspectral images can be fully captured, achieving fast and accurate hyperspectral pixel classification. Summary of the Invention

[0006] The technical problem solved by the present invention is: the present invention provides a hyperspectral image classification method based on a spatial-spectral multi-order gated aggregation network with bidirectional interactive fusion, which is used to solve the problem of the expressive but often overlooked interactive relationship between the spatial domain and the spectral domain. The method of the present invention minimizes feature loss while effectively retaining key information, thereby achieving fast and accurate hyperspectral image classification.

[0007] The technical solution of the present invention is: a hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network, the method comprising: using a constructed bidirectional interactive fusion spatial-spectral multi-order gated aggregation based hyperspectral image classification network to perform hyperspectral image classification;

[0008] The hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation includes four core components: preliminary feature extraction module, spatial-spectral feature extraction module, spatial-spectral feature fusion module and classifier module;

[0009] The preliminary feature extraction module is used to extract local-global spectral features and local spatial features of the input hyperspectral image block to generate primary features;

[0010] The spatial-spectral feature extraction module is used to aggregate primary features through multiple stages and output aggregated features;

[0011] The spatial-spectral feature fusion module is used to fuse the output aggregated features to obtain the final fused features;

[0012] The classifier module is used to classify the final fused features to obtain a category label for each pixel of the hyperspectral image block.

[0013] Furthermore, the specific steps of the method include:

[0014] Step 1: Prepare multiple public hyperspectral image datasets as experimental data for network training;

[0015] Step 2: Preprocess the hyperspectral image, extract hyperspectral image blocks centered on each pixel, and divide the extracted hyperspectral image blocks into non-overlapping hyperspectral training sample sets, verification sample sets, and test sample sets;

[0016] Step 3: Construct a hyperspectral image classification network based on bidirectional interactive fusion of spatial and spectral multi-order gated aggregation;

[0017] Step 4: Use the hyperspectral training sample set to train the hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation. After each training batch is completed, the performance of the trained network is verified using the hyperspectral validation sample set to evaluate the current training status and convergence.

[0018] Step 5. Input the hyperspectral test sample set into the trained hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation to obtain the category label of each pixel in the test sample set, thereby completing the hyperspectral image classification task.

[0019] Furthermore, the specific steps of Step 2 are as follows:

[0020] Step 2.1. Hyperspectral images have three-dimensional characteristics, and their original data are represented as Where H and W represent the spatial size of the image, and C represents the number of spectral bands of the image. The original hyperspectral image is filled with pixels with a value of 0 around the edges, and then the hyperspectral image block is extracted from the filled image.

[0021] Step 2.2: According to the category of the central pixel of the hyperspectral image block, the corresponding image block is classified into the category set to which it belongs;

[0022] Step 2.3: For data sets of different sizes, divide the samples in each category set according to the preset ratio: first, select a certain proportion of hyperspectral image blocks as the training sample set, then select the same proportion of hyperspectral image blocks as the verification sample set, and finally select the remaining hyperspectral image blocks as the test sample set.

[0023] Furthermore, the specific steps of Step 3 are as follows:

[0024] Step 3.1. Build a preliminary feature extraction module, which is composed of a spectral feature optimization module and a neighborhood pixel embedding module in series, and is used to extract local-global spectral features and local spatial features of the input hyperspectral image block to generate primary features;

[0025] First, a spectral feature refinement module is constructed, which includes a two-dimensional convolutional layer, a three-dimensional convolutional layer, a normalization layer, and a ReLU activation function layer.

[0026] At the same time, a neighborhood pixel embedding module is designed, which includes a two-dimensional convolutional layer, a normalization layer, a ReLU activation function layer, and a neighborhood channel embedding block.

[0027] Step 3.2: Build a spatial-spectral feature extraction module. This module adopts a three-stage structural design. Each stage includes a spatial aggregation module and a spectral aggregation module. By integrating the convolution operation and the gated aggregation mechanism into a compact module, a spatial aggregation module for modeling spatial dimension information and a spectral aggregation module for extracting spectral dimension features are constructed respectively, realizing multi-level spatial-spectral feature extraction and expression. After the primary features are sequentially input into each stage, each stage outputs the aggregated features X1, X2, and X3 respectively.

[0028] Step 3.3, build a spatial-spectral feature fusion module and design a bidirectional interactive fusion block based on the bidirectional cross attention mechanism; first, the aggregated features output by the second stage are adjusted to the spectral dimension through a 1×1 convolution layer and the aggregated features output by the third stage, and then input into the first bidirectional interactive fusion module for fusion; then, the fused features output by the first bidirectional interactive fusion module and the aggregated features X1 output by the first stage are input into the second bidirectional interactive fusion module to obtain the final fused features X f ;

[0029] Step 3.4, build a classifier module consisting of a Dropout layer, a global average pooling layer and a linear classification layer in series, and combine the final fusion feature X f The module inputs the category label of each pixel of the hyperspectral image patch.

[0030] Furthermore, in Step 3.1, the two-dimensional convolution layer and the three-dimensional convolution layer of the spectral feature refinement module include two two-dimensional convolution modules and three three-dimensional convolution modules respectively; the convolution module structure is: convolution layer → normalization layer → ReLU activation function layer; the structure of the spectral feature refinement module is: first two-dimensional convolution module → first three-dimensional convolution module → second three-dimensional convolution module → third three-dimensional convolution module → second two-dimensional convolution module; wherein:

[0031] The convolution kernel size of the two-dimensional convolution module is set to 1×1, the convolution kernel sizes of the first three-dimensional convolution module and the third three-dimensional convolution module are set to 1×1×3, and the convolution kernel size of the second three-dimensional convolution module is set to 1×1×7.

[0032] Furthermore, in Step 3.1, the neighborhood pixel embedding module specifically includes a normalization layer, a 3×3 convolution module, a 1×1 convolution layer, and a neighborhood channel embedding module; the structure of the convolution module is fixed as follows: convolution layer → normalization layer → ReLU activation function layer; the structure of the neighborhood pixel embedding module is fixed as follows: normalization layer → 3×3 convolution module → neighborhood channel embedding module → 1×1 convolution layer; wherein:

[0033] The neighborhood channel embedding module includes multiple parallel 1×1 convolution branches. For the input feature map, each spectral branch processes a spectral band and its neighborhood spectral subset. Specifically, for the i-th spectrum, the input of the current spectral branch is intercepted with the spectrum as the center and one adjacent spectrum before and after it. The input first passes through the corresponding 1×1 convolution layer, and then passes through the ReLU activation function layer and the normalization layer. After all spectral branches independently complete the above operations, their output features are spliced ​​along the spectral dimension to form a complete feature map.

[0034] Furthermore, Step 3.2 includes:

[0035] Step 3.2.1, build the spatial aggregation module;

[0036] The spatial aggregation block includes a normalization layer, a feature decomposition unit, and a multi-order spatial gating aggregation unit. Its structure is fixed as follows: normalization layer → feature decomposition unit → multi-order spatial gating aggregation unit; it can be expressed as:

[0037] x spa =x+MoGA(FD(Norm(x)))

[0038] Among them, x represents the input features of each stage, x spa Represents the features output by the spatial aggregation block, Norm represents the normalization layer, FD represents the feature decomposition unit, and MoGA represents the multi-order spatial gated aggregation unit;

[0039] The feature decomposition unit directly extracts local features and global statistical features through a 1×1 convolution layer and a global average pooling layer respectively. The feature decomposition unit formula is expressed as:

[0040]

[0041] Among them, x FD Represents the features output by the feature decomposition unit, Conv 1×1 represents a 1×1 convolutional layer, δ a is a learnable scaling factor, with the initial value set to 0; GAP represents the global average pooling layer, and GELU represents the GELU activation function;

[0042] The multi-order spatial gating aggregation unit consists of a gating branch φ(g) and a dynamic feature integration branch ψ(d); a depthwise separable convolution DWConv is introduced into ψ(d) to extract multi-order features, and three DWConv layers with different expansion rates r∈{1,2,3} are used; first, a 5×5 DWConv is used to extract low-order features; then, the features output by the feature decomposition unit are decomposed into Satisfy C l +C m +C h =C; then, and Input to 5×5DWConv and and Directly retained as the identity mapping; finally, the three branch features are spliced ​​along the spectral dimension as In order to adaptively aggregate the features in ψ(d), the SiLU activation function is used in φ(g), and the multi-order spatial gated aggregation unit is finally expressed as:

[0043]

[0044] Step 3.2.2, build the spectrum aggregation module;

[0045] The spectrum aggregation block includes a normalization layer, a 1×1 convolution layer, a 3×3 depth-separable convolution layer, a spectrum redistribution unit, and a GELU activation function layer. Its structure is fixed as follows: normalization layer → 1×1 convolution layer → 3×3 depth-separable convolution layer → GELU activation function layer → spectrum redistribution unit → 1×1 convolution layer; it can be expressed as:

[0046] x′ spa =GELU(DWConv 3×3 (Conv 1×1 (Norm(x spa ))))

[0047] x spe =Conv 1×1 (SR(x′ spa ))+x′ spa

[0048] Among them, Norm represents the normalization layer, DWConv 3×3 represents a 3×3 depth-wise separable convolutional layer, SR represents a spectral redistribution unit, and W is projected through dimensionality reduction: And the GELU activation function collects and redistributes spectral information; the specific formula is as follows:

[0049] SR(x′ spa ) = x′ spa +δ e ⊙(x′ spa -GELU(x′ spa W))

[0050] Among them, δ e is the spectral scaling factor; the spectral redistribution unit redistributes the spectral features through complementary interactions, thereby enhancing the model's ability to represent spectral information that was previously not captured.

[0051] Furthermore, in Step 3.3, the bidirectional interactive fusion module includes a first bidirectional interactive fusion module and a second bidirectional interactive fusion module, and the bidirectional interactive fusion module is based on two inputs and includes two parallel branches;

[0052] For the first bidirectional interactive fusion module, its input features are the aggregated feature X2 output from the second stage of the spatial-spectral feature extraction module and the aggregated feature X3 output from the third stage. The upper branch is dominated by X2, and the lower branch is dominated by X3. The two branches implement information interaction between features through a bidirectional cross-attention mechanism, and obtain attention maps corresponding to X2 and X3 respectively. Then, learnable weights are introduced to the attention maps, and residual fusion is performed with the original features X2 and X3. Finally, the two fused features are spliced ​​along the spectral dimension, and the fused features are output.

[0053] For the second bidirectional interactive fusion module, its input features are the aggregated features X1 output by the first stage of the spatial-spectral feature extraction module and the fused features output by the first bidirectional interactive fusion module. The rest of the processing flow is the same as that in the first bidirectional interactive fusion module. Finally, the second bidirectional interactive fusion module outputs the final fused features.

[0054] Furthermore, the Step 3.3 also includes:

[0055] The bidirectional interactive fusion module takes two input features X Aand X B is the input, where the input feature X A and X B The values ​​of are the input of the first two-way interactive fusion module or the input of the second two-way interactive fusion module, which are first normalized and then pass through two independent linear mappings. The specific formula is as follows:

[0056] Q A =W A (Norm(X A )),Q B =W B (Norm(X B ))

[0057] Among them, W A and W B Represents the linear mapping weight matrix, Q A and Q B Indicates that from X A and X B The extracted query features; at the same time, two independent linear mappings are also used to transform X A and X B Mapped to corresponding values ​​V A and V B ; Then Q A , Q B 、V A and V B A bidirectional cross-attention mechanism is used to fuse the two features:

[0058]

[0059] X′ A =ζ A M B +X A

[0060] X′ B =ζ B M A +X B

[0061] Among them, M A and M B Represents the feature X A and X B Attention map of the image, C represents the number of spectral bands, ζ A and ζ B It is a learnable channel scaling factor that can dynamically adjust the weight of the bidirectional fusion feature. Finally, the two features obtained are concatenated along the spectral dimension to generate the fused feature, which is used as the final feature for classification.

[0062] Furthermore, the Step 4 includes:

[0063] The stochastic gradient descent algorithm is used to update the model parameters, and the loss value is calculated by the cross entropy loss function, which is expressed as:

[0064] l(a,y)=L={l1,…,l N} T ,

[0065]

[0066] Where L represents the total loss, a represents the total input, y represents the label value, N represents the batch size, and w z represents the category weight of category z, Z represents the total number of categories, a n,z Represents the observed sample a n The predicted probability of belonging to category z, y n,z Represents a label vector element. The beneficial effects of the present invention are:

[0067] 1. This paper constructs a new spatial-spectral multi-order gated aggregation network for hyperspectral image classification tasks; it can more accurately identify the types of objects in hyperspectral images, thereby effectively improving classification performance;

[0068] 2. To focus on the expressive but often overlooked interaction between the spatial and spectral domains, we propose spatial aggregation blocks and spectral aggregation blocks, enabling the model to effectively perform adaptive contextual feature extraction;

[0069] 3. In order to optimize the feature fusion process, a bidirectional interactive fusion block based on a bidirectional cross-attention mechanism is proposed to minimize feature loss while effectively retaining key information;

[0070] 4. The method of the present invention can achieve efficient feature extraction and classify hyperspectral images with high accuracy, and can be widely used in remote sensing application fields such as hyperspectral image ground object category detection and identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0072] Figure 2 This figure compares the neighborhood channel embedding block in the present invention with the ordinary two-dimensional convolution; Figure 2 (a) is an illustration of ordinary two-dimensional convolution; Figure 2 (b) is a diagram of the neighborhood pixel embedding block in the present invention;

[0073] Figure 3 It is a diagram of the spatial aggregation block and the spectral aggregation block in the present invention; Figure 3(a) is a diagram of the spatial aggregation block in the present invention; Figure 3 (b) is a diagram of the spectrum aggregation block in the present invention;

[0074] Figure 4 It is a diagram of the bidirectional interactive fusion block in the present invention;

[0075] Figure 5 The diagram shows the classification results of the present invention and two other existing advanced methods on the Pavia University dataset; Figure 5 (a) is the classification result diagram of the Pavia University dataset using the SSRN method; Figure 5 (b) is the classification result diagram of the Pavia University dataset using the SSFTT method; Figure 5 (c) is a diagram showing the classification results of the present invention on the Pavia University dataset;

[0076] Figure 6 The diagram shows the classification results of the present invention and two other existing advanced methods on the WHU-Hi-LongKou dataset; Figure 6 (a) is the classification result diagram of the WHU-Hi-LongKou dataset using the SSRN method; Figure 6 (b) is the classification result diagram of the WHU-Hi-LongKou dataset using the SSFTT method; Figure 6 (c) is the classification result diagram of the present invention on the WHU-Hi-LongKou dataset. DETAILED DESCRIPTION

[0077] Example 1: Figures 1-6 As shown, a hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network, the specific steps of the method include:

[0078] Step 1: Prepare multiple public hyperspectral image datasets as experimental data for network training;

[0079] Step 2: Preprocess the hyperspectral image, extract hyperspectral image blocks centered on each pixel, and divide the extracted hyperspectral image blocks into non-overlapping hyperspectral training sample sets, verification sample sets, and test sample sets;

[0080] The specific steps of Step 2 are as follows:

[0081] Step 2.1. Hyperspectral images have three-dimensional characteristics, and their original data are represented as Where H and W represent the spatial size of the image, and C represents the number of spectral bands of the image. The original hyperspectral image is filled with pixels of size n and pixel value 0 around the edges, and then the hyperspectral image block is extracted from the filled image. The data is represented as Where P represents the spatial size of the extracted hyperspectral image block, that is, a hyperspectral image block with a spatial size of (2n+1)×(2n+1) and a channel number of C is selected with each original pixel as the center; in this example, n is taken but not limited to 5;

[0082] Step 2.2: According to the category of the central pixel of the hyperspectral image block, the corresponding image block is classified into the category set to which it belongs;

[0083] Step 2.3: For data sets of different sizes, divide the samples in each category set according to the preset ratio: first, select a certain proportion of hyperspectral image blocks as the training sample set, then select the same proportion of hyperspectral image blocks as the verification sample set, and finally select the remaining hyperspectral image blocks as the test sample set.

[0084] Step 3: Construct a hyperspectral image classification network based on bidirectional interactive fusion of spatial and spectral multi-order gated aggregation;

[0085] Reference Figure 1 ,The hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation includes four core components: preliminary ,feature extraction module, spatial-spectral feature extraction module, spatial-spectral feature fusion module and classifier module;

[0086] The preliminary feature extraction module is used to extract local-global spectral features and local spatial features of the input hyperspectral image block to generate primary features;

[0087] The spatial-spectral feature extraction module is used to aggregate primary features through multiple stages and output aggregated features;

[0088] The spatial-spectral feature fusion module is used to fuse the output aggregated features to obtain the final fused features;

[0089] The classifier module is used to classify the final fused features to obtain a category label for each pixel of the hyperspectral image block.

[0090] Furthermore, the specific steps of Step 3 are as follows:

[0091] Step 3.1. Build a preliminary feature extraction module, which is composed of a spectral feature optimization module and a neighborhood pixel embedding module in series, and is used to extract local-global spectral features and local spatial features of the input hyperspectral image block to generate primary features;

[0092] First, a spectral feature refinement module is constructed, which includes a two-dimensional convolutional layer, a three-dimensional convolutional layer, a normalization layer, and a ReLU activation function layer.

[0093] At the same time, a neighborhood pixel embedding module is designed, which includes a two-dimensional convolutional layer, a normalization layer, a ReLU activation function layer, and a neighborhood channel embedding block.

[0094] Step 3.2: Build a spatial-spectral feature extraction module. This module adopts a three-stage structural design. Each stage includes a spatial aggregation module and a spectral aggregation module. By integrating the convolution operation and the gated aggregation mechanism into a compact module, a spatial aggregation module for modeling spatial dimension information and a spectral aggregation module for extracting spectral dimension features are constructed respectively, realizing multi-level spatial-spectral feature extraction and expression. After the primary features are sequentially input into each stage, each stage outputs the aggregated features X1, X2, and X3 respectively.

[0095] Step 3.3, build a spatial-spectral feature fusion module and design a bidirectional interactive fusion block based on the bidirectional cross attention mechanism; first, the aggregated features output by the second stage are adjusted to the spectral dimension through a 1×1 convolution layer and the aggregated features output by the third stage, and then input into the first bidirectional interactive fusion module for fusion; then, the fused features output by the first bidirectional interactive fusion module and the aggregated features X1 output by the first stage are input into the second bidirectional interactive fusion module to obtain the final fused features X f ;

[0096] Step 3.4, build a classifier module consisting of a Dropout layer, a global average pooling layer and a linear classification layer in series, and combine the final fusion feature X f The module inputs the category label of each pixel of the hyperspectral image patch.

[0097] Furthermore, in Step 3.1, the two-dimensional convolution layer and the three-dimensional convolution layer of the spectral feature refinement module include two two-dimensional convolution modules and three three-dimensional convolution modules respectively; the convolution module structure is: convolution layer → normalization layer → ReLU activation function layer; the structure of the spectral feature refinement module is: first two-dimensional convolution module → first three-dimensional convolution module → second three-dimensional convolution module → third three-dimensional convolution module → second two-dimensional convolution module; wherein:

[0098] The convolution kernel size of the two-dimensional convolution module is set to 1×1, the convolution kernel sizes of the first three-dimensional convolution module and the third three-dimensional convolution module are set to 1×1×3, and the convolution kernel size of the second three-dimensional convolution module is set to 1×1×7.

[0099] Furthermore, in Step 3.1, the neighborhood pixel embedding module specifically includes a normalization layer, a 3×3 convolution module, a 1×1 convolution layer, and a neighborhood channel embedding module; the structure of the convolution module is fixed as follows: convolution layer → normalization layer → ReLU activation function layer; the structure of the neighborhood pixel embedding module is fixed as follows: normalization layer → 3×3 convolution module → neighborhood channel embedding module → 1×1 convolution layer; wherein:

[0100] Reference Figure 2 The neighborhood channel embedding module includes multiple parallel 1×1 convolution branches; for the input feature map, each spectral branch processes a spectral band and its neighborhood spectral subset; specifically, for the i-th spectrum, the spectrum centered on it and one adjacent spectrum before and after it is intercepted as the input of the current spectral branch; the input first passes through the corresponding 1×1 convolution layer, and then passes through the ReLU activation function layer and the normalization layer; after all spectral branches complete the above operations independently, their output features are spliced ​​along the spectral dimension to form a complete feature map.

[0101] In Step 3.2, the spatial-spectral feature extraction module includes three stages of spatial aggregation modules and spectral aggregation modules, and its structure is fixed as follows: first spatial aggregation module → first spectral aggregation module → 1×1 convolution layer → second spatial aggregation module → second spectral aggregation module → 1×1 convolution layer → third spatial aggregation module → third spectral aggregation module.

[0102] Furthermore, Step 3.2 includes:

[0103] Step 3.2.1, build the spatial aggregation module;

[0104] Reference Figure 3 The spatial aggregation block includes a normalization layer, a feature decomposition unit, and a multi-order spatial gating aggregation unit. Its structure is fixed as follows: normalization layer → feature decomposition unit → multi-order spatial gating aggregation unit; it can be expressed as:

[0105] x spa =x+MoGA(FD(Norm(x)))

[0106] Among them, x represents the input features of each stage, x spa Represents the features output by the spatial aggregation block, Norm represents the normalization layer, FD represents the feature decomposition unit, and MoGA represents the multi-order spatial gated aggregation unit;

[0107] The feature decomposition unit directly extracts local features and global statistical features through a 1×1 convolution layer and a global average pooling layer respectively. The feature decomposition unit formula is expressed as:

[0108]

[0109] Among them, x FD Represents the features output by the feature decomposition unit, Conv 1×1 represents a 1×1 convolutional layer, δ a is a learnable scaling factor, with an initial value of 0; GAP represents the global average pooling layer, and GELU represents the GELU activation function; the feature decomposition unit combines local and global complementary interaction components through weighted fusion, which not only suppresses redundant information and alleviates interaction strength bias, but also enhances the diversity and representation ability of spatial features;

[0110] The multi-order spatial gating aggregation unit consists of a gating branch φ(g) and a dynamic feature integration branch ψ(d); a depthwise separable convolution DWConv is introduced into ψ(d) to extract multi-order features, and three DWConv layers with different expansion rates r∈{1,2,3} are used; first, a 5×5 DWConv is used to extract low-order features; then, the features output by the feature decomposition unit are decomposed into Satisfy C l +C m +C h =C; then, and Input to 5×5DWConv (expansion rate d=2) and 3×3DWConv (expansion rate d=3), respectively, and Directly retained as the identity mapping; finally, the three branch features are spliced ​​along the spectral dimension as In order to adaptively aggregate the features in ψ(d), the SiLU activation function is used in φ(g), and the multi-order spatial gated aggregation unit is finally expressed as:

[0111]

[0112] Step 3.2.2, build the spectrum aggregation module;

[0113] Reference Figure 3 The spectrum aggregation block includes a normalization layer, a 1×1 convolution layer, a 3×3 depth-separable convolution layer, a spectrum redistribution unit, and a GELU activation function layer. Its structure is fixed as follows: normalization layer → 1×1 convolution layer → 3×3 depth-separable convolution layer → GELU activation function layer → spectrum redistribution unit → 1×1 convolution layer; it can be expressed as:

[0114] x′ spa =GELU(DWConv 3×3 (Conv 1×1 (Norm(x spa ))))

[0115] x spe=Conv 1×1 (SR(x′ spa ))+x′ spa

[0116] Among them, Norm represents the normalization layer, DWConv 3×3 represents a 3×3 depth-wise separable convolutional layer, SR represents a spectral redistribution unit, and W is projected through dimensionality reduction: And the GELU activation function collects and redistributes spectral information; the specific formula is as follows:

[0117] SR(x′ spa ) = x′ spa +δ e ⊙(x′ spa -GELU(x′ spa W))

[0118] Among them, δ e is the spectral scaling factor; the spectral redistribution unit redistributes the spectral features through complementary interactions, thereby enhancing the model's ability to represent spectral information that was previously not captured.

[0119] Furthermore, in Step 3.3, the bidirectional interactive fusion module includes a first bidirectional interactive fusion module and a second bidirectional interactive fusion module, and the bidirectional interactive fusion module is based on two inputs and includes two parallel branches;

[0120] For the first bidirectional interactive fusion module, its input features are the aggregated feature X2 output from the second stage of the spatial-spectral feature extraction module and the aggregated feature X3 output from the third stage. The upper branch is dominated by X2, and the lower branch is dominated by X3. The two branches implement information interaction between features through a bidirectional cross-attention mechanism, and obtain attention maps corresponding to X2 and X3 respectively. Then, learnable weights are introduced to the attention maps, and residual fusion is performed with the original features X2 and X3. Finally, the two fused features are spliced ​​along the spectral dimension, and the fused features are output.

[0121] For the second bidirectional interactive fusion module, its input features are the aggregated features X1 output by the first stage of the spatial-spectral feature extraction module and the fused features output by the first bidirectional interactive fusion module. The rest of the processing flow is the same as that in the first bidirectional interactive fusion module. Finally, the second bidirectional interactive fusion module outputs the final fused features.

[0122] Furthermore, the Step 3.3 also includes:

[0123] Reference Figure 4The bidirectional interactive fusion block supports bidirectional information interaction by sharing the attention matrix, thereby achieving simultaneous information exchange and effectively enhancing the feature fusion effect; the bidirectional interactive fusion module takes two input features X A and X B is the input, where the input feature X A and X B The values ​​of are the input of the first two-way interactive fusion module or the input of the second two-way interactive fusion module, which are first normalized and then pass through two independent linear mappings. The specific formula is as follows:

[0124] Q A =W A (Norm(X A )),Q B =W B (Norm(X B ))

[0125] Among them, W A and W B Represents the linear mapping weight matrix, Q A and Q B Indicates that from X A and X B The extracted query features; at the same time, two independent linear mappings are also used to transform X A and X B Mapped to corresponding values ​​V A and V B ; Then Q A , Q B 、V A and V B A bidirectional cross-attention mechanism is used to fuse the two features:

[0126]

[0127] X′ A =ζ A M B +X A

[0128] X′ B =ζ B M A +X B

[0129] Among them, M A and M B Represents the feature X A and X B Attention map of the image, C represents the number of spectral bands, ζ A and ζ BIt is a learnable channel scaling factor that can dynamically adjust the weights of the bidirectional fusion features, thereby enhancing the robustness of the model. Finally, the two obtained features are concatenated along the spectral dimension to generate the fused features, which are used as the final features for classification.

[0130] Step 4: Use the hyperspectral training sample set to train the hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation. After each training batch is completed, the performance of the trained network is verified using the hyperspectral validation sample set to evaluate the current training status and convergence.

[0131] Furthermore, the Step 4 includes:

[0132] The stochastic gradient descent algorithm is used to update the model parameters, and the loss value is calculated by the cross entropy loss function, which is expressed as:

[0133] l(a,y)=L={l1,…,l N} T ,

[0134]

[0135] Where L represents the total loss, a represents the total input, y represents the label value, N represents the batch size, and w z represents the category weight of category z, Z represents the total number of categories, a n,z Represents the observed sample a n The predicted probability of belonging to category z, y n,z Represents the label vector element, if the sample x n The true category is equal to c and takes 1, otherwise it takes 0.

[0136] Step 5. Input the hyperspectral test sample set into the trained hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation to obtain the category label of each pixel in the test sample set, thereby completing the hyperspectral image classification task.

[0137] The present invention generally first determines the data set: multiple public hyperspectral image data sets are selected as experimental data; data preprocessing: sample blocks are extracted from the hyperspectral data set and divided into training set, validation set and test set; network construction: a spatial-spectral multi-order gated aggregation network based on bidirectional interactive fusion is constructed for hyperspectral image classification; network training: the hyperspectral samples in the training set are input into the constructed network in batches for training, and after each training batch is completed, the classification performance is evaluated using the validation set samples; sample classification: the hyperspectral samples in the test set are input into the trained classification network to obtain the final classification results.

[0138] Furthermore, in this embodiment, the learning rate of the network is set to 0.005, the batch size is 64, and training is performed for 100 rounds of iterations, thereby obtaining the bias parameters and weight file of the final network.

[0139] The hardware platform for the simulation experiment in this embodiment is: a 13th Gen Intel(R) Core(TM) i7-13700H CPU and an NVIDIA GeForce RTX 3090 GPU with 24GB of memory. The configured virtual environment includes: Python 3.9.19, PyTorch 2.3.1 and CUDA 11.5, etc.

[0140] The present invention sends the hyperspectral test sample set into the above-mentioned trained hyperspectral classification network to calculate three common evaluation indicators: overall classification accuracy (OA), average accuracy (AA), and Kappa coefficient. The larger the three indicators, the better the classification effect.

[0141] In order to evaluate the effectiveness of the proposed method, two existing advanced methods, SSRN and SSFTT, were used to classify ground objects in two public hyperspectral datasets, Pavia University and WHU-Hi-LongKou, respectively.

[0142] The SSRN method is a hyperspectral classification method proposed by Z. Zhong et al. in “Spectral–Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework”, referred to as SSRN.

[0143] The SSFTT method is a hyperspectral classification method proposed by L. Sun et al. in “Spectral–Spatial Feature Tokenization Transformer for Hyperspectral Image Classification”, referred to as SSFTT.

[0144] The comparison of the classification results of the three networks under the two data sets is shown in the following table:

[0145]

[0146] Experiments on two mainstream datasets show that the proposed method performs more powerfully than advanced hyperspectral image classification methods and can more accurately predict the pixel sample categories of hyperspectral images.

[0147] Figure 5(a) is the classification result diagram of the Pavia University dataset using the SSRN method;

[0148] Figure 5 (b) is the classification result diagram of the Pavia University dataset using the SSFTT method;

[0149] Figure 5 (c) is a diagram showing the classification results of the present invention on the Pavia University dataset;

[0150] Figure 6 (a) is the classification result diagram of the WHU-Hi-LongKou dataset using the SSRN method;

[0151] Figure 6 (b) is the classification result diagram of the WHU-Hi-LongKou dataset using the SSFTT method;

[0152] Figure 6 (c) is a diagram showing the classification results of the present invention on the WHU-Hi-LongKou dataset;

[0153] It can be clearly seen from the figure that the present invention has the least misclassified pixels and achieves good classification results in both the class with noise points and the boundary area, showing a strong classification ability compared with other methods.

[0154] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network, characterized by: The method comprises: using a constructed hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation to classify hyperspectral images; The hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation includes four core components: preliminary feature extraction module, spatial-spectral feature extraction module, spatial-spectral feature fusion module and classifier module; The preliminary feature extraction module is used to extract local-global spectral features and local spatial features of the input hyperspectral image block to generate primary features; The spatial-spectral feature extraction module is used to aggregate primary features through multiple stages and output aggregated features; The spatial-spectral feature fusion module is used to fuse the output aggregated features to obtain the final fused features; The classifier module is used to classify the final fused features to obtain a category label for each pixel of the hyperspectral image block.

2. The hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network according to claim 1 is characterized by: The specific steps of the method include: Step 1: Prepare multiple public hyperspectral image datasets as experimental data for network training; Step 2: Preprocess the hyperspectral image, extract hyperspectral image blocks centered on each pixel, and divide the extracted hyperspectral image blocks into non-overlapping hyperspectral training sample sets, verification sample sets, and test sample sets; Step 3: Construct a hyperspectral image classification network based on bidirectional interactive fusion of spatial and spectral multi-order gated aggregation; Step 4: Use the hyperspectral training sample set to train the hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation. After each training batch is completed, the performance of the trained network is verified using the hyperspectral validation sample set to evaluate the current training status and convergence. Step 5. Input the hyperspectral test sample set into the trained hyperspectral image classification network based on bidirectional interactive fusion of spatial-spectral multi-order gated aggregation to obtain the category label of each pixel in the test sample set, thereby completing the hyperspectral image classification task.

3. The hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network according to claim 2 is characterized by: The specific steps of Step 2 are as follows: Step 2.

1. Hyperspectral images have three-dimensional characteristics, and their original data are represented as Where H and W represent the spatial size of the image, and C represents the number of spectral bands of the image. The original hyperspectral image is filled with pixels with a value of 0 around the edges, and then the hyperspectral image block is extracted from the filled image. Step 2.2: According to the category of the central pixel of the hyperspectral image block, the corresponding image block is classified into the category set to which it belongs; Step 2.3: For data sets of different sizes, divide the samples in each category set according to the preset ratio: first, select a certain proportion of hyperspectral image blocks as the training sample set, then select the same proportion of hyperspectral image blocks as the verification sample set, and finally select the remaining hyperspectral image blocks as the test sample set.

4. The hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network according to claim 2 is characterized by: The specific steps of Step 3 are as follows: Step 3.

1. Build a preliminary feature extraction module, which is composed of a spectral feature optimization module and a neighborhood pixel embedding module in series, and is used to extract local-global spectral features and local spatial features of the input hyperspectral image block to generate primary features; First, a spectral feature refinement module is constructed, which includes a two-dimensional convolutional layer, a three-dimensional convolutional layer, a normalization layer, and a ReLU activation function layer. At the same time, a neighborhood pixel embedding module is designed, which includes a two-dimensional convolutional layer, a normalization layer, a ReLU activation function layer, and a neighborhood channel embedding block. Step 3.2: Build a spatial-spectral feature extraction module. This module adopts a three-stage structural design, each of which includes a spatial aggregation module and a spectral aggregation module. By integrating the convolution operation and the gated aggregation mechanism into a compact module, a spatial aggregation module for modeling spatial dimension information and a spectral aggregation module for extracting spectral dimension features are constructed respectively, thus achieving multi-level spatial-spectral feature extraction and expression. After the primary features are input into each stage in sequence, each stage outputs the aggregated features X1, X2 and X3 respectively; Step 3.3, build a spatial-spectral feature fusion module and design a bidirectional interactive fusion block based on the bidirectional cross attention mechanism; first, the aggregated features output by the second stage are adjusted to the spectral dimension through a 1×1 convolution layer and the aggregated features output by the third stage, and then input into the first bidirectional interactive fusion module for fusion; then, the fused features output by the first bidirectional interactive fusion module and the aggregated features X1 output by the first stage are input into the second bidirectional interactive fusion module to obtain the final fused features X f ; Step 3.4, build a classifier module consisting of a Dropout layer, a global average pooling layer and a linear classification layer in series, and combine the final fusion feature X f The module inputs the category label of each pixel of the hyperspectral image patch.

5. The hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network according to claim 4 is characterized by: In Step 3.1, the two-dimensional convolution layer and the three-dimensional convolution layer of the spectral feature refinement module include two two-dimensional convolution modules and three three-dimensional convolution modules respectively; the convolution module structure is: convolution layer → normalization layer → ReLU activation function layer; the structure of the spectral feature refinement module is: first two-dimensional convolution module → first three-dimensional convolution module → second three-dimensional convolution module → third three-dimensional convolution module → second two-dimensional convolution module; wherein: The convolution kernel size of the two-dimensional convolution module is set to 1×1, the convolution kernel sizes of the first three-dimensional convolution module and the third three-dimensional convolution module are set to 1×1×3, and the convolution kernel size of the second three-dimensional convolution module is set to 1×1×7.

6. The hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network according to claim 4 is characterized by: In Step 3.1, the neighborhood pixel embedding module specifically includes a normalization layer, a 3×3 convolution module, a 1×1 convolution layer, and a neighborhood channel embedding module; the structure of the convolution module is fixed as follows: convolution layer → normalization layer → ReLU activation function layer; the structure of the neighborhood pixel embedding module is fixed as follows: normalization layer → 3×3 convolution module → neighborhood channel embedding module → 1×1 convolution layer; wherein: The neighborhood channel embedding module includes multiple parallel 1×1 convolution branches. For the input feature map, each spectral branch processes a spectral band and its neighborhood spectral subset. Specifically, for the i-th spectrum, the input of the current spectral branch is intercepted with the spectrum as the center and one adjacent spectrum before and after it. The input first passes through the corresponding 1×1 convolution layer, and then passes through the ReLU activation function layer and the normalization layer. After all spectral branches independently complete the above operations, their output features are spliced ​​along the spectral dimension to form a complete feature map.

7. The hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network according to claim 4 is characterized by: Step 3.2 includes: Step 3.2.1, build the spatial aggregation module; The spatial aggregation block includes a normalization layer, a feature decomposition unit, and a multi-order spatial gating aggregation unit. Its structure is fixed as follows: normalization layer → feature decomposition unit → multi-order spatial gating aggregation unit; it can be expressed as: x spa =x+MoGA(FD(Norm(x))) Among them, x represents the input features of each stage, x spa Represents the features output by the spatial aggregation block, Norm represents the normalization layer, FD represents the feature decomposition unit, and MoGA represents the multi-order spatial gated aggregation unit; The feature decomposition unit directly extracts local features and global statistical features through a 1×1 convolution layer and a global average pooling layer respectively. The feature decomposition unit formula is expressed as: Among them, x FD Represents the features output by the feature decomposition unit, Conv 1×1 represents a 1×1 convolutional layer, δ a is a learnable scaling factor, with the initial value set to 0; GAP represents the global average pooling layer, and GELU represents the GELU activation function; The multi-order spatial gating aggregation unit consists of a gating branch φ(g) and a dynamic feature integration branch ψ(d); a depthwise separable convolution DWConv is introduced into ψ(d) to extract multi-order features, and three DWConv layers with different expansion rates r∈{1,2,3} are used; first, a 5×5 DWConv is used to extract low-order features; then, the features output by the feature decomposition unit are decomposed into Satisfy C l +C m +C h =C; then, and Input to 5×5DWConv and 3×3DWConv respectively, and Directly retained as the identity mapping; finally, the three branch features are spliced ​​along the spectral dimension as In order to adaptively aggregate the features in ψ(d), the SiLU activation function is used in φ(g), and the multi-order spatial gated aggregation unit is finally expressed as: Step 3.2.2, build the spectrum aggregation module; The spectrum aggregation block includes a normalization layer, a 1×1 convolution layer, a 3×3 depth-separable convolution layer, a spectrum redistribution unit, and a GELU activation function layer. Its structure is fixed as follows: normalization layer → 1×1 convolution layer → 3×3 depth-separable convolution layer → GELU activation function layer → spectrum redistribution unit → 1×1 convolution layer; it can be expressed as: x′ spa =GELU(DWConv 3×3 (Conv. 1×1 (Norm(x) spa )))) x spe =Conv 1×1 (SR(x′ spa ))+x′ spa Among them, Norm represents the normalization layer, DWConv 3×3 represents a 3×3 depth-wise separable convolutional layer, SR represents a spectral redistribution unit, and W is projected through dimensionality reduction: And the GELU activation function collects and redistributes spectral information; the specific formula is as follows: SR(x′ spa )=x′ spa +δ e ⊙(x′ spa -GEL(x′ spa W)) Among them, δ e is the spectral scaling factor; the spectral redistribution unit redistributes the spectral features through complementary interactions, thereby enhancing the model's ability to represent spectral information that was previously not captured.

8. The hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network according to claim 4 is characterized by: In Step 3.3, the bidirectional interactive fusion module includes a first bidirectional interactive fusion module and a second bidirectional interactive fusion module. The bidirectional interactive fusion module is based on two inputs and includes two parallel branches. For the first bidirectional interactive fusion module, its input features are the aggregated feature X2 output from the second stage of the spatial-spectral feature extraction module and the aggregated feature X3 output from the third stage. The upper branch is dominated by X2, and the lower branch is dominated by X3. The two branches implement information interaction between features through a bidirectional cross-attention mechanism, and obtain attention maps corresponding to X2 and X3 respectively. Then, learnable weights are introduced to the attention maps, and residual fusion is performed with the original features X2 and X3. Finally, the two fused features are spliced ​​along the spectral dimension, and the fused features are output. For the second bidirectional interactive fusion module, its input features are the aggregated features X1 output by the first stage of the spatial-spectral feature extraction module and the fused features output by the first bidirectional interactive fusion module. The rest of the processing flow is the same as that in the first bidirectional interactive fusion module. Finally, the second bidirectional interactive fusion module outputs the final fused features.

9. The hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network according to claim 4 is characterized by: The Step 3.3 also includes: The bidirectional interactive fusion module takes two input features X A and X B is the input, where the input feature X A and X B The values ​​of are the input of the first two-way interactive fusion module or the input of the second two-way interactive fusion module, which are first normalized and then pass through two independent linear mappings. The specific formula is as follows: Q A =W A (Norm(X A )),Q B =W1(Norm(X B )) Among them, W A and W B Represents the linear mapping weight matrix, Q A and Q B Indicates that from X A and X B The extracted query features; at the same time, two independent linear mappings are also used to transform X A and X B Mapped to corresponding values ​​V A and V B ; Then Q A , Q B 、V A and V B A bidirectional cross-attention mechanism is used to fuse the two features: X′ A =ζ A M B +X A X′ B =ζ B M A +X B Among them, M A and M B Represents the feature X A and X B Attention map of the image, C represents the number of spectral bands, ζ A and ζ B It is a learnable channel scaling factor that can dynamically adjust the weight of the bidirectional fusion feature. Finally, the two features obtained are concatenated along the spectral dimension to generate the fused feature, which is used as the final feature for classification.

10. The hyperspectral image classification method based on a bidirectional interactive fusion spatial-spectral multi-order gated aggregation network according to claim 1, characterized in that: Step 4 includes: The stochastic gradient descent algorithm is used to update the model parameters, and the loss value is calculated by the cross entropy loss function, which is expressed as: l(a,y)=L={l1,…,l N } T , Where L represents the total loss, a represents the total input, y represents the label value, N represents the batch size, and w z represents the category weight of category z, Z represents the total number of categories, a n,z Represents the observed sample a n The predicted probability of belonging to category z, y n,z Represents a label vector element.

Images