Spatial-spectral combined hyperspectral image classification method based on multi-scale adaptive attention mechanism

Abstract

The invention discloses a spatial-spectral combined hyperspectral image classification method based on a multi-scale adaptive attention mechanism, and the method comprises the steps: firstly carrying out the extraction and fusion of features of different scales through a multi-scale feature fusion module, and comprehensively capturing the detailed spectral information of a hyperspectral image; secondly, dynamically adjusting a feature extraction process by using an efficient self-adaptive spectrum-space feature extraction module, and optimizing fusion and representation of spectrum-space features; and finally, enhancing the ability of the model to understand the context relationship in the image by using the context-aware fusion network. The three modules provided by the method are based on a deep learning framework, a convolutional neural network in deep learning can utilize multiband rich spectral information, and the method is suitable for hyperspectral remote sensing image data sets with different spatial resolutions, including a foreign hyperspectral data set and a domestic hyperspectral data set. The method has high adaptability and transportability, and can meet the requirement of image classification more easily.

Description

Technical Field

[0001] The present invention relates to an image classification method, in particular to a spatial-spectral combined hyperspectral image classification method based on a multi-scale adaptive attention mechanism. Background Art

[0002] Hyperspectral images provide rich information about materials and objects in the image scene by analyzing the spectral spatial characteristics of images captured by multiple narrow continuous bands on the electromagnetic spectrum. However, in addition to the large amount of hyperspectral image data, high dimensionality, and information redundancy, it is difficult to identify different land cover categories from limited labeled samples during model training, which affects the classification accuracy of the model.

[0003] In recent years, deep learning techniques have shown great potential in the field of hyperspectral image classification and have achieved remarkable results. Using these techniques, researchers can extract comprehensive features from hyperspectral images to facilitate accurate classification and recognition. Initially, researchers only used convolutional layers to solve hyperspectral image classification tasks such as 1D-CNN, 2D-CNN, and 3D-CNN. Then, with the design of more complex and deeper network models, the performance of hyperspectral image classification tasks changed.

[0004] In the current study, although many models can effectively extract spectral spatial information from hyperspectral images, the practical limitation of insufficient labeled samples in hyperspectral datasets emphasizes the necessity of network models that can robustly extract spectral spatial features under such constraints. In addition, existing Transformer-based models, although powerful, often require a lot of computation and may not effectively capture fine-grained spatial details and scale variations, resulting in poor performance in hyperspectral image classification. Summary of the invention

[0005] The object of the present invention is to provide a spatial-spectral joint hyperspectral image classification method based on a multi-scale adaptive attention mechanism, which can improve the classification performance of hyperspectral images.

[0006] The invention is implemented as follows:

[0007] Firstly, a multi-scale feature fusion module is used to extract and fuse features of different scales to fully capture the detailed spectral information of the hyperspectral image.

[0008] Secondly, an efficient adaptive spectral-spatial feature extraction module is used to dynamically adjust the feature extraction process and optimize the fusion and representation of spectral-spatial features;

[0009] Finally, a context-aware fusion network is used to enhance the model’s ability to understand contextual relationships within images.

[0010] The present invention may also include:

[0011] 1. The image is a hyperspectral image.

[0012] 2. The use of a multi-scale feature fusion module to extract and fuse features of different scales specifically includes:

[0013] The multi-scale feature fusion module consists of four parallel branches, each of which focuses on extracting spectral features at different frequency scales. Specifically, assuming the input tensor is X∈R N×C×D×μ×W , N is the batch size, C is the number of input channels, D is the depth (i.e., spectral dimension), H is the height, and W is the width. Then, the number of channels is divided into four parts, and 1×1×3, 1×1×5, 1×1×7, and 1×1×9 three-dimensional depth-separable convolution extraction is performed respectively. The convolution operation is followed by batch normalization and ReLU6 activation function to promote faster convergence and improved feature representation;

[0014] When the model is initialized, self.c is one-fourth of the number of input channels C, which is used to set the number of output channels of each convolution branch. This is usually done to reduce computational complexity and memory consumption. The number of channels is divided into:

[0015]

[0016] At this time, the divided input tensor is For each convolution branch, operations are performed using different convolution kernel sizes and grouped convolutions, where the operation of each branch is as follows:

[0017]

[0018] Among them, ReLU6 is a variant of ReLU, and the formula is:

[0019] ReLU6(x)=min(max(0,x),6)

[0020] The formula for BatchNorm3D is:

[0021]

[0022] where μ is the mean of each channel, ξ 2 is the variance of each channel, is a small constant, γ and β are learnable scaling and offset parameters;

[0023] Finally, the outputs of the four different convolutional branches are concatenated along the channel dimension (i.e., the second dimension):

[0024]

[0025] in These are the outputs after 1×1×3, 1×1×5, 1×1×7, and 1×1×9 convolutions respectively;

[0026] The concatenated output tensor is Y∈R N×C×D×H×W , the number of channels is 4×self.c (i.e. C).

[0027] 3. The efficient adaptive spectral-spatial feature extraction module specifically includes:

[0028] The spectral-spatial adaptive attention mechanism module and the convolutional multi-layer perceptron module, in which the spectral-spatial adaptive attention mechanism module first reduces the dimension of the input tensor by a 1×1 convolution layer, and applies group normalization and ReLU activation function; then enhances the features by a 3×3 convolution layer, and applies batch normalization again. Then, the weights of the feature map are calculated using an adaptive average pooling layer and a Softmax function; finally, the result is added to the original input through residual cascade to obtain the final output.

[0029] This convolutional multilayer perceptron module is used for nonlinear transformation of input features. First, the feature transformation is performed through a 1×1 convolution layer, and then the activation function and Dropout layer are applied to nonlinear transformation and regularization.

[0030] The spectral-spatial adaptive attention mechanism module first generates spatial attention on the input feature map through a 1×1 convolutional layer to generate a spatial attention map. Then, the ReLU activation function is applied to the nonlinear transformation of the convolution output to obtain the activated attention map. The activated attention is then normalized using the Softmax function to obtain the spatial attention weights. Next, these normalized attention weights are applied to the input feature map and weighted to generate a weighted feature map. Finally, the weighted feature map is added element-by-element to the original feature map to obtain the final enhanced feature map, thereby completing the feature enhancement process of the spatial adaptive attention mechanism.

[0031] Assume that the input feature map is I∈R H×W×C , where H is the height, W is the width, and C is the number of channels; a 1×1 convolution operation is used to generate a spatial attention map. The purpose of this step is to map each position of the feature map to an attention value. Assume that the weight of the convolution kernel is W∈R 1×1×C×1 , then the convolution operation is:

[0032] A=Conv 1×1 (I)∈R H×W×1

[0033] in, Among them A i，j is the spatial attention value of position (i, j).

[0034] Non-linear activation of convolution output:

[0035] A raw =ReLU(I)∈R H×W×1

[0036] The activated spatial attention is normalized by the Softmax function to obtain the attention weight;

[0037] A norm =softmax(A raw )∈R H×W×1

[0038] The softmax operation is performed at each spatial position as follows:

[0039] I′ i，j =A raw，i，j ×I i，j , I′∈R H×W×C

[0040] Apply the normalized spatial attention map to the input feature map for weighting:

[0041] I″=I+I′,I″∈R H×W×C

[0042] The efficient adaptive spectral-spatial feature extraction module combines the spectral-spatial adaptive attention mechanism module and the convolutional multi-layer perceptron module to realize an efficient adaptive spatial fusion mechanism; the integration module utilizes residual connection and adaptive spatial attention mechanism, and through the synergy of multiple sub-modules, it can effectively extract and fuse the spatial information of input features, thereby improving the performance of the model.

[0043] 4. The context-aware fusion network specifically includes:

[0044] It can adapt to different frequency bands, optimize feature representation, reduce redundant information, and enhance the importance of key features, thereby significantly improving the performance of the model on complex data sets. The context-aware network consists of three convolutional layers, a Conv2D layer with two 1×1 convolutional kernels and a PWConv2D layer with a 3×3 convolutional kernel. Assume that the input data is Y in ∈R H×W×D First, Y in The input Conv2D layer performs feature mapping and then uses the GELU activation function for nonlinear operations. The PWConv2D layer then maps each input by grouping features, further uses the GELU activation function, and uses the Conv2D layer mapping to produce the final output;

[0045] The above process can be expressed as:

[0046] Y = Conv2D1×1 (PWConv2D 3×3 (Conv2D 1×1 (Y in )))

[0047] This paper proposes a new algorithm based on a spatial-spectral joint hyperspectral image classification method based on a multi-scale adaptive attention mechanism, which is used to enhance the classification of hyperspectral images. Traditional methods often face problems such as scale change, multi-scale feature integration and contextual relationship fusion, which limit the classification effect. In order to solve these problems, this paper adopts a multi-scale adaptive fusion strategy, which integrates three advanced modules: a multi-scale feature fusion module, an efficient adaptive spectral-spatial feature extraction module and a convolutional multi-layer perceptron. The multi-scale feature fusion module plays an important role in capturing and integrating spectral spatial information of various scales, enabling the model to identify fine-grained spectral spatial details. This method dynamically adapts to different scales and enhances feature representation by effectively fusing spectral spatial information at different resolutions. At the same time, the multi-layer perceptron focuses on improving the model's ability to understand and utilize contextual relationships in the data, further improving classification accuracy.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) A multi-scale feature fusion module is proposed, which captures and fuses the multi-angle features of different frequency bands of the input data by introducing multiple three-dimensional convolution operations of different scales. This strategy effectively enhances the model's ability to extract complex features from hyperspectral data, enabling the model to more accurately distinguish and classify different types of objects when processing hyperspectral images with multi-band information.

[0050] (2) An efficient adaptive spectral-spatial feature extraction module is proposed. First, 1×1 convolution is used for dimension transformation and feature preservation. Second, the spectral-spatial adaptive attention mechanism module is combined to achieve spatial adaptive attention enhancement. Finally, the convolutional multi-layer perceptron module is used to further improve the nonlinear expression ability of features. This strategy effectively enhances the model's ability to extract and characterize complex hyperspectral data features, thereby improving classification accuracy and generalization performance.

[0051] (3) The present invention is experimentally verified on four benchmark hyperspectral image datasets: Houston2013, Botswana, WHU-Hi-HanChuan and WHU-Hi-HongHu. The experimental results of the present invention show that the proposed hyperspectral image classification architecture is comparable to or even better than some of the most advanced algorithms in this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1This is a flow chart of the spatial-spectral joint hyperspectral image classification method based on the multi-scale adaptive attention mechanism;

[0053] Figure 2 This is a framework diagram of the spatial-spectral joint hyperspectral image classification method based on the multi-scale adaptive attention mechanism;

[0054] Figure 3 This is a framework diagram of the multi-scale feature fusion module;

[0055] Figure 4 It is a framework diagram of an efficient adaptive spectral-spatial feature extraction module;

[0056] Figure 5(a) is the real distribution map of 15 types of features in the Houston2013 dataset;

[0057] Figure 5(b) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm HybridSN;

[0058] Figure 5(c) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm SSFTT;

[0059] Figure 5(d) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm LSFAT;

[0060] Figure 5(e) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm LSGA;

[0061] Figure 5(f) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm morphFormer;

[0062] Figure 5(g) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm MRSST;

[0063] Figure 5(h) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm DBCT;

[0064] Figure 5(i) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm DCTN;

[0065] Figure 5(j) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm MASSFormer;

[0066] Figure 5(k) is the overall classification accuracy graph of the Houston2013 dataset based on the comparison algorithm MCAM;

[0067] Figure 6(a) is the actual distribution map of 14 types of features in the Botswana dataset;

[0068] Figure 6(b) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm HybridSN;

[0069] Figure 6(c) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm SSFTT;

[0070] Figure 6(d) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm LSFAT;

[0071] Figure 6(e) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm LSGA;

[0072] Figure 6(f) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm morphFormer;

[0073] Figure 6(g) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm MRSST;

[0074] Figure 6(h) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm DBCT;

[0075] Figure 6(i) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm DCTN;

[0076] Figure 6(j) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm MASSFormer;

[0077] Figure 6(k) is the overall classification accuracy graph of the Botswana dataset based on the comparison algorithm MCAM;

[0078] Figure 7(a) is the real distribution map of 16 types of objects in the WHU-Hi-HanChuan dataset;

[0079] Figure 7(b) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm HybridSN;

[0080] Figure 7(c) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm SSFTT;

[0081] Figure 7(d) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm LSFAT;

[0082] Figure 7(e) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm LSGA;

[0083] Figure 7(f) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm morphFormer;

[0084] Figure 7(g) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm MRSST;

[0085] Figure 7(h) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm DBCT;

[0086] Figure 7(i) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm DCTN;

[0087] Figure 7(j) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm MASSFormer;

[0088] Figure 7(k) is the overall classification accuracy graph of the WHU-Hi-HanChuan dataset based on the comparison algorithm MCAM;

[0089] Figure 8(a) is the real distribution map of 22 types of features in the WHU-Hi-HongHu dataset;

[0090] Figure 8(b) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm HybridSN;

[0091] Figure 8(c) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm SSFTT;

[0092] Figure 8(d) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm LSFAT;

[0093] Figure 8(e) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm LSGA;

[0094] Figure 8(f) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm morphFormer;

[0095] Figure 8(g) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm MRSST;

[0096] Figure 8(h) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm DBCT;

[0097] Figure 8(i) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm DCTN;

[0098] Figure 8(j) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm MASSFormer;

[0099] Figure 8(k) is the overall classification accuracy graph of the WHU-Hi-HongHu dataset based on the comparison algorithm MCAM;

[0100] Figure 9(a) shows the OA of the 10 algorithms using 2%-10% training sample ratio on the Houston2013 dataset;

[0101] Figure 9(b) shows the OA of the 10 algorithms using 2%-10% training sample ratio on the Botswana dataset;

[0102] Figure 9(c) shows the OA of the 10 algorithms using 2%-10% training sample ratio on the WHU-Hi-HanChuan dataset;

[0103] Figure 9(d) shows the OA of the 10 algorithms using 2%-10% training sample ratio on the WHU-Hi-HongHu dataset;

[0104] Figure 10(a) shows the overall classification accuracy of the MCAM algorithm on the Houston2013 dataset using different learning rates;

[0105] Figure 10(b) shows the overall classification accuracy of the MCAM algorithm on the Botswana dataset using different learning rates;

[0106] Figure 10(c) shows the overall classification accuracy of the MCAM algorithm on the WHU-Hi-HanChuan dataset using different learning rates;

[0107] Figure 10(d) shows the overall classification accuracy of the MCAM algorithm on the WHU-Hi-HongHu dataset using different learning rates;

[0108] Figure 11(a) shows the overall classification accuracy of the MCAM algorithm using different block sizes on the Houston 2013 dataset;

[0109] Figure 11(b) shows the overall classification accuracy of the MCAM algorithm using different block sizes on the Botswana dataset;

[0110] Figure 11(c) shows the overall classification accuracy of the MCAM algorithm on the WHU-Hi-HanChuan dataset using different block sizes;

[0111] Figure 11(d) shows the overall classification accuracy of the MCAM algorithm using different block sizes on the WHU-Hi-HongHu dataset. DETAILED DESCRIPTION

[0112] The present invention is described in more detail with reference to the following examples.

[0113] Firstly, a multi-scale feature fusion module is used to extract and fuse features of different scales to fully capture the detailed spectral information of the hyperspectral image.

[0114] Secondly, an efficient adaptive spectral-spatial feature extraction module is used to dynamically adjust the feature extraction process and optimize the fusion and representation of spectral-spatial features;

[0115] Finally, a context-aware fusion network is used to enhance the model’s ability to understand contextual relationships within an image;

[0116] The method of the present invention is applicable to hyperspectral remote sensing image datasets of different spatial resolutions, including foreign hyperspectral datasets and domestic hyperspectral datasets, has strong adaptability and portability, and can more easily meet the needs of image classification.

[0117] The present invention specifically includes:

[0118] 1. The multi-scale feature fusion module proposed in the present invention extracts and fuses features of different scales, and the contents are as follows:

[0119] The multi-scale feature fusion module consists of four parallel branches, each of which focuses on extracting spectral features at different frequency scales. Specifically, assuming the input tensor is X∈R N×C×D×H×W , N is the batch size, C is the number of input channels, D is the depth (i.e., spectral dimension), H is the height, and W is the width. Then, the number of channels is divided into four parts, and 1×1×3, 1×1×5, 1×1×7 and 1×1×9 three-dimensional depth-separable convolution extraction are performed respectively. The convolution operation is followed by batch normalization and ReLU6 activation function to promote faster convergence and improved feature representation.

[0120] When the model is initialized, self.c is one-fourth of the number of input channels C, which is used to set the number of output channels of each convolution branch. This is usually done to reduce computational complexity and memory consumption. The number of channels is divided into:

[0121]

[0122] At this time, the divided input tensor is For each convolution branch, we operate with different convolution kernel sizes and use grouped convolution, where the operation of each branch is as follows:

[0123]

[0124] Among them, ReLU6 is a variant of ReLU, and the formula is:

[0125] ReLU6(x)=min(max(0,x),6)

[0126] The formula for BatchNorm3D is:

[0127]

[0128] where μ is the mean of each channel, ξ 2 is the variance of each channel, is a small constant, and γ and β are learnable scaling and bias parameters.

[0129] Finally, the outputs of the four different convolutional branches are concatenated along the channel dimension (i.e., the second dimension):

[0130]

[0131] in They are the outputs after 1×1×3, 1×1×5, 1×1×7, and 1×1×9 convolutions respectively.

[0132] The concatenated output tensor is Y∈R N×C×D×H×W , the number of channels is 4×self.c (i.e. C).

[0133] 2. The efficient adaptive spectral-spatial feature extraction module proposed in the present invention specifically includes:

[0134] The spectral-spatial adaptive attention mechanism module and the convolutional multi-layer perceptron module are used. The spectral-spatial adaptive attention mechanism module first reduces the dimension of the input tensor by a 1×1 convolution layer and applies group normalization and ReLU activation function. The features are then enhanced by a 3×3 convolution layer and batch normalization is applied again. Then, the weights of the feature maps are calculated using an adaptive average pooling layer and a Softmax function. Finally, the result is added to the original input through residual cascade to obtain the final output.

[0135] This convolutional multilayer perceptron module is used for nonlinear transformation of input features. First, the feature transformation is performed through a 1×1 convolution layer, and then the activation function and Dropout layer are applied to nonlinear transformation and regularization.

[0136] The spectral-spatial adaptive attention mechanism module first generates spatial attention on the input feature map through a 1×1 convolutional layer to generate a spatial attention map. Then, the ReLU activation function is applied to the nonlinear transformation of the convolution output to obtain the activated attention map. The activated attention is then normalized using the Softmax function to obtain the spatial attention weights. Next, these normalized attention weights are applied to the input feature map and weighted to generate a weighted feature map. Finally, the weighted feature map is added element-by-element to the original feature map to obtain the final enhanced feature map, thereby completing the feature enhancement process of the spatial adaptive attention mechanism.

[0137] Assume that the input feature map is I∈R H×W×C , where H is the height, W is the width, and C is the number of channels. A 1×1 convolution operation is used to generate a spatial attention map. The purpose of this step is to map each position of the feature map to an attention value. Assume that the weight of the convolution kernel is W∈R 1×1×C×1 , then the convolution operation is:

[0138] A=Conv 1×1 (I)∈R H×W×1

[0139] in, Among them A i，j is the spatial attention value of position (i, j).

[0140] Non-linear activation of convolution output:

[0141] A raw =ReLU(I)∈R H×W×1

[0142] The activated spatial attention is normalized by the Softmax function to obtain the attention weight.

[0143] A norm =softmax(A raw )∈R H×W×1

[0144] The softmax operation is performed at each spatial position as follows:

[0145] I′ i，j =A raw，i，j ×I i，j , I′∈R H×W×C

[0146] Apply the normalized spatial attention map to the input feature map for weighting:

[0147] I″=I+I′,I″∈R H×W×C

[0148] The efficient adaptive spectral-spatial feature extraction module combines the spectral-spatial adaptive attention mechanism module and the convolutional multi-layer perceptron module to realize an efficient adaptive spatial fusion mechanism. The integrated module utilizes residual connection and adaptive spatial attention mechanism. Through the synergy of multiple sub-modules, it can effectively extract and fuse the spatial information of input features, thereby improving the performance of the model.

[0149] 3. The context-aware fusion network proposed in the present invention specifically includes:

[0150] It can adapt to different frequency bands, optimize feature representation, reduce redundant information, and enhance the importance of key features, thereby significantly improving the performance of the model on complex data sets. The context-aware network consists of three convolutional layers, a Conv2D layer with two 1×1 convolutional kernels and a PWConv2D layer with a 3×3 convolutional kernel. Assume that the input data is Y in ∈R H×W×D , Yin is first input into the Conv2D layer for feature mapping, and then the GELU activation function is used for nonlinear operation. The PWConv2D layer then maps each input through the grouped feature map, then GELU activation, and then another Conv2D layer mapping to produce the final output.

[0151] The above process can be expressed as:

[0152] Y = Conv2D 1×1 (PWConv2D 3×3 (Conv2D 1×1 (Y in )))

[0153] Figure 1 This is a flow chart of the spatial-spectral joint hyperspectral image classification method based on the multi-scale adaptive attention mechanism. The process includes first using the multi-scale feature fusion module to extract and fuse features of different scales; secondly, dynamically adjusting the feature extraction process, combining the adaptive attention mechanism to enhance the attention to spectral features; finally, using the context-aware network enhancement model to understand the contextual relationship.

[0154] Figure 2 This is a framework diagram of the spatial-spectral joint hyperspectral image classification method based on the multi-scale adaptive attention mechanism. The basic framework includes a multi-scale feature fusion module, an efficient adaptive spectral-spatial feature extraction module and a context-aware fusion network.

[0155] Figure 3 It is a multi-scale feature fusion module that obtains rich feature information through four different convolution branches and then fuses the features after convolution together.

[0156] Figure 4It is an efficient adaptive spectral-spatial feature extraction module that combines the spectral-spatial adaptive attention mechanism module and the convolutional multi-layer perceptron module to realize an efficient adaptive spatial fusion mechanism. The integrated module utilizes residual connection and adaptive spatial attention mechanism. Through the synergy of multiple sub-modules, it can effectively extract and fuse the spatial information of input features, thereby improving the performance of the model.

[0157] Figure 5(a)-Figure 5(k) The Houston2013 dataset is a real hyperspectral image data. It was collected using the CASI1500 sensor in and around the University of Houston, Texas, USA. The image resolution of the dataset is 349×1905 pixels and contains 144 spectral bands with a wavelength range from 0.38 to 1.05 microns (μm). It includes data from 15 different ground feature categories. Figure 5(b)-Figure 5(k) Represents the classification results of the other 10 algorithms on the Houston2013 dataset. These 10 algorithms are HybridSN, SSFTT, LSFAT, LSGA, morphFormer, MRSST, DBCT, DCTN, MASSFormer and MCAM. From these 10 classification result graphs, it can be seen that the proposed MCAM algorithm obtains the best classification graph (i.e., Figure 5(k)).

[0158] Figure 6(a)-Figure 6(k) It is a real hyperspectral image data. The Botswana dataset was acquired by NASA's EO-1 satellite over the Okavango Delta in Botswana. The dataset consists of a 7.7 km long data strip collected by the Hyperion sensor at a resolution of 30 million pixels in 242 bands, covering the 400-2500nm spectral part of the 10nm window. The absorption band is removed, leaving 145 spectral bands. The dataset can be divided into 14 categories. Figure 6(b)-Figure 6(k) Represents the classification results of the other 10 algorithms on the Botswana dataset. These 10 algorithms are HybridSN, SSFTT, LSFAT, LSGA, morphFormer, MRSST, DBCT, DCTN, MASSFormer and MCAM. From these 10 classification result graphs, it can be seen that the proposed MCAM algorithm obtains the best classification graph (i.e., Figure 6(k)).

[0159] Figure 7(a)-Figure 7(k)It is a real hyperspectral image data. The WHU-Hi-HanChuan data was captured by a 17mm focal length Headwall Nano-Hyperspec sensor mounted on a Leica AibotX6 drone V1. The weather was clear and sunny, with a temperature of about 30 degrees Celsius and a relative humidity of 70%. The surveyed area is an urban-rural junction where buildings, water and cultivated land are mixed together, with seven main crops: strawberry, cowpea, soybean, sorghum, water spinach, watermelon and vegetables. The drone operates at an altitude of 250 meters, producing images with a size of 1217×303 pixels, covering 274 spectral bands from 400 to 1000nm, with a spatial resolution of about 0.109 meters. Due to the low elevation angle of the sun in the afternoon, many shadow areas are included in the image. Figure 7(b)-Figure 7(k) Represents the classification results of the other 10 algorithms on the WHU-Hi-HanChuan dataset. These 10 algorithms are HybridSN, SSFTT, LSFAT, LSGA, morphFormer, MRSST, DBCT, DCTN, MASSFormer and MCAM. It can be seen from these 10 classification result graphs that the proposed MCAM algorithm obtains the best classification graph (i.e., Figure 7(k)).

[0160] Figure 8(a)-Figure 8(k) This is a real hyperspectral image data, WHU-Hi-HongHu data collection, using a 17mm focal length Headwall Nano-Hyperspec sensor mounted on a DJI matrix 600Pro drone. The weather during data collection was cloudy, with a temperature of about 8°C and a relative humidity of 55%. The area studied is a diverse agricultural landscape featuring a variety of crops, including different varieties of Chinese cabbage, Chinese cabbage, Brassica rapa, and Brassica rapa. The drone operates at an altitude of 100 meters and captures images at a resolution of 940×475 pixels in 270 spectral bands from 400 to 1000nm, with a spatial resolution of about 0.043 meters. Figure 8(b)-Figure 8(k) Represents the classification results of the other 10 algorithms on the WHU-Hi-HongHu dataset. These 10 algorithms are HybridSN, SSFTT, LSFAT, LSGA, morphFormer, MRSST, DBCT, DCTN, MASSFormer and MCAM. From these 10 classification result graphs, it can be seen that the proposed MCAM algorithm obtains the best classification graph (i.e., Figure 8(k)).

[0161] Figure 9(a)-Figure 9(d) The variation of classification accuracy among ten different algorithms using different proportions of training samples on four datasets is shown.

[0162] Reference Figure 10(a)-Figure 10(d), the changes of overall classification accuracy under different learning rates are studied and analyzed. In Figure 10(a), for the Houston2013 dataset, the best overall classification accuracy is obtained when the learning rate is 0.0002. In Figure 10(b), for the Botswana dataset, the best overall classification accuracy is obtained when the learning rate is 0.0004. In Figure 10(c), for the WHU-Hi-HanChuan dataset, the best overall classification accuracy is obtained when the learning rate is 0.0002. In Figure 10(d), for the WHU-Hi-HongHu dataset, the best overall classification accuracy is obtained when the learning rate is 0.0003.

[0163] Reference Figure 11(a)-Figure 11(d) , the changes of overall classification accuracy under different block sizes are studied and analyzed. In Figure 11(a), for the Houston2013 dataset, the optimal overall classification accuracy is obtained when the block size is 11×11. In Figure 11(b), for the Botswana dataset, the optimal overall classification accuracy is obtained when the block size is 17×17. In Figure 11(c), for the WHU-Hi-HanChuan dataset, the optimal overall classification accuracy is obtained when the block size is 15×15. In Figure 11(d), for the WHU-Hi-HongHu dataset, the optimal overall classification accuracy is obtained when the block size is 15×15.

[0164] The above is a specific embodiment of the present invention and is not intended to limit the present invention. The spatial-spectral joint hyperspectral image classification method based on the multi-scale adaptive attention mechanism provided by the present invention is also applicable to the classification of other non-hyperspectral images. Without departing from the essence and scope of the present invention, some adjustments and optimizations can be made, and the protection scope of the present invention shall be subject to the claims.

Claims

1. A spatial-spectral joint hyperspectral image classification method based on a multi-scale adaptive attention mechanism, characterized by: Firstly, a multi-scale feature fusion module is used to extract and fuse features of different scales to fully capture the detailed spectral information of the hyperspectral image. Secondly, an efficient adaptive spectral-spatial feature extraction module is used to dynamically adjust the feature extraction process and optimize the fusion and representation of spectral-spatial features; Finally, a context-aware fusion network is used to enhance the model’s ability to understand contextual relationships within an image; The use of the multi-scale feature fusion module to extract and fuse features of different scales specifically includes: The multi-scale feature fusion module consists of four parallel branches, each of which focuses on extracting spectral features at different frequency scales. Specifically, assuming the input tensor is X∈R N×C×D×H×W , N is the batch size, C is the number of input channels, D is the depth (i.e., spectral dimension), H is the height, and W is the width. Then, the number of channels is divided into four parts, and 1×1×3, 1×1×5, 1×1×7, and 1×1×9 three-dimensional depth-separable convolution extraction is performed respectively. The convolution operation is followed by batch normalization and ReLU6 activation function to promote faster convergence and improved feature representation; When the model is initialized, self.c is one-fourth of the number of input channels C, which is used to set the number of output channels of each convolution branch. This is usually done to reduce computational complexity and memory consumption. The number of channels is divided into: At this time, the divided input tensor is For each convolution branch, operations are performed using different convolution kernel sizes and grouped convolutions, where the operation of each branch is as follows: Among them, ReLU6 is a variant of ReLU, and the formula is: ReLU6(x)=min(max(0,x),6) The formula for BatchNorm3D is: where μ is the mean of each channel, ξ 2 is the variance of each channel, is a small constant, γ and β are learnable scaling and offset parameters; Finally, the outputs of the four different convolutional branches are concatenated on the channel dimension (i.e., the second dimension); in These are the outputs after 1×1×3, 1×1×5, 1×1×7, and 1×1×9 convolutions respectively; The concatenated output tensor is Y∈R N×C×D×H×W , the number of channels is 4×self.c (i.e. C).

2. The spatial-spectral joint hyperspectral image classification method based on multi-scale adaptive attention mechanism according to claim 1 is characterized by: The efficient adaptive spectral-spatial feature extraction module includes a spectral-spatial adaptive attention mechanism module and a convolutional multi-layer perceptron module. The spectral-spatial adaptive attention mechanism module first reduces the dimension of the input tensor by a 1×1 convolution layer and applies group normalization and ReLU activation function. It then enhances the features by a 3×3 convolution layer and applies batch normalization again. Then, it calculates the weights of the feature map using an adaptive average pooling layer and a Softmax function. Finally, the result is added to the original input through residual cascade to obtain the final output. This convolutional multilayer perceptron module is used for nonlinear transformation of input features. First, the feature transformation is performed through a 1×1 convolutional layer, and then the activation function and Dropout layer are applied to nonlinear transformation and regularization; The spectral-spatial adaptive attention mechanism module first generates spatial attention on the input feature map through a 1×1 convolution layer to generate a spatial attention map. Then, the ReLU activation function is applied to the nonlinear transformation of the convolution output to obtain the activated attention map. The activated attention is then normalized using the Softmax function to obtain the spatial attention weights. Next, these normalized attention weights are applied to the input feature map and weighted to generate a weighted feature map. Finally, the weighted feature map is added element by element to the original feature map to obtain the final enhanced feature map, thereby completing the feature enhancement process of the spatial adaptive attention mechanism. Assume that the input feature map is I∈RH× W ×C, where H is the height, W is the width, and C is the number of channels; a 1×1 convolution operation is used to generate a spatial attention map. The purpose of this step is to map each position of the feature map to an attention value. Assume that the weight of the convolution kernel is W∈R 1×1×C×1 , then the convolution operation is: A=Conv 1×1 (I)∈RH ×W×1 in, Among them A i，j is the spatial attention value of position (i, j); Non-linear activation of convolution output: A raw =ReLU(I)∈R H×W×1 The activated spatial attention is normalized by the Softmax function to obtain the attention weight; A norm =softmax(A raw )∈R H×W×1 The softmax operation is performed at each spatial position as follows: I′ i，j =A raw，i，j ×I i，j ,I′∈R H×W×C Apply the normalized spatial attention map to the input feature map for weighting: I″=I+I′,I″∈R H×W×C The efficient adaptive spectral-spatial feature extraction module combines the spectral-spatial adaptive attention mechanism module and the convolutional multi-layer perceptron module to achieve an efficient adaptive spatial fusion mechanism. The integrated module uses residual connections and adaptive spatial attention mechanisms, and through the synergy of multiple sub-modules, it can effectively extract and fuse the spatial information of input features, thereby improving the performance of the model.

3. The spatial-spectral joint hyperspectral image classification method based on multi-scale adaptive attention mechanism according to claim 2 is characterized by: The context-aware fusion network can adapt to different frequency bands, optimize feature representation, reduce redundant information, and enhance the importance of key features, thereby significantly improving the performance of the model on complex data sets. The context-aware network consists of three convolutional layers, a Conv2D layer with two 1×1 convolutional kernels and a PWConv2D layer with a 3×3 convolutional kernel. Assume that the input data is Y in ∈R H×W×D First, Y in The input Conv2D layer performs feature mapping and then uses the GELU activation function for nonlinear operations. The PWConv2D layer then maps each input by grouping features, further uses the GELU activation function, and uses the Conv2D layer mapping to produce the final output; The above process can be expressed as: Y=Conv2D 1×1 (PWConv2D 3×3 (Conv2D 1×1 (Y in ))) 4. The spatial-spectral joint hyperspectral image classification method based on multi-scale adaptive attention mechanism according to claims 1, 2 and 3 is characterized by: The image is a hyperspectral image.

Images