Remote sensing image fusion classification method based on attention feature interaction enhancement

Abstract

The present application relates to a kind of remote sensing image fusion classification methods based on attention feature interaction enhancement, belong to multi-modal remote sensing image classification field.The method of the present application first reads multi-modal remote sensing image data;Data pre-processing and create training set and test set;Hyperspectral image data and synthetic aperture radar image data are respectively through corresponding feature extraction network, the last convolution layer of these two feature extraction networks shares weight;The classification result output by feature network is fused and sent into spatial attention mechanism, then the output result and single modal classification result are decision fusion;Loss function based on weighted summation method is used to optimize model, finally the classification result is obtained.The complementary features of feature interaction extraction of the present application, capture remote dependence, reach higher classification accuracy.

Description

Technical Field

[0001] This invention relates to a remote sensing image fusion classification method based on attention feature interaction enhancement, which is applicable to the collaborative classification of hyperspectral and synthetic aperture radar remote sensing data and belongs to the field of multimodal remote sensing image classification. Background Technology

[0002] Remote sensing image classification is an important task in Earth observation, widely used in agricultural monitoring, urban planning, and mineral exploration. With the continuous optimization and maturation of remote sensing imaging technology, different attributes of remote sensing data can be collected from the same area using multiple sensors, providing a platform and technical support for intelligent interpretation of remote sensing images. Hyperspectral (HS) imaging and synthetic aperture radar (SAR) imaging are two powerful remote sensing technologies that can provide complementary information about objects on the Earth's surface. Hyperspectral images can capture various spectral bands, enabling detailed characterization of surface materials, while SAR images excel at providing information about surface structure and moisture content, unaffected by weather conditions or lighting, and possessing all-weather, all-time characteristics. Therefore, utilizing the complementarity of different modal data and co-processing related data can improve the accuracy and reliability of land cover classification.

[0003] In recent years, deep learning-based methods have been widely used for the fusion and classification of multi-source remote sensing data, achieving relatively ideal results, but some problems still exist. 1) Multimodal data have different focuses on feature representation: HS data has rich spectral and spatial information, while SAR data can provide geometric information about amplitude and phase. This leads to an imbalance between the information from multiple sources, reducing the effectiveness of feature interaction. Therefore, utilizing attention mechanisms to further acquire the unique advantages of each source and enhance features to achieve cross-modal interaction is a crucial issue in the field. 2) Some existing methods only use convolutional kernels of different sizes for multi-scale information extraction, and then only use concatenation or alignment for feature fusion. This cannot guarantee the effectiveness of feature extraction and fusion, but instead increases the parameters and complexity of the model. Summary of the Invention

[0004] The purpose of this invention is to address the problem of insufficient classification accuracy caused by the imbalance between heterogeneous data information in the prior art. This invention provides a remote sensing image fusion classification method based on attention feature interaction enhancement. This method can better utilize complementary information, extract global dependencies from complementary data, enhance local important information, and combine with single-modal features to achieve data cross-attention while reducing the number of parameters, thereby achieving higher land cover classification accuracy.

[0005] The objective of this invention is achieved through the following technical solutions.

[0006] A remote sensing image fusion and classification method based on attention feature interaction enhancement includes the following steps:

[0007] Step 1: Obtain a multimodal remote sensing image dataset containing hyperspectral (HS) and synthetic aperture radar (SAR) images of the same region, and preprocess the data to divide it into training and test sets.

[0008] Specifically, hyperspectral images and synthetic aperture radar (SAR) images of the same region are read to obtain hyperspectral data and SAR data, referred to as bimodal data. The bimodal data are normalized separately, and then principal component extraction (PCI) is performed on the hyperspectral data, retaining D principal components. The processed bimodal data is then divided into square pixel blocks of the same size, and then divided into training and test sets.

[0009] Step 2: Design feature extraction networks for hyperspectral data and synthetic aperture radar data respectively. The last convolutional layer of these two networks shares weights. Input the hyperspectral data and synthetic aperture radar data into the feature extraction networks to obtain dual-modal deep abstract features, and then obtain the prediction result y corresponding to the deep abstract features of the hyperspectral data. HS The prediction result y corresponding to the deep abstract features of synthetic aperture radar SAR .

[0010] The feature extraction network Net1 for hyperspectral images consists of a residual feature enhancement network (FEM) module based on a spatial spectral attention mechanism and two convolutional layers (CBRM and CBRMD layers). The FEM module uses global max pooling and global average pooling to extract the global maximum and average features, then uses the Softmax function to obtain the weight matrix, which is applied to the original features. The features are then enhanced using the LeakyReLU activation function. In the two convolutional layers, deep semantic features of the hyperspectral image are obtained through convolution, batch normalization, and pooling.

[0011]

[0012] F FEM =σ LR (F CBR (F AFEM (F CBR (I i )))+I i )

[0013] F CBR =σ(BN(Conv(I) i W i )))

[0014] FCBRM =F MP (σ(BN(Conv(I i W i ))))

[0015] F CBRMD =dropout(F MP (σ(BN(Conv(I i W i )))))

[0016] Where · denotes matrix multiplication; × denotes the weight matrix and the corresponding expanded two-dimensional feature I. i Multiplication; F AFEM F FEM F CBR F CBRM F CBRMD These represent the operations of the AFEM module, FEM module, CBR module, CBRM layer, and CBRMD layer, respectively; I∈R B ×C×H×W Given the input feature cube, B, C, H, and W represent the batch size, feature channels, and height and width of the feature map, respectively. Since the bimodal data was preprocessed into square pixel blocks in step one, the values ​​of H and W are the same; i I represents the unfolded 2D feature map; max(·) represents taking the maximum value along the channel dimension; ||·||2 is the L2 norm; Conv(·) represents 2D convolution; F MP F AP W represents max pooling and average pooling operations, respectively; i Let W be the weight parameter matrix for the i-th convolution operation. i These are learnable parameters that are adaptively changed through forward and backward propagation of the network; BN(·) represents the batch normalization function; σ(·), σ LR (·) represent the ReLU and LeakyReLU activation functions, respectively; Softmax represents the Softmax function; and dropout is the dropout operation.

[0017] The hyperspectral training data obtained in step one is processed through Net1 to obtain deep abstract features of the hyperspectral data. Then, it is processed through an unfolded layer and a fully connected layer to obtain the prediction result y. HS .

[0018] The feature extraction network Net2 for synthetic aperture radar contains two convolutional layers (CBRM layer and CBRMD layer). Net2 and Net1 share the CBRMD convolutional layer, meaning that the last convolutional layer of Net1 and Net2 shares the weights.

[0019] The synthetic aperture radar training set data obtained in step one is processed through Net2 to obtain the deep abstract features of synthetic aperture radar. Then, it is processed through an expansion layer and a fully connected layer to obtain the prediction result y. SAR .

[0020] Step 3: Extract the bimodal deep abstract features f from Step 2. HS and f SAR The fusion is performed to obtain the fused feature f. dual :

[0021] f dual =concat(f HS ,f SAR )

[0022] Here, `concat(·)` represents feature concatenation. It combines the fused features f... dual In the input spatial attention module, the processed features can be improved to enhance the feature representation of important regions while retaining the enhanced key features.

[0023] F SA =f dual ×δ(Conv(concat(max(f dual ),avg(f dual )),W i ))

[0024] Among them, F SA (·) indicates that the signal has been processed by the spatial attention module; δ(·) represents the sigmoid activation function; max(·) and avg(·) represent taking the maximum and average values ​​along the channel dimension, respectively;

[0025] The enhanced features are expanded and then processed through a fully connected layer to output the prediction result y. dual :

[0026] y dual =ψ(F SA (f dual ))

[0027] Here, ψ(·) represents the unroll and fully connected (FC) operations.

[0028] Step 4: Calculate the predicted value y HS y SAR and y dual Cross-entropy loss value with the true label HS loss SAR loss dual :

[0029] loss HS =CE(yHS (Label)

[0030] loss SAR =CE(y SAR (Label)

[0031] loss dual =CE(y dual (Label)

[0032] Where CE(·) is the Cross Entropy Loss function, and Label is the true label value.

[0033] Then, decision fusion is performed, with the loss function being:

[0034] Loss = αloss HS +βloss SAR +loss dual

[0035] Each loss HS loss SAR loss dual Substituting these values ​​into the loss function formula, where α and β are hyperparameters, both taking a value of 0.01.

[0036] Step 5: Input the training set from Step 1 into Steps 2 through 4. That is, input the training set data into the feature extraction networks Net1 and Net2 in Step 2, and then process it through the fusion operation and spatial attention module in Step 3 to obtain the output. Then, calculate the model loss using the loss function Loss in Step 4, and update and optimize the parameters in the model through backpropagation. Test the model's accuracy at this point, iterate for a certain number of rounds, and save the model with the best accuracy. After training, obtain the final overall classification accuracy, average classification accuracy, kappa coefficient, and classification result map.

[0037] Beneficial effects:

[0038] 1. This invention provides a remote sensing image fusion classification method based on attention feature interaction enhancement. During data preprocessing, the data is normalized to avoid problems such as non-convergence and excessively long training time that may occur during training. At the same time, it utilizes a two-branch feature extraction model to design appropriate feature extraction networks for the characteristics of two heterogeneous data (HS and SAR) to better adapt to the dual-modal image classification task.

[0039] 2. This invention provides a remote sensing image fusion and classification method based on attention-based feature interaction enhancement. The method uses the global maximum feature and global average feature of the feature map as the feature tensors of the attention-based feature enhancement module (AFEM). The global maximum feature can extract texture features, obtain global contextual relationships in the feature map, better capture long-distance dependencies, and reduce the influence of useless information; the global average feature can ensure the integrity of data information in the feature map and highlight background features. This module can achieve adaptive selection of spatial spectral features, reducing data redundancy.

[0040] 3. This invention provides a remote sensing image fusion classification method based on attention feature interaction enhancement. The last CBRMD layer in the feature extraction networks of HS and SAR shares weights, which can realize the interaction of multimodal complementary features, reduce the number of learnable parameters in the feature extraction network, and improve the network training speed. The dropout layer used in the CBRMD layer can prevent the network from overfitting and enhance the generalization ability of the model.

[0041] 4. This invention provides a remote sensing image fusion and classification method based on attention feature interaction enhancement. It connects the deep features extracted from each modality into composite features through feature-level fusion, and then uses a spatial attention mechanism to focus on important information while suppressing unimportant features, thereby achieving feature enhancement.

[0042] 5. This invention provides a remote sensing image fusion classification method based on attention-based feature interaction enhancement. It employs a dual fusion strategy in multimodal data fusion, including feature-level image fusion and decision-level image fusion. Feature-level fusion first extracts deep abstract features from the original data and then establishes new composite features. This reduces the amount of data while retaining representative information, which is beneficial for improving network training speed, reducing information redundancy, and effectively compressing multimodal data information. However, feature-level fusion may lose some original information, thereby reducing the model's accuracy and robustness. Decision-level image fusion can effectively compensate for the shortcomings of feature-level fusion. Decision-level fusion has greater tolerance and error correction for classification results, maximizing the accuracy of classification results and exhibiting good adaptability and anti-interference capabilities. Attached Figure Description

[0043] Figure 1 This is a flowchart of the remote sensing image fusion and classification method based on attention feature interaction enhancement in this invention;

[0044] Figure 2 This is a specific model diagram of the attention feature-based interaction enhancement network in this invention;

[0045] Figure 3The images shown are the results of the fusion and classification of the Berlin dataset provided by this invention; where (a) is a pseudo-color image of HS data, (b) is a pseudo-color image of SAR data, (c) is a ground truth image, and (d) is a classification result image.

[0046] Figure 4 The images shown are the results of the fusion and classification of the Augsburg dataset provided by this invention; where (a) is a pseudo-color image of the HS data, (b) is a pseudo-color image of the SAR data, (c) is a ground truth image, and (d) is the classification result image. Detailed Implementation

[0047] To make the technical solution of the present invention clearer and more understandable, the present invention will now be described in detail with reference to the accompanying drawings in the embodiments.

[0048] This invention provides a remote sensing image fusion and classification method based on attention-based feature interaction enhancement, which uses an attention-based feature interaction enhancement network to classify land cover in multimodal data (HS-SAR).

[0049] like Figure 1 As shown, a remote sensing image fusion and classification method based on attention feature interaction enhancement includes the following steps:

[0050] Step 1: Obtain a multimodal remote sensing image dataset containing hyperspectral (HS) and synthetic aperture radar (SAR) images of the same region, such as the Berlin dataset or the Augsburg dataset. Then, preprocess the data. Specifically, normalize the two modalities separately, mapping them to the [0,1] interval. Next, perform principal component extraction on the spectrally rich HS data, retaining 20 principal components to reduce the dimensionality of the HS data while preserving important information. Finally, crop the HS and SAR data into 11×11 pixel blocks and divide them into training and test sets.

[0051] Step 2: Design feature extraction networks Net1 and Net2 for hyperspectral data and synthetic aperture radar data respectively, such as... Figure 2 As shown.

[0052] A feature extraction network, Net1, was designed for hyperspectral (HS) data. It includes a residual feature enhancement network (FEM) module based on spatial spectral attention, CBRM convolutional layers, and CBRMD convolutional layers. The FEM module is a residual module that includes a CBR module and an attention-based feature enhancement module (AFEM). Figure 2As shown, AFEM first performs an unfold operation to transform the input feature cube into a two-dimensional feature tensor and then performs L2 norm normalization. Next, global max pooling and global average pooling are used to extract the global maximum and average features. The fused maximum and average features are then sequentially subjected to convolution and L2 norm normalization operations, and finally multiplied with the normalized two-dimensional feature tensor. A weight matrix is ​​obtained through the Softmax function, which is applied to the original two-dimensional feature tensor. Finally, the LeakyReLU activation function is used to enhance the features. Figure 2 The reshape operation represents dimension adjustment to accommodate matrix multiplication and activation operations; the CBR layer includes convolution, batch normalization (BN), and ReLU activation operations; the CBRM layer includes convolution, batch normalization, ReLU activation, and max pooling (MP) operations; and the CBRMD layer includes convolution, batch normalization, ReLU activation, max pooling, and Dropout operations.

[0053]

[0054] F FEM =σ LR (F CBR (F AFEM (F CBR (I i )))+I i )

[0055] F CBR =σ(BN(Conv(I) i W i ))) (1)

[0056] F CBRM =F MP (σ(BN(Conv(I i W i ))))

[0057] F CBRMD =dropout(F MP (σ(BN(Conv(I i W i )))))

[0058] Where · denotes matrix multiplication; × denotes the weight matrix and the corresponding expanded two-dimensional feature I. i Multiplication; F AFEM F FEM F CBR F CBRM F CBRMDThese represent the operations of the AFEM module, FEM module, CBR module, CBRM layer, and CBRMD layer, respectively; I∈R B ×C×H×W Given the input feature cube, where B, C, H, and W represent the batch size, feature channels, and height and width of the feature map, respectively, this invention crops the input feature pixel block to a size of 11×11, therefore H and W are both 11; i I represents the unfolded 2D feature map; max(·) represents taking the maximum value along the channel dimension; ||·||2 is the L2 norm; Conv(·) represents 2D convolution; F MP F AP W represents max pooling and average pooling operations, respectively; i Let W be the weight parameter matrix for the i-th convolution operation. i These are learnable parameters that are adaptively changed through forward and backward propagation of the network; BN(·) represents the batch normalization function; σ(·), σ LR (·) represent the ReLU and LeakyReLU activation functions, respectively. Softmax represents the Softmax function, and dropout is the dropout operation with a dropout rate of 0.5.

[0059] A feature extraction network, Net2, was designed for synthetic aperture radar (SAR) data, comprising CBRM convolutional layers and CBRMD convolutional layers. For example... Figure 2 As shown, the two networks Net1 and Net2 share the CBRMD convolutional layer, that is, the last convolutional layer of Net1 and Net2 shares the weights, which extracts deep abstract features of the two modalities while reducing the number of learnable parameters.

[0060] Step 3: Extract the dual-modal deep features f from Net1 and Net2 in Step 2. HS and f SAR The fusion is performed to obtain the fused feature f. dual :

[0061] f dual =concat(f HS ,f SAR (2)

[0062] Here, `concat(·)` represents feature concatenation. It combines the fused features f... dual The processing of equation (3) in the spatial attention module can improve the feature representation of important regions and retain the enhanced key features:

[0063] F SA =f dual ×δ(Conv(concat(max(f dual ),avg(fdual )),W i (3)

[0064] Among them, F SA (·) indicates that the signal has been processed by the spatial attention module; δ(·) represents the sigmoid activation function; max(·) and avg(·) represent taking the maximum and average values ​​along the channel dimension, respectively;

[0065] The enhanced features are expanded and then processed through a fully connected layer to output the prediction result y. dual :

[0066] y dual =ψ(F SA (f dual (4)

[0067] Here, ψ(·) represents the expansion and fully connected operations.

[0068] Step 4: Calculate the predicted value y HS y SAR and y dual Cross-entropy loss value with the true label HS loss SAR loss dual :

[0069] loss HS =CE(y HS (Label)

[0070] loss SAR =CE(y SAR (5)

[0071] loss dual =CE(y dual (Label)

[0072] Where CE(·) is the Cross Entropy Loss function, and Label is the true label value.

[0073] Each loss HS loss SAR and loss dual Substitute into the loss function formula:

[0074] Loss = αloss HS +βloss SAR +loss dual (6)

[0075] α and β are hyperparameters, both with a value of 0.01.

[0076] The overall process of the proposed remote sensing image fusion classification method is as follows:

[0077] Step 5: Input the training set from Step 1 into Steps 2 to 4, that is, input the training set data into the feature extraction networks Net1 and Net2 in Step 2, and then process it through the fusion operation and spatial attention module in Step 3 to obtain the output. Then, calculate the model loss using the loss function Loss designed in Step 4, and then update and optimize the parameters in the model through backpropagation. The parameters include the weight parameter matrix of the convolutional layer in the feature extraction network Net1 for hyperspectral data and the feature extraction network Net2 for synthetic aperture radar data in Step 2, and the weight parameter matrix of the convolutional layer in the spatial attention mechanism module in Step 3. Then, test the accuracy of the model at this time, iterate for a certain number of rounds, save the model with the best accuracy, and after training, obtain the final overall classification accuracy, average classification accuracy, and kappa coefficient, and generate the remote sensing image classification result map.

[0078] This invention was used to classify two publicly available HS-SAR image datasets, Berlin and Augsburg. The overall classification accuracy (OA), average classification accuracy (AA), and kappa coefficient (k) of the obtained datasets are shown in Table 1. This invention achieved overall classification accuracies of 80.04% and 90.07% on the Berlin and Augsburg datasets, respectively. The classification results for the two datasets are shown in the figures below. Figure 3 , Figure 4 As shown.

[0079] Table 1: Classification accuracy for different datasets

[0080]

[0081] The above detailed description further explains the background, content, and beneficial effects of the present invention. It should be noted that the present invention has many other embodiments, and those skilled in the art can make a series of modifications based on the present invention, but all such corresponding changes should fall within the protection scope of the present invention.

Claims

1. A remote sensing image fusion and classification method based on attention feature interaction enhancement, characterized in that, Includes the following steps: Step 1: Obtain a multimodal remote sensing image dataset containing hyperspectral and synthetic aperture radar images of the same region, and preprocess the data, dividing it into training and test sets; Step 2: Design feature extraction networks for hyperspectral data and synthetic aperture radar data respectively, with the last convolutional layer of these two networks sharing weights; input the hyperspectral data and synthetic aperture radar data into the feature extraction networks to obtain bimodal deep abstract features, and then obtain the prediction results corresponding to the deep abstract features of the hyperspectral data. Prediction results corresponding to the deep abstract features of synthetic aperture radar ; Step 3: Fuse the dual-modal deep abstract features obtained in Step 2 and input them into the spatial attention mechanism module to obtain enhanced features, thereby obtaining the prediction result. The input space attention mechanism module is used to locate and output important features, while suppressing useless features. Step 4: Predicted Results , and Input the loss function into the design to obtain the total loss; Step 5: Substitute the training set into steps 2 to 4 to obtain the model's loss. Then, update and optimize the parameters in the model through backpropagation, and test the model's accuracy at this point. Iterate for a certain number of rounds, save the model with the best accuracy, and after training, obtain the final classification result.

2. The remote sensing image fusion and classification method based on attention feature interaction enhancement according to claim 1, characterized in that, Step 1 includes the following sub-steps: Step 1.1: Read the hyperspectral image and synthetic aperture radar image of the same area to obtain hyperspectral data and synthetic aperture radar data, which are called dual-mode data; normalize the dual-mode data respectively, and then perform principal component extraction on the hyperspectral data, retaining D principal components; Step 1.2: Divide the bimodal data processed in 1.1 into square pixel blocks of the same size, and then divide them into training set and test set.

3. The remote sensing image fusion and classification method based on attention feature interaction enhancement according to claim 1, characterized in that, Step 2 includes the following sub-steps: Step 2.1: Feature extraction network for hyperspectral images It includes a residual feature enhancement network (FEM) module based on spatial spectral attention mechanism, CBRM convolutional layers, and CBRMD convolutional layers. The residual feature enhancement network based on spatial spectral attention mechanism uses global max pooling and global average pooling to extract the global maximum and average features, and then obtains the weight matrix through the Softmax function. The weight matrix is ​​applied to the original features, and then the LeakyReLU activation function is used to enhance the features. In the two convolutional layers, deep semantic features of hyperspectral images are obtained through convolution, batch normalization, and pooling. (1) in, Represents matrix multiplication; Represents the weight matrix and its corresponding expanded two-dimensional features. Multiply; , , , , These are the operations for the AFEM module, FEM module, CBR module, CBRM layer, and CBRMD layer, respectively. The input feature cube is defined by B, C, H, and W, which represent the batch size, feature channel, height, and width of the feature map, respectively. Since the bimodal data is preprocessed into square pixel blocks in step 1.2, the values ​​of H and W are the same. for The unfolded two-dimensional feature map; This indicates taking the maximum value along the channel dimension; It is an L2 norm; Represents two-dimensional convolution; , These represent max pooling and average pooling operations, respectively. Let be the weight parameter matrix for the i-th convolution operation. These are learnable parameters that are adaptively changed through forward and backward propagation of the network. This represents the batch normalization function; , These represent the ReLU and LeakyReLU activation functions, respectively. This represents the Softmax function. For discarding operations; The hyperspectral training set data obtained in step 1.2 is processed... This process yields deep abstract features of the hyperspectral data, which are then processed through an expansion layer and a fully connected layer to obtain the prediction results. ; Step 2.2: Feature Extraction Network for Synthetic Aperture Radar It consists of CBRM convolutional layers and CBRMD convolutional layers. and in step 2.1 Shared CBRMD convolutional layer, i.e. and The last convolutional layer shares weights; The synthetic aperture radar training set data obtained in step 1.2 is processed... This process yields deep abstract features of the synthetic aperture radar, which are then processed through an expansion layer and a fully connected layer to obtain the prediction results. .

4. The remote sensing image fusion and classification method based on attention feature interaction enhancement according to claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3.1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] and Extracted deep abstract features and The features are spliced ​​and blended to obtain the blended features. : (2) in, Indicates feature concatenation; the fused features The processing of equation (3) in the spatial attention module can improve the feature representation of important regions and retain the enhanced key features; (3) in, This indicates that the data has been processed by the spatial attention module; This represents the sigmoid activation function; , These represent taking the maximum value and average value along the channel dimension, respectively. Step 3.2: The enhanced features are expanded and then processed through a fully connected layer to output the prediction results. : (4) in, This indicates the operations of expanding and fully connected.

5. The remote sensing image fusion and classification method based on attention feature interaction enhancement according to claim 1, characterized in that, Step 4 includes the following sub-steps: Step 4.1: Calculate the predicted value , and Cross-entropy loss value with the true label , , ; (5) in, The Cross Entropy Loss function is used. The actual label value; Step 4.2: The loss function is: (6) Each time , and Substituting into the loss function formula, where These are hyperparameters, and their values ​​are all 0.

01.

6. The remote sensing image fusion and classification method based on attention feature interaction enhancement according to claim 1, characterized in that, Step 5 includes the following sub-steps: The training set from step 1 is then fed into steps 2 through 4, meaning the training set data is input into the feature extraction network of step 2. and In the next step, after the fusion operation and spatial attention module processing in step 3, the output is obtained. Then, the loss of the model is calculated by the loss function Loss in step 4. The parameters in the model are updated and optimized by backpropagation, and the accuracy of the model at this time is tested. After a certain number of iterations, the model with the best accuracy is saved. After training, the final overall classification accuracy, average classification accuracy, kappa coefficient and classification result map are obtained.

Images