A Multimodal Remote Sensing Image Classification Method Based on Heterogeneous Feature Learning Networks

Abstract

This invention relates to a multimodal remote sensing image classification method based on a heterogeneous feature learning network, belonging to the field of image classification technology. The classification model of this invention employs a hybrid structure of Transformer and CNN branches. The Transformer branch models global dependencies along the spectral dimension, while the CNN branch extracts local contextual features from multimodal remote sensing data. This hybrid feature learning mode effectively extracts heterogeneous features from the multimodal data. Simultaneously, a heterogeneous feature coupling module integrates feature maps from the CNN branch and feature embeddings from the Transformer branch. Heterogeneous spectral and spatial features are continuously fused interactively through feature coupling, thereby bridging structural and semantic gaps. Furthermore, by increasing the number of feature maps in the CNN branch and the number of heads in the Transformer branch to represent more complex local spatial and global spectral features, this ensures the full integration of these heterogeneous features, ultimately improving classification performance.

Description

Technical Field

[0001] This invention relates to a multimodal remote sensing image classification method based on heterogeneous feature learning networks, belonging to the field of image classification technology. Background Technology

[0002] As a non-contact sensing method, remote sensing technology can acquire the spectral characteristics of observed ground objects from spaceborne and airborne platforms. Subsequent data processing methods mainly utilize the spectral emission and reflectance characteristics of these objects to interpret the observed scene. With the explosive development of remote sensing data acquisition technology, multimodal remote sensing data often exists within the same observation scene, providing diverse and complementary information for understanding and perceiving the geospatial environment. Therefore, the joint processing of multimodal remote sensing data will be a promising paradigm in the field of remote sensing image interpretation.

[0003] Hyperspectral images contain nearly continuous spectral and spatial information and have been successfully applied to fine-grained land cover classification. Commonly used deep neural network models, such as convolutional neural networks, recurrent neural networks, generative adversarial networks, and graph convolutional neural networks, have been applied to hyperspectral image classification tasks. These deep neural networks can achieve satisfactory results in hyperspectral images of simple scenes, but in complex observation scenarios, the ability of hyperspectral data for fine-grained identification and classification of land features remains limited, especially for land features with the same spatial distribution or similar spectral features but belonging to different categories.

[0004] Utilizing other modalities of remote sensing imagery can improve the classification performance of hyperspectral images for more complex scenes. Several studies have been proposed to advance land cover classification using multimodal data. LiDAR data records elevation information of land features, providing complementary elevation information for hyperspectral image classification. Therefore, joint classification of hyperspectral and LiDAR data has become a common multimodal remote sensing data classification model. Other collaborative classification methods for multimodal remote sensing data also exist. However, these techniques have limited capabilities in feature extraction and representation of multimodal remote sensing data, affecting the performance of subsequent accurate land cover classification. In general, current multimodal remote sensing image classification methods have two main drawbacks: feature representation of multimodal remote sensing data and heterogeneous feature fusion. Since multimodal remote sensing data contains features with different modalities, using the same network structure to extract features from various modalities will severely affect the representation of heterogeneous features. Furthermore, current fusion methods generally employ a stacked feature fusion mode, which cannot dynamically fuse heterogeneous features based on their content, resulting in low accuracy in the final classification results. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal remote sensing image classification method based on heterogeneous feature learning networks to solve the problem of inaccurate multimodal remote sensing image classification.

[0006] To address the aforementioned technical problems, this invention provides a multimodal remote sensing image classification method based on heterogeneous feature learning networks. This classification method includes the following steps:

[0007] 1) Construct a remote sensing image classification model, which includes a Transformer branch, a CNN branch, a heterogeneous feature coupling module, and a classification layer. The Transformer branch is used to extract global spectral features along the spectral dimension of hyperspectral image patches. The CNN branch is used to extract local spatial features from multimodal remote sensing data according to a hierarchical CNN structure. The heterogeneous feature coupling module is used to interactively fuse local spatial features and global spectral features. The classification layer is used to perform pooling operations on the local spatial features in the CNN branch to obtain spatial abstract features, perform linear dimensionality reduction on the global features in the Transformer branch to obtain spectral features, and use these heterogeneous features for final collaborative classification.

[0008] 2) Train the constructed classification model;

[0009] 3) Use the trained classification model to classify the multimodal remote sensing images to be classified.

[0010] This invention employs a heterogeneous feature network structure that simultaneously incorporates Transformer and CNN for feature extraction from multi-source remote sensing data. Leveraging the strong advantages of CNN in spatial feature extraction, the heterogeneous feature coupling module combines the two, integrating the advantages of Transformer and CNN in global feature representation and local feature extraction. This avoids the limitations of using the Transformer structure alone in spatial local feature extraction and the problem of a large number of parameters, thereby improving the learning and joint classification performance of heterogeneous features in multi-source remote sensing images.

[0011] Furthermore, both the Transformer branch and the CNN branch include four stages. In the first stage, the Transformer branch uses a single Transformer layer to extract initial sequence features from hyperspectral image patches, while the CNN branch uses max pooling and convolution to extract initial local features from multimodal remote sensing data. In the second, third, and fourth stages, the Transformer branch and the CNN branch in each stage fuse the two types of features through a heterogeneous feature coupling module.

[0012] This invention employs a multi-level network architecture to extract and fuse heterogeneous features from multimodal remote sensing data. Both the Transformer and CNN branches include four stages, with fusion between branches at each stage. By utilizing the increasing number of feature maps in the CNN and the number of heads in the Transformer to represent more complex local spatial features and global spectral features, the heterogeneous features are fully integrated, further improving the subsequent classification performance.

[0013] Furthermore, the heterogeneous feature coupling module is used to convert the feature map of the spatial features extracted by the CNN branch into sequence features and send it to the Transformer branch of the corresponding stage. The Transformer branch of the corresponding stage fuses the received sequence features with the sequence features extracted in that stage. The heterogeneous feature coupling module is also used to convert the feature map of the sequence features extracted by the Transformer branch into spatial features and send it to the CNN branch of the corresponding stage. The CNN branch of the corresponding stage fuses the received spatial features with the spatial features extracted in that stage.

[0014] Furthermore, in each stage, the heterogeneous feature coupling module is used to keep the number of feature maps consistent with the number of feature sequences through a 1×1 convolution operation, to linearly transform each feature map into a feature sequence using a convolution operation with a kernel size equal to the feature map size, and finally to normalize the spatial sequence features using a layer normalization operation and superimpose them with the spectral sequence features to realize the conversion of spatial features into sequence features.

[0015] Furthermore, the heterogeneous feature coupling module in each stage is used to perform transformation operations on the feature sequence, converting the feature sequence into a feature map of size S×S. Then, a 1×1 convolution operation is performed to keep the number of feature sequences consistent with the number of feature maps. An interpolation function is used to convert the feature map of size S×S into a feature map of size H×W. After normalization, it is added to the feature map to realize the conversion of sequence features into spatial features.

[0016] This invention utilizes a heterogeneous feature coupling module to achieve the conversion between two heterogeneous features, enabling the effective fusion of global and local spectral features at each stage.

[0017] Furthermore, the Transformer branch employs a Transformer structure including a multi-head self-attention layer, a multilayer perceptron, and a normalization layer. The multi-head attention layer is used to enable the Transformer structure to learn different dependencies from multiple subspaces, which can be formally expressed as:

[0018] MultiHead(Q,K,V)=Concat(y1,y2,…,yh W O

[0019]

[0020] Where h is the number of heads in the multi-head attention layer, and W O Let d represent the learnable transformation matrix. k =d model / h represents the feature dimension within each head, Q i =X i W i q K i =X i W i k and V i =X i W i v Let Q, K, and V represent the subspace matrices respectively, and W... i q W i k and W i v Let X represent three different transformation parameter matrices. i Let be the eigenvalue of the i-th subspace.

[0021] This invention draws inspiration from the Visual Transformer (ViT), adding a category sequence to the sequence features after linear embedding features to represent the category information of spectral features, and using a multi-head self-attention layer (MHSA) and a multilayer perceptron (MLP) to extract global features.

[0022] Furthermore, each self-attention layer and the multilayer perceptron are connected via a residual structure.

[0023] This invention introduces residual connections in each self-attention layer and multilayer perceptron to improve the trainability of the model and fully utilize the features of different stages for learning. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the Transformer structure used in this invention;

[0025] Figure 2 This is a schematic diagram of the heterogeneous feature coupling module structure used in this invention;

[0026] Figure 3 This is a network architecture diagram of the remote sensing image classification model constructed in this invention;

[0027] Figure 4This is a comparison chart of the classification results of the present invention and different methods. Detailed Implementation

[0028] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0029] First, a remote sensing image classification model is constructed, comprising a Transformer branch, a CNN branch, a heterogeneous feature coupling module, and a classification layer. The constructed classification model is then trained. Finally, the trained model is used to classify the multimodal remote sensing images to be classified. The classification model of this invention employs a hybrid structure of Transformer and CNN branches. The Transformer branch models global dependencies along the spectral dimension, while the CNN branch extracts local contextual features from multimodal remote sensing data. A hybrid feature learning mode is used to effectively extract heterogeneous features from multimodal data. Simultaneously, a heterogeneous feature coupling module integrates feature maps from the CNN branch and feature embeddings from the Transformer branch. Heterogeneous spectral and spatial features are continuously fused interactively through feature coupling, thereby bridging structural and semantic gaps. Furthermore, the increasing number of feature maps in the CNN branch and the increasing number of heads in the Transformer branch represent more complex local spatial and global spectral features, ensuring that these heterogeneous features are fully integrated.

[0030] The classification model constructed in this invention includes a Transformer branch, a CNN branch, a heterogeneous feature coupling module, and a classification layer. The Transformer branch is used to extract global spectral features along the spectral dimension of hyperspectral image patches. The CNN branch is used to extract local spatial features from multimodal remote sensing data according to a hierarchical CNN structure. The heterogeneous feature coupling module is used to interactively fuse local spatial features and global spectral features. The classification layer is used to perform pooling operations on the local spatial features in the CNN branch to obtain spatial abstract features, linearly reduce the dimensionality of the global features in the Transformer branch to obtain spectral features, and use these heterogeneous features for final collaborative classification.

[0031] Specifically, this classification model employs a concurrent network structure, such as... Figure 3As shown, the system comprises four stages. In the first stage, the initial module of the CNN branch is a convolution with a stride of 2, followed by max pooling with a stride of 1, and a 3×3 convolution is used to extract initial local features. The initial module of the Transformer branch consists of sequence feature embedding and addition of class features, and a Transformer layer is used to extract initial sequence features. In the second, third, and fourth stages, the concurrent structure allows the CNN and Transformer branches to extract local spatial features and global spectral representations, respectively. The heterogeneous feature coupling module acts as a bridging module, interactively fusing local and global spectral features in each stage. Finally, the classification layer pools the features from the CNN branch to obtain spatial abstract features and linearly reduces the spectral features from the class features in the Transformer branch to obtain spectral features. These heterogeneous features are then used for final co-classification. Detailed parameters of this heterogeneous feature learning network for multimodal remote sensing images are shown in Table 1.

[0032] Table 1

[0033]

[0034] The Transformer structure used in the Transformer branch is as follows: Figure 1 As shown, this structure borrows from the Visual Transformer (ViT) and mainly consists of a multi-head self-attention layer (MHSA) and a multilayer perceptron (MLP). The multi-head attention layer enables the Transformer structure to learn different dependencies from multiple subspaces, which can be formally expressed as:

[0035] MultiHead(Q,K,V)=Concat(y1,y2,…,y h W O

[0036]

[0037] Where h is the number of heads in the multi-head attention layer, and W O Let d represent the learnable transformation matrix. k =d model / h represents the feature dimension within each head. Q i =X i W i q K i =X i W i k and V i =X i W i v Let Q, K, and V represent the subspace matrices respectively, and W...i q W i k and W i v Let X represent three different transformation parameter matrices. i Let be the feature value of the i-th subspace. In the Transformer architecture, layer normalization is performed before MHSA and MLP operations, and residual connections are introduced in each self-attention layer and multilayer perceptron to improve the trainability of the model and fully utilize features from different stages for learning.

[0038] Because the Transformer excels at extracting global dependencies in sequence features, while CNNs effectively extract locally invariant spatial features, but the feature maps extracted by CNNs and the sequence features extracted by Transformers differ significantly in structure and content, a heterogeneous feature coupling module is used to transform these two heterogeneous features, thereby enabling the effective fusion of global and local spectral features. A schematic diagram of the heterogeneous feature coupling module is shown below. Figure 2 As shown. In the process of converting feature maps representing spatial features into sequence features, firstly, a 1×1 convolution operation is performed to ensure the number of feature maps matches the number of feature sequences. Then, a convolution operation with a kernel size equal to the feature map size is used to linearly transform each feature map into a feature sequence. Finally, a layer normalization operation is used to normalize the spatial sequence features and then superimpose them with the spectral sequence features. In the process of converting spectral sequence features into spatial feature maps, firstly, the feature sequences are transformed into feature maps of size S×S. Then, a 1×1 convolution operation is performed to ensure the number of feature sequences matches the number of feature maps. An interpolation function is used to transform the S×S feature maps into feature maps of size H×W. After normalization, these are added to the feature maps.

[0039] Since the global spectral features described by serialization and the local spatial features described by feature maps differ greatly in structure and content, in order to fully integrate the two heterogeneous features, the entire network structure is divided into multiple stages, and the heterogeneous features are integrated in each stage, so as to fully integrate the interaction and fusion of the two types of features.

[0040] Experimental verification

[0041] To further illustrate the effectiveness of the classification method of this invention, a simulation experiment was conducted. In this experiment, the classification model was implemented using the Python programming language within the PyTorch framework. The main classification evaluation coefficients, namely overall accuracy (OA), average accuracy (AA), and kappa coefficient (κ), were used to quantitatively evaluate the classification performance. A classification graph was also used to qualitatively evaluate the experimental results. To increase the reliability and credibility of the experimental results, ten trials were conducted for each classification experiment.

[0042] Experimental data

[0043] This experiment uses the Houston 2018 dataset, which covers the University of Houston campus and surrounding urban areas. It includes hyperspectral imagery, VHR images, and DSMs generated from LiDAR data. This dataset was publicly released during the 2018 GRSS Data Fusion Competition. The experiment uses the training portion of the hyperspectral data and resamples the corresponding LiDAR-derived DSMs and ground conditions to maintain spatial resolution consistency. The hyperspectral data contains a total of 48 spectral bands, ranging from 380 nm to 1050 nm. The dataset contains 380 × 2384 pixels, corresponding to a spatial resolution of 1 m ground sampling distance (GSD). The image coverage includes 20 distinguishable land cover types; land cover categories and detailed sample sizes are shown in Table 2.

[0044] Table 2

[0045]

[0046] To evaluate the classification performance of the classification method of this invention, several state-of-the-art methods were compared in this experiment. These models included deep learning-based models (i.e., CDCNN, SSRN, SSUN, DBDA, HResNet, SSTN, ViT, EP-CNN, and HRWN) as well as SVM with radial basis function kernels. To ensure a fairer comparison, the same number of training samples were used in all methods, and 20% of the training samples were set as validation samples.

[0047] Experimental Results and Analysis

[0048] The experimental results of the classification method and the selected comparison method of the present invention are shown in Table 3. Table 3 shows the detailed comparative experimental results of different classification methods in terms of average OA, AA, kappa and accuracy of each category, as well as the root mean square error of OA, AA and kappa.

[0049] Table 3

[0050]

[0051]

[0052] Based on the experimental results shown in Table 3, we can conclude that:

[0053] First, in supervised classification experiments, deeper networks generally exhibit higher classification accuracy. Comparing classification results from neural networks of different depths, models with more hidden layers (such as SSUN, HResNet, and SSTN) typically outperform models with fewer hidden layers (such as CDCNN and EP-CNN) on benchmark datasets. The reason behind this phenomenon is that deeper models can extract more discriminative and complex features, which contribute more significantly to land cover classification.

[0054] Secondly, classification methods that simultaneously utilize spatial and spectral features (e.g., SSRN, SSUN, and SSTN) generally achieve higher classification accuracy than those using single-modal features (e.g., CDCNN and ViT). This is because heterogeneous features are diverse and complementary, and effectively utilizing spectral and spatial features can significantly improve classification performance.

[0055] Third, the introduction of self-attention mechanisms can improve classification performance to some extent. There are two methods that utilize self-attention mechanisms: the first is to introduce independent attention blocks in the CNN architecture (i.e., DBDA and HResNet) to highlight information features; the second is to be entirely based on self-attention mechanisms (i.e., SSTN and ViT). Both methods can improve classification accuracy when the network structures are similar, but plug-in self-attention mechanism models have better classification performance, mainly due to the powerful spatial feature extraction capabilities of the CNN structure.

[0056] Furthermore, content-aware multimodal remote sensing classification methods (e.g., HRWN and HFLN) typically achieve higher classification accuracy than methods based on connection-based feature fusion patterns (e.g., EP-CNN). This is because feature-level fusion models struggle to address the fusion problem when there are significant structural and semantic differences between heterogeneous features. Content-aware feature fusion methods can dynamically and adaptively fuse heterogeneous features based on the content of elements, thus resolving the content imbalance problem between heterogeneous features.

[0057] Furthermore, the network architecture in a classification model significantly impacts classification performance. Generally, CNN-based classification methods (such as SSRN, DBDA, and HResNet) achieve higher classification accuracy, primarily due to their powerful spatial feature extraction capabilities. The proposed HFLN utilizes both CNN and transformer architectures, achieving higher classification accuracy on benchmark remote sensing datasets than CNN-only architectures. This is because this hybrid model employs a concurrent network structure to effectively extract heterogeneous features, and the dynamic fusion module further facilitates the fusion of these heterogeneous features, resulting in superior classification outcomes.

[0058] In addition to using evaluation metrics for quantitative assessment, this experiment also used classification maps obtained from different models for visual comparison, such as... Figure 4 The diagram illustrates classification maps obtained through several comparison methods and the classification results of this invention on a benchmark dataset. (a) represents hyperspectral imagery, (b) represents DSM imagery data, (c) represents ground truth representation, (d) represents the classification result using the SVM model, (e) represents the classification result using the CDCNN model, (f) represents the classification result using the SSRN model, (g) represents the classification result using the SSUN model, (h) represents the classification result using the DBDA model, (i) represents the classification result using the HResNet model, (j) represents the classification result using the SSTN model, (k) represents the classification result using the ViT model, (l) represents the classification result using the EP-CNN model, (m) represents the classification result using the HTWN model, and (n) represents the classification result using the HFLN model (this invention). In these diagrams, each color corresponds to a subject category, and ground truth maps are also used for better comparison. These figures demonstrate that classification methods using single-modal features produce poor classification maps, while those fusing multiple features are more uniform. Classification methods using hybrid networks (i.e., SSUN and HFLN) yield smoother classification maps with fewer noisy pixels. Furthermore, the classification map obtained by the HFLN method contains mostly occasional noisy pixels, which provides solid support for the quantitative analysis above. This is because the heterogeneous feature learning architecture proposed in this invention can use different networks to extract more discriminative heterogeneous features from multimodal remote sensing data and effectively fuse these features to obtain better classification results.

[0059] In summary, this invention employs a classification model based on a hybrid architecture of Transformer and CNN, capable of extracting global spectral and local spatial features from multimodal remote sensing images. Furthermore, to eliminate structural and semantic gaps between heterogeneous features, it integrates spectral and spatial features from different stages through heterogeneous feature coupling, and utilizes an increasing number of feature maps and self-attention heads to represent more complex heterogeneous features. Feature extraction and co-classification were also conducted on benchmark multimodal remote sensing datasets, and comparative and ablation experiments further demonstrated the effectiveness and superiority of the proposed classification method.

Claims

1. A multimodal remote sensing image classification method based on heterogeneous feature learning networks, characterized in that, This classification method includes the following steps: 1) Construct a remote sensing image classification model, which includes a Transformer branch, a CNN branch, a heterogeneous feature coupling module, and a classification layer. The Transformer branch is used to extract global spectral features along the spectral dimension of hyperspectral image patches. The CNN branch is used to extract local spatial features from multimodal remote sensing data according to a hierarchical CNN structure. The heterogeneous feature coupling module is used to convert the feature map of the spatial features extracted by the CNN branch into sequence features and send it to the corresponding Transformer branch. The corresponding Transformer branch fuses the received sequence features with the sequence features extracted at that stage. The heterogeneous feature coupling module is also used to convert the feature map of the sequence features extracted by the Transformer branch into spatial features and send it to the corresponding CNN branch. The corresponding CNN branch fuses the received spatial features with the spatial features extracted at that stage. The classification layer is used to perform pooling operations on the local spatial features in the CNN branch to obtain spatial abstract features, perform linear dimensionality reduction on the global features in the Transformer branch to obtain spectral features, and use these heterogeneous features for final collaborative classification. 2) Train the constructed classification model; 3) Use the trained classification model to classify the multimodal remote sensing images to be classified.

2. The multimodal remote sensing image classification method based on heterogeneous feature learning networks according to claim 1, characterized in that, Both the Transformer branch and the CNN branch consist of four stages. In the first stage, the Transformer branch uses a single Transformer layer to extract initial sequence features from hyperspectral image patches, while the CNN branch uses max pooling and convolution to extract initial local features from multimodal remote sensing data. In the second, third, and fourth stages, the Transformer branch and the CNN branch in each stage fuse the two types of features through a heterogeneous feature coupling module.

3. The multimodal remote sensing image classification method based on heterogeneous feature learning networks according to claim 2, characterized in that, In the first stage, the initial module of the CNN branch is a convolution with a stride of 2, and a 3×3 convolution is used to extract initial local features. The initial module of the Transformer branch consists of sequence feature embedding and adding class features.

4. The multimodal remote sensing image classification method based on heterogeneous feature learning networks according to claim 1, characterized in that, The heterogeneous feature coupling module in each stage is used to keep the number of feature maps consistent with the number of feature sequences through 1×1 convolution operations, and to linearly transform each feature map into a feature sequence using convolution operations with a kernel size equal to the feature map size. Finally, a layer normalization operation is used to normalize the spatial sequence features and superimpose them with the spectral sequence features to realize the conversion of spatial features into sequence features.

5. The multimodal remote sensing image classification method based on heterogeneous feature learning networks according to claim 1, characterized in that, The heterogeneous feature coupling module in each stage is used to perform transformation operations on the feature sequence, converting the feature sequence into a size of S×S The feature maps are then processed through a 1×1 convolution operation to ensure the number of feature sequences matches the number of feature maps. An interpolation function is then used to transform the feature maps of size [missing information]. S×S The feature map is transformed into a size of H×W The feature map is normalized and then added to the feature map to convert the sequence features into spatial features.

6. The multimodal remote sensing image classification method based on heterogeneous feature learning networks according to any one of claims 1-5, characterized in that, The Transformer branch described above employs a Transformer structure including a multi-head attention layer, a multilayer perceptron, and a normalization layer. The multi-head attention layer enables the Transformer structure to learn different dependencies from multiple subspaces, which can be formally expressed as: ； in, h It refers to the number of heads in the multi-head attention layer. Represents a learnable transformation matrix. This represents the feature dimension within each head. , and Representing the subspace Q , k and v matrix, , and These represent three different transformation parameter matrices. For the first i The eigenvalues ​​of each subspace.

7. The multimodal remote sensing image classification method based on heterogeneous feature learning networks according to claim 6, characterized in that, Each attention layer and the multilayer perceptron are connected via a residual structure.

Images