A multi-modal remote sensing image classification method based on self-supervised pre-training

Abstract

The present application relates to a kind of multi-modal remote sensing image classification methods based on self-supervised pre-training, belong to remote sensing image classification processing technical field.The present application is pre-trained by using the self-supervised learning model to multi-modal remote sensing image without label, obtains the trained encoder and cross attention module, the trained encoder and cross attention module are used as multi-modal feature extractor, to establish multi-modal remote sensing image classification model, and using a small amount of multi-modal remote sensing image with label to the classification model fine-tuning training, the multi-modal remote sensing image to be classified is input into the classification model after fine-tuning training, and remote sensing image classification can be realized.The method does not need a large number of labeled images, and the encoder can learn the key features of remote sensing images through pre-training, while the cross attention module is used to realize the information interaction between multi-modal features, fully utilize each modal data, and improve the classification accuracy of the image.

Description

Technical Field

[0001] This invention relates to a multimodal remote sensing image classification method based on self-supervised pre-training, belonging to the field of remote sensing image classification and processing technology. Background Technology

[0002] With the continuous development of remote sensing technology, a large number of multimodal remote sensing images are available for understanding the Earth's environment. Passive remote sensing technology can obtain high spatial and spectral resolution information of ground objects; therefore, the obtained hyperspectral (HSI), multispectral (MSI), and ultra-high resolution (VHR) images contain rich information that can be used for interpreting the observed scene. Active remote sensing technology monitors targets by transmitting and receiving echo information; synthetic aperture radar (SAR) and airborne laser data (LiDAR) record the specific electromagnetic properties of the measured objects. These multimodal remote sensing images contain both general and specific information about the observed scene, making land cover classification a very challenging task.

[0003] Supervised classification methods are the most commonly used approach for land cover classification. Early research focused primarily on image analysis methods, such as band selection, feature extraction, and classifier design. Due to the powerful feature extraction capabilities of deep learning models, typical deep networks have been used for image classification, particularly in the field of hyperspectral image classification. Convolutional Neural Networks (CNNs) can extract local deep features for recognition tasks, with 1D CNNs, 2D CNNs, and 3D CNNs used for land cover classification. Recurrent Neural Networks (RNNs) utilize the sequence information contained in hyperspectral images for classification; recently, Transformers have been studied to further improve classification performance in order to explore long-range dependencies. In addition to the model-driven methods mentioned above, several machine learning strategies have been employed for hyperspectral image classification, such as attention mechanisms, knowledge distillation, multi-scale learning, and sparse representation. For multimodal remote sensing image co-classification, there are hyperspectral and LiDAR data classification, hyperspectral and multispectral image classification, and hyperspectral and SAR image co-classification. These supervised methods are data-driven, and their accuracy largely depends on the number of training samples. However, labeling high-quality samples is laborious; therefore, addressing the scarcity of labeled samples is a major problem in the field of remote sensing image classification.

[0004] Semi-supervised learning paradigms offer a feasible solution to the small-sample classification problem by utilizing labeled and unlabeled samples. Graph-based learning is a typical semi-supervised classification mode for remote sensing data. Graph convolutional neural networks and generative adversarial networks can extract deep features from labeled and unlabeled data for land cover classification. However, these semi-supervised classification methods have limitations in solving sample generation and large image classification problems. Another important research direction to address the scarcity of annotated samples is the small-sample learning paradigm. However, this pre-training method is supervised, and the process also requires a large number of labeled samples for feature learning.

[0005] While current supervised and semi-supervised models have achieved significant performance, these classification schemes still fail to effectively address the most prominent problem: the sheer volume of unlabeled multimodal data coupled with a limited number of labeled samples. As a novel learning paradigm, self-supervised learning leverages the inherent features of unlabeled data to learn salient features and applies these learned feature representations to downstream recognition tasks. Based on the learning objectives in the designed interface task, self-supervised learning can be broadly categorized into contrastive and generative approaches. Contrastive learning aims to learn a latent space where similar sample pairs cluster together, while different sample pairs are separated, thereby learning invariant and discriminative feature representations. Contrastive self-supervised learning typically employs CNNs as the base feature extractor to extract high-level abstract features, but it suffers from limitations in long-range dependency extraction and multimodal data processing.

[0006] Generative self-supervised learning achieves its feature learning goal by recovering artificially damaged data. The motivation is that once the model can recover the original signal from the damaged data, it means the model has learned the key features representing the original signal. Generative self-supervised pre-training schemes have been very successful in natural language processing, but their application in the visual domain has lagged behind. This is mainly because images are natural signals with a large amount of spatial redundancy, making it difficult to learn high-level key information from them. This is especially true in remote sensing image processing, where multimodal remote sensing images exist within the same observation scene. These heterogeneous images are rich in complementary information; however, current generative self-supervised models cannot fully utilize these heterogeneous images, exhibiting limited multimodal feature learning capabilities, leading to relatively low final accuracy. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal remote sensing image classification method based on self-supervised pre-training, so as to solve the problem of low classification accuracy caused by the inability of current self-supervised models to make full use of heterogeneous images.

[0008] To address the aforementioned technical problems, this invention provides a multimodal remote sensing image classification method based on self-supervised pre-training, which includes the following steps:

[0009] 1) Acquire remote sensing images of at least two modalities, and perform block processing on each modal remote sensing image. Divide each modal remote sensing image into regular non-overlapping blocks, perform random masking processing on the obtained image blocks, obtain the embedding feature information of the unmasked image blocks, and record the features of the masked image blocks.

[0010] 2) The embedded feature information of the obtained unmasked image patch is input into the encoder, which learns the input embedded feature information and exchanges the results between the modal features through the cross attention module;

[0011] 3) Input the learning results of the cross-attention module and the features of the masked image blocks into the decoder. The decoder reconstructs the masked image blocks in each modality of remote sensing image. The difference between the reconstructed block and the corresponding masked image block is used as the loss. The encoder and cross-attention module are trained using this loss.

[0012] 4) Use the trained encoder and cross-attention module as a multimodal feature extractor to build a lightweight classifier. Use the multimodal feature extractor and lightweight classifier as a multimodal remote sensing image classification model, and fine-tune the classification model.

[0013] 5) Input the remote sensing images of each modality to be classified into the fine-tuned classification model to achieve the classification of multimodal remote sensing images.

[0014] This invention utilizes unlabeled multimodal remote sensing imagery to pre-train a self-supervised learning model, obtaining a trained encoder and cross-attention module. This trained encoder and cross-attention module are then used as a multimodal feature extractor to establish a multimodal remote sensing image classification model. This model is then fine-tuned using a small number of labeled multimodal remote sensing images. The multimodal remote sensing images to be classified are then input into the fine-tuned model to achieve remote sensing image classification. This method does not require a large number of labeled images; pre-training enables the encoder to learn the key features of remote sensing images. Furthermore, the cross-attention module facilitates information interaction between multimodal features, fully utilizing data from each modality and improving image classification accuracy.

[0015] Furthermore, in step 4), the mean square error between the masked image block and the reconstructed block is used as the loss function during fine-tuning training.

[0016] This invention calculates the reconstruction loss only on the masked image blocks and the corresponding reconstruction blocks during pre-training, which ensures the accuracy of training and avoids the problem of large computational load caused by calculating on all image blocks.

[0017] Furthermore, the encoder and decoder adopt a Transformer structure; the encoder includes a position encoding module and multiple Transformer structures; the decoder includes N decoding units, where N is the number of modalities of the remote sensing image, and each decoding unit includes two Transformer structures and one multilayer perceptron.

[0018] Furthermore, the Transformer structure includes a multilayer perceptron, a layer normalization module, and a multi-head attention.

[0019] This invention employs a multilayer perceptron, a layer normalization module, and multi-head attention to construct the Transformer structure.

[0020] Furthermore, the cross-attention module employs a multi-head cross-attention mechanism.

[0021] After the encoder learns the features, this invention uses a cross-attention module to exchange information between heterogeneous features to enhance the representational ability of the learned features, and achieves cross-fusion between different modal features through the multi-head cross-attention (MCA) mechanism.

[0022] Furthermore, the lightweight classifier is a support vector machine classifier.

[0023] This invention uses a support vector machine classifier as a lightweight classifier, which can achieve classification quickly and efficiently. Attached Figure Description

[0024] Figure 1 This is a flowchart of the multimodal remote sensing image classification method based on self-supervised pre-training of the present invention;

[0025] Figure 2 This is a network structure diagram of the self-supervised learning model constructed in this invention;

[0026] Figure 3 This is a schematic diagram of the multimodal self-supervised pre-training and fine-tuning scheme adopted in this invention;

[0027] Figure 4 A schematic diagram of the experimental data used in the experimental verification process;

[0028] Figure 5 This is a comparison chart of the classification methods of this invention and existing classification methods on a benchmark dataset. Detailed Implementation

[0029] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0030] This invention first employs a self-supervised learning model composed of an asymmetric encoder and decoder for pre-training. This model first utilizes the encoder to learn the high-level generalized information contained in the multimodal data, and then uses a cross-attention module to exchange features between different modalities to learn more complementary information from the multimodal remote sensing data. Finally, the decoder is used for reconstruction to train the encoder and cross-attention module. Then, the trained encoder and cross-attention module are used as a multimodal feature extractor to construct a lightweight classifier. This multimodal feature extractor and lightweight classifier form a multimodal remote sensing image classification model, which is then fine-tuned. Finally, the remote sensing images to be classified are input into the fine-tuned classification model, thus achieving the classification of multimodal remote sensing images. The implementation principle of this method is as follows: Figure 1 As shown below, a detailed explanation will be provided with specific examples.

[0031] 1. Construct a self-supervised learning model.

[0032] like Figure 2 As shown, the self-supervised learning model constructed in this invention adopts an asymmetric encoder-decoder structure, with both the encoder and decoder employing a Transformer architecture. The Transformer structure includes a Layer Normalization module, residual blocks, a Multilayer Perceptron (MLP), and a Multi-Head Attention (MHA) structure. MHA is used to construct multiple subspaces from multimodal labels to learn complex dependencies. The residual connection structure can comprehensively utilize features from different stages and promote model training. Layer Normalization performs a normalization operation on the extracted features. The formula for MHA is expressed as:

[0033] MHA(Q,K,V)=Concat(h1,h2,…,h h W O

[0034]

[0035] Q i K i and V i Let represent the query, index, and content value of the i-th header, respectively, and let h be the number of headers.

[0036] The encoder includes a positional encoding module and multiple Transformer structures. The input to the Transformer encoder is multimodal visible markers. Class markers represent information from the corresponding modal data and are used to learn general information from the multimodal data. Relative positional encoding is used to represent the positional information of the features, and class markers are also added to each modal data for feature T. v It means that T vThis represents the stacked embedding features of unmasked image sub-blocks from various types of remote sensing imagery. After the embedding operation, the visible embedding features T... v The input is fed into the encoder for learning, which can be represented as:

[0037] T e =Encoder(T) v )

[0038] To enrich the learned multimodal features, a cross-attention module is used to exchange information after the encoder. The cross-attention module is as follows: Figure 2 As shown, a multi-head cross-attention (MCA) structure is adopted. During feature fusion of the three modalities, features from any two modalities are fused based on cross-attention. The class labels from one modality are then... As a proxy, with patch tags from another modality. A connection is represented as follows:

[0039]

[0040] like Figure 2 As shown, the multi-head cross-attention (MCA) mechanism is used to... and The processing can be represented as follows.

[0041]

[0042] Where Q, K, and V represent the query, index, and content values ​​in the cross-attention operation, and C represents the embedded feature dimension. Through the cross-attention layer, pairs of heterogeneous features are integrated into the content-aware pattern, thereby enhancing representational capabilities.

[0043] The decoder is used to reconstruct a specific modality based on multimodal features and masking labels. The decoder consists of N decoding units, where N is the number of modalities in the remote sensing image. Each decoding unit includes two Transformer structures and one multilayer perceptron. The input to each decoding unit is the unmasked embedded features. and mask features Location embedding is used to provide location information. It can be expressed as:

[0044]

[0045] in, and These represent the visual and mask feature embeddings of the i-th modality data, respectively. In the last layer of each decoding module, the prediction head reconstructs image patches in pixel space, using a fully connected layer as the prediction head. The prediction head of the decoding module will... Its projection is onto a vector with the same dimension as the input sample. Then, through matrix transformation operations, the prediction block P is obtained. i It can be expressed as follows:

[0046]

[0047] In this embodiment, the acquired multimodal data refers to images in three modalities: HIS (hyperspectral imagery), DSM (digital surface model imagery), and VHR (high-resolution imagery). In other implementations, images in other modalities can also be used. These three modalities of image data contain rich and complementary information, allowing for self-supervised feature learning. Before self-learning, preprocessing of each modal data is required.

[0048] The preprocessing process is as follows: A local image of size H×W is cropped from each pixel in the image and used as a self-supervised pre-trained processing unit. Let H represent a sample of the i-th modal data, where H, W, and B are... i Let T represent the height, width, and depth of the sample, respectively. The samples for each modality are divided into regular non-overlapping blocks, and subsets of these blocks are randomly masked at a ratio of m. Unmasked patches are embedded with trainable linear projections. The concatenation results of the embedding labels for all modalities are used as input to a unified encoder, denoted as T. v Each mask token is represented by a trainable vector, which is used as a placeholder for the decoder, denoted as T. m .

[0049] 2. Pre-train the constructed self-supervised learning model.

[0050] Similar to masked autoencoders (MAE), this invention normalizes the original and reconstructed blocks, calculates the mean squared error (MSE) between them, and computes the reconstruction loss only on the mask block and the corresponding reconstructed block. The total reconstruction loss is the sum of the errors for each modality. Based on the total reconstruction loss, the constructed self-supervised learning model is pre-trained, without the need for labeled training samples.

[0051] 3. Construct a multimodal remote sensing image classification model using a pre-trained encoder and cross-attention module.

[0052] After the pre-training process, the trained encoder and cross-attention module are used as a multimodal feature extractor. This multimodal feature extractor and a lightweight classifier are then used to construct a multimodal remote sensing image classification model. In this embodiment, the lightweight classifier is a Support Vector Machine (SVM). Since the self-supervised pre-training process can convert local remote sensing image patches into feature vectors, other classifiers, such as nearest neighbor classification models, can also be used. Figure 3 As shown, this invention uses a trained encoder and a cross-attention layer to extract features from the input multimodal remote sensing images, obtains the corresponding features, normalizes the class labels of these multimodal features and connects them with the corresponding spectral information before inputting them into an SVM for classification. Before the classifier, a small number of labeled samples are used to fine-tune the classification model. The fine-tuned classification model can then classify multimodal remote sensing images.

[0053] Since the features learned by the encoder and cross-attention module in the pre-training stage are highly discriminative, only a small number of labeled samples are needed to train a lightweight classifier in the fine-tuning stage when performing classification. This alleviates the urgent need for labeled samples and eliminates the need to rely on a large number of labeled samples.

[0054] Experimental verification

[0055] To further verify the effectiveness of the present invention, the model proposed in this invention was implemented in the PyTorch framework using the Python programming language. The main classification evaluation coefficients, namely overall accuracy (OA), average accuracy (AA), and kappa coefficient (κ), were used to quantitatively evaluate the classification performance. At the same time, classification graphs were also used to qualitatively evaluate the experimental results.

[0056] This experiment selected the Berlin dataset as the experimental data. This dataset contains hyperspectral and PolSAR data of the Berlin area. The hyperspectral data consists of 244 bands, with a spectral range of 400 nm to 2500 nm; the spatial size is 380 × 2384 pixels, and the spatial resolution is 30 m; the image coverage area contains 8 distinguishable ground features, such as... Figure 4 As shown.

[0057] To evaluate the classification performance of the proposed method, this experiment uses commonly used classification models for comparison, including supervised deep learning models (i.e., SVM, CDCNN, SSUN, and SSRN), semi-supervised learning models (i.e., TSVM and CEGCN), and few-sample learning methods (i.e., DFSL and DMVL). In the experiment, to verify the classification accuracy of different models in the case of few samples, 20 samples were selected for classification of each land cover class.

[0058] Based on the experimental results, the average OA, AA, kappa coefficients and corresponding variances of the present invention (referred to as MultiSSL) and other methods, as well as the classification accuracy of each land cover class, are shown in Table 1.

[0059] Table 1

[0060]

[0061] Table 1 presents detailed comparative experimental results of different classification methods in terms of mean OA, AA, kappa, and accuracy for each class. These tables also report the root mean square error of OA, AA, and kappa. Through the above quantitative comparison, it can be determined that:

[0062] When using a small number of labeled samples for classification, semi-supervised and self-supervised methods outperform supervised methods. In supervised classification methods, a small number of labeled samples can lead to overfitting, severely impacting classification performance. Semi-supervised learning methods utilize both labeled and unlabeled samples, achieving higher classification accuracy than supervised models. Self-supervised learning methods learn key feature representations within the designed interface task, effectively improving classification performance. DMVL, in particular, based on a contrastive learning paradigm, learns meaningful features from unlabeled data, resulting in classification accuracy generally higher than other contrastive methods.

[0063] This invention employs a multimodal self-supervised pre-training and fine-tuning scheme, achieving the best classification performance in terms of both principal evaluation coefficients and individual classification accuracy when using the same number of labeled samples, with an overall classification accuracy of 70.34% on the benchmark dataset. For the Berlin dataset selected in this experiment, existing learning paradigms achieve low classification accuracy with a small number of labeled samples, while the method proposed in this invention achieves higher accuracy in complex scenarios. This is because the model proposed in this invention utilizes multimodal remote sensing data for self-supervised pre-training, thereby learning the key features for subsequent classification tasks and effectively alleviating the high dependence on labeled samples.

[0064] Classification diagrams obtained using different classification methods on the benchmark dataset are as follows: Figure 5 As shown, from left to right, the classification results are obtained by GT, SVM, CDCNN, SSUN, SSRN, TSVM, CEGCN, 3DCAE, DFSL, DMVL, and finally MultiSSL (this invention). In these images, different ground objects are represented by different colors. It can be observed that semi-supervised and self-supervised learning methods obtain more uniform classification images, while the classification image obtained by the model proposed in this invention has fewer noisy pixels and more accurate details in magnified views. This indicates that the MultiSSL method of this invention can learn more discriminative features from multimodal remote sensing images.

[0065] This invention constructs a self-supervised learning model consisting of an asymmetric encoder and decoder. The model is pre-trained using unlabeled multimodal remote sensing images to obtain a trained encoder and cross-attention module. This trained encoder and cross-attention module are then used as a multimodal feature extractor. A lightweight classifier and the multimodal feature extractor are then used as a multimodal remote sensing image classification model. This model is further fine-tuned using a small number of labeled multimodal remote sensing images. Comprehensive experiments on multimodal benchmark datasets further demonstrate the effectiveness and superiority of the proposed classification method, providing a feasible solution to the small-sample classification problem.

Claims

1. A multimodal remote sensing image classification method based on self-supervised pre-training, characterized in that, This classification method includes the following steps: 1) Acquire remote sensing images of at least two modalities, and perform block processing on each modal remote sensing image. Divide each modal remote sensing image into regular non-overlapping blocks, perform random masking processing on the obtained image blocks, obtain the embedding feature information of the unmasked image blocks, and record the features of the masked image blocks. 2) The embedded feature information of the obtained unmasked image patch is input into the encoder, which learns the input embedded feature information and exchanges the results between the modal features through the cross attention module; 3) Input the learning results of the cross-attention module and the features of the masked image blocks into the decoder. The decoder reconstructs the masked image blocks in each modality of remote sensing image. The difference between the reconstructed block and the corresponding masked image block is used as the loss. The encoder and cross-attention module are trained using this loss. 4) Use the trained encoder and cross-attention module as a multimodal feature extractor to build a lightweight classifier. Use the multimodal feature extractor and lightweight classifier as a multimodal remote sensing image classification model, and fine-tune the classification model. 5) Input the remote sensing images of each modality to be classified into the fine-tuned classification model to achieve the classification of multimodal remote sensing images.

2. The multimodal remote sensing image classification method based on self-supervised pre-training according to claim 1, characterized in that, In step 4), the mean square error between the masked image block and the reconstructed block is used as the loss function during fine-tuning training.

3. The multimodal remote sensing image classification method based on self-supervised pre-training according to claim 1 or 2, characterized in that, The encoder and decoder adopt a Transformer architecture. The encoder includes a position encoding module and multiple Transformer structures. The decoder includes N decoding units, where N is the number of modalities in the remote sensing image. Each decoding unit includes two Transformer structures and one multilayer perceptron.

4. The multimodal remote sensing image classification method based on self-supervised pre-training according to claim 3, characterized in that, The Transformer structure includes a multilayer perceptron, a layer normalization module, and a multi-head attention.

5. The multimodal remote sensing image classification method based on self-supervised pre-training according to claim 1, characterized in that, The aforementioned cross-attention module employs a multi-head cross-attention mechanism.

6. The multimodal remote sensing image classification method based on self-supervised pre-training according to claim 1, characterized in that, The lightweight classifier is a support vector machine classifier.

Images